Waypoint Runners

Waypoint utilizes "runners" to execute operations. A runner is a program executing waypoint runner agent that is configured with the details of the Waypoint server.

Every operation that Waypoint runs (build, deploy, release, etc) is executed by a Waypoint runner. The Waypoint server maintains a queue of jobs that the server hands out to connected runners. The server never connects directly to runners, runners always connect to the server. This is important because runners are designed to be run in the same environment that your application is deployed into because the plugins such as Deploy need to talk directly to your deployment platform APIs.

In most deployments, there are two kinds of runners:

Static Runner

Static runners are long-lived runners that are installed into your deployment environment once, likely when you setup Waypoint for the first time.

The waypoint install and waypoint runner install commands can install these runners for you, using details about your deployment environment.

One issue with static runners is that they have fixed capacity: they can only run so many operations at a time. Additionally to run certain plugins like Docker, we need to have uniquely configured environments.

For that reason, Waypoint makes heavy usage of on-demand runners:

On-Demand Runners

While the static runner is capable of performing operations on its own, it also supports the ability to spawn on-demand runners for a given platform. These on-demand runners are ephemeral container instances that perform one operation only. They can perform container builds without any privileged access, offer more isolation between operations, and allow Waypoint to scale further.

On-demand runners: see on-demand runners for more information.

Runner Architecture

This diagram shows the relationship between Waypoint and remote static and on-demand runners, on your infrastructure of choice:

Other Runners

One runner behavior is used less often, the CLI runner:

CLI Runner

When you trigger an action with the CLI, i.e. by running waypoint deploy, and either specify -local or do not have any runners registered on the server, the CLI itself acts as a runner capable of performing your operation. If your deployment needs AWS credentials, or a valid kubectl context, it can access the credentials from the environment that is performing the CLI command.

This diagram shows a CLI runner, as well as remote static and on-demand runners:

Runner Targeting

Runner targeting was designed with multi-environment and multi-region workflows in mind. Static runners can be installed with user-defined ID and labels. You can then set runner profiles, which tell the server which static runner to target to handle operations, and the configurations of the short-lived on-demand runner that will execute jobs. Finally, specify which runner profile to use for an application via the profile option in the runner stanza of the waypoint.hcl file.

Runner Targeting Example

If you install a runner with the runner install or server install commands, it comes with a preset runner profile and adopted static runner.

The following example illustrates a custom local setup of a Waypoint runner and profile.

$ waypoint runner agent -name=test -label=org=hashicorp -label=env=dev

Static runners need a valid runner token and server adoption in order to receive jobs. For more information on runner adoption, see adoption workflow.

With waypoint runner list, you can see the full list of available runners and their labels.

$ waypoint runner list
    ID   |   STATE    |  KIND  |         LABELS         | LAST REGISTERED
---------+------------+--------+------------------------+------------------
  test   | preadopted | remote | org:hashicorp env:dev  | 6 seconds ago

$ waypoint runner adopt test
Runner "test" adopted.

$ waypoint runner list
    ID   |  STATE  |  KIND  |         LABELS         | LAST REGISTERED
---------+---------+--------+------------------------+------------------
  test   | adopted | remote | org:hashicorp env:dev  | 1 minute ago

You can now set a runner profile to target that specific static runner by ID, or any static runner that has matching labels. When there is a runner profile specified in the waypoint.hcl configurations, only a matching static runner can receive jobs for that application and generate on-demand runners to execute them.

$ waypoint runner profile set -name=docker-hashicorp-dev -plugin-type=docker -target-runner-label=org=hashicorp -target-runner-label=env=dev
✓ Runner profile created

                  NAME                  | PLUGIN TYPE |               OCI URL                |              TARGET RUNNER              | DEFAULT
----------------------------------------+-------------+--------------------------------------+-----------------------------------------+----------
  docker-hashicorp-dev                  | docker      | hashicorp/waypoint-odr:latest        | labels: {"env":"dev","org":"hashicorp"} |

Specify the runner profile to use with the runner stanza in the waypoint.hcl file. In this example, a user variable "org" and the built-in workspace variable are interpolated into the runner profile name. At the time of execution of remote operations, Waypoint uses the desired runner profile based on the workspace and configuration variables.

project = "example-nodejs"

app "example-nodejs" {
  runner {
    profile = "docker-${var.org}-${workspace.name}"
  }

  build {
    # ...
  }

  deploy {
    # ...
  }

  release {
    # ...
  }
}

variable "org" {
  type = string
}

Our runner targeting is all set up! An example operation that would use this runner profile:

waypoint build -w=dev -var=org=hashicorp -local=false.

Runner Targeting Advantages

Multi-Cluster, Multi-Environment, Multi-Region

As shown in the example above, enabling multi-environment operations is the most straightforward use-case for runner targeting. Waypoint dynamically interpolates the "dev" workspace into the runner profile name at operation runtime, which then targets the matching static runner for the "dev" environment. You could create additional runners and runner profiles for the "prod" and "test" environments, and Waypoint would target the correct runners automagically with a simple -workspace flag.

Runner targeting is also useful for workflows where you deploy to multiple clusters on a platform. A static runner pod in one Kubernetes cluster would only perform remote operations within that cluster. With runners in multiple Kubernetes clusters, you can easily target and deploy to any cluster with Waypoint remote operations.

High Availability and Performance

In a production environment, runner targeting may be essential to achieving high availability. Deploying multiple runners with the same labels and a runner profile that targets those same labels will cause Waypoint to "round-robin" between the available runners. This approach reduces bottleneck of one static runner handling a large job queue, and ensures continued availability if one of the runners goes down. As long as one of the static runners with matching labels remains online, remote operations on the application continue uninterrupted.

Disabling Runners

Runners can be disabled if desired. With runners disabled, Waypoint executes all operations via the local CLI (even when communicating with a remote server).

Unavailable Functionality

With runners disabled, certain functionality becomes unavailable:

Remote operations
Git polling
Status Checks

In addition to disabled functionality, consider that all clients must now have proper credentials for the target platforms (such as AWS, GCP, etc.) since without runners they must execute all operations locally.

The reason this functionality is not available is the server creates jobs to perform these operations itself and without a long lived runner to execute them, the functionality is not available.

Disabling Runner Installation

Prior to installation, runners can be disabled by specifying the -runner=false flag to the waypoint install command. This will not install a runner.

Removing a Runner

If there is a runner which you'd no longer like your server to use for remote operations, this runner can be "invalidated" for use by the server with the waypoint runner reject command, with the ID of the runner provided.

With the runner forgotten, the server will no longer send any jobs to it. To cleanup the resources being used by that runner though, there are two options:

To uninstall a runner that is already installed, there are two options:

You may also manually go into your platform and delete the runner. For instance with kubernetes, you'd delete the StatefulSet that maintains the static runner.
Uninstall and reinstall Waypoint with a data snapshot and restore. This is a fairly substantial action and should only be undertaken if there is no other options.