Secure Consul with Access Control Lists (ACLs)

Consul uses Access Control Lists (ACLs) to secure access to the UI, API, CLI, service communications, and agent communications. When securing your datacenter you should configure the ACLs first. At the core, ACLs operate by grouping rules into policies, then associating one or more policies with a token. The ACL documentation introduces basic concepts and syntax for the ACL system, and we recommend that you read it before you begin this tutorial.

In this tutorial, you will bootstrap the ACL system and then learn how to create tokens with minimum privileges for:

• Consul Server and Client Agents
• Services
• Consul DNS
• Consul KV
• Consul UI

Bootstrap the ACL system

You will bootstrap the ACL system in two steps, enabling ACLs and creating the bootstrap token.

Enable ACLs on the agents

To enable ACLs, add the following ACL parameters to the agent's configuration file and then restart the Consul service. In order for ACL configuration to be applied correctly you will need to apply the same parameters to every server and every client in your datacenter. If you want to reduce Consul client restarts, you can enable the ACLs on them when you apply the token.

acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

In this example, you configured the default policy of "deny", which means you are in a mode where only explicitly named resources are allowed to access the datacenter. You also enabled token persistence when using the HTTP API. With persistence enabled, tokens will be persisted to disk and reloaded when an agent restarts.

Create the initial bootstrap token

To create the initial bootstrap token, use the acl bootstrap command on one of the servers.

$ consul acl bootstrap

The output gives you important information about the token, including the associated policy global-management and SecretID.

While you are setting up the ACL system, temporarily set the CONSUL_HTTP_TOKEN environment variable to the bootstrap token on one Consul server agent, using export to ensure it will not persist once you have closed the session.

$ export CONSUL_HTTP_TOKEN=<your_token_secret_id_here>

This gives you the necessary privileges to call all of the following commands in this tutorial from the same Consul agent to create additional ACL tokens and policies.

Apply individual tokens to agents

Adding tokens to agents is a two step process.

1. Create the agent token
2. Add the token to the agent

Server agents must always have their agent tokens applied by operators as described here. However, Consul can automatically apply the agent tokens needed by client agents with its auto-config feature. Auto-config involves issuing and validating secure JSON Web Tokens (JWTs) using an OIDC Identity Provider. To learn more, follow the Automate Consul Agent Security with Auto-Config tutorial.

Create the agent token

Create an individual token for each Consul agent by specifying a node identity with the node's name and datacenter. A node identity grants the privileges necessary to register a node by that name and read any service in the catalog.

$ consul acl token create -description "<node-name> agent token" \
  -node-identity "<node-name>:<datacenter-of-node>"

This command returns the token information, including the description and node identity information.

Repeat this process for each agent. It is the responsibility of the operator to save tokens in a secure location; we recommend Vault.

Add the token to the agent

Finally, apply the tokens to the agents.

Start by applying the agent token to the servers and ensure they are working correctly before applying the agent token to the clients.

You can setup the token in three ways: CLI, API or using the agent configuration file.

$ export CONSUL_HTTP_TOKEN="<bootstrap token here>"

$ consul acl set-agent-token agent "<agent token secret id here>"

{
  "Token": "<agent token secret id here>"
}

$ curl \
    --request PUT \
    --data @payload.json \
    --header "X-Consul-Token: <<bootstrap token secret id here>" \
    http://127.0.0.1:8500/v1/agent/token/acl_token

acl {
  tokens {
    agent  = "<agent token secret id here>"
  }
}

$ export CONSUL_HTTP_TOKEN="<agent token secret id here>"

$ consul reload

You will need to complete this process on every agent.

At this point, every agent that has a token can once again read and write information to Consul, but only for node-related actions. Actions for individual services are not yet allowed.

Apply individual tokens to the services

Service tokens are necessary for managing the service's information in the catalog, including registration and health checks.

The token creation and application process for services is similar to agents.

1. Create the token for the service
2. Provide the token to the service in its service definition. If you are using service mesh, you must also provide the token to the Envoy sidecar proxy when launched with consul connect envoy.

Create the service token

First, create a token for the service by specifying a service identity with the service's name. A service identity grants the privileges necessary to register a service instance by that name and an associated service mesh sidecar proxy, and to read any service in the catalog.

$ consul acl token create -description "Token for <service-name>" \
  -service-identity "<service-name>"

The command will return information about the token, including the description and service identity information. As usual, save the token to a secure location.

Alternatively, you can automate service token creation using Consul auth methods. Auth methods map attributes from a trusted source of identity such as Kubernetes, JWT, OIDC, or AWS-IAM to ACL tokens with properly scoped policies. For example, a verified JWT can be used to generate a service token by mapping a JWT attribute to a service identity through an auth method with a service-type binding rule. The operator is still responsible for setting up external automation to provide the service ACL tokens generated by Consul auth methods to the services and their sidecar proxies.

Add the token to the service definition

Next, add the token to the service definition.

If the service is already running, re-register it with the token. Register the service with Consul through one of the following mechanisms:

• In a service definition file read by a Consul agent on startup or reload
• The service registration HTTP API endpoint, with the service token passed as a request header as described in HTTP API Authentication
• The service registration CLI command

Below is an example service definition file that specifies the service token created above, for both service mesh and service discovery (non-mesh) use cases.

service {
  name = "dashboard",
  port = 9002,

  # This service instance's sidecar proxy should be given the same ACL token
  # when launched with "consul connect envoy".
  token = "57c5d69a-5f19-469b-0543-12a487eecc66",

  connect {
    sidecar_service {}
  }

  check {
    id = "dashboard-check",
    http = "http://localhost:9002/health",
    method = "GET",
    interval = "1s",
    timeout = "1s"
  }
}

{
  "service": {
    "name": "dashboard",
    "port": 9002,

    "token": "57c5d69a-5f19-469b-0543-12a487eecc66",

    "connect": {
      "sidecar_service": {}
    },

    "check": {
      "id": "dashboard-check",
      "http": "http://localhost:9002/health",
      "method": "GET",
      "interval": "1s",
      "timeout": "1s"
    }
  }
}

service {
  name = "dashboard",
  port = 9002,

  token = "57c5d69a-5f19-469b-0543-12a487eecc66",

  check {
    id = "dashboard-check",
    http = "http://localhost:9002/health",
    method = "GET",
    interval = "1s",
    timeout = "1s"
  }
}

{
  "service": {
    "name": "dashboard",
    "port": 9002,

    "token": "57c5d69a-5f19-469b-0543-12a487eecc66",

    "check": {
      "id": "dashboard-check",
      "http": "http://localhost:9002/health",
      "method": "GET",
      "interval": "1s",
      "timeout": "1s"
    }
  }
}

For service mesh: add the token to the sidecar proxy

Lastly, if the service instance is participating in the Consul service mesh, its sidecar proxy requires the service token. The token must be provided directly to the sidecar proxy; it cannot be retrieved from the service definition.

Provide the service token when launching the Envoy sidecar proxy instance with consul connect envoy through either:

• Environment variables CONSUL_HTTP_TOKEN or CONSUL_HTTP_TOKEN_FILE
• CLI flags -token or -token-file

Token for DNS

To access services using Consul's DNS interface, the token used for DNS must have policy rules for nodes and services. For more details, refer to the DNS documentation. You should apply the token to the Consul agent serving DNS requests. When that agent's DNS server makes a request to a Consul server agent to fulfill the DNS query, it will include the token in that request. Consul can either authorize or revoke the request, depending on the token's privileges.

The token creation process for DNS is a three step process:

1. Create an ACL policy appropriate for your use case
2. Create an ACL token that uses that policy
3. Apply that ACL token to the Consul agent serving DNS requests

Create the DNS policy

First, create a user-defined ACL policy appropriate for your use case.

Note that this step of creating an ACL policy was skipped for agents and services. That's because node and service identities apply an appropriate synthetic policy for those token types, making the creation of unique, explicit policies unnecessary.

Below is an example policy file suitable for typical use of Consul's DNS interface that specifies read privileges for all services and nodes.

node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "read"
}

Create the ACL policy from that policy file:

$ consul acl policy create -name "dns-requests" -rules @dns-request-policy.hcl

Create the DNS token

Next, create the token.

$ consul acl token create -description "Token for DNS Requests" -policy-name dns-requests

Apply the DNS token

Finally, apply the token to the Consul agent serving DNS request in default token ACL configuration parameter.

$ consul acl set-agent-token default "<dns token>"

Note, if you have multiple agents serving DNS requests you can use the same policy to create individual tokens for all of them if they are using the same rules.

Consul KV tokens

The process of creating tokens for Consul KV follows the same three step process as the token for DNS:

1. Create a policy
2. Create a token that uses that policy
3. Apply or use that token

However, unlike tokens for agents and services, Consul KV has many varied use cases.

• Services may need to access configuration data in the key-value store.
• You may want to store distributed lock information for sessions.
• Operators may need access to update configuration values in the key-value store.

The rules for KV have four policy levels; deny, write, read, and list. Let's review several examples of read and write.

Depending on the use case, the token will be applied differently. For services you will add the token to the HTTP client. For operators use, the operator will use the token when issuing commands, either with the CLI or API.

Recursive reads

key_prefix "redis/" {
  policy = "read"
}

In the above example, you are allowing any key with the prefix redis/ to be read. If you issued the command consul kv get -recurse redis/ -token=<your token> you would get a list of key/values for redis/.

This type of policy is good for allowing operators to recursively read configuration parameters stored in the KV. Similarly, a "write" policy with the same prefix would allow you to update any keys that begin with "redis/".

Write privileges for one key

key "dashboard-app" {
  policy = "write"
}

In the above example, you are allowing read and write privileges to the dashboard-app key. This allows for get, delete, and put operations.

This type of token would allow an application to update and read a value in the KV store. It would also be useful for operators who need access to set specific keys.

Read privileges for one key

key "counting-app" {
  policy = "read"
}

In the above example, you are setting a read privileges for a single key, "counting-app". This allows for only get operations.

This type of token allows an application to read from a key to get the value. This is useful for configuration parameter updates.

Consul UI token

Once you have bootstrapped the ACL system, access to the UI will be limited. The anonymous token grants UI access if no default token is set on the Consul agent serving the UI, and all operations will be denied, including viewing nodes and services.

You can re-enable UI features (with flexible levels of access) by creating and distributing tokens to operators. Once you have a token, you can use it in the login form to access the UI features:

After saving a new token, it will be shown in the Consul UI.

The browser stores tokens that you add to the UI. This allows you to distribute different levels of privileges to operators. Like other tokens, it's up to the operator to decide the per-token privileges.

Below is an example of policy that will allow the operator to have read access to the UI for services, nodes, key/values, and intentions. You need to have "acl = read" to view policies and tokens. Otherwise you will not be able to access the ACL section of the UI, not even to view the token you used to access the UI.

service_prefix "" {
  policy = "read"
}
key_prefix "" {
  policy = "read"
}
node_prefix "" {
  policy = "read"
}

The ACL system is described here is one of several mechanisms available to secure Consul UI access. Additional mechanisms include enabling HTTPS access and restricting HTTP writes to a list of CIDRs. For more information on additional mechanisms, consult the "Secure UI Access" entry in the security model recommendations.

Rotate a token

Eventually, you will need to rotate your tokens. To do so, you must create a new token, update any configuration that references the old token with the new token, and then delete the old token. You can rotate tokens used to register services and agents, for DNS queries, and for accessing Consul data.

To keep the same privileges as the previous token, use the clone command with the existing token's AccessorID.

$ consul acl token clone -description "Clone of <token_you_are_cloning>" -id 6a1253d2-1785-24fd-91c2-f8e78c745511

A response will be returned similar to the one below.

Token cloned successfully.
AccessorID:   dcfa52ed-9288-b3ff-056d-255ef69d2d88
SecretID:     0005d17e-5bb2-7e8b-7bfa-15f2eee9ad14
Description:  Clone of Super User
Local:        false
Create Time:  2018-10-22 16:26:02.909096 -0400 EDT
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

Apply tokens to the agent or to the services as needed.

Finally, delete the old token from Consul.

consul acl token delete -id 6a1253d2-1785-24fd-91c2-f8e78c745511

This workflow can also be performed via the HTTP API.

Next steps

In this tutorial, you bootstrapped the ACL system for Consul and applied tokens to agents and services. You assigned tokens for DNS, Consul KV, and the Consul UI, and learned how to rotate a token.

To learn more about Consul's security model read the internals documentation. You can find commands relating to ACLs in our reference documentation.