Vault installation to minikube via Helm with TLS enabled

13min
|
Vault

This tutorial walks through setting up end-to-end TLS on a HA Vault cluster in Kubernetes. You will create a private key and a wildcard certificate using the Kubernetes CA. You will store the certificate and the key in the Kubernetes secrets store. Finally you will configure the helm chart to use the kubernetes secret.

Prerequisites

This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, minikube, the Vault Helm chart, and the additional configuration to bring it all together.

First, follow the directions to install minikube, including VirtualBox or similar.

Next, install kubectl CLI and helm CLI.

NOTE: This tutorial was last tested in November 2022 on a macOS 12.6.1 using this configuration.

Docker version.

$ docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
  ## ...

Display minikube version.

$ minikube version
minikube version: v1.27.1
commit: fe869b5d4da11ba318eb84a3ac00f336411de7ba

Helm version.

$ helm version
version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

Start minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters running inside Virtual Machines (VM) on your local system.

Start a Kubernetes cluster.

$ minikube start
😄  minikube v1.27.1 on Darwin 12.6.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.25.2 preload ...
 > preloaded-images-k8s-v18-v1...:  320.84 MiB / 320.84 MiB  100.00% 23.57 M
 > gcr.io/k8s-minikube/kicbase:  348.47 MiB / 348.47 MiB  100.00% 16.91 MiB 
 > gcr.io/k8s-minikube/kicbase:  0 B [________________________] ?% ? p/s 21s
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.25.2 on Docker 20.10.18 ...
 ▪ Generating certificates and keys ...
 ▪ Booting up control plane ...
 ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
 ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the minikube cluster.

$ minikube status
minikube
type: Control Plane 
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Install the Vault Helm chart

Vault manages the secrets that are written to these mountable volumes. To provide these secrets a single Vault server is required. For this demonstration Vault can be run in development mode to automatically handle initialization, unsealing, and setup of a KV secrets engine.

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Update all the repositories to ensure helm is aware of the latest versions.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

To verify, search repositories for vault in charts.

$ helm search repo hashicorp/vault
NAME            CHART VERSION   APP VERSION     DESCRIPTION
hashicorp/vault 0.22.1          1.12.0          Official HashiCorp Vault Chart

Create the certificate

create a working directory
```
$ mkdir /tmp/vault
```

Export the working directory location and the naming variables.

$ export VAULT_K8S_NAMESPACE="vault" \
export VAULT_HELM_RELEASE_NAME="vault" \
export VAULT_SERVICE_NAME="vault-internal" \
export K8S_CLUSTER_NAME="cluster.local" \
export WORKDIR=/tmp/vault

Generate the private key

$ openssl genrsa -out ${WORKDIR}/vault.key 2048
Generating RSA private key, 2048 bit long modulus
....................+++
....+++

Create the Certificate Signing Request (CSR).

Create the CSR configuration file

$ cat > ${WORKDIR}/vault-csr.conf <<EOF
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[ kubelet_serving ]
O = system:nodes
CN = system:node:*.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${VAULT_SERVICE_NAME}
DNS.2 = *.${VAULT_SERVICE_NAME}.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}
DNS.3 = *.${VAULT_K8S_NAMESPACE}
IP.1 = 127.0.0.1
EOF

Generate the CSR

$ openssl req -new -key ${WORKDIR}/vault.key -out ${WORKDIR}/vault.csr -config ${WORKDIR}/vault-csr.conf

Issue the Certificate.

Create the csr yaml file to send it to Kubernetes.

$ cat > ${WORKDIR}/csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
   name: vault.svc
spec:
   signerName: kubernetes.io/kubelet-serving
   expirationSeconds: 8640000
   request: $(cat ${WORKDIR}/vault.csr|base64|tr -d '\n')
   usages:
   - digital signature
   - key encipherment
   - server auth
EOF

Send the CSR to Kubernetes

$ kubectl create -f ${WORKDIR}/csr.yaml
certificatesigningrequest.certificates.k8s.io/vault.svc created

Approve the CSR in Kubernetes.

$ kubectl certificate approve vault.svc
certificatesigningrequest.certificates.k8s.io/vault.svc approved

Confirm the certificate was issued

$ kubectl get csr vault.svc
NAME        AGE   SIGNERNAME                      REQUESTOR       REQUESTEDDURATION   CONDITION
vault.svc   16s   kubernetes.io/kubelet-serving   minikube-user   100d                Approved,Issued

Store the certificates and Key in the Kubernetes secrets store

Retrieve the certificate

$ kubectl get csr vault.svc -o jsonpath='{.status.certificate}' | openssl base64 -d -A -out ${WORKDIR}/vault.crt

Retrieve Kubernetes CA certificate

$ kubectl config view \
--raw \
--minify \
--flatten \
-o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
| base64 -d > ${WORKDIR}/vault.ca

Create the Kubernetes namespace

$ kubectl create namespace $VAULT_K8S_NAMESPACE
namespace/vault created

Create the TLS secret

$ kubectl create secret generic vault-ha-tls \
   -n $VAULT_K8S_NAMESPACE \
   --from-file=vault.key=${WORKDIR}/vault.key \
   --from-file=vault.crt=${WORKDIR}/vault.crt \
   --from-file=vault.ca=${WORKDIR}/vault.ca

Output:

# secret/vault-ha-tls created

Deploy the vault cluster via helm with overrides

Create the overrides.yaml file.

$ cat > ${WORKDIR}/overrides.yaml <<EOF
global:
   enabled: true
   tlsDisable: false
injector:
   enabled: true
server:
   extraEnvironmentVars:
      VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
      VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
      VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
   volumes:
      - name: userconfig-vault-ha-tls
        secret:
         defaultMode: 420
         secretName: vault-ha-tls
   volumeMounts:
      - mountPath: /vault/userconfig/vault-ha-tls
        name: userconfig-vault-ha-tls
        readOnly: true
   standalone:
      enabled: false
   affinity: ""
   ha:
      enabled: true
      replicas: 3
      raft:
         enabled: true
         setNodeId: true
         config: |
            cluster_name = "vault-integrated-storage"
            ui = true
            listener "tcp" {
               tls_disable = 0
               address = "[::]:8200"
               cluster_address = "[::]:8201"
               tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
               tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
               tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
            }
            storage "raft" {
               path = "/vault/data"
            }
            disable_mlock = true
            service_registration "kubernetes" {}
EOF

Recommendation

If you are using Prometheus for monitoring and alerting, we recommend to set the cluster_name in the HCL configuration. With the Vault Helm chart, this is accomplished with the config parameter.

Deploy the Cluster

$ helm install -n $VAULT_K8S_NAMESPACE $VAULT_HELM_RELEASE_NAME hashicorp/vault -f ${WORKDIR}/overrides.yaml

Example output:

NAME: vault
LAST DEPLOYED: Thu Nov  3 19:47:36 2022
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/

Your release is named vault. To learn more about the release, try:

   $ helm status vault
   $ helm get manifest vault

Display the pods in the namespace that you created for vault

$ kubectl -n $VAULT_K8S_NAMESPACE get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          22s
vault-1                                 0/1     Running   0          22s
vault-2                                 0/1     Running   0          22s
vault-agent-injector-6679b665fd-92n5q   1/1     Running   0          23s

Initialize vault-0 with one key share and one key threshold.
```
$ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > ${WORKDIR}/cluster-keys.json
```
The operator init command generates a root key that it disassembles into key shares -key-shares=1 and then sets the number of key shares required to unseal Vault -key-threshold=1. These key shares are written to the output as unseal keys in JSON format -format=json. Here the output is redirected to a file named cluster-keys.json.
Display the unseal key found in cluster-keys.json.
```
$ jq -r ".unseal_keys_b64[]" ${WORKDIR}/cluster-keys.json
ezMnsvK9j6aArz1sP0kQNBnTW5PH5u15iFtwPp4Jdzg=
```
Insecure operation
Do not run an unsealed Vault in production with a single key share and a single key threshold. This approach is only used here to simplify the unsealing process for this demonstration.
Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key.
```
$ VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" ${WORKDIR}/cluster-keys.json)
```
After initialization, Vault is configured to know where and how to access the storage, but does not know how to decrypt any of it. Unsealing is the process of constructing the root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

Unseal Vault running on the vault-0 pod.

$ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

Insecure operation

Providing the unseal key with the command writes the key to your shell's history. This approach is only used here to simplify the unsealing process for this demonstration.

The operator unseal command reports that Vault is initialized and unsealed.

Example output:

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.12.0
Build Date              2022-10-10T18:14:33Z
Storage Type            raft
Cluster Name            vault-cluster-5b1b8251
Cluster ID              2e2f214b-c4e7-39e7-52b5-39bffba1aa12
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2022-11-04T13:08:48.505370004Z
Raft Committed Index    36
Raft Applied Index      36

The Vault server is initialized and unsealed.

Join `vault-1` and `vault2` pods to the Raft cluster

Start an interactive shell session on the vault-1 pod.
```
$ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-1 -- /bin/sh
/ $
```
Your system prompt is replaced with a new prompt / $.

Join the vault-1 pod to the Raft cluster.

$ vault operator raft join -address=https://vault-1.vault-internal:8200 -leader-ca-cert="$(cat /vault/userconfig/vault-ha-tls/vault.ca)" -leader-client-cert="$(cat /vault/userconfig/vault-ha-tls/vault.crt)" -leader-client-key="$(cat /vault/userconfig/vault-ha-tls/vault.key)" https://vault-0.vault-internal:8200

Example output:

Key       Value
---       -----
Joined    true

Exit the vault-1 pod.
```
$ exit
```

Unseal vault-1.

$ kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.12.0
Build Date         2022-10-10T18:14:33Z
Storage Type       raft
HA Enabled         true

Start an interactive shell session on the vault-2 pod.
```
$ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-2 -- /bin/sh
/ $
```
Your system prompt is replaced with a new prompt / $.

Join the vault-2 pod to the Raft cluster.

$ vault operator raft join -address=https://vault-2.vault-internal:8200 -leader-ca-cert="$(cat /vault/userconfig/vault-ha-tls/vault.ca)" -leader-client-cert="$(cat /vault/userconfig/vault-ha-tls/vault.crt)" -leader-client-key="$(cat /vault/userconfig/vault-ha-tls/vault.key)" https://vault-0.vault-internal:8200

Example output:

Key       Value
---       -----
Joined    true

Exit the vault-2 pod.
```
$ exit
```

Unseal vault-2.

$ kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-2 -- vault operator unseal $VAULT_UNSEAL_KEY

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.12.0
Build Date         2022-10-10T18:14:33Z
Storage Type       raft
HA Enabled         true

Export the cluster root token

$ export CLUSTER_ROOT_TOKEN=$(cat ${WORKDIR}/cluster-keys.json | jq -r ".root_token")

$ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault login $CLUSTER_ROOT_TOKEN

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.lF3yPwVSCl6AwMI7XdrsirdH
token_accessor       JaQ0MUL3R1zjbTSkUMCKARmk
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

List the raft peers.

$ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator raft list-peers

Node       Address                        State       Voter
----       -------                        -----       -----
vault-0    vault-0.vault-internal:8201    leader      true
vault-1    vault-1.vault-internal:8201    follower    true
vault-2    vault-2.vault-internal:8201    follower    true

Print the HA status

$ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault status

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.12.0
Build Date              2022-10-10T18:14:33Z
Storage Type            raft
Cluster Name            vault-cluster-0a1e62ef
Cluster ID              95972f8a-c38e-8200-9a67-0a910b34b691
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2022-11-04T14:21:05.668816678Z
Raft Committed Index    42
Raft Applied Index      42

You now have a working 3 node cluster with TLS enabled at the pod level. Next you will create a secret and retrieve it via and API call to confirm TLS is working as expected.

Create a secret

Start an interactive shell session on the vault-0 pod.
```
$ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-0 -- /bin/sh
/ $
```
Note
Your system prompt is replaced with a new prompt / $.

Enable the kv-v2 secrets engine

$ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

Create a secret at the path secret/tls/apitest with a username and a password

$ vault kv put secret/tls/apitest username="apiuser" password="supersecret"

===== Secret Path =====
secret/data/tls/apitest

======= Metadata =======
Key                Value
---                -----
created_time       2022-11-07T17:00:23.335480752Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Verify that the secret is defined at the path secret/tls/apitest

$ vault kv get secret/tls/apitest

===== Secret Path =====
secret/data/tls/apitest

======= Metadata =======
Key                Value
---                -----
created_time       2022-11-07T17:00:23.335480752Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    supersecret
username    apiuser

Exit the vault-0 pod.
```
$ exit
```

Expose the vault service and retrieve the secret via the API

The Helm chart defined a Kubernetes service named vault that forwards requests to its endpoints (i.e. The pods named vault-0, vault-1, and vault-2).

Confirm the Vault service configuration

$ kubectl -n $VAULT_K8S_NAMESPACE get service vault

NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
vault   ClusterIP   10.110.24.17   <none>        8200/TCP,8201/TCP   68m

In another terminal, port forward the vault service.

$ kubectl -n vault port-forward service/vault 8200:8200

Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200

In the original terminal, perform a HTTPS curl request to retrieve the secret you created in the previous section.

$ curl --cacert $WORKDIR/vault.ca \
   --header "X-Vault-Token: $CLUSTER_ROOT_TOKEN" \
   https://127.0.0.1:8200/v1/secret/data/tls/apitest | jq .data.data

Example output:

{
"password": "supersecret",
"username": "apiuser"
}

The secret you created earlier is displayed back to us.

Clean up

Stop the running local Kubernetes cluster.

$ minikube stop

✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 node stopped.

This deactivates minikube, and all pods still exist at this point.

Delete the local Kubernetes cluster.

Be aware that minikube delete removes the minikube deployment including all pods. Be sure you want everything removed before continuing.

$ minikube delete

🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /Users/username/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

Remove the files and the working directory you created.
```
$ rm -r $WORKDIR
```

Next steps

You launched Vault in high-availability via a Helm chart with TLS enabled. Learn more about the Vault Helm chart by reading the documentation or exploring the project source code.

Then you created and API call and requested a secret directly from Vault. Explore how pods can retrieve secrets through the Vault Injector service via annotations or secrets mounted on ephemeral volumes.

Vault installation to minikube with Consul

Vault installation to Amazon EKS via Helm

Prerequisites

Start minikube

Install the Vault Helm chart

Create the certificate

Create the Certificate Signing Request (CSR).

Issue the Certificate.

Store the certificates and Key in the Kubernetes secrets store

Deploy the vault cluster via helm with overrides

Join vault-1 and vault2 pods to the Raft cluster

Create a secret

Expose the vault service and retrieve the secret via the API

Clean up

Next steps

Join `vault-1` and `vault2` pods to the Raft cluster