Deploying Vault on K8s via Kustomize and using Raft Integrated Storage as Backend
Table of Contents generated with DocToc
- vault.hcl configMap
- vault-role
- Services
- Initializing Vault
- High Availability with Integrated Storage
- Useful Resources
conf/vault.hcl
is used as the configuration file for vault, it contains the necessary stanzas to define storage backend in use (Integrated storage in this case)- The kubernetes service registeration section allows tagging vault pods, this is useful for identifying the state of each pod in the cluster and based on it the service
vault-service-active
was created in order to map only to the active leader
- Required with permissions to at least
get
,update
andpatch
pods in order for kubernetes service registeration to function - Results in having the following label definitions
- vault-active
- vault-initialized
- vault-perf-standby
- vault-sealed
- vault-version
- Kubernetes service that uses the dynamic labels of Vault to route traffic only to the active vault leader via the selector
vault-active: "true"
- Initially when vault pod is up and running this service will not map to it since it's not yet initialized (There's no EndPoint), in order for the service to route to the pod, vault should be initialized
- A service is becomes headless by specifying
clusterIP: None
- It's the service referenced by the StatefulSet
serviceName
key - headless service returns list of IPs for the associated pods
- A StatefulSet needs a headlessService for Pods discovery and to maintain the Pods sticky network identity
kubectl exec -it vault-0 -n vault /bin/sh
# Generates the 5 unseal keys used to consturct the master key
# Generates vault root token
vault operator init
# Unseals vault resulting in the pod having vault-active: true label
vault operator unseal <3 keys>
- Setting StatefulSet replicas to
3
then applying the kubernetes resources will result in initiallyvault-0
pod being scheduled, it needs to be intialized and vault needs to be unsealed - After unsealing the pod
vault-1
will also be scheduled but the readiness probe will fail as it needs to be initialized (We'll make it joinvault-0
) as follows
kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-headless:8200
kubectl exec -ti vault-1 -- vault operator unseal <3-keys>
Key Value
--- -----
Joined true
- Same procedure will be done to
vault-2
and we can list all the available nodes with the command
kubectl exec -ti vault-0
export VAULT_TOKEN=<token>
vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
8bc7e17f-2640-bd3b-76d2-f8440f22c10a vault-0.vault-headless:8201 leader true
b2507758-85d4-c4f5-bf19-575eb900bf8f vault-1.vault-headless:8201 follower true
b1a97d24-ad93-1fb4-4429-e4fbae3f8eef vault-2.vault-headless:8201 follower true