How We Built a Centralized Secrets Management System on AWS EKS Using HashiCorp Vault

One Vault cluster. Multiple teams. Zero secrets in Git. Here is the complete setup, flow, and onboarding guide.

---

Why We Built This

Every growing engineering team hits the same wall. Secrets start in .env files, move to config maps, then someone accidentally pushes a token to Git and now you have a real problem.

We needed one place where all secrets live — one source of truth — with proper access control, automatic rotation, audit logs, and zero manual work when a new app joins.

This is what we built.

---

The Big Picture

Before going into steps, understand the shape of the system:

AWS KMS
  └── Auto-unseals Vault on every restart (no human needed)

Central Vault Cluster (3 nodes on EKS)
  ├── vault-0  →  Leader
  ├── vault-1  →  Follower
  └── vault-2  →  Follower
       └── Raft storage (built-in HA, no external database)

External Secrets Operator (ESO)
  └── Watches Vault, creates K8s secrets automatically

Application Clusters (cluster-1, cluster-2 ... cluster-N)
  └── Each has its own ESO + K8s secrets
  └── All connect to the same central Vault

One central Vault serves all clusters. Each cluster has its own ESO instance that authenticates to Vault and pulls secrets into its namespaces. Application pods never talk to Vault directly — they just read environment variables from Kubernetes secrets.

---

Secret Path Structure

We agreed on this folder structure before storing a single secret. A clear structure makes policies, access control, and onboarding much easier.

secret/
└── mycompany/
    ├── stg/
    │   └── <namespace>/
    │       ├── normal/     → non-sensitive: URLs, feature flags, config
    │       └── critical/   → sensitive: passwords, API keys, DB creds, tokens
    └── prod/
        └── <namespace>/
            ├── normal/
            └── critical/

Examples:

• mycompany/stg/backend/normal/payments-service

• mycompany/prod/backend/critical/payments-service

• mycompany/stg/aiml/normal/model-server

The normal vs critical split matters for RBAC. Developers can see and edit normal in non-prod. They cannot see values in critical for production — only DevOps can.

---

RBAC: Who Can See What

Role	Non-prod normal	Non-prod critical	Prod normal	Prod critical
devops	Full	Full	Full	Full
dev	Full	Full	Write only	Keys only — no values
qa	Full	Full	Read only	Keys only — no values
readonly	Read	Read	Read	Keys only — no values

"Keys only — no values" means a developer can see that a secret called DB_PASSWORD exists but cannot read its value. This protects production credentials without blocking day-to-day work.

---

Part 1 — Setting Up the Central Vault Cluster

Do this once. After this, the central cluster runs itself.

Step 1 — Namespace and Helm Repo

helm repo add hashicorp https://helm.releases.hashicorp.com
kubectl create namespace vault

Step 2 — TLS Certificate

Vault requires TLS. We use a self-signed cert valid for 10 years covering all internal DNS names.

openssl req -newkey rsa:2048 -nodes \
  -keyout vault.key \
  -x509 -days 3650 \
  -out vault.crt \
  -subj "/CN=vault" \
  -addext "subjectAltName=\
DNS:vault,\
DNS:vault.vault.svc,\
DNS:vault.vault.svc.cluster.local,\
DNS:vault-0.vault-internal,\
DNS:vault-1.vault-internal,\
DNS:vault-2.vault-internal,\
IP:127.0.0.1"

kubectl create secret generic vault-tls \
  --namespace vault \
  --from-file=tls.crt=vault.crt \
  --from-file=tls.key=vault.key \
  --from-file=ca.crt=vault.crt

Step 3 — AWS KMS Key

This key is what allows Vault to unseal itself automatically after a restart. No human needs to enter unseal keys ever again.

aws kms create-key \
  --description "Vault Auto Unseal" \
  --query 'KeyMetadata.KeyId' \
  --output text
# Save the key ID — you will need it in helm-values.yaml

Step 4 — IAM Policy and Role

Create this in the AWS Console.

IAM Policy — KMS permissions only:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/YOUR-KEY-ID"
    }
  ]
}

IAM Role — Web Identity trust with OIDC:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/YOUR-OIDC-ID"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.us-east-1.amazonaws.com/id/YOUR-OIDC-ID:sub": "system:serviceaccount:vault:vault",
        "oidc.eks.us-east-1.amazonaws.com/id/YOUR-OIDC-ID:aud": "sts.amazonaws.com"
      }
    }
  }]
}

Step 5 — Kubernetes Service Account (IRSA)

IRSA lets Vault authenticate to AWS without storing credentials anywhere. The pod gets a short-lived OIDC token that it exchanges for temporary AWS credentials.

kubectl create serviceaccount vault --namespace vault

kubectl annotate serviceaccount vault \
  --namespace vault \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/VaultEKSRole

Test it before moving on:

kubectl run test-aws --rm -it --restart=Never \
  --image=amazon/aws-cli \
  --overrides='{"spec":{"serviceAccountName":"vault"}}' \
  -n vault \
  -- sts get-caller-identity
# Must return account 123456789012

Step 6 — Deploy Vault via Helm

Key change settings in default file of the helm-values.yaml:

ha:
  enabled: true
  replicas: 3
  raft:
    enabled: true
    setNodeId: true
    config: |
      ui = true
      listener "tcp" {
        tls_disable = 0
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-tls/tls.crt"
        tls_key_file  = "/vault/userconfig/vault-tls/tls.key"
        tls_client_ca_file = "/vault/userconfig/vault-tls/ca.crt"
      }
      storage "raft" {
        path = "/vault/data"
        retry_join {
          leader_api_addr = "https://vault-0.vault-internal:8200"
          leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
        }
        retry_join {
          leader_api_addr = "https://vault-1.vault-internal:8200"
          leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
        }
        retry_join {
          leader_api_addr = "https://vault-2.vault-internal:8200"
          leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
        }
      }
      seal "awskms" {
        region     = "us-east-1"
        kms_key_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      }
      service_registration "kubernetes" {}

helm upgrade --install vault hashicorp/vault \
  --namespace vault \
  --values helm-values.yaml \
  --set ui.serviceType=ClusterIP

kubectl get pods -n vault -w
# Wait for all 3 pods to show 1/1 Running

Step 7 — Initialize Vault (Run Once Only)

⚠️ This is the most important step. You only do this once. Save everything immediately.

kubectl exec -n vault vault-0 -- vault operator init \
  -recovery-shares=5 \
  -recovery-threshold=3 \
  -format=json > vault-init.json

# Save to AWS Secrets Manager immediately
aws secretsmanager create-secret \
  --name vault/recovery-keys \
  --secret-string file://vault-init.json \
  --region us-east-1

Post-init setup (also run once):

ROOT_TOKEN=$(cat vault-init.json | jq -r '.root_token')
kubectl exec -n vault vault-0 -- vault login $ROOT_TOKEN

# Enable Kubernetes auth method
kubectl exec -n vault vault-0 -- vault auth enable kubernetes
kubectl exec -n vault vault-0 -- vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"

# Enable KV v2 secrets engine
kubectl exec -n vault vault-0 -- vault secrets enable -path=secret kv-v2

# Enable username/password login for the UI
kubectl exec -n vault vault-0 -- vault auth enable userpass

# Enable audit logging
kubectl exec -n vault vault-0 -- vault audit enable file \
  file_path=/vault/audit/vault-audit.log

# Verify all 3 nodes are in the Raft cluster
kubectl exec -n vault vault-0 -- vault operator raft list-peers

Step 8 — Expose via Ingress

Deploy an internal ALB so the Vault UI is accessible from inside the VPC and VPN.

kubectl apply -f helm-ingress.yaml
kubectl get ingress -n vault
# Create Route53 CNAME: vault.yourdomain.com → ALB DNS

Step 9 — Install External Secrets Operator

One ESO instance in the vault namespace. It serves ALL namespaces in the cluster.

helm repo add external-secrets https://charts.external-secrets.io

helm upgrade --install external-secrets external-secrets/external-secrets \
  -n vault

kubectl apply -f external-pod-secret-operator.yaml
kubectl get clustersecretstore
# STATUS must show: Valid

Store the root token for ESO to use:

kubectl create secret generic vault-root-token \
  --namespace vault \
  --from-literal=token=$ROOT_TOKEN

Step 10 — Namespace Access Control

Control which namespaces ESO can pull secrets into. Edit the ALLOWED_NAMESPACES list:

- name: ALLOWED_NAMESPACES
  value: "backend,frontend,aiml,staging"

kubectl apply -f vault-namespace-policies.yaml
kubectl exec -n vault vault-0 -- vault policy list

Step 11 — RBAC and User Management

Apply all RBAC policies:

kubectl apply -f vault-rbac.yaml
kubectl logs -n vault -l job-name=vault-rbac-setup

Add users by editing the USERS section:

- name: USERS
  value: |
    alice:Pass@123:devops:
    bob:Pass@123:dev:backend
    carol:Pass@123:qa:backend+frontend
    dave:Pass@123:readonly:

kubectl apply -f vault-user.yaml

Central cluster is fully live. You never need to touch most of these files again.

---

Part 2 — Onboarding a New Cluster

Repeat these steps for every new cluster you want to connect to central Vault.

On the New Cluster

Step 1 — Switch context and install ESO:

aws eks update-kubeconfig --region us-east-1 --name your-new-cluster

helm upgrade --install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

kubectl get pods -n external-secrets -w
# Wait for 1/1 Running

Step 2 — Apply setup file:

Edit newcluster-eso-setup.yaml — change only this line:

mountPath: "kubernetes-your-new-cluster-name"

kubectl apply -f newcluster-eso-setup.yaml

Step 3 — Export service account token and CA cert:

kubectl get secret external-secrets-token -n external-secrets \
  -o jsonpath='{.data.token}' | base64 -d > /tmp/sa-token.txt

kubectl config view --raw --minify --flatten \
  -o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
  | base64 -d > /tmp/new-cluster-ca.crt

On the Vault Cluster

Step 4 — Switch context and copy files:

aws eks update-kubeconfig --region us-east-1 --name your-vault-cluster

kubectl cp /tmp/sa-token.txt vault/vault-0:/tmp/sa-token.txt
kubectl cp /tmp/new-cluster-ca.crt vault/vault-0:/tmp/new-cluster-ca.crt

Step 5 — Register the new cluster in Vault:

kubectl exec -n vault vault-0 -- vault write \
  auth/kubernetes-your-new-cluster-name/config \
  kubernetes_host="https://NEW-CLUSTER-API-SERVER" \
  kubernetes_ca_cert=@/tmp/new-cluster-ca.crt \
  token_reviewer_jwt=@/tmp/sa-token.txt \
  disable_iss_validation=true

# Test the login — must return a valid token
kubectl exec -n vault vault-0 -- vault write \
  auth/kubernetes-your-new-cluster-name/login \
  role="external-secrets-role" \
  jwt=@/tmp/sa-token.txt

Back on the New Cluster

Step 6 — Verify:

aws eks update-kubeconfig --region us-east-1 --name your-new-cluster
kubectl get clustersecretstore
# STATUS must show: Valid — True

The cluster is now connected. It can pull any secret it has permission for.

---

Part 3 — Adding a New App Secret

Repeat these 5 steps for every new app. No policy changes needed.

Step 1 — Store the Secret in Vault UI

Go to https://vault.yourdomain.com and log in.

Navigate to Secrets Engines → secret/ → Create secret

For non-sensitive config (URLs, feature flags, ports):

• Path: mycompany/stg/backend/normal/payments-service

For sensitive credentials (passwords, API keys, tokens):

• Path: mycompany/stg/backend/critical/payments-service

Toggle JSON on and paste your key-value pairs. Save.

Step 2 — Create an ExternalSecret File

# apps/backend/payments-service.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-service
  namespace: backend
spec:
  refreshInterval: "5m"
  secretStoreRef:
    name: vault-backend        # do not change
    kind: ClusterSecretStore
  target:
    name: payments-service-secret
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: mycompany/stg/backend/normal/payments-service
  - extract:
      key: mycompany/stg/backend/critical/payments-service

Step 3 — Push to Manifest Repo

git add apps/backend/payments-service.yaml
git commit -m "add payments-service external secret"
git push

ArgoCD picks it up and applies it. ESO creates the Kubernetes secret automatically.

Step 4 — Update the App Deployment

# In your deployment or values.yaml
envFrom:
  - secretRef:
      name: payments-service-secret

Step 5 — Verify

kubectl get externalsecret payments-service -n backend
# READY: True   STATUS: SecretSynced

kubectl get secret payments-service-secret -n backend
# Should show the secret with all your keys

From this point, if you update a secret in Vault, the K8s secret updates itself within 5 minutes. No restarts. No manual commands.

Force sync if you want it immediately:

kubectl annotate externalsecret payments-service -n backend \
  force-sync=$(date +%s) --overwrite

---

Day-2 Operations

Upgrading Vault

Always restart followers first, leader last:

helm upgrade --install vault hashicorp/vault \
  --namespace vault --values helm-values.yaml

kubectl delete pod vault-2 -n vault   # wait 1/1 Ready
kubectl delete pod vault-1 -n vault   # wait 1/1 Ready
kubectl delete pod vault-0 -n vault   # wait 1/1 Ready

Vault Pod Crash

Nothing to do. KMS auto-unseals on restart.

kubectl get pods -n vault -w
kubectl exec -n vault vault-0 -- vault status
# Sealed: false  ← auto-unsealed

Recover Root Token (if lost)

aws secretsmanager get-secret-value \
  --secret-id vault/recovery-keys \
  --region us-east-1 \
  --query 'SecretString' \
  --output text | jq -r '.root_token'

Check All Secret Sync Status

kubectl get externalsecret --all-namespaces
# All should show: SecretSynced

---

What to Watch Out For

Save the root token immediately after init. If you lose it and your recovery keys, recovering Vault is very painful. Store it in AWS Secrets Manager before doing anything else.

Test IRSA before deploying Vault. Most auto-unseal failures come from IRSA being misconfigured. Run the STS test in Step 5 and confirm it returns the right account.

The ClusterSecretStore uses a Vault token. If that token expires, all syncs stop silently. Use a long-lived token or a renewable one, and monitor it.

TLS cert expiry reminder. The self-signed cert is valid 10 years but put a calendar reminder for year 9.

One Vault token per cluster for ESO. Each cluster's ESO authenticates via Kubernetes auth, not the root token. The root token stored in the vault namespace is only for initial setup and management.

---

Quick Reference

What	How
Vault UI	https://vault.yourdomain.com
Add a namespace	Edit ALLOWED_NAMESPACES in vault-namespace-policies.yaml, re-apply
Add a user	Edit USERS in vault-user.yaml, re-apply
Add an org or env	Edit ORGS, NONPROD_ENVS, NAMESPACES in vault-rbac.yaml, re-apply
Onboard new cluster	Follow Part 2 above
Add new app secret	Follow Part 3 above
Check cluster status	vault operator raft list-peers
Check secret sync	kubectl get externalsecret --all-namespaces
Force resync	kubectl annotate externalsecret <name> -n <ns> force-sync=$(date +%s) --overwrite
Recover root token	AWS Secrets Manager → vault/recovery-keys

---

Wrapping Up

This setup took a few days to get right but runs completely on its own now. The things that made the biggest difference:

KMS auto-unseal — zero incidents from sealed Vault after pod restarts.

ESO — teams stop asking DevOps to "copy the secret to this namespace". It just appears.

The path structure — when every secret follows the same pattern, writing policies and onboarding new namespaces is trivial.

RBAC roles — four roles covers every real use case. Developers get what they need, production credentials stay protected.

If you are running workloads on EKS and secrets are scattered across config maps, environment variables, and Git — this stack gives you one clean place to manage all of it.

Drop any questions in the comments. Happy to go deeper on any part of this.

---

Tags: DevOps, Kubernetes, HashiCorp Vault, AWS EKS, Secrets Management, Cloud Security, IRSA, External Secrets Operator