Secrets

A Secret holds small pieces of sensitive data: passwords, tokens, TLS keys, registry credentials. It exists so that credentials live in the cluster’s API rather than in an image or a manifest in git.

It is worth being clear about what a Secret does and does not give you. The data is base64-encoded, not encrypted. Anyone who can read the Secret can read the value. The protection comes from three other things: RBAC deciding who may read it, encryption at rest on the API server’s storage, and the discipline not to put secrets anywhere else.

Reference

• https://kubernetes.io/docs/concepts/configuration/secret/

Terminology

• data — Base64-encoded values. echo -n hunter2 | base64 is the whole of the encoding.
• stringData — Write-only convenience: you supply plain text, the API server encodes it into data for you. Useful when hand-writing manifests, and the field is never returned on read.
• immutable — Once set, the contents cannot be changed; the Secret must be deleted and recreated. Immutability lets the kubelet stop watching the Secret, which is a real performance win at scale.
• Volume mount — The Secret appears as files under a directory in the container. Contents update when the Secret changes.
• Environment variable — Read once when the container starts. Does not update, and is visible in kubectl describe pod and in any process dump.
• imagePullSecret — A Secret of type kubernetes.io/dockerconfigjson, used to authenticate to a private registry.

Types You Will Meet

Creating One

kubectl create secret generic db --from-literal=password=hunter2
kubectl create secret generic db --from-file=./tls.key
kubectl create secret generic db --from-env-file=./db.env
kubectl create secret generic db --from-literal=password=hunter2 --dry-run=client -o yaml > db.yaml

That last form is the one to adopt: generate the YAML, then keep it somewhere that is not git. If a Secret manifest must live in a repository, it should be sealed or encrypted — Sealed Secrets, SOPS, or an external secret operator. A plaintext Secret in git is a leaked credential, no matter how private the repository is.

Consuming One

As environment variables — simple, but static and visible:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db
        key: password
envFrom:
  - secretRef:
      name: db              # every key becomes an environment variable

As mounted files — the better default for anything that may rotate:

volumeMounts:
  - name: db
    mountPath: /etc/db
    readOnly: true
volumes:
  - name: db
    secret:
      secretName: db
      defaultMode: 0400     # owner read-only

Exercises

1. Create and inspect.

   kubectl create secret generic db --from-literal=password=hunter2
   kubectl get secret db
   kubectl get secret db -o jsonpath='{.data.password}' | base64 -d; echo

   Note that kubectl describe secret deliberately hides the values while kubectl get -o yaml does not. Neither is encryption.

2. Consume it both ways, and prove which one updates. Create a Pod with the Secret mounted at /etc/db and also as an environment variable. Then change the Secret:

   kubectl create secret generic db --from-literal=password=newvalue --dry-run=client -o yaml | kubectl apply -f -
   kubectl exec <pod> -- cat /etc/db/password      # new value, within a minute
   kubectl exec <pod> -- printenv DB_PASSWORD      # still the old one

   That difference decides the design: credentials that rotate must be mounted as files, and the application must re-read them.

3. Mount only the key you need, rather than the whole Secret, with items:

   volumes:
     - name: db
       secret:
         secretName: db
         items:
           - key: password
             path: db-password

Gotchas Worth Knowing

• Base64 is not encryption. kubectl get secret -o jsonpath gives anyone with read access the plaintext.
• Environment variables leak. They appear in describe, in crash dumps, and to any process in the container.
• A Secret snapshot is a credential dump. An etcd backup contains every Secret in the cluster in plaintext — see etcd backups on Day 3.
• Secrets are namespaced and can only be referenced by Pods in the same namespace.
• kubectl apply on a Secret you generated elsewhere can silently revert a rotation if your copy is stale.

What to Take Away

• A Secret keeps credentials out of images and out of Pod specs. It is not encryption.
• Mount them as files if they rotate; use environment variables only for values that are fixed for the life of the Pod.
• Protect them with RBAC, encryption at rest, and by keeping them out of git.