Describe secure secret injection into Kubernetes containers during CI/CD

WHAT THIS TESTS: The interviewer wants to know if you understand the full attack surface of secrets in a containerized CI/CD pipeline. This includes build-time exposure, transit security, storage at rest in etcd, runtime visibility inside containers, and access control. They are looking for defense-in-depth rather than a single use Kubernetes Secrets answer.

A GOOD ANSWER COVERS: First, secret generation and storage should live outside the cluster in a dedicated secret manager such as HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault, and the CI/CD pipeline should never touch the plaintext value directly. Second, use an operator like External Secrets Operator or Sealed Secrets to sync secrets into Kubernetes Secret objects automatically during deployment, avoiding any manual kubectl steps or Git storage. Third, mount secrets as read-only volumes under tmpfs rather than injecting them as environment variables, because environment variables are visible in procfs, process listings, and application crash dumps. Fourth, enable encryption at rest for etcd so that Secret data is not stored in plaintext in the underlying cluster database. Fifth, lock down RBAC so only the specific ServiceAccount running the workload can read the Secret in its own namespace, following least privilege. Sixth, implement rotation through your secret manager and trigger rolling updates or use dynamic secret leasing so credentials expire automatically.

COMMON WRONG ANSWERS: Baking secrets into container images via Docker build arguments or layer files. Storing plaintext secrets in Git repositories even if private. Using ConfigMaps for sensitive data because they are not designed for confidentiality. Injecting secrets as environment variables without acknowledging the runtime exposure risk. Giving cluster-admin or broad Secret read access to CI/CD service accounts. Forgetting etcd encryption at rest. Assuming Kubernetes Secrets are encrypted by default without enabling the EncryptionConfiguration.

LIKELY FOLLOW-UPS: How would you rotate a database password without dropping active connections? What happens to a secret when you delete a Pod but not the Deployment? How do you audit who read a Secret? Would you ever use environment variables over volume mounts, and why? How do you handle secrets in a multi-tenant cluster where users should not read each others credentials?

ONE CONCRETE EXAMPLE: A pipeline builds a generic image with no credentials. ArgoCD or Flux deploys a Helm chart that references an ExternalSecret object. The External Secrets Operator fetches the database password from Vault and writes it to a Kubernetes Secret in the payments namespace. The Deployment mounts that Secret as a volume at /etc/secrets/db-password with readOnly true. The container application reads the file on startup. RBAC grants only the payments ServiceAccount permission to get that Secret. The cluster etcd is configured with a KMS provider to encrypt Secret resources at rest. When Vault rotates the password, the operator syncs the new value and the Deployment rolls out automatically.

Source: kubernetes.io