Compare Kubernetes Secrets versus environment variables for Pod credentials

WHAT THIS TESTS: This question probes your ability to reason about defense in depth inside a cluster. The interviewer wants to see that you understand the difference between configuration management and secrets management, and that you can trace a credential from the developer laptop through version control, the API server, etcd, the kubelet, and finally the container runtime and process table. They are looking for awareness of RBAC, node compromise, etcd encryption, and the practical reality that Kubernetes Secrets are a distribution mechanism rather than a vault.

A GOOD ANSWER COVERS: First, plain environment variables defined directly in a Deployment manifest mean the secret is stored in version control, visible to anyone with manifest read access, and embedded in the container process environment where it can be leaked via ps, proc, or debug sidecars. Second, Kubernetes Secrets decouple the credential from the workload manifest, allowing independent RBAC, rotation, and admission control, but they are stored in etcd as base64 by default and are readable by the kubelet and any identity with Secret get or list in that namespace. Third, the node threat model matters because if an attacker gains root on the worker node, they can read Secrets from the local kubelet cache or filesystem unless additional protections like encryption at rest or external secret stores are used. Fourth, best practices such as mounting Secrets as volumes rather than environment variables to reduce exposure in process listings, enabling etcd encryption providers, using external secret management systems like Vault or cloud KMS integrations, and restricting Secret access via RBAC and network policies.

COMMON WRONG ANSWERS: A major red flag is claiming that Kubernetes Secrets are encrypted by default or that the name Secret implies security. Another is saying environment variables are fine for production because the cluster is private or behind a firewall. Some candidates also miss the node-level exposure entirely and focus only on the API server. Failing to mention that base64 is encoding rather than encryption signals shallow knowledge. Finally, suggesting that Git history is not a concern because the repo is private shows poor secrets hygiene.

LIKELY FOLLOW-UPS: The interviewer may ask how you would rotate a database password without restarting pods, how you would handle secrets in a multi-tenant cluster, or what happens to a Secret when it is updated while mounted as a volume. They might also ask for a comparison with external secret operators or how to encrypt etcd at rest.

ONE CONCRETE EXAMPLE: Imagine a Deployment that sets DB_PASSWORD directly in the env block. A junior engineer commits this to Git. Even after removing it, the value persists in Git history. The value also appears in kubectl describe pod output for anyone with pod read access. In contrast, creating a Secret and referencing it with envFrom keeps the value out of the manifest, but the pod still exposes it via proc. A senior candidate would recommend mounting the Secret as a file under tmpfs, reading it at application startup, and enabling etcd encryption with a KMS provider so that the value is encrypted before hitting the underlying disk on the control plane node.

Source: kubernetes.io