Describe two secure methods for providing secrets to a running container

WHAT THIS TESTS: This question evaluates whether you understand the container security boundary and runtime secret injection. Interviewers want to see that you know images are immutable artifacts that should never contain credentials, and that you can distinguish between build-time variables and runtime secrets. In ML workflows, this is critical because models often need cloud storage or database access, and leaking keys means retraining pipelines or revoked API access.

A GOOD ANSWER COVERS: First, Docker secrets mounted as files at runtime, either through Docker Swarm secrets or Kubernetes secrets mounted as tmpfs volumes, so credentials exist only in memory and disappear when the container stops. Second, environment variable injection by the orchestrator at runtime, not during build, which keeps secrets out of image layers but requires care since env vars can be exposed via procfs or logging. Third, cloud-native IAM integration such as AWS IRSA, GCP Workload Identity, or Azure Managed Identity, which eliminates static credentials entirely by granting the container an OIDC-token-based role. Fourth, secret manager sidecars or init containers that fetch credentials from HashiCorp Vault or cloud secret managers and write them to a shared tmpfs volume before the main ML process starts.

COMMON WRONG ANSWERS: A major red flag is suggesting a dot-env file copied into the image during build, because layer history retains that data even if you delete it in a later layer. Another is using build args for secrets without BuildKit secrets mounting, since old Docker versions stored build args in image metadata. Some candidates propose mounting the host dot-env file as a bind volume, which is better than baking it in but still leaves plaintext on disk and ignores rotation. Finally, suggesting plaintext secrets in docker-compose files committed to git shows a lack of operational security awareness.

LIKELY FOLLOW-UPS: The interviewer may ask how you would rotate a database password without restarting the container, which tests whether you know that file-based secrets can be updated if the orchestrator remounts them, whereas env vars are immutable after startup. They might also ask how BuildKit secret mounts differ from build args, expecting you to explain that secret mounts expose credentials only to a specific RUN instruction without persisting in layers. Another follow-up is comparing Kubernetes Secrets to Sealed Secrets or external secret operators, probing your familiarity with GitOps security.

ONE CONCRETE EXAMPLE: Suppose your training container needs an AWS access key. Instead of embedding it, you define a Docker Compose secret that sources from a local file excluded by gitignore. In production, you deploy to Amazon ECS with a task role attached via IAM roles for tasks; the container retrieves temporary credentials from the ECS metadata endpoint at 169.254.170.2, so no static key ever touches disk or image layers. If you must use a static key, you store it in AWS Secrets Manager and run a sidecar that writes it to runc secrets under dev shm, which your Python training script reads on startup and keeps in memory only.