How would you implement zero-downtime secrets rotation?

WHAT THIS TESTS: This question tests whether you can design an operational workflow that changes critical credentials without breaking running applications. The interviewer cares about your understanding of secret lifecycle management, application coupling, and safe deployment practices rather than just naming a secrets manager.

A GOOD ANSWER COVERS: A strong answer walks through six phases in order. First, inventory every secret including database credentials, API keys, and service tokens, paying special attention to secrets shared across multiple services. Second, document how each application handles secrets, specifically whether it caches them in memory, polls for updates, or fails immediately on invalid credentials, because this determines if you need in-app reload logic. Third, establish a monitoring baseline for authentication failures, database connection metrics, and application error rates so you can detect issues during the rotation. Fourth, build rotation infrastructure that supports versioning, automation scripts, and clear rollback procedures for emergencies. Fifth, implement a dual-phase rotation where you generate and distribute new secrets while keeping old ones active, update applications to use the new secrets, verify success, and only then revoke the old credentials. Sixth, practice the full rotation in development and staging environments before touching production to surface hidden dependencies.

COMMON WRONG ANSWERS: The biggest red flag is proposing a single-phase swap where you revoke the old secret and immediately restart all services to pick up the new one. This ignores in-memory caching, drops active database connections, and assumes all services restart atomically. Another weak pattern is skipping the monitoring baseline and relying on users to report outages. Saying you will just use a secrets manager without explaining the cutover mechanics is also insufficient at the senior level.

LIKELY FOLLOW-UPS: Interviewers often push deeper with questions like how you would handle a secret shared by fifty microservices without restarting all of them, how you would rotate a credential that an external vendor controls, or what your rollback looks like if the new secret works in staging but fails in production. They may also ask how often you rotate and how you automate the rotation of dynamic database credentials.

ONE CONCRETE EXAMPLE: Suppose you rotate a PostgreSQL password used by a web tier. In phase one, you create the new database user credential in your secrets platform and configure the database to accept both old and new passwords. Your application instances, using a secrets manager SDK with a polling interval, pick up the new credential over the next sixty seconds and begin opening new connections with it. You monitor connection error rates and query latency. After five minutes of healthy metrics, you revoke the old database password. If error rates spike, you halt the revocation and roll the application configuration back to the old secret while investigating.

Source: doppler.com