Dynamic database credential rotation for microservices

WHAT THIS TESTS The interviewer wants a design that delivers least-privilege, auditable, short-lived credentials and rotates them without the downtime of restarting services, which rules out static secrets baked in at boot.

A GOOD ANSWER COVERS The centerpiece is a secrets manager such as HashiCorp Vault or a cloud equivalent that can generate dynamic database credentials. Rather than a shared static password, each microservice requests its own credential at runtime. The service first authenticates to the secrets manager using its workload identity, for example a Kubernetes service account or cloud IAM role, so no bootstrap secret is stored. The manager then creates a unique database user with a short lease, returns those credentials, and tracks the lease. Because credentials are leased, the service must renew the lease or fetch fresh credentials before expiry; a sidecar or the application's client library does this in the background and swaps the connection pool's credentials live, so connections are re-established with new credentials without a process restart. The manager periodically rotates its own privileged root credential to the database too. Leases can be revoked instantly on compromise, cutting access without redeploying. Every issuance and renewal is logged, giving a full audit trail. The net effect is per-service, least-privilege, automatically expiring credentials with zero-downtime rotation.

COMMON WRONG ANSWERS Injecting one static database password through an environment variable at startup, which cannot rotate without a restart and is shared across services. Storing credentials in the image. Rotating by redeploying every service, which is the downtime the question forbids.

LIKELY FOLLOW-UPS How does the service authenticate without a bootstrap secret? How does the connection pool pick up rotated credentials? How are leases revoked on compromise?

ONE CONCRETE EXAMPLE A payments service authenticates to Vault with its Kubernetes service account, receives a one-hour database credential, and a sidecar renews it; when a host is compromised, the lease is revoked in seconds and only that service's short-lived user is affected.