Smoke test fails after canary deployment. Design the automated rollback.

What's really being asked

The interviewer wants to see if you can design a self-healing delivery pipeline that minimizes blast radius when a canary fails. They care about your understanding of automated verification gates, atomic rollback mechanisms, and the operational boundary between deployment tooling and runtime safety. Specifically they are looking for whether you treat rollback as a first-class pipeline stage rather than an afterthought, and whether you understand that a failed smoke test in production is a signal to revert traffic, not to patch forward.

The full answer

A strong answer walks through four sequential components. First, traffic isolation: the canary progression must freeze instantly so no additional users hit the bad version. This means stopping the canary scheduler and either draining existing canary pods or shifting traffic back to the stable baseline before any rollback begins. Second, the rollback trigger: the pipeline should automatically initiate a new rollout using the last successfully deployed release without waiting for human approval. The reference architecture shows this is done by configuring verification at the pipeline stage level so that a failed verification job directly drives the rollback automation. Third, rollback verification: after the old release is restored you must run the same smoke tests or health checks against it to prove the previous version is still viable and that the failure was caused by the new code rather than an environmental issue. Fourth, forensics and alerting: the failed rollout and its logs must remain accessible for debugging, the on-call engineer should be paged with context, and the pipeline should emit metrics on rollback duration and frequency.

The mistakes people make

A major red flag is suggesting manual approval for the rollback. The question asks for an automated process, so introducing a human gate breaks the safety contract. Another red flag is proposing to mutate live containers or edit code in place to fix the smoke test. Interviewers also get nervous when candidates casually mention rolling back database schema changes as part of the same automated flow, since that couples stateful and stateless recovery. Finally, describing a rollback that simply redeploys without verifying the old version is still healthy shows shallow operational thinking.

What usually comes next

The interviewer may ask how you handle a rollback that also fails, which tests whether you keep a known good pool warm or have a circuit breaker to halt all deployments. They might ask how long you retain the failed canary artifacts, or how you prevent a rollback storm when the underlying issue is actually a degraded dependency. Another common follow-up is how you coordinate this process across multiple regions or shards.

A concrete example

Imagine a Kubernetes pipeline using a delivery tool with stage-level verification. After a canary deploy to production, a verification container runs smoke tests and times out after five minutes. The test fails. The pipeline stage has verify set to true, so the delivery system automatically creates a new rollout pointing to the previous container image and configuration. Traffic shifts back to the stable replica set. Once the old pods pass health checks, the pipeline marks the rollback successful and pages the on-call team with a link to the failed verification logs. The bad release remains in the cluster but receives zero traffic so engineers can exec into it for debugging.