How would you structure a backend architecture A/B test and define metrics?

WHAT THIS TESTS: This question evaluates your ability to apply causal inference to infrastructure rather than product features. Interviewers want to see if you understand that backend A/B tests violate standard assumptions because users share caches, databases, and network paths, and because latency distributions are highly skewed. The goal is to see if you can isolate architecture from deployment noise, user heterogeneity, and time-based confounders.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, define the unit of randomization and enforce sticky routing so that a single user consistently hits the same backend variant for the duration of the experiment, preventing cross-contamination. Second, control for covariates by stratifying on region, device type, or time-of-day before randomization, because latency varies massively across these dimensions. Third, isolate the architecture from the deployment by running a parallel canary or a holdback group at identical traffic percentages, ensuring that any observed difference stems from code paths rather than rollout transient errors or cache warm-up. Fourth, define paired primary metrics: a system-level metric like P99 latency or P99.9 latency measured at the edge, and a user-facing guardrail metric like task completion rate or perceived load time, because architecture changes can improve milliseconds while breaking user flows.

COMMON WRONG ANSWERS: Common wrong answers include suggesting simple 50-50 traffic splitting without user stickiness, which creates a terrible user experience when stateful requests bounce between architectures. Another red flag is relying solely on mean latency instead of tail latencies, since architecture changes often hurt the long tail while leaving the average untouched. A third mistake is ignoring SUTVA, the assumption that one user's treatment does not affect another's outcome, which fails when variants share a database or cache pool and one variant hogs resources.

LIKELY FOLLOW-UPS: Interviewers often push deeper with questions like: how would you detect cache pollution between variants, what sample size is needed to detect a 20 millisecond P99 improvement, how do you handle rollback if the guardrail metric degrades but the latency metric improves, or how would you adapt this framework to a multi-region active-active setup.

ONE CONCRETE EXAMPLE: Imagine migrating from a monolithic synchronous API to an async event-driven service. You would bucket users by a hash of their session ID modulo 100, route 5 percent to the async variant, and keep them sticky for two weeks. You would stratify by AWS region to balance network distance. Primary objectives would be P99 latency on the checkout endpoint dropping from 800 ms to under 600 ms, error rate staying below 0.1 percent, and checkout completion rate remaining flat or improving. You would pre-register a rollback rule: if error rate exceeds 0.15 percent at any point, the test stops automatically.

Source: Wikipedia: A/B testing