Designing a self-serve experimentation platform

WHAT THIS TESTS The interviewer wants to see platform thinking. Can you move experimentation from a bespoke service that bottlenecks on one team into self-serve infrastructure with safety baked in? They are probing for systems judgment around correctness, isolation, and failure containment, not textbook t-tests.

A GOOD ANSWER COVERS Start with an SDK that hides the hard parts: deterministic bucketing by a stable unit id, automatic exposure logging at the decision point, and safe defaults that fall back to control if the service is unreachable. Add a config layer with mutually exclusive layers so overlapping experiments do not collide, plus traffic budgets per surface. Cover pre-launch checks that validate metric wiring, minimum detectable effect, sample size, and that the experiment compiles against a schema. Then runtime guardrails: sample-ratio-mismatch alarms, guardrail metrics like crash rate and latency, automatic rollback on regression, and a global kill switch.

COMMON WRONG ANSWERS Focusing only on statistical significance while ignoring operational risk. Assuming teams will read documentation rather than enforcing correctness in code. Forgetting that a misconfigured experiment can ship a bug to one hundred percent of users.

LIKELY FOLLOW-UPS How do you detect a sample ratio mismatch and what causes it? How do you prevent peeking and inflated false positives? How do you handle interaction effects between concurrent experiments? How do you log exposure without biasing results?

ONE CONCRETE EXAMPLE A team launches a new checkout button. The SDK assigns users via hash(user_id + experiment_salt), logs exposure only when the button actually renders, and reports to a layer that already reserves twenty percent of traffic. A pre-launch lint catches that they never declared a primary metric. After launch, an SRM alarm fires because logged-out users were excluded from treatment only, revealing an assignment bug before any decision is made.