Design an automated A/B test reporting system

WHAT THIS TESTS: Whether you can move experimentation from artisanal one-off decks to a repeatable, trustworthy system, and whether you know which components prevent the common analysis mistakes that erode trust in A/B results.

A GOOD ANSWER COVERS: Start with standardized, centrally defined metrics so every test computes conversion, revenue, and guardrails the same way, removing per-analyst drift. Automate the statistics: point estimate of lift, confidence intervals or Bayesian posteriors, and p-values, but emphasize effect size and uncertainty over a bare significant or not significant verdict. Build in validity checks that run automatically, especially a sample-ratio mismatch test to catch broken randomization, plus pre-registered guardrail metrics so a primary win that harms retention or latency is flagged. Add segment breakdowns for a few key dimensions to spot heterogeneous effects, and data-quality warnings for missing or anomalous data. Visualizations should show distributions and intervals, not just bars. Finish with an automated, plain-language recommendation: ship, do not ship, or keep running, with the reasoning surfaced.

COMMON WRONG ANSWERS: A dashboard that prints raw p-values and nothing else. Omitting sample-ratio and guardrail checks. Allowing each analyst to redefine metrics, so results are not comparable. Encouraging peeking without correction.

LIKELY FOLLOW-UPS: How would you prevent peeking or handle sequential testing? How do you correct for many simultaneous tests? Where does the metric definition layer live?

ONE CONCRETE EXAMPLE: An analyst configures a test by name; the system pulls assignments, runs a sample-ratio check, computes lift with confidence intervals for the primary metric and guardrails, breaks results out by platform, flags a latency guardrail regression, and emits a one-line recommendation to hold, all without a custom deck.