What experiment artifacts and metrics do you present to decide shipping?

WHAT THIS TESTS: The interviewer wants to know if you treat experimentation as a disciplined engineering process or as a casual dashboard exercise. Shipping decisions should be anchored in artifacts created before the experiment launched, not improvised during the review. The question probes your ability to separate signal from noise, protect the business from regressions, and drive a binary decision with pre-defined criteria.

A GOOD ANSWER COVERS: First, present the experiment design document itself, which should include the original description, screenshots or feature context, and the hypothesis stated in a confirmable way. Second, bring the scorecard organized by metric tiers: primary goal metrics that the experiment hoped to move, primary guardrail metrics that must not regress, and secondary metrics that reveal trade-offs or cannibalization effects. Third, show the power analysis and duration context so the team knows the readout hit the required duration and that the minimum detectable effect was reasonable. Fourth, give a crisp recommendation that maps each primary and secondary metric result back to the original hypothesis, explicitly calling out whether the evidence supports shipping, killing, or extending the test.

COMMON WRONG ANSWERS: A major red flag is walking in with a grab bag of every metric that moved and asking the room what they think. Another is ignoring guardrail metrics entirely or dismissing a negative top-line result because a secondary metric looked good. Retrospectively changing the hypothesis to match what won is also a fatal error. Finally, presenting results before the pre-calculated duration has been reached shows you do not understand statistical maturity.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle a situation where the primary metric is flat but a secondary metric is strongly positive, or how you would structure a follow-up holdout to measure long-term effects. They might also probe what you do when guardrail metrics regress slightly but goal metrics are strongly positive, or how you prioritize which metrics belong in the primary versus secondary bucket.

ONE CONCRETE EXAMPLE: Imagine you ran a checkout flow redesign. In the review, you present the design doc that hypothesized a two percent lift in purchase conversion with no regression in refund rate. The scorecard shows a two point five percent lift in purchases with a flat refund rate, while secondary metrics reveal average order value dropped one percent. You note the experiment ran for the full fourteen days dictated by the power analysis, then recommend shipping with a post-launch holdout to monitor whether the lower average order value eventually cannibalizes lifetime revenue.

Source: statsig.com