Design a measurement framework and experimentation plan for a risky feature rollout

What's really being asked

The interviewer wants to see if you can structure ambiguity. They are looking for a repeatable process that treats product metrics and system counter-metrics as equally important, uses staged rollout to isolate failure modes, and makes go or no-go decisions data-driven rather than political.

The full answer

First, metric taxonomy. You should separate the success metric, engagement lift, from guardrail metrics such as p99 latency, error rate, availability, and infrastructure cost per user or per request. Second, experimental sequencing. Start with a shadow launch or dark launch to measure resource overhead without user impact. Then run a canary release on a small percentage of traffic to detect stability anomalies. Only then run a powered A/B test to measure engagement causally. Third, statistical rigor. Define the minimum detectable effect for engagement, the maximum acceptable degradation for each guardrail, and the sample size or runtime required before calling a result. Fourth, operational readiness. Assign metric owners, set automated alerts and rollback thresholds, and have an incident playbook ready before any user-facing traffic increases.

The mistakes people make

A major red flag is skipping the canary phase and going straight to a 50-50 A/B test, which conflates user behavior with system risk. Another is choosing guardrails but not defining rollback thresholds, leaving the team to debate whether a 15 percent cost increase is acceptable during an outage. Some candidates also propose measuring engagement and stability in the same phase without acknowledging that canary durations are hours while A/B tests may need weeks; mixing these timelines creates false confidence.

What usually comes next

The interviewer may ask how you would handle a scenario where the A/B test shows plus 5 percent engagement but p99 latency rises by 30 milliseconds. They might also probe how you would attribute infrastructure cost increases to the feature versus seasonal traffic, or ask what you would do if guardrails breach during a canary but the root cause is unclear.

A concrete example

Imagine adding a real-time recommendation carousel to a homepage. The success metric is click-through rate on the module. Guardrails are p99 homepage load time under 200 milliseconds, error rate under 0.1 percent, and compute cost per homepage request under 0.5 cents. Week one is a shadow launch where the service runs in parallel but results are discarded, validating that cost per request stays under 0.5 cents. Week two is a 1 percent canary in a single region; if latency or error rate breaches guardrails for more than 10 minutes, automatic rollback triggers. Week three through six is a 10 percent A/B test powered to detect a 2 percent relative lift in click-through rate. The team pre-registers that engagement must rise by at least 2 percent with 95 percent confidence and no guardrail may degrade beyond its threshold for the test to proceed to 100 percent rollout.