How would you implement an A/B test for a redesigned dashboard?

WHAT THIS TESTS: This question evaluates whether you can design a rigorous randomized experiment rather than simply toggling a feature flag. Interviewers want to see that you understand statistical validity, instrumentation across the full stack, and the difference between product metrics and system health guardrails. Senior candidates are expected to mention stable randomization, pre-registration of hypotheses, and power analysis rather than vague plans to ship the variant with more clicks.

A GOOD ANSWER COVERS: First, randomization mechanics: assign users to control or treatment using a stable hash of user ID so the same user always sees the same variant, avoiding cookie-based assignment that refreshes on logout. Second, primary and secondary metrics: pick one north-star engagement metric such as weekly active usage of the dashboard, time to complete a key workflow, or feature adoption rate; instrument this on the backend with immutable event logs rather than frontend pings alone. Third, guardrail metrics: monitor page load time, API error rate, and total support tickets to ensure the redesign does not harm performance or reliability. Fourth, statistical setup: define the minimum detectable effect upfront, calculate required sample size and experiment duration based on baseline traffic, and agree on a success criterion before peeking at results. Fifth, instrumentation details: frontend should log exposure events when the variant renders, while backend should log business outcomes and revenue events tied to the same user ID for clean joinability.

COMMON WRONG ANSWERS: A major red flag is proposing to randomize by session or device, which splits users and dilutes the treatment effect. Another is relying solely on frontend analytics without backend validation, creating data loss when users block trackers. Suggesting a one-day test or checking results daily without a fixed stopping rule introduces peeking bias and inflates false-positive rates. Finally, ignoring guardrail metrics can let a performance regression slip through because the engagement metric looked positive.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle network effects if the dashboard has collaborative features, how to run the test when traffic is too low for standard power, or what you would do if the primary metric is flat but guardrails degrade. They might also probe how you would segment results by user tenure or device type without inflating family-wise error rates.

ONE CONCRETE EXAMPLE: Suppose the current dashboard has a 20 percent seven-day retention rate and you expect the redesign to lift it to 22 percent. With a baseline of 10,000 daily active users, a significance level of 0.05, and 80 percent power, you need roughly 6,400 users per variant, so you run the test for two weeks. Frontend instruments an exposure event called dashboard_redesign_exposure with variant name and timestamp; backend instruments dashboard_core_action_completed and support_ticket_created. You review the engagement lift alongside a guardrail showing that average API latency stayed under 200 milliseconds. If both criteria pass, you schedule a staged rollout.

Source: Wikipedia: A/B testing