How would you A/B test a redesigned dashboard?

What's really being asked

This question assesses your product sense and technical depth. The interviewer wants to see if you can take a vague business request ("increase engagement") and translate it into a concrete, measurable, and technically sound experiment. It tests your ability to think end-to-end, from defining metrics and user segmentation to instrumenting data collection and interpreting results with statistical rigor. They are looking for a partner to the PM, not just a code implementer.

The full answer

A strong answer has four parts. First, clarify the hypothesis by defining "engagement" with specific, measurable metrics. This includes a primary metric (e.g., clicks on key dashboard widgets) and secondary/guardrail metrics (e.g., page load time, overall session duration, user retention). Second, describe the technical implementation. This involves a user bucketing mechanism (e.g., hashing a stable user ID modulo 2 to assign to control/variant) and using a feature flag service to serve the correct dashboard version. Third, detail the instrumentation. Specify which events to log on the frontend (e.g., widget_click, dashboard_view) and what data might be needed from the backend (e.g., data processing latency for dashboard widgets). Fourth, briefly mention the analysis phase: calculating statistical significance (e.g., aiming for a p-value < 0.05), determining the required sample size and experiment duration, and planning for a phased rollout (e.g., 1%, 10%, 50%, 100%).

The mistakes people make

A major red flag is jumping straight into the UI implementation without first defining success. Candidates who say "I'd build the new dashboard in React..." are missing the point. Another common mistake is only suggesting vanity metrics like page views. A senior answer considers counter-metrics; for example, did engagement on the dashboard increase but overall user retention decrease? A third pitfall is neglecting the backend. Forgetting to instrument backend performance or data-related metrics shows a limited, frontend-only perspective. Finally, answers that don't mention statistical significance or sample size feel incomplete and suggest a lack of experience with running real experiments.

What usually comes next

Expect questions like: "How do you ensure a user consistently sees the same variant?"; "What if the results are flat or inconclusive?"; "How would you handle the 'novelty effect' where users click new things just because they're new?"; "What if a key metric goes up but a guardrail metric (like latency) gets worse?".

A concrete example

To measure "engagement," we'll define a primary metric: the click-through rate (CTR) on the "Create New Project" button. We'll also track secondary metrics like time-to-first-click and guardrail metrics like page load time (P95) and the rate of users leaving the app within 30 seconds (bounce rate). We'll bucket users based on hash(user_id) % 100. Users 0-49 get the control (A), and users 50-99 get the new dashboard (B). We need to run the test for 2 weeks to get enough data for statistical significance and to smooth out weekly usage patterns. We'll consider the test a success if the CTR for B is 5% higher than A with a p-value < 0.05, and our P95 page load doesn't increase by more than 10%.