How would you A/B test a redesigned dashboard?
Tests translating a vague goal ('more engagement') into a concrete engineering plan. A good answer defines key metrics first, then outlines user bucketing, instrumentation, and statistical analysis.
What's really being asked
This question tests your ability to bridge product requirements and technical implementation. The interviewer isn't just looking for the definition of an A/B test. They are evaluating your product sense, your data literacy, and your systems thinking. A senior answer demonstrates ownership by turning a vague request ('more engagement') into a rigorous, measurable, and technically sound experiment. It shows you can think about the entire lifecycle: hypothesis, implementation, measurement, and decision-making.
The full answer
A strong answer addresses four areas in order. First, clarifying the hypothesis by defining 'engagement' with specific, measurable metrics. This includes a primary metric (e.g., number of key actions taken per session) and secondary/guardrail metrics (e.g., session duration, latency, error rates). Second, the implementation plan, including the user splitting mechanism (e.g., 50/50 split based on a hash of user ID), the technology used (e.g., a feature flagging service), and how to ensure a consistent user experience. Third, the instrumentation plan, detailing the specific frontend and backend events and performance timers to be captured. Finally, the analysis and decision-making framework, including required sample size, test duration (e.g., two full business weeks), and the statistical threshold for success (e.g., p-value < 0.05).
The mistakes people make
A major red flag is jumping directly into implementation details ('I'll use a feature flag...') without first defining what success looks like. Another is choosing vanity metrics like 'page views', which might increase simply because a new design is confusing. A junior answer states that if variant B gets more clicks, it wins, ignoring statistical significance, variance, or confidence intervals. Forgetting to mention guardrail metrics is also a common omission; a successful feature that doubles server costs or increases page load time by 300ms is not a success.
What usually comes next
Expect follow-ups on statistical concepts: 'How do you determine the required sample size or test duration?' or 'What is a p-value and what does a value of 0.04 actually mean?'. You might also be asked about edge cases: 'What if the results are inconclusive?' or 'How would your approach change for a native mobile app versus a web app?'. A system design follow-up could be 'Describe the high-level architecture of the feature flagging service you mentioned.'
A concrete example
To test a new dashboard, let's hypothesize it will increase the creation of 'custom reports' by 10%. We'll run a 50/50 test, splitting users based on a hash of their user ID. The primary metric is a backend event custom_report_created. Guardrail metrics include the P95 dashboard load time and the rate of JavaScript errors on the frontend. To detect a 10% lift with 95% confidence, we might need 200,000 users per variant. At our traffic levels, this means running the test for 14 days to smooth out weekly usage patterns. We'll declare a winner only if the primary metric shows a statistically significant lift and guardrail metrics do not significantly regress.
Interview question
When planning an A/B test for a redesigned dashboard to improve 'engagement,' what is the most critical initial action to take?
- a.Define 'engagement' with specific metrics and formulate a testable hypothesis.Correct
- b.Calculate the necessary sample size and test duration to achieve statistical significance.
- c.Configure a feature flag system to split traffic 50/50 between the old and new designs.
- d.Add tracking events for every clickable element on both the old and new dashboards.
Why? this is the answer
The most critical first step is defining what 'engagement' means in measurable terms. Without a clear hypothesis and metrics, implementation (A, D) is premature and statistical planning (B) is impossible.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #a/b testing
- #product engineering
- #data
- #metrics
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on a/b testing — each one lists the topics its interview covers.
See open roles