Set up a cohort analysis for a new onboarding flow

What's really being asked

This question tests your ability to translate a product change into a rigorous, data-driven experiment. The interviewer wants to see if you can move beyond a simple definition to design an analysis that isolates the impact of a specific feature launch. They are evaluating your grasp of experimental design, your precision with metrics, and your ability to connect product work to business outcomes like retention.

The full answer

A strong answer has four parts. First, a concise definition: a cohort is a group of users sharing a common characteristic, like their sign-up date, tracked over time. Second, the experimental setup: define an acquisition cohort of "treatment" users who signed up on or after Jan 1st and a "control" cohort from before the launch (e.g., all of December). Third, the success metric: choose a specific, quantifiable metric tied to onboarding success, like "Week 1 Retention" or "Day 7 Activation Rate" (defined as completing a specific key action). Fourth, the analysis: track these cohorts for 4-8 weeks and compare their retention curves, looking for a clear, sustained lift in the January cohort's metrics versus the December cohort's baseline.

The mistakes people make

The biggest red flag is failing to define a control group. Analyzing the January cohort in isolation tells you nothing about the new flow's impact. Another common mistake is using vague metrics like "improved engagement." A senior answer names a specific, measurable KPI like "W1 retention rate." Comparing January vs. February is also weaker than a clean before/after split around the launch date, as it introduces more confounding variables like seasonality. Just defining the term without explaining the practical setup is a junior-level answer.

What usually comes next

"What if you see a 2% lift? Is that good?" (Tests understanding of statistical significance and business context). "What confounding variables might affect your analysis?" (Tests awareness of seasonality, marketing campaigns, or concurrent feature launches). "How would you visualize this data for a product manager?" (Tests communication skills, expecting a description of the classic cohort retention table/heatmap).

A concrete example

We'd compare the December 2023 cohort (control) to the January 2024 cohort (treatment). Our primary metric would be Week 1 retention. If the December cohort had a W1 retention of 35% and the January cohort shows 42%, that's a positive signal. We would continue to track both cohorts for at least 8 weeks to see if this lift is sustained. We'd present this as a table with rows for each cohort and columns for Week 0, Week 1, Week 2, etc., showing the percentage of users remaining active.