Cohort Analysis for a New Onboarding Flow

What's really being asked

This question tests your ability to apply a core analytics concept to a practical business problem. The interviewer isn't just looking for a definition; they want to see if you can design a sound experiment to measure the impact of a product change. It shows you can move beyond misleading aggregate metrics to find the real signal in user behavior data.

The full answer

An excellent answer has four parts. First, define a user cohort clearly: a group of users who share a common characteristic, such as their sign-up date, whose behavior is tracked over time. Second, specify the type of analysis: an acquisition cohort analysis. Third, define the specific cohorts for comparison: the 'treatment' group of users who signed up on or after January 1st and experienced the new onboarding, and a 'control' group of users from a comparable period before the launch, like December. Fourth, state the primary metric and timeframe: you will track user retention (e.g., Week 1, Week 2, Week 4) for both cohorts to see if the post-launch group retains at a higher rate.

The mistakes people make

A common mistake is to suggest looking at the overall retention rate before and after January 1st. Aggregate metrics like this are noisy and can be influenced by many factors other than the onboarding flow, hiding the true impact. Another red flag is comparing incomparable groups, for instance, comparing January sign-ups to sign-ups from a year ago without accounting for seasonality or business growth. A weak answer also fails to mention tracking the behavior 'over time', which is the defining characteristic of cohort analysis.

What usually comes next

Expect questions that test your analytical depth. For example: "What if Week 1 retention improves, but your activation rate drops? How do you decide if the change was successful?" or "How would you ensure your results are statistically significant?" They might also ask about other types of cohorts, such as behavioral cohorts (users who performed a specific action), to test the breadth of your knowledge.

A concrete example

To make your answer concrete, use numbers. You could say: "We'd create a cohort chart. If our December cohort (pre-launch) had a Week 1 retention of 32% and our January cohort (post-launch) has a Week 1 retention of 45%, that 13-point lift is a strong positive signal. We would then continue tracking to see if this improvement holds at Week 4 and Week 8, proving a long-term impact on user value."