Cohort analysis for an onboarding change

WHAT THIS TESTS Whether you can define a cohort precisely and design a comparison that attributes a retention change to the onboarding flow rather than to time, mix, or seasonality.

A GOOD ANSWER COVERS A cohort is a set of users who share a common starting characteristic within a defined window, most often an acquisition cohort grouped by signup date. Cohort analysis tracks how that fixed group behaves as it ages, which separates the effect of tenure from calendar effects. To measure the January 1 onboarding launch, form two acquisition cohorts: users who signed up in a window before January 1 experienced the old flow, and users who signed up after experienced the new flow. For each cohort, compute retention and a leading activation metric such as reaching the aha moment at equal ages, day 1, day 7, day 30, and plot retention curves on a relative time axis of days since signup so the cohorts are compared at the same lifecycle stage. Improvement appears as the new cohort's curve sitting above the old one.

COMMON WRONG ANSWERS Comparing January total active users to December total active users, which conflates cohorts of every age and is polluted by seasonality and acquisition volume. Aligning on calendar date instead of cohort age. Ignoring that marketing or holiday traffic may change the type of user acquired, a selection confound. Looking only at a single retention point instead of the curve.

LIKELY FOLLOW-UPS How long must you wait to read day-30 retention? How do you control for a simultaneous marketing campaign? Could you randomize instead for cleaner causality?

ONE CONCRETE EXAMPLE You bucket signups by week, label them pre and post January 1, and chart day-7 retention by cohort age. The post cohorts show day-7 retention rising from 22 to 31 percent and holding, suggesting the new onboarding improved activation, while you also inspect February cohorts to rule out a January-only seasonal bump before concluding the change worked.