What metrics and models link onboarding to long-term retention?

WHAT THIS TESTS: This question evaluates whether you can design a rigorous longitudinal study that connects a product intervention to a lagged business outcome. The interviewer wants to see that you understand the difference between short-term proxy metrics and actual retention, that you recognize censoring and temporal confounding as core threats to validity, and that you can select statistical machinery appropriate for time-to-event data rather than defaulting to standard A/B test tooling.

A GOOD ANSWER COVERS: First, a metric hierarchy. You should name activation proxies such as onboarding completion rate, time-to-first-value, or feature adoption within session one, then ladder up to hard retention outcomes like Day 7, Day 30, and Day 90 return rates. Second, an experimental or quasi-experimental design. If randomized, a staggered rollout by cohort preserves power while mitigating risk; if observational, you need propensity scoring or a synthetic control. Third, a survival model. Propose Cox proportional hazards to estimate hazard ratios between treatment and control while handling right-censoring, or an Accelerated Failure Time model if the proportional hazards assumption fails. Mention time-varying covariates to capture changing user context. Fourth, a difference-in-differences or panel component if seasonality or network effects threaten parallel trends, and explicitly state how you would test the parallel trends assumption. Fifth, practical controls: segment by platform, traffic source, and user tenure, and account for seasonality by including calendar fixed effects.

COMMON WRONG ANSWERS: A red flag is proposing a simple pre-post t-test or a logistic regression on a binary thirty-day retention flag because these ignore the continuous nature of time and the fact that many users have not yet reached the thirty-day window. Another red flag is listing vanity metrics like page views or clicks without tying them to a retention mechanism. Suggesting ordinary least squares on raw retention days is also problematic because it mishandles censored users and violates distributional assumptions.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle staggered rollouts where different cohorts receive the treatment at different times, how you would power the study given low base rates for ninety-day retention, or how you would interpret a non-proportional hazard. They might also probe whether you would use a multi-armed bandit instead of a fixed horizon experiment, or how you would attribute retention to the onboarding flow versus later product changes.

ONE CONCRETE EXAMPLE: Suppose you redesign onboarding for a fintech app. You randomize ten thousand new users per week into treatment and control. Your primary outcome is days until first lapse, defined as seven consecutive days without opening the app. You fit a Cox model with treatment as the main predictor, adding time-varying covariates for push-notification opt-in and first deposit. You include week-of-entry fixed effects to absorb seasonality. If the hazard ratio is zero point seven five and the ninety-five percent confidence interval excludes one, you conclude the new onboarding reduces lapse risk by twenty-five percent, but you validate this with a Kaplan-Meier plot and a log-rank test before presenting to leadership.