How would you validate that early Project creation drives retention?

WHAT THIS TESTS: Your ability to structure a testable growth hypothesis using the four required components, design a technically sound validation around a baseline and success metric, and avoid the common mistake of confusing correlation with causation. Interviewers want to see that you define the expected change, metric, timeframe, and baseline before analyzing data, and that you know when to move from observational analysis to an A/B test.

A GOOD ANSWER COVERS: First, define the hypothesis in a testable statement using the four components: the expected change is creating a first Project within 24 hours, the success metric is long-term retention such as D30 or D90, the timeframe is the observation window after signup, and the baseline is the current retention rate for users who do not meet the 24-hour threshold. Second, identify the exact data needed: user signup timestamp, first Project creation timestamp, user attributes such as acquisition channel or role, and any pre-existing intent signals. Third, describe the analysis. Cohort users by weekly signup buckets to establish a stable baseline, then compare retention curves for the 24-hour group versus the control group. Because users who create a Project quickly may have higher intent, you must control for confounders rather than treating the raw correlation as causal. Fourth, present absolute and relative lift against the baseline, include confidence intervals and sample sizes to show statistical rigor, and check for segment heterogeneity. Fifth, close the causal gap by recommending an A/B nudge experiment, such as an in-app prompt to create a Project, so you compare outcomes to the baseline under randomized conditions and decide whether to scale, tweak, or drop the idea.

COMMON WRONG ANSWERS: Jumping straight to building a feature without defining the four hypothesis components or establishing a baseline. Running a simple query that counts retained users by a 24-hour flag and calling it proof while ignoring self-selection bias. Failing to check sample size and statistical power, which Growth-onomics notes is a common testing error. Testing multiple changes at once, such as simultaneously redesigning onboarding and measuring Project creation. Using vague retention definitions that make the hypothesis untestable.

LIKELY FOLLOW-UPS: How would you design the A/B nudge experiment, and what would be the primary and guardrail metrics? What is the minimum sample size you need if baseline D30 retention is 20 percent and you expect a 2 to 5 percent lift? How would you handle weekday versus weekend signups in your cohort analysis? Would you use survival analysis instead of point-in-time retention, and why?

ONE CONCRETE EXAMPLE: Suppose your baseline D30 retention for users who do not create a Project within 24 hours is 20 percent. You cohort 10,000 January signups by week and find the 24-hour group retains at 35 percent. After controlling for acquisition channel and role, the marginal lift is 10 percentage points with a 95 percent confidence interval of 6 to 14 points. You present the cohort retention curves, the controlled lift, and the sample size calculation, then recommend an A/B onboarding prompt test to measure causal impact before scaling.

Source: growth-onomics.com