How do you measure impact while accounting for the novelty effect?

WHAT THIS TESTS: This question assesses your depth in experimental design beyond a simple A/B test. It probes your understanding of threats to external validity, specifically time-dependent effects like novelty (initial excitement) and primacy (learning curve). The interviewer wants to see if you can design an experiment that produces a trustworthy estimate of the long-term impact, which is crucial for making durable product decisions.

A GOOD ANSWER COVERS: A strong answer has three parts. First, acknowledge the novelty effect as a significant risk that can inflate initial metrics for the new newsfeed. Second, propose a specific experimental design to mitigate this. The primary method is a long-running A/B test, lasting several weeks or even months, not just 1-2 weeks. Third, describe the analysis. This involves plotting the treatment effect on session duration over time. A key technique is to segment users into weekly cohorts based on when they first entered the experiment. You then compare the metric for the treatment vs. control group within each cohort over the experiment's lifetime. A stable long-term effect is found if the lift observed in later weeks for early cohorts is similar to the lift observed for newer cohorts.

COMMON WRONG ANSWERS: The biggest red flag is proposing a standard, short-duration (e.g., 2-week) A/B test without mentioning the novelty effect at all. This suggests a lack of experience with real-world product changes. Another weak answer is acknowledging the novelty effect but not proposing a concrete way to measure or control for it, simply saying "we should be careful." A senior candidate must provide a specific analytical framework. Failing to mention cohorting by join date is a missed opportunity to show depth.

LIKELY FOLLOW-UPS: "How long is long enough? How do you decide when to stop the experiment?" (Answer: Monitor the cohort analysis; when the weekly lift stabilizes for several consecutive weeks, you have a good estimate). "What if we can't afford to run a multi-month experiment? Are there other methods?" (Answer: This is where you can bring in observational methods like Difference-in-Differences on post-launch data to estimate the learning effect, comparing early vs. late adopters. This has higher statistical power but relies on assumptions about time-based interactions).

ONE CONCRETE EXAMPLE: We launch the new feed algorithm in an A/B test for 8 weeks. We create 8 weekly cohorts of users. For Cohort 1 (users who joined in week 1), we see a +15% lift in session duration in their first week. By week 4, their lift has decayed to +4%. For Cohort 4 (users who joined in week 4), we see a similar +14% lift in their first week. If, by week 8, all early cohorts have stabilized at a ~4% lift, we can be confident the true long-term impact is +4%, not the initial +15% from the novelty effect.