Design an experiment to isolate long-term impact from novelty effect

What's really being asked

This question tests your understanding that short-term experiment results can be misleading. The interviewer is looking for your ability to design experiments that produce trustworthy, long-term metrics. It's a test of statistical maturity and your awareness of second-order effects like user learning (novelty and primacy effects), which are critical for making durable product decisions rather than just chasing short-term lift.

The full answer

First, explicitly acknowledge the problem: a simple A/B test comparing group means over two weeks might capture a temporary novelty effect, leading to an overestimation of the true, sustainable impact. Mention that the opposite, a primacy effect, can also occur where a feature's value grows as users overcome a learning curve.

Second, propose an experimental design focused on time. The most direct method is a long-running A/B test (e.g., 4-8 weeks instead of 2). The key is to analyze results not just in aggregate, but by cohorting users based on their tenure in the experiment (e.g., day 1 users, day 2 users, etc.). This allows you to plot the treatment effect over time.

Third, detail the statistical analysis. Instead of a simple t-test on the overall means, you would model the treatment effect as a function of time. A good approach is to plot the daily or weekly lift and look for when it stabilizes. A more advanced answer would propose a regression model with an interaction term between the treatment dummy variable and a time variable (e.g., session_duration ~ treatment * days_in_experiment). This formally tests if the treatment effect changes over time and estimates its final, stable value.

Fourth, define the long-term impact. The long-term impact is the asymptotic value of the treatment effect after the initial novelty or primacy period has worn off. In your time-series plot, this is the value where the curve flattens out.

The mistakes people make

A major red flag is simply describing a standard A/B test without addressing the time component. This shows the candidate missed the entire point of the question. Another common mistake is suggesting a very short experiment duration (e.g., "one week is enough") or being unable to propose a specific analytical method beyond "we'll watch the graph." Confusing novelty effects with weekly seasonality is another sign of inexperience; a good answer would control for seasonality while isolating the novelty decay curve.

What usually comes next

How long is "long enough" to run the test? The answer is not a fixed number of weeks, but "until the treatment effect curve stabilizes." You monitor the daily lift estimates and their confidence intervals until they stop trending up or down. What if we can't afford a long-running experiment? A sophisticated answer mentions observational methods like Difference-in-Differences (DiD) on pre/post exposure data, which can help estimate long-term effects from shorter experiments, though they come with their own assumptions.

A concrete example

For the new newsfeed, we run a 6-week A/B test. In week 1, the treatment group shows a +12% lift in session duration. We plot the daily lift. By week 4, the lift has decayed and stabilized around +3.5%. We can confidently say the initial +12% was inflated by a large novelty effect. The true, long-term expected impact is +3.5%. If we had made a ship decision based on the week 1 results, we would have grossly overestimated the feature's value and potentially misallocated future resources.