Design an A/B test separating novelty from true long-term impact

What's really being asked

Whether the candidate understands that treatment effects are not static when users learn and adapt. Novelty, a temporary spike from curiosity, and primacy, a gradual ramp-up from adoption, both violate the stable unit treatment value assumption over time. The interviewer wants to see experimental design thinking that goes beyond a standard two-week A/B test and statistical fluency in separating time-treatment interactions from average treatment effects.

The full answer

First, design mechanics: propose a staggered rollout or a cohort-based holdout where different user segments receive the new newsfeed at different times, creating multiple treatment and control periods. Second, the analytical framework: apply a difference-in-differences estimator to compare early treated users against later treated users, using the later group as a control for the earlier group during the pre-treatment period. Third, functional form: model session duration as a function of treatment indicator, time since first exposure, and their interaction, allowing the coefficient to decay or grow. Fourth, power and sample size: acknowledge that detecting a small novelty decay requires more time or users than a standard t-test, and that the minimum detectable effect widens if you split data into weekly cohorts. Fifth, decision criteria: define a policy threshold based on the asymptotic long-term effect rather than the initial lift.

The mistakes people make

A major red flag is suggesting you simply run the experiment for 8 weeks and take the final week average. This ignores that the control group also experiences seasonality and maturation. Another error is comparing users who saw the feature early against users who saw it late without adjusting for calendar time, which confounds treatment with temporal trends. Some candidates propose surveying users for self-reported novelty; subjective metrics do not solve the causal identification problem. Finally, ignoring spillover is a mistake: if the newsfeed change affects downstream referral behavior, the control group may be contaminated.

What usually comes next

The interviewer may ask how you would validate the parallel trends assumption required for difference-in-differences when you only have observational data. They might also probe what you would do if the novelty effect interacts with seasonality, such as a holiday week. Another follow-up is how to trade off statistical power against user experience when holding back a beneficial treatment from the control group for an extended period. You should also be ready to discuss whether a simple linear time interaction is sufficient or if you need a non-parametric approach.

A concrete example

Suppose you launch the personalized newsfeed to 1 percent of users in week 1, 2 percent in week 2, and so on. For the week-1 cohort, weeks 1 through 4 are treated, while the week-4 cohort is still control during those weeks. You estimate the treatment effect for the week-1 cohort relative to the week-4 cohort in week 1, then again in week 4. If the gap shrinks from 30 seconds to 5 seconds, the difference-in-differences coefficient captures a novelty decay of 25 seconds. You then project the asymptote and compare it against the engineering cost of maintaining the new algorithm.