The Novelty Effect: When New Isn't Always Better
The Novelty Effect is a temporary metric spike from a feature's newness, not its inherent value. It often appears in A/B tests for high-frequency products, inflating short-term metrics. The footgun is mistaking this initial excitement for a long-term win.
The mental model
The Novelty Effect is a temporary metric spike caused by a feature's newness, not its inherent value. Think of it as user curiosity. A redesigned button might get more clicks just because it's different, not because it's functionally better. This initial excitement often fades as the newness wears off, revealing the feature's true, sustained impact.
How it works
A novelty effect is a real treatment effect, not a statistical error or bias. It's a genuine user response to change. The danger lies in extrapolation. If you run a one-week experiment and see a 20% lift, you might incorrectly assume that lift will hold for a year. In reality, the effect might be strong initially but disappear completely as users get accustomed to the change. The best way to identify it is by plotting the treatment effect over time and observing if the initial lift stabilizes or decays.
When to look for it
Look for novelty effects in high-frequency products where users encounter changes often, like social media feeds or e-commerce sites. The effect is most visible in surface-level, short-term metrics like click-through rates (CTR). A sudden increase in CTR can mean users found value, but it can also just mean they were attracted by something new and shiny. Disentangling the two is key.
When not to use it
Do not use short-term results dominated by novelty to make long-term product decisions. More importantly, do not create incentive structures, like KPIs, that reward teams for generating novelty spikes. This encourages superficial changes over meaningful improvements. Finally, never try to 'correct' for novelty using statistical methods. It is a real behavior to be understood and observed over a longer duration, not an error to be removed from your data.
One canonical example
A restaurant you pass daily improves its menu and service by 100%. You might not even notice. But if it changes its name, you'll likely be intrigued and go inside to see what's new. A responsible owner wouldn't change the name every week. But if a manager's bonus is tied to a quarterly KPI for daily visits, they are incentivized to make superficial changes that create novelty spikes rather than investing in fundamental improvements.
Interview question
Which approach is most effective for identifying if an A/B test's metric lift is a Novelty Effect?
- a.Applying statistical adjustments to the initial data to normalize for newness.
- b.Relying on qualitative user feedback to assess initial user excitement.
- c.Plotting the treatment effect over an extended period to observe its stability or decay.Correct
- d.Ending the experiment early to capture the highest observed metric increase.
Why? this is the answer
The card states, "The best way to identify it is by plotting the treatment effect over time and observing if the initial lift stabilizes or decays." This method directly reveals if the initial excitement is temporary. Option A is incorrect because the card explicitly advises against trying to 'correct' for novelty using statistical methods.
Just read this? Test yourself on what you have been reading.
Read the original → statsig.com
- #a/b testing
- #product analytics
- #metrics
- #user behavior
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on a/b testing — each one lists the topics its interview covers.
See open roles