Null Hypothesis: Assume No Effect Until Proven Otherwise
The null hypothesis is your default assumption: nothing changed. You run experiments to gather enough evidence to reject this default. In A/B testing, the null is that your new feature has no effect, while the alternative is that it does.
Why it exists
The null hypothesis framework was created to bring scientific rigor to testing claims. It protects us from confirmation bias—the tendency to see patterns in random noise. By forcing us to assume there is no effect by default, it sets a high bar for proving that a change actually made a difference.
The mental model
Think of a court trial. The null hypothesis is the defendant, presumed innocent (meaning, no effect occurred). The alternative hypothesis is the prosecution's claim that the defendant is guilty (an effect did occur). Your experimental data is the evidence. You need to present enough evidence to convince the jury 'beyond a reasonable doubt' (i.e., reach statistical significance) to reject the presumption of innocence.
How it works
You start by defining two opposing statements. The null hypothesis (H₀) states there is no difference between groups (e.g., 'the new button has the same click rate as the old one'). The alternative hypothesis (H₁) states there is a difference. After collecting data, you perform a statistical test to determine the probability (the p-value) that you'd see your results if the null hypothesis were true. If this probability is very low (typically under 5%), you reject the null hypothesis.
When to use it
This is the foundation of all statistical hypothesis testing, especially A/B testing. Use it whenever you need to determine if an observed effect is real or just due to random chance. It's for answering questions like, 'Did this new feature actually increase engagement?' or 'Is this new algorithm truly faster?'.
When not to use it
This formal framework isn't necessary for purely exploratory analysis where you're just looking for interesting patterns without a specific question. It's also overkill if an effect is so large and obvious that the outcome is certain without statistical validation.
One canonical example
An engineering team wants to know if a new caching strategy reduces API response time. Null Hypothesis (H₀): The average response time with the new cache is the same as the old cache. Alternative Hypothesis (H₁): The average response time with the new cache is different from the old cache. After a test, they find the new cache is faster and the result has a p-value of 0.01. Since 0.01 is less than their significance threshold of 0.05, they reject the null hypothesis and conclude the new caching strategy has a statistically significant effect on performance.
Interview question
What is the fundamental role of the null hypothesis in scientific testing?
- a.To provide a default position of no effect or no difference that requires strong evidence to overturn.Correct
- b.To ensure that the experiment is designed to minimize bias and maximize accuracy.
- c.To confirm that the experimental results are statistically significant.
- d.To predict the specific outcome that the researcher expects to find.
Why? this is the answer
The null hypothesis establishes a baseline assumption that there is no effect or difference, acting as a default that must be disproven with sufficient evidence. Option D describes the alternative hypothesis, which represents the effect the researcher aims to demonstrate.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on experimentation — each one lists the topics its interview covers.
See open roles