A/B Testing: Making Decisions with Data, Not Guesses
A/B testing is a controlled experiment pitting two versions of a product against each other with real users. It's used to see if a change, like a new button color, improves a metric like clicks.
The mental model
A/B testing replaces guesswork with data by running a scientific experiment on your users. Instead of debating whether a red or blue button is better, you can show the red button to one group of users and the blue button to another, then measure which group clicks more. It is a direct application of two-sample hypothesis testing to product development.
How it works
The process involves a few key steps. First, you form a hypothesis, such as "Changing the 'Sign Up' button from green to orange will increase registrations." Next, you create two versions: Variant A (the control, green button) and Variant B (the challenger, orange button). Then, you randomly assign incoming users to either see Variant A or Variant B. You collect data on the key metric for both groups. Finally, you use statistical analysis to determine if the observed difference in performance is statistically significant, meaning it's unlikely to be due to random chance.
When to use it
Use A/B testing for optimizing existing features where you can isolate a single variable and measure a clear outcome. It is ideal for testing changes to user interfaces, such as button colors, headlines, or page layouts. It's also effective for testing different email subject lines to improve open rates or for comparing the performance of different recommendation algorithms.
When not to use it
A/B testing is not suitable for validating entirely new products or features where there is no existing 'control' version to compare against. It's also ineffective for changes whose impact is not immediately measurable or for testing multiple variables at once, which is the domain of multivariate testing. Drastic redesigns are also poor candidates, as you can't isolate which specific change caused the outcome.
One canonical example
An e-commerce site wants to improve its conversion rate. Their hypothesis is that changing the "Buy Now" button text to "Add to Cart" will increase clicks. Variant A (control) shows "Buy Now" to 50% of traffic. Variant B (challenger) shows "Add to Cart" to the other 50%. After running the test on 20,000 users, the team finds that Variant B had a 15% click-through rate, while Variant A had 12%. Statistical analysis confirms this 3% lift is significant, so the company rolls out "Add to Cart" to all users.
Interview question
Which scenario is LEAST suited for an A/B test?
- a.Deciding between two distinct user interface layouts for a major product redesignCorrect
- b.Comparing two different headlines for a landing page
- c.Testing if a new button color increases click-through rates
- d.Evaluating the impact of a new email subject line on open rates
Why? this is the answer
A/B testing is most effective for isolating single variable changes. Major product redesigns typically involve multiple simultaneous changes, making it difficult to determine which specific element caused an observed outcome, as stated in the card.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on analytics — each one lists the topics its interview covers.
See open roles