Control vs. Variation: The Core of A/B Testing
To know if a change works, compare it against the original. The original is your 'control,' and the new version is your 'variation.' This is the core of A/B testing, used to test new button colors or email subjects.
Why it exists
Without a control, you're just guessing. If sales go up after you change your website, was it your change, a holiday, or a competitor's outage? A control group isolates the impact of your change from random noise, letting you prove cause and effect instead of just correlation.
The mental model
Think of a classic science experiment. The 'control' is your baseline, the existing version you don't change. The 'variation' is the new idea you want to test. By randomly showing the control to one group of users and the variation to another similar group at the same time, you can confidently attribute any difference in behavior directly to your change.
How it works
A/B testing is the most common application. First, you define a hypothesis, like 'Changing the checkout button from blue to green will increase purchases.' Second, you create two versions: Version A (the control) with the blue button, and Version B (the variation) with the green button. Third, you randomly direct a portion of your users to each version. Finally, you use statistical analysis to determine if the difference in purchases is significant enough to declare a winner.
When to use it
Use this framework for any change where you want to measure its impact on a specific user behavior. This is ideal for optimizing a single, measurable goal (a 'key metric') like click-through rate, conversion rate, or sign-ups. Examples include testing website copy, button placement, email subject lines, and pricing models.
When not to use it
Avoid this for changes that can't be easily isolated or measured, like a complete brand redesign where everything changes at once. It's also ineffective for very low-traffic sites, as you won't have enough data to get a statistically significant result. The goal is to test one variable at a time; testing a new headline and a new image simultaneously makes it impossible to know which one drove the result.
One canonical example
An e-commerce site wants to test if adding customer reviews to product pages increases sales. The control group sees the page without reviews. The variation group sees the same page with reviews included. The site then compares the conversion rate between the two groups. If the variation group's conversion rate is statistically higher, the change is implemented for all users.
Interview question
What is the primary purpose of comparing a 'control' against a 'variation' in an A/B test?
- a.To identify and fix any technical issues present in the new version.
- b.To ensure that the test collects a sufficient amount of data.
- c.To allow users to express their preference between the two options.
- d.To isolate the true cause-and-effect relationship of the change.Correct
Why? this is the answer
The core purpose of comparing a control and a variation is to isolate the specific impact of the change, allowing you to prove cause and effect rather than just correlation. While collecting sufficient data is important, it's not the reason for the comparison itself.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on a/b testing — each one lists the topics its interview covers.
See open roles