Experimentation
189 bites tagged Experimentation — interview questions with model answers, and 60-second explainers.
Novelty and Learning Effects in A/B Testing
The novelty effect is a temporary metrics lift from curious users exploring a new feature. The learning effect is the opposite: a dip as users struggle with a change. Both can mislead A/B tests if you don't run them long enough to see the true.
Randomization: The Key to Trustworthy Experiments
Randomization ensures experimental groups are similar before a test begins, like shuffling cards before dealing. This lets you confidently attribute differences in outcomes to your changes, whether in A/B tests or clinical trials.
Null Hypothesis: Assume No Effect Until Proven Otherwise
The null hypothesis is your default assumption: nothing changed. You run experiments to gather enough evidence to reject this default. In A/B testing, the null is that your new feature has no effect, while the alternative is that it does.
Minimum Detectable Effect: How Small a Change Can You See?
Minimum Detectable Effect (MDE) is the smallest change your A/B test can reliably see. You calculate it *before* a test to determine the sample size needed.
Hypothesis-Driven Development: Test Your Ideas Before You Build
Hypothesis-Driven Development treats product work as a series of experiments, not a to-do list. You state a testable belief ("If we build X, users will do Y") before writing code. This de-risks new features by validating ideas early.
The Endowment Effect: We Overvalue What We Already Own
We irrationally value things more simply because we own them. This appears in free trials that create a sense of ownership, making users less likely to cancel. The footgun is assuming users judge value objectively; they don't, and will resist switching.
Variable Rewards: The Engine of Habit
Variable rewards make products sticky by creating unpredictable payoffs, like a slot machine. This drives repeat actions in social feeds or games. The footgun is that overuse can feel manipulative and lead to user burnout or accusations of addictive design.
Nudge Theory: Shaping Choices Without Forcing Them
Nudge theory influences behavior by subtly redesigning the environment where choices are made. It's used in product design to guide users toward desired actions, like setting smarter defaults.
Anchoring Bias: The First Number You See Matters Most
The first number you see acts like a mental anchor, warping all subsequent judgments. This is used in pricing, where a high "original" price makes a sale price seem better, and in negotiations.
Fogg Behavior Model: Why Users Act (or Don't)
The Fogg Behavior Model states a behavior only happens when Motivation, Ability, and a Prompt converge (B=MAP). Use it to diagnose why a feature fails or to design new ones. The footgun is blaming low motivation when the real issue is low ability.
Social Proof: People Copy People
Social proof is the tendency to copy others' actions, assuming they reflect the correct behavior. It’s why we trust a product with thousands of positive reviews or a restaurant with a long line. The footgun: the crowd can be wrong or even faked.
Attribution Modeling: Who Gets Credit for a Conversion?
Attribution modeling decides how to credit different marketing channels for a conversion. It's like splitting a sales commission: does the first call, the final meeting, or everyone involved get credit? The footgun is using a simplistic model like last-touch.
The Pivot: A Structured Change in Strategy, Not Vision
A pivot is a structured course correction, not a restart. You change strategy based on validated learning from your MVP when data shows your initial hypothesis was wrong. The footgun is pivoting on a whim instead of data, or failing to pivot despite it.
ICE Score: A Quick Framework for Prioritizing Ideas
The ICE score is a quick framework for ranking ideas by asking: what's the potential Impact, our Confidence in the outcome, and the Ease of implementation? Growth teams use it to prioritize experiments, balancing big bets with quick wins.
High-Tempo Testing: Move Faster Than Your Channels Decay
High-tempo testing treats growth as a continuous experiment, not a one-time campaign. Since marketing channels decay quickly, this lets you find new wins across the entire user journey.
Proxy Metrics: Estimate Long-Term Impact Now
A proxy metric uses a model to estimate a slow, long-term outcome, like annual revenue. It lets you quickly judge an A/B test's impact without waiting months for the true result. The footgun is trusting a biased model or ignoring its error, giving you false.
A/B Testing: Calculating Your Sample Size
Calculating A/B test sample size ensures your results are statistically significant, not just random noise. It's vital before testing anything from button colors to headlines. The biggest footgun is stopping a test early when a variant looks like it's winning.
Control vs. Variation: The Core of A/B Testing
To know if a change works, compare it against the original. The original is your 'control,' and the new version is your 'variation.' This is the core of A/B testing, used to test new button colors or email subjects.
Multivariate Testing: Finding the Best Combination
Multivariate testing (MVT) finds the best *combination* of changes, not just the best single change. It tests multiple elements at once, like three headlines and two button colors, to see how they interact.
Sample Ratio Mismatch (SRM): When Your A/B Test Is Broken
Sample Ratio Mismatch (SRM) means your A/B test's traffic split is broken, violating random assignment. For example, a 50/50 split results in a statistically significant imbalance.
A/B Testing: Making Decisions with Data, Not Guesses
A/B testing is a controlled experiment pitting two versions of a product against each other with real users. It's used to see if a change, like a new button color, improves a metric like clicks.
Get Experimentation bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.