Experimentation
189 bites tagged Experimentation — interview questions with model answers, and 60-second explainers.
What a Head of Growth Actually Does
A Head of Growth is a systems architect for revenue, deciding where to place strategic bets, not just a campaign manager. They align product, engineering, and marketing around shared metrics and build the experimentation engine.
Growth Engineer: The Entrepreneur Inside the Company
A growth engineer is an entrepreneurial developer using code to directly drive business metrics. They work across the product, optimizing onboarding or running experiments to boost activation.
Holdback Groups: Measure Your True Cumulative Impact
A holdback group is your product's long-term control, shielding a small set of users from all new features. Use it to measure the cumulative impact of many small changes over months or years, beyond what individual A/B tests can show.
User Bucketing: The Engine of A/B Testing
User bucketing is a deterministic hash that assigns users to experiment groups. It ensures a user sees the same variation every time, which is critical for valid A/B tests. The main footgun is using an unstable user ID, which can re-bucket users and ruin your.
Guardrail Metrics: Don't Win the Battle to Lose the War
Guardrail metrics are your experiment's safety net, ensuring a win in one area doesn't cause a loss elsewhere. During A/B tests, you monitor them to catch negative side effects on site speed or revenue.
The PIE Framework: Prioritizing What to Test Next
The PIE framework ranks A/B test ideas by scoring them on Potential, Importance, and Ease. It helps teams decide where to experiment first, focusing limited resources on high-impact changes.
Managing Growth with an Experiment Backlog
An experiment backlog is a prioritized list of testable ideas that turns strategy into action. Agile teams use it to manage projects, using business intelligence to measure performance and ensure ideas deliver value. The footgun is a lack of governance.
Growth Loops: The Engine of Product Growth
A growth loop is a closed system where outputs are reinvested to generate more inputs, creating compounding growth. Unlike linear funnels, this model unifies product, marketing, and monetization. The footgun is using funnels, which creates silos.
Growth Hacking: Marketing via Rapid Experimentation
Growth hacking treats marketing like a science experiment, using rapid, low-cost tests to find what grows a user base instead of relying on big, slow campaigns. It's used by startups to A/B test ideas before investing resources.
Growth Squads: Centralized vs. Decentralized
A growth squad is a trade-off between speed and cultural change. A centralized team optimizes for velocity with dedicated members, while a decentralized (embedded) team spreads the growth mindset by borrowing staff.
Experiment Knowledge Base: Don't Repeat Failed Tests
An Experiment Knowledge Base is your company's collective memory for A/B tests, centralizing hypotheses, results, and learnings. It prevents re-running old tests and surfaces past insights.
The Independent Growth Team Model
An independent growth team is a startup-within-a-company, given autonomy to run experiments across the funnel. Use it to break silos and accelerate learning. The main footgun is isolation, creating a rogue unit whose wins are difficult to integrate.
Growth Meeting Cadence: Focus on Learnings, Not Updates
Run your weekly growth meeting like a learning synthesizer, not a status report. Focus on extracting insights from experiments to drive future impact. The biggest mistake is wasting time on "what" you're doing; handle status updates asynchronously.
Growth Product Manager: Driving Metrics, Not Just Features
A Growth PM is a business optimizer for an existing product. They focus on moving a single metric like user activation or retention, often through rapid experimentation. This role is key in product-led companies where the product must sell itself.
What is an Experimentation Stats Engine?
A stats engine is the brain of an A/B testing platform, turning raw data into reliable 'which version won?' decisions. It powers tools that analyze feature rollouts, ensuring statistical rigor.
Server-Side Experimentation: Testing Your Backend Logic
Server-side experimentation renders A/B test variations on the server before sending the page. Use it for testing deep backend logic like search algorithms or to avoid the visual 'flicker' of client-side tests. The footgun: it requires developer cycles.
Causal Impact: Measuring Effects Without an A/B Test
Causal Impact estimates an intervention's effect by modeling a 'what if' counterfactual. It's used to measure lift from a new feature or ad campaign when a clean A/B test isn't possible.
Survival Analysis: Predicting When, Not Just If
Survival analysis predicts *when* an event like user churn will happen, not just *if*. It's used to model customer lifetime or hardware failure rates. The key mistake is using simple averages, which are skewed by users who haven't churned yet (censored data).
In-Product Discovery: Finding Growth Inside Your App
In-product discovery is finding your next growth lever by observing users within your live product, not just in pre-launch research. It's used to scale existing products via in-app experiments.
Product Bumps: Testing Prices with Temporary Increases
A product bump is a temporary price increase for an in-app purchase to test user price sensitivity. It helps you find the optimal price without permanently changing it for everyone. The footgun is misinterpreting statistical noise from small test groups.
Conjoint Analysis: What Features Do Users *Really* Value?
Stop asking users what they want; make them choose. Conjoint analysis reveals true priorities by forcing trade-offs between product features, like price vs. battery life. It's used for pricing and roadmapping.
Price Elasticity: How Price Changes Affect Demand
Price elasticity measures how sensitive sales are to price changes. An elasticity of -2 means a 1% price increase causes a 2% drop in quantity sold. It's key for forecasting revenue from price tests, but the biggest footgun is assuming this ratio is constant.
Bonferroni Correction: Raising the Bar for Significance
The Bonferroni correction prevents finding false positives when running many tests by making your significance threshold stricter for each one. It's used in A/B tests with multiple variants.
Regression to the Mean: Why Outliers Settle Down
Extreme results are part skill, part luck. Regression to the mean is the principle that luck evens out, so a follow-up measurement will be closer to the average. This impacts A/B tests and performance analysis.
Get Experimentation bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.