Experimentation
189 bites tagged Experimentation — interview questions with model answers, and 60-second explainers.
What is a p-value in A/B tests and what does threshold mean?
P-value is the chance of this or more extreme result if the null is true; the 5% threshold is the accepted false-positive rate. your grasp of frequentist testing mechanics. saying it is the probability the variant wins.
How do you set up a button color A/B test?
Tests whether you can define a valid experiment, not just a color change. A strong answer covers hypothesis, randomization, primary metric, sample size, and significance threshold.
WAU is flat despite positive A/B tests; why and how to diagnose
This tests distinguishing real impact from statistical artifacts. Strong answers cite false positives from low base rates, peeking, novelty, and local-global mismatches. Diagnose with long-term holdouts, audits, and causal bridges.
How would you structure your growth team's experimentation portfolio?
3 asset classes (iterative 30-70%, tech investments, big bets 20-40%), use expected value per week, and evolve the mix. Capital allocation across risk classes in growth. Gut voting or flat effort without expected ROI.
When increasing CTA clicks, what side-effects and guardrails should you consider?
This tests balancing growth with business safety. Name guardrails like retention, revenue, fraud, load time; set NI thresholds pre-launch; and include SRM checks. A red flag is treating guardrails as optional success metrics instead of hard stop limits.
Develop a testable hypothesis for a 40% email verification drop-off
This tests structured hypothesis formation under uncertainty. Strong answers: segment the 40% drop by device and latency; build a Customer Theory from data; isolate one lever; draft a four-part MECLABS hypothesis. Red flag: skipping diagnosis to guess fixes.
What framework decides between low-effort/low-impact and high-effort/high-impact experiments?
This tests structured experiment sequencing beyond gut instinct. A strong answer picks ICE, RICE, or PIE; scores both experiments by impact, confidence, and effort or reach; then weighs opportunity cost and bandwidth.
Which three data sources would you analyze to improve activation?
This tests whether you ground hypotheses in diverse evidence before experimenting. A strong answer names qualitative feedback, funnel metrics, and behavioral analytics as distinct inputs.
Design a system that detects choice paralysis and dynamically simplifies the interface
Track hover entropy, scroll jitter, and time-to-click; use a contextual bandit to select simplification tiers. real-time behavioral inference with safe UI adaptation.
Design an A/B test for loss aversion versus gain framing at checkout
Tests whether you can isolate framing effects from checkout confounders. Strong answers detail user-level randomization, event logging, and guardrail metrics like revenue per visitor. Red flag: a conversion-only analysis with no unit of diversion defined.
How do you architect a global notification holdback group?
Tests persistent control-group isolation without breaking critical flows. Strong answers use deterministic sticky bucketing by user ID, separate marketing and transactional namespaces, and audit holdout bleed.
How do you track events and calculate funnel drop-off?
Tests event mapping and funnel math. Answer: one event per step (page view, email submit, profile done); compute relative step conversion and overall rate from top; deduplicate users. Red flag: raw pageviews instead of uniques.
How do you attribute signups to Facebook, Google, and blog campaigns?
This tests URL-based campaign attribution and session persistence. A strong answer covers UTM tags, landing page parsing, cookie storage, and writing the source to the user record at signup.
How would you architect long-term holdback experiment groups?
Tests longitudinal causal inference and engineering tradeoffs for multi-month isolation. A strong answer covers bucketing, delayed metrics, and cross-experiment guards. Red flag: daily re-randomization or ignoring survivorship bias in aging cohorts.
How do you design allocation logic to minimize concurrent A/B test interactions?
Compare mutual exclusion with layered randomization via layers and reservations; stress isolation vs throughput. Designing allocation to isolate concurrent experiments.
Describe RICE scoring and architect data pipelines for Reach and Effort
Define RICE; automate Reach via event streams with time windows; automate Effort from PM tool estimates with calibration. Bridging prioritization and data systems for Reach and Effort.
What fields belong in an experiment tracking event?
Tests disciplined schema thinking over random fields. Strong answers cite a tracking plan with event and user properties, environment separation, and consistency. Red flag: dumping data without a schema or single source of truth.
What counter metrics track health of weekly active users?
Tests whether you can spot growth-at-all-costs blind spots. A strong answer pairs WAU with 7-day retention, sessions per user, and error rate, mapping each to churn, shallow engagement, or bugginess.
Embedded Growth Teams: Experimentation as a Squad Muscle
Growth talent embedded in product squads spreads experimentation beyond a central team. It fits multi-surface products needing data-driven culture. The footgun is letting embedded specialists become sole experimenters while squad PMs and engineers disengage.
Log Exposure, Not Just Assignment
Exposure logging records when a user actually sees a variant, not just assignment. Counting assigned but unexposed users dilutes treatment effects and biases lift. The common footgun is logging assignment as exposure, which silently underpowers your test.
Revenue Recognition in Growth Experiments
Revenue is earned as you deliver value, not when cash arrives. In growth experiments, booking an annual prepayment as immediate revenue makes a pricing test look like a massive win. Teams often confuse bookings with revenue and credit experiments incorrectly.
Golden Path: The One Journey That Matters
Golden Path is the single user journey that drives core value. In growth, you optimize this highway before fixing side roads. The footgun is A/B testing edge cases while your main funnel leaks users.
Bayesian vs. Frequentist A/B Testing
Frequentist testing asks how surprising a result is if nothing changed; Bayesian asks probability B is better. Frequentist fixes sample size to control false positives, while Bayesian lets you peek.
Scarcity Principle: Limit Availability to Drive Action
People want what they might lose. In growth, scarcity amplifies conversion by framing offers as limited in time, quantity, or access. The footgun is fake scarcity: invented limits destroy trust and backfire permanently.
Get Experimentation bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.