Experimentation
189 bites tagged Experimentation — interview questions with model answers, and 60-second explainers.
Designing an in-house A/B testing platform
Config service, deterministic bucketing SDK, metrics pipeline, analysis engine; fix peeking with fixed samples or sequential tests. System design plus statistical rigor.
Sampling 5% of users for a one-time survey
Hash user ID for the 5% gate, persist a 'shown' flag, race-safe single display. Deterministic sampling plus durable seen-state at scale.
Consistent A/B bucketing across sessions and devices
Hash a stable user ID with experiment salt, not random or cookie-only; beware caches keyed without variant. Deterministic, identity-based bucketing.
Client-side versus server-side A/B testing
Client-side flickers and is bypassable; server-side assigns before response, hides logic, suits backend changes. Where variant assignment and rendering happen.
Why A/B-only, no upfront research, costs engineers more
A/B testing optimizes within a chosen direction but cannot reveal the right problem; building wrong wastes engineering cycles and tech debt. defending upfront research on engineering-cost grounds.
Confidence intervals versus p-values, explained simply
Define a CI as a plausible range for the true effect with its precision, contrast it with a binary p-value, and read [-1%, 7%] as inconclusive because it spans zero. statistical literacy for A/B decisions.
Using mission to prioritize debt vs new feature
Map each task to mission impact, time horizon, and reversibility; weigh learning value against risk. Whether you can prioritize with a strategic lens, not gut feel.
Design an automated A/B test reporting system
Standardized metric definitions, automated stats with confidence intervals and guardrails, segment breakdowns, a clear ship recommendation. scalable experiment reporting design.
Framing ad-load tradeoffs: revenue versus retention
Define revenue plus guardrail engagement metrics, run a long-enough experiment to see retention effects, and weigh short-term lift against lifetime-value erosion. balancing competing metrics over time.
Does forcing profile completion cause retention?
Name the confounder (engaged users self-select into completing profiles), warn that forcing it may not transfer the effect, and propose a randomized experiment. distinguishing correlation from causation.
Build a simple A/B test for a headline
Assign each visitor a sticky bucket, serve the matching headline variant, log impressions and conversions per variant. basic A/B test mechanics. re-randomizing on every load so a user sees both variants.
Use Difference-in-Differences without an A/B test
Give a scenario like a region-wide launch, apply Difference-in-Differences comparing treated vs control over time, and state the parallel-trends assumption. causal inference when randomization is impossible.
Determine A/B test sample size
Define baseline rate, minimum detectable effect, significance (alpha), and power (1-beta); smaller effects and stricter thresholds need more users. the inputs to a power calculation. ignoring power or treating MDE as fixed.
Interpret a p-value in an A/B test
Define it as P(data this extreme | null true), interpret 0.03 against a 0.05 threshold, and state what it is NOT. correct meaning of a p-value. claiming a 3% chance the result is wrong or that the effect is.
Implementing a consistent-assignment A/B test
Need an assignment service, exposure logging, and event tracking; ensure stickiness by hashing a stable user id; analyze conversion per variant. End-to-end experiment engineering. Bucketing on session or device.
The multiple comparisons problem in A/B testing
Many tests at alpha 0.05 inflate the chance of a false positive; mitigate with Bonferroni or FDR control plus pre-registered metrics. Statistical rigor at scale. Cherry-picking whichever metric crosses p<0.05.
Designing a self-serve experimentation platform
SDK with sane defaults, automated pre-launch validation, sample-ratio and guardrail-metric checks. building safe experimentation as a platform, not a service.
Dynamic personalized onboarding architecture
A segmentation pipeline, a serving layer choosing task order per segment, an experimentation engine, and a feedback loop measuring activation. closing the loop from data to decision to learning.
Components of a testable A/B hypothesis
A specific change, a predicted directional effect on one primary metric, a rationale, and a measurable success threshold. experimental rigor before launch.
Design a centralized experimentation service
A config/assignment API, deterministic SDK-side bucketing, and a separate exposure-logging pipeline. platform thinking for shared experimentation.
Set up a client-side button color A/B test
Stable hashing of a persistent ID into buckets, conditional rendering of the variant, exposure plus click event logging. practical experiment wiring on the client. re-randomizing on each load so users flicker between variants.
Handle interaction effects on a shared page
Combined variants may produce effects neither has alone; use mutual exclusion for likely interactions, orthogonal designs with interaction monitoring otherwise. reasoning about interaction effects and mitigation.
Prevent conflicting experiments with layers
Group conflicting experiments into one layer so a user's per-layer bucket maps to at most one of them; orthogonal layers can overlap. enforcing mutual exclusion via layered assignment.
Run concurrent experiments without interference
Independent non-interacting tests can share traffic through orthogonal layers; interacting ones need mutual exclusion in a shared layer. designing for parallel experiments via layered infrastructure.
Get Experimentation bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.