Set up a client-side button color A/B test
WHAT IT TESTS: practical experiment wiring on the client. OUTLINE: stable hashing of a persistent ID into buckets, conditional rendering of the variant, exposure plus click event logging. RED FLAG: re-randomizing on each load so users flicker between variants.
Attribute a mobile install to a desktop ad
WHAT IT TESTS: cross-device attribution methods and their trade-offs. OUTLINE: deterministic matching via a shared login is accurate but needs auth on both ends; probabilistic fingerprinting scales without login but is noisy and privacy-fraught.
Handle interaction effects on a shared page
WHAT IT TESTS: reasoning about interaction effects and mitigation. OUTLINE: combined variants may produce effects neither has alone; use mutual exclusion for likely interactions, orthogonal designs with interaction monitoring otherwise.
Why repeatedly extending a test inflates false positives
WHAT IT TESTS: understanding the peeking problem and inflated false positives. OUTLINE: repeatedly checking and extending until significance is p-hacking via optional stopping, which inflates the false-positive rate; fix with fixed sample sizes or sequential…
Architect a configurable, goal-based onboarding flow
WHAT IT TESTS: building a data-driven, configurable flow rather than hardcoded branches. OUTLINE: capture the goal, let the backend return a server-driven flow definition mapping goal to steps and content, render generic components on the client.
Increase experiment velocity for non-engineers
WHAT IT TESTS: decoupling experiment changes from release cycles safely. OUTLINE: server-driven config, feature flags, and a self-serve UI let non-engineers ship copy or layout variants instantly; add guardrails and metric checks.
Prevent conflicting experiments with layers
WHAT IT TESTS: enforcing mutual exclusion via layered assignment. OUTLINE: group conflicting experiments into one layer so a user's per-layer bucket maps to at most one of them; orthogonal layers can overlap.
Design a contamination-safe pricing experiment
WHAT IT TESTS: choosing a randomization unit that hides price differences. OUTLINE: user-level price tests leak via fairness perception, so use geo holdouts or time-based cohorts where everyone in a unit sees one price.
Run concurrent experiments without interference
WHAT IT TESTS: designing for parallel experiments via layered infrastructure. OUTLINE: independent non-interacting tests can share traffic through orthogonal layers; interacting ones need mutual exclusion in a shared layer.
Design a referral feature's lifecycle and races
WHAT IT TESTS: modeling a stateful flow with idempotency and concurrency safety. OUTLINE: a referral entity with explicit states, a unique constraint on the invited user, and atomic transactions plus idempotency to prevent double credits.
Why consistent user IDs matter in experiments
WHAT IT TESTS: understanding identity for stable assignment and clean measurement. OUTLINE: the user id seeds deterministic bucketing and ties events to one person across devices; achieve it via authenticated ids and anonymous-to-known stitching.
Trace an event from click to analysis
WHAT IT TESTS: end-to-end understanding of an analytics event pipeline. OUTLINE: client SDK captures and batches, a collection endpoint ingests, a stream and ETL enrich and load into a warehouse for analysis.
Communicate forecast uncertainty with prediction intervals
WHAT IT TESTS: quantifying and communicating forecast uncertainty. OUTLINE: a point estimate hides risk; produce a prediction interval via model error, simulation, or scenarios, and state assumptions.
When user-level A/B tests get contaminated
WHAT IT TESTS: recognizing interference that breaks the independence assumption. OUTLINE: network or marketplace spillover violates SUTVA, so randomize by cluster (geo, group, time) and analyze at that level.
Build an opportunity-sizing model before building
WHAT IT TESTS: quantifying upside before investing. OUTLINE: locate the affected funnel step, estimate addressable population times a bounded conversion lift times value per user, then sanity-check against a realistic ceiling.
Combine qualitative and quantitative data for hypotheses
WHAT IT TESTS: mixed-methods reasoning to build strong hypotheses. OUTLINE: quant reveals what and where, qual reveals why, then triangulate into a falsifiable hypothesis with a metric. RED FLAG: treating anecdotes as proof or analytics as self-explanatory.
Explain RICE scoring and its Confidence factor
WHAT IT TESTS: understanding RICE and the role of Confidence. OUTLINE: score equals Reach times Impact times Confidence divided by Effort; Confidence discounts uncertain estimates; ground it in evidence tiers.
Instrument a first-full-song activation event
WHAT IT TESTS: precise event definition and reliable instrumentation. OUTLINE: define 'full song' server-side, emit a typed event with user, song, and context, dedupe the first-time flag.
Architect an experimentation dashboard for culture
WHAT IT TESTS: product thinking about experimentation as an organizational system, not just stats. OUTLINE: searchable experiment repository, structured hypotheses, results regardless of outcome, and cross-team discovery.
Resurrection Campaign
A resurrection campaign is a targeted effort to win back dormant or churned users by re-engaging them with relevant value, often via email or push. It matters because reactivating known users is usually cheaper than acquiring new ones.