Experimentation
189 bites tagged Experimentation — interview questions with model answers, and 60-second explainers.
Design a pre-aggregation architecture for low-latency experiment results
Tests OLAP-at-scale trade-offs. Strong answers design streaming rollups into a real-time OLAP store, use partial cubes for high-cardinality dimensions, and retain raw events.
Design a system to reduce large client-side experiment payload size
Tests edge evaluation and payload compression. Use server-side pre-evaluation or edge nodes sending only assigned variants; compact bucketing indexes or Bloom filters; lazy-load noncritical experiments. Never do full client-side evaluation of every flag rule.
Compare PostgreSQL versus a columnar warehouse for raw event data
Contrast row vs column storage, compression, and scan speed; note Postgres suits OLTP and point lookups while columnar stores excel at aggregations. data orientation tradeoffs.
Should we build or buy an A/B testing platform?
This tests build-vs-buy judgment for experimentation infrastructure. Strong answers cover build for warehouse metrics and cache control; buy for proven stats and front-end speed. Red flag: answering with cost alone or assuming in-house is always superior.
Design an assignment service for an experimentation platform
This tests deterministic hashing and independence in randomization. A strong answer lists user ID, experiment salt, and allocations; uses salted hash for consistency; and warns that separate exposure and bucket hashes with mod can cause sample ratio mismatch.
How would you implement a simple feature flag system?
Tests runtime config separation and distributed hygiene. Outline: control service, datastore, API, SDK, and update mechanism; local evaluation with cached state; short-lived flags. Red flag: static config files requiring redeployment are not feature flags.
How would you adapt a growth model for network effects and k-factor?
Define K as invites x conversion; K over 1.0 explodes, yet K over 0.7 with fast cycle time still compounds; anchor at peak delight. Moving beyond linear funnels to viral growth.
How would you model a 10% Day-1 retention improvement's impact on LTV?
Tests whether you model retention as a survival curve, not a single point. A strong answer builds a cohort curve, propagates D1 lift to D30/D90 via decay, and sums revenue. Red flag: claiming 10% D1 gain equals 10% LTV growth without curve assumptions.
What differentiates leading and lagging indicators for subscriptions?
Leading indicators forecast; lagging indicators confirm. Subscriptions: activation rate leads, MRR lags. Predictive vs historical metrics in growth. Using raw signups as leading without proven revenue link.
Why is extrapolating 5% weekly growth naive for annual forecasting?
This tests knowledge of extrapolation limits. A strong answer cites market saturation, seasonality, and channel exhaustion as invalidating factors, and notes that compounding 5% over 52 weeks magnifies error.
What data and approach for a simple 30-day DAU forecast?
Tests forecasting from sessionized logs without overengineering. Cite timestamped events, a 30 min session rule, and a regression baseline with day-of-week, recent totals, scored with MAE. Red flag: deep learning before a baseline or ignoring privacy hashing.
What data pipelines and infrastructure feed a viral user acquisition model?
Tests causal attribution architecture. Great answers cover invite instrumentation with identity resolution, streaming pipelines that split organic and viral signups, and feature stores for network-state features.
Design a multivariate experimentation platform with collision-free concurrent bucketing and cross-device consistency
Tests orthogonal layers and cross-session assignment persistence. Cover: deterministic hashing per layer, a user profile service for sticky bucketing, and stable ID resolution across devices. Red flag: random bucketing or local storage breaking consistency.
Implement a 10% onboarding tutorial feature flag and track success
Tests lightweight experiment design and metric selection beyond vanity numbers. Strong answers cover user-ID bucketing, conversion tracking, and guardrail metrics like drop-off. Red flag: manual toggles or click-through without downstream outcomes.
How do you ensure consistent A/B test variants across sessions and devices?
Tests stable identity and delayed attribution. Fix: one stable user ID for SDK bucketing, persisted across devices via login or first-party cookies, attached to every conversion event. Never use per-device randomization or third-party cookies.
Design an A/B test for a Buy Now button
This tests experiment plumbing: deterministic bucketing, sticky storage, and logging. A strong answer covers user-ID hashing, cookie persistence, and impression-plus-conversion events.
Long-term onboarding holdback: technical and data integrity challenges
This tests the engineering cost of year-long holdbacks in growth. A strong answer covers feature-flag entropy, pipeline drift, survivorship bias, and counterfactual validity. Red flag: treating the holdback as static config that never rots.
Design a role-based personalized onboarding system
Tests separation of content and logic for scalable personalization. Strong answer: CMS-backed rule engine, multi-channel delivery, event-driven triggers, and per-segment metrics. Red flag: hardcoding role-specific UI components in the client.
How would you validate that early Project creation drives retention?
Tests causal rigor on behavioral predictors. Good answer: define D30 retention and the 24-hour treatment; pull timestamps and covariates; cohort-compare with propensity matching; show lift with confidence intervals and propose an A/B nudge.
How do you determine if a user is 'new' for a setup guide?
This tests whether you separate account age from user state for onboarding. Good answers compare created_at (brittle) with a persistent flag (idempotent) and consider milestones. A red flag is using a timestamp as a permanent new proxy without managing reruns.
Architect an A/B test for paid-ad signup flows
Tests pre-auth bucketing and funnel attribution. Hash a stable anonymous ID for fast assignment; stream events via Kafka into hourly aggregates; run t-tests on signup rates. Red flag: assigning after signup starts or DB lookups per assignment.
How would you implement a last-touch attribution model for user signups?
Tests your ability to translate marketing concepts into warehouse SQL. A strong answer covers UTM/pageview events, sessionized tables, and a windowed join for the last touch within 30 days of signup.
What is the multiple comparisons problem and how to correct?
This tests your grasp of family-wise error inflation across many tests. A strong answer defines the problem, contrasts per-comparison and family-wise error, and names corrections like Bonferroni or FDR.
Explain statistical power, MDE, and sample size impact
Power is the chance of detecting a true effect; MDE is the smallest lift worth measuring; raising power or shrinking MDE inflates sample size. This checks whether you grasp the power-MDE-sample-size trade-off.
Get Experimentation bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.