Interview questions in Growth & Experimentation, page 3
Explain statistical power, MDE, and sample size impact
Power is the chance of detecting a true effect; MDE is the smallest lift worth measuring; raising power or shrinking MDE inflates sample size.
When user-level A/B tests get contaminated
Network or marketplace spillover violates SUTVA, so randomize by cluster (geo, group, time) and analyze at that level.
What is the multiple comparisons problem and how to correct?
This tests your grasp of family-wise error inflation across many tests. A strong answer defines the problem, contrasts per-comparison and family-wise error, and names corrections like Bonferroni or FDR.
How would you implement a last-touch attribution model for user signups?
Tests your ability to translate marketing concepts into warehouse SQL. A strong answer covers UTM/pageview events, sessionized tables, and a windowed join for the last touch within 30 days of signup.
Architect an A/B test for paid-ad signup flows
Tests pre-auth bucketing and funnel attribution. Hash a stable anonymous ID for fast assignment; stream events via Kafka into hourly aggregates; run t-tests on signup rates. Red flag: assigning after signup starts or DB lookups per assignment.
Design a programmatic SEO system for 1 million landing pages
Tests data infrastructure thinking, not content generation. Covers one-row-one-page schema, template rendering with edge caching, hierarchical routing, and crawl-budget controls via sitemaps. Red flag: AI bulk writing without structured data or caching.
How would you instrument a 4-step onboarding wizard?
Track Step Started and Step Completed with step_index and flow_variant; tie via distinct_id.
How do you determine if a user is 'new' for a setup guide?
This tests whether you separate account age from user state for onboarding. Good answers compare created_at (brittle) with a persistent flag (idempotent) and consider milestones. A red flag is using a timestamp as a permanent new proxy without managing reruns.
How would you validate that early Project creation drives retention?
Tests causal rigor on behavioral predictors. Good answer: define D30 retention and the 24-hour treatment; pull timestamps and covariates; cohort-compare with propensity matching; show lift with confidence intervals and propose an A/B nudge.
How would you design resumable multi-step onboarding state management?
This tests cross-device onboarding resume. A strong answer uses debounced server sync for cross-device resume with localStorage fallback, covers anonymous users, and handles conflicts. Red flag: pure client or server storage ignoring offline gaps or privacy.
Design a role-based personalized onboarding system
Tests separation of content and logic for scalable personalization. Strong answer: CMS-backed rule engine, multi-channel delivery, event-driven triggers, and per-segment metrics. Red flag: hardcoding role-specific UI components in the client.
Long-term onboarding holdback: technical and data integrity challenges
This tests the engineering cost of year-long holdbacks in growth. A strong answer covers feature-flag entropy, pipeline drift, survivorship bias, and counterfactual validity. Red flag: treating the holdback as static config that never rots.
Describe the end-to-end data flow for tracking a 'Share' button click
Payload carries event type, user ID, timestamp, device, content; client batches with retry; backend validates and lands in a partitioned store.
Describe the data model and backend logic for a daily login bonus.
This tests streak state machines and calendar edge cases. A strong answer stores last_login_utc and streak_count, uses UTC day buckets, resolves timezones per user tz, and needs no leap-year logic.
Design an A/B test for a Buy Now button
This tests experiment plumbing: deterministic bucketing, sticky storage, and logging. A strong answer covers user-ID hashing, cookie persistence, and impression-plus-conversion events.
Outline architecture for a weekly email digest of unread notifications
This tests batch processing and scheduled delivery at scale. Pre-aggregate unread counts, shard digest jobs across a distributed scheduler, and cache unsubscribes for fast filtering. Never scan the notifications table at send time for millions of users.
Compare fan-out-on-write vs fan-out-on-read for an activity feed
Tests whether you tie feed architecture to read/write ratios and follower distribution. Strong answers contrast push O(1) reads with celebrity storms against pull O(1) writes with read amplification, then propose a hybrid threshold.
Design a real-time personalized notification trigger system
Stream events to a delayed queue, expose a rule UI to non-technical users, and deliver idempotently.
What schema changes are needed to add a Pro subscription tier?
This tests normalization of billing data versus hardcoding tiers. Add a plans table with integer cents pricing, link subscriptions via plan_id, and leave users untouched. Red flag: adding a tier string column to users or storing prices in subscriptions.
Model a 3-month 20% promo discount and apply it at billing
This tests separating coupon rules from per-user redemptions. Good answers use a coupons table for the 20%/3-month rule, a redemptions table for usage, and apply the discount to the first three invoices. A red flag is hard-coding the discount on the user row.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles