Intermediate interview questions in Growth & Experimentation, page 2
Design a role-based personalized onboarding system
Tests separation of content and logic for scalable personalization. Strong answer: CMS-backed rule engine, multi-channel delivery, event-driven triggers, and per-segment metrics. Red flag: hardcoding role-specific UI components in the client.
Design an A/B test for a Buy Now button
This tests experiment plumbing: deterministic bucketing, sticky storage, and logging. A strong answer covers user-ID hashing, cookie persistence, and impression-plus-conversion events.
Outline architecture for a weekly email digest of unread notifications
This tests batch processing and scheduled delivery at scale. Pre-aggregate unread counts, shard digest jobs across a distributed scheduler, and cache unsubscribes for fast filtering. Never scan the notifications table at send time for millions of users.
Compare fan-out-on-write vs fan-out-on-read for an activity feed
Tests whether you tie feed architecture to read/write ratios and follower distribution. Strong answers contrast push O(1) reads with celebrity storms against pull O(1) writes with read amplification, then propose a hybrid threshold.
How do you ensure consistent A/B test variants across sessions and devices?
Tests stable identity and delayed attribution. Fix: one stable user ID for SDK bucketing, persisted across devices via login or first-party cookies, attached to every conversion event. Never use per-device randomization or third-party cookies.
Design a system to handle subscription renewals
States include active, past_due, and cancelled; transitions are payment success, dunning exhaustion, and grace expiry; use idempotent webhooks.
How would you instrument an application to calculate Customer Lifetime Value?
Tests whether you tie revenue and cost events to persistent identity and model cohort retention. Outline: track purchases, CAC, and churn with persistent IDs; project value via retention curves.
What are the key architectural differences between freemium and free trial models?
Contrast tiered entitlements against trial timers and revocation; discuss free-user overhead and conversion tracking.
Design referral tracking from invite to conversion
Tests data modeling for multi-stage conversion tracking. Strong answers separate codes from conversion events, model status transitions, and enforce unique constraints. Weak answers merge invite and reward into one table or store denormalized counts on users.
How would you enforce a 3-project freemium limit and handle upgrades?
Tests entitlement and growth tradeoffs. Strong answers use API-level enforcement, atomic checks to prevent concurrent overages, soft upsell prompts preserving context, and async billing webhooks. Red flag: UI checks or limits in the projects table.
Walk me through a magic link login system and its security considerations
Tests auth architecture and threat modeling for passwordless flows. Strong answers map request-token-email-verify-session, then harden with short expiry, single-use tokens, rate limits, and device binding.
How would you model a 10% Day-1 retention improvement's impact on LTV?
Tests whether you model retention as a survival curve, not a single point. A strong answer builds a cohort curve, propagates D1 lift to D30/D90 via decay, and sums revenue. Red flag: claiming 10% D1 gain equals 10% LTV growth without curve assumptions.
How would you incorporate external events into a time-series forecast?
This tests exogenous regressor design. A strong answer proposes binary or continuous regressors for holidays and campaigns in Prophet or ARIMA, then validates lift via backtesting. A red flag is dropping outlier days or applying post-hoc manual adjustments.
Communicate forecast uncertainty with prediction intervals
A point estimate hides risk; produce a prediction interval via model error, simulation, or scenarios, and state assumptions.
Design an assignment service for an experimentation platform
This tests deterministic hashing and independence in randomization. A strong answer lists user ID, experiment salt, and allocations; uses salted hash for consistency; and warns that separate exposure and bucket hashes with mod can cause sample ratio mismatch.
Should we build or buy an A/B testing platform?
This tests build-vs-buy judgment for experimentation infrastructure. Strong answers cover build for warehouse metrics and cache control; buy for proven stats and front-end speed. Red flag: answering with cost alone or assuming in-house is always superior.
Compare PostgreSQL versus a columnar warehouse for raw event data
Contrast row vs column storage, compression, and scan speed; note Postgres suits OLTP and point lookups while columnar stores excel at aggregations.
What experiment artifacts and metrics do you present to decide shipping?
This tests structured experiment discipline. A strong answer covers the pre-approved design doc, scorecard results for primary goal and guardrail metrics plus secondary breakdowns versus the hypothesis, and duration context.
How do you systematically manage and pay down experiment debt?
Tests sustainable velocity through experiment lifecycle hygiene. Strong answers cover isolated experiment directories, TTLs on feature flags, and recurring cleanup sprints. Red flag: banning experiments or treating all experiment code as permanent.
How do you prevent concurrent onboarding and navigation experiments from polluting results?
This tests experiment isolation via layer-based traffic allocation. A strong answer covers hashing users into independent layers with one variant per layer, and assigning each experiment to a distinct layer.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles