More in Growth & Experimentation — page 7
Propose a strategy to enforce a consistent analytics event schema
WHAT IT TESTS: scalable data governance preventing schema drift across teams. ANSWER: schema registry with CI validation, typed SDK wrappers blocking bad builds, plus ingestion-time rejection. RED FLAG: docs or manual reviews without automated guardrails.
Describe the SQL and data model for weekly cohort retention
WHAT IT TESTS: Anchoring activity to signup cohort. ANSWER OUTLINE: Join users and activity, compute week offset per user, group by cohort and offset for retention. RED FLAG: Counting active users globally without cohort anchoring.
How do you track events and calculate funnel drop-off?
Tests event mapping and funnel math. Answer: one event per step (page view, email submit, profile done); compute relative step conversion and overall rate from top; deduplicate users. Red flag: raw pageviews instead of uniques.
How do you attribute signups to Facebook, Google, and blog campaigns?
This tests URL-based campaign attribution and session persistence. A strong answer covers UTM tags, landing page parsing, cookie storage, and writing the source to the user record at signup.
How would you architect long-term holdback experiment groups?
Tests longitudinal causal inference and engineering tradeoffs for multi-month isolation. A strong answer covers bucketing, delayed metrics, and cross-experiment guards. Red flag: daily re-randomization or ignoring survivorship bias in aging cohorts.
How would you structure an event schema for funnel and cohort analysis?
This tests analytical event modeling. Use an immutable log with one row per event; attach context user_id, session_id, campaign; use a wide atomic table plus extensible contexts for funnel and cohort queries without joins.
How do you design allocation logic to minimize concurrent A/B test interactions?
WHAT IT TESTS: Designing allocation to isolate concurrent experiments. ANSWER OUTLINE: Compare mutual exclusion with layered randomization via layers and reservations; stress isolation vs throughput.
Describe RICE scoring and architect data pipelines for Reach and Effort
WHAT IT TESTS: Bridging prioritization and data systems for Reach and Effort. ANSWER OUTLINE: Define RICE; automate Reach via event streams with time windows; automate Effort from PM tool estimates with calibration.
What fields belong in an experiment tracking event?
Tests disciplined schema thinking over random fields. Strong answers cite a tracking plan with event and user properties, environment separation, and consistency. Red flag: dumping data without a schema or single source of truth.
How do you instrument a marketing funnel versus a product-led growth loop?
This tests if you distinguish linear attribution from compounding systems. A strong answer contrasts stage-tracking and CAC with viral-coefficient instrumentation, cycle-time velocity, plus identity resolution.
What counter metrics track health of weekly active users?
Tests whether you can spot growth-at-all-costs blind spots. A strong answer pairs WAU with 7-day retention, sessions per user, and error rate, mapping each to churn, shallow engagement, or bugginess.
Embedded Growth Teams: Experimentation as a Squad Muscle
Growth talent embedded in product squads spreads experimentation beyond a central team. It fits multi-surface products needing data-driven culture. The footgun is letting embedded specialists become sole experimenters while squad PMs and engineers disengage.
Log Exposure, Not Just Assignment
Exposure logging records when a user actually sees a variant, not just assignment. Counting assigned but unexposed users dilutes treatment effects and biases lift. The common footgun is logging assignment as exposure, which silently underpowers your test.
Revenue Recognition in Growth Experiments
Revenue is earned as you deliver value, not when cash arrives. In growth experiments, booking an annual prepayment as immediate revenue makes a pricing test look like a massive win. Teams often confuse bookings with revenue and credit experiments incorrectly.
Revenue Churn: Dollars Lost, Not Logos
Revenue churn tracks dollars lost from your existing base, not just headcount. In subscriptions, one enterprise downgrade can dwarf ten small cancellations. Teams often celebrate low logo churn while ignoring revenue churn that silently erodes growth.
Golden Path: The One Journey That Matters
Golden Path is the single user journey that drives core value. In growth, you optimize this highway before fixing side roads. The footgun is A/B testing edge cases while your main funnel leaks users.
Bayesian vs. Frequentist A/B Testing
Frequentist testing asks how surprising a result is if nothing changed; Bayesian asks probability B is better. Frequentist fixes sample size to control false positives, while Bayesian lets you peek.
Framing Effect: Presentation Rewires Decisions
The same fact hits differently depending on its wrapper. In growth, framing decides whether users see "90% uptime" or "10% downtime," swinging conversions without changing the product. Teams obsess over the offer yet ship copy that frames value as a loss.
Commitment and Consistency: The Identity Ratchet
People follow through on what they already said yes to. Use it to turn a tiny user action into sustained engagement, or to get a team to ship by making a public deadline. A coerced yes creates resentment, not consistency.
Scarcity Principle: Limit Availability to Drive Action
People want what they might lose. In growth, scarcity amplifies conversion by framing offers as limited in time, quantity, or access. The footgun is fake scarcity: invented limits destroy trust and backfire permanently.