All bites
The whole library, newest first. Filter by what you are here for, or pick a topic if you already know.
8668 bites
Page 157
What is the multiple comparisons problem and how to correct?
This tests your grasp of family-wise error inflation across many tests. A strong answer defines the problem, contrasts per-comparison and family-wise error, and names corrections like Bonferroni or FDR.
Explain statistical power, MDE, and sample size impact
Power is the chance of detecting a true effect; MDE is the smallest lift worth measuring; raising power or shrinking MDE inflates sample size.
What is a p-value in A/B tests and what does threshold mean?
P-value is the chance of this or more extreme result if the null is true; the 5% threshold is the accepted false-positive rate.
How do you set up a button color A/B test?
Tests whether you can define a valid experiment, not just a color change. A strong answer covers hypothesis, randomization, primary metric, sample size, and significance threshold.
WAU is flat despite positive A/B tests; why and how to diagnose
This tests distinguishing real impact from statistical artifacts. Strong answers cite false positives from low base rates, peeking, novelty, and local-global mismatches. Diagnose with long-term holdouts, audits, and causal bridges.
How would you structure your growth team's experimentation portfolio?
3 asset classes (iterative 30-70%, tech investments, big bets 20-40%), use expected value per week, and evolve the mix.
When increasing CTA clicks, what side-effects and guardrails should you consider?
This tests balancing growth with business safety. Name guardrails like retention, revenue, fraud, load time; set NI thresholds pre-launch; and include SRM checks. A red flag is treating guardrails as optional success metrics instead of hard stop limits.
Develop a testable hypothesis for a 40% email verification drop-off
This tests structured hypothesis formation under uncertainty. Strong answers: segment the 40% drop by device and latency; build a Customer Theory from data; isolate one lever; draft a four-part MECLABS hypothesis. Red flag: skipping diagnosis to guess fixes.
What framework decides between low-effort/low-impact and high-effort/high-impact experiments?
This tests structured experiment sequencing beyond gut instinct. A strong answer picks ICE, RICE, or PIE; scores both experiments by impact, confidence, and effort or reach; then weighs opportunity cost and bandwidth.
Which three data sources would you analyze to improve activation?
This tests whether you ground hypotheses in diverse evidence before experimenting. A strong answer names qualitative feedback, funnel metrics, and behavioral analytics as distinct inputs.
Design a system that detects choice paralysis and dynamically simplifies the interface
Track hover entropy, scroll jitter, and time-to-click; use a contextual bandit to select simplification tiers.
How would you use ML to optimize habit-loop notifications?
Tests blending behavioral psychology and ML to personalize cues without coercion. Good answers use contextual bandits with user-state features and reward habit formation over clicks.
Design the data model and backend for a 7-day trial at scale
Tests state machine design for time-bound entitlements at scale. A strong answer covers: an idempotent enrollment API, a trial ledger with timezone-aware expiration, and an event-driven expiration pipeline.
Design a variable daily-login reward system with anti-gaming controls
Tests server-side reward probabilities and idempotency in distributed systems. Strong answers cover: configurable weights, idempotent tokens with DB unique constraints, rolling windows, and server-side grants.
Design an A/B test for loss aversion versus gain framing at checkout
Tests whether you can isolate framing effects from checkout confounders. Strong answers detail user-level randomization, event logging, and guardrail metrics like revenue per visitor. Red flag: a conversion-only analysis with no unit of diversion defined.
How would you design a near real-time social proof notification system?
This tests low-latency event pipeline design with bounded load. A strong answer uses a fire-and-forget beacon, stream processor, and cache with TTL; it favors approximate counts and windowed aggregation.
How would you implement a timezone-safe, tamper-proof offer countdown?
Tests distrust of the client and server-side UTC enforcement. Outline: server owns canonical end time; client syncs clock offset to render remaining time; checkout re-validates expiry. Red flag: using local Date.now or localStorage.
How would you instrument events and query a 3-invite aha moment?
Tests taxonomy and stateful aggregation across sessions. Strong answers instrument Teammate Invited with timestamps, compute 7-day per-user counts via stream or SQL windowing, and materialize cohorts.
Design a near real-time user interaction tracking and analytics system
Tests decoupling ingestion from querying with justified tech choices. Outline: client → Kafka → Flink → ClickHouse → API; budget sub-30s latency and backpressure per stage. Red flag: one monolithic RDBMS or batch ETL handling both writes and reads.
Explain user identity stitching across devices and SDK roles
Tests event-level identity resolution. A strong answer covers anonymousId per device, the identify call binding anonymous events to userId, and backend merge handling out-of-order events. Red flag: claiming cookies alone stitch mobile and web.