All bites
The whole library, newest first. Filter by what you are here for, or pick a topic if you already know.
4330 bites
Page 67
Design a pre-aggregation architecture for low-latency experiment results
Tests OLAP-at-scale trade-offs. Strong answers design streaming rollups into a real-time OLAP store, use partial cubes for high-cardinality dimensions, and retain raw events.
How do you mitigate peeking in experiment infrastructure?
Lock results behind minimum samples; auto-correct via sequential testing; hide early metrics and require stop approval.
Lifecycle of a feature flag experiment from creation to cleanup
Tests operational rigor across the full flag lifecycle. A strong answer covers six stages: SDK instrumentation with event tracking, phased rollout, monitored experiment, ship/kill decision, and code cleanup.
What experiment artifacts and metrics do you present to decide shipping?
This tests structured experiment discipline. A strong answer covers the pre-approved design doc, scorecard results for primary goal and guardrail metrics plus secondary breakdowns versus the hypothesis, and duration context.
How do you systematically manage and pay down experiment debt?
Tests sustainable velocity through experiment lifecycle hygiene. Strong answers cover isolated experiment directories, TTLs on feature flags, and recurring cleanup sprints. Red flag: banning experiments or treating all experiment code as permanent.
How do you prevent concurrent onboarding and navigation experiments from polluting results?
This tests experiment isolation via layer-based traffic allocation. A strong answer covers hashing users into independent layers with one variant per layer, and assigning each experiment to a distinct layer.
Compare server-side and client-side experimentation architectures
This tests rendering-layer architecture. A strong answer contrasts server-side zero-flicker and algorithm tests against client-side marketer agility and SEO safety, mapping each to release cycles. A red flag is claiming one approach dominates every dimension.
Describe cluster or switchback randomization for network-effect A/B tests.
Tests SUTVA violation, cluster-switchback tradeoffs. Outline: cluster (geo, teams) or switchback (time slices) isolation; cover pipeline changes, 10x inflation, and correlated error. Red flag: user-level randomization with post-hoc fixes or ignoring spillover.
Design a referral system: data models, APIs, attribution, self-referral prevention
Tests data modeling with fraud guardrails and idempotent rewards. Cover: Users with nullable referred_by, ReferralEvents state table, async ledger attribution, and device-fingerprint self-referral blocks. Red flag: bare integer credit with no audit trail.
Build a system to measure viral coefficient and attribute invites to signups
Tests if you can map K=i×c to logged events and resilient pipeline. Good answers define invite_sent, click, signup events with referral tokens; sketch stream joins; and flag cross-device and organic attribution gaps. Red flag: assuming perfect attribution.
Explain the difference between statistical and practical significance
Define statistical vs practical significance; note large samples make tiny effects significant; give a real example.
Design a referral feature's lifecycle and races
A referral entity with explicit states, a unique constraint on the invited user, and atomic transactions plus idempotency to prevent double credits.
Run concurrent experiments without interference
Independent non-interacting tests can share traffic through orthogonal layers; interacting ones need mutual exclusion in a shared layer.
Design a contamination-safe pricing experiment
User-level price tests leak via fairness perception, so use geo holdouts or time-based cohorts where everyone in a unit sees one price.
Prevent conflicting experiments with layers
Group conflicting experiments into one layer so a user's per-layer bucket maps to at most one of them; orthogonal layers can overlap.
Increase experiment velocity for non-engineers
Server-driven config, feature flags, and a self-serve UI let non-engineers ship copy or layout variants instantly; add guardrails and metric checks.
Architect a configurable, goal-based onboarding flow
Capture the goal, let the backend return a server-driven flow definition mapping goal to steps and content, render generic components on the client.
Why repeatedly extending a test inflates false positives
Repeatedly checking and extending until significance is p-hacking via optional stopping, which inflates the false-positive rate; fix with fixed sample sizes or sequential…
Handle interaction effects on a shared page
Combined variants may produce effects neither has alone; use mutual exclusion for likely interactions, orthogonal designs with interaction monitoring otherwise.
Attribute a mobile install to a desktop ad
Deterministic matching via a shared login is accurate but needs auth on both ends; probabilistic fingerprinting scales without login but is noisy and privacy-fraught.