Metrics
400 bites tagged Metrics — interview questions with model answers, and 60-second explainers.
What is the difference between a primary metric and a guardrail metric?
Tests whether you distinguish success criteria from safety checks in experiments. A strong answer defines primary metrics as the target outcome, guardrails as protective thresholds, and gives a concrete scenario where a primary lift does not justify shipping…
What is the 'novelty effect' in experimentation?
Tests whether you separate temporary curiosity from durable value. A strong answer defines novelty effect as short-term behavior change triggered by new elements, notes it inflates early experiment lift, and proposes longer runtimes or lagged cohort analysis.
Design an A/B test for a 'Buy Now' button color change
Tests structured experiment design from hypothesis to metric. Strong answers: define a falsifiable hypothesis; pick purchase conversion as primary; size the sample and duration; randomize by user; pre-commit to stopping rules.
Explain Simpson's Paradox and construct a user engagement scenario
Tests whether you spot trends reversing when population mixes differ. Good answers define the paradox, give a numerical example with per-segment wins but aggregate loss, and warn against segment-only decisions.
Determine if a 10% DAU drop is statistically significant
Tests signal vs noise in stable metrics. Good answers define a null hypothesis, compute a test statistic from historical variance, compare to a critical value at set alpha, and check seasonality. Red flag: calling a large drop real without baseline variance.
Explain the difference between correlation and causation with a software example.
Tests whether you distinguish association from causation to avoid blaming production issues. A strong answer defines both concepts, names a confounding variable, and gives a software example with a common cause. Red flag: claiming correlation is causation.
A/B test p-value 0.08, PM wants to ship. How do you advise?
Tests statistical rigor versus business pragmatism. A strong answer covers pre-registered thresholds, false positive risk, statistical power, confidence intervals, and the business cost of being wrong. Red flag: shipping without quantifying downside risk.
Mean or median for API response times?
Tests if you know latency is skewed and outlier-sensitive. Pick median over mean because hiccups distort the mean, but note median hides tail latency. Advocate for p95, p99, and max. Red flag: defending mean as representative or ignoring tail behavior.
How would you validate a feature's conversion impact given self-selection bias?
Tests causal inference for opt-in features. Strong answers use quasi-experiments like propensity matching or diff-in-diff to compare similar users and verify pre-trends. Red flag: a raw t-test between adopters and non-adopters ignoring selection bias.
Average latency up 50ms but p99 flat: diagnose the discrepancy
Tests if you know mean reflects full distribution while p99 is a threshold. Strong answers hypothesize body shift like cache misses or traffic mix changes, and demand histograms and segmentation by endpoint. Red flag: blaming outliers, which would raise p99.
What user segments do you check first after a 10% DAU drop?
Validate by time, platform, and geography; then slice by new vs returning, channel, and feature usage to isolate the bleeding cohort. Structured triage of a metric drop through user segmentation.
How would you structure a user engagement dashboard for PMs?
Tests narrative sequencing of metrics across abstraction layers. Strong answer: DAU headline for health, retention cohorts for pattern diagnosis, feature adoption funnels for root-cause drill-down.
Compare five companies' market share: bar or line chart?
This tests categorical vs. temporal encoding. Pick a bar chart because companies are discrete categories, not a time series; line charts falsely imply sequence or trend. Calling a line chart acceptable is a red flag.
Why prefer median and p95 over mean for API latency?
This tests statistical intuition for skewed distributions. A strong answer notes that median captures typical experience, p95 captures tail suffering, and mean hides outliers. A red flag is claiming mean alone is sufficient.
Conversion metric dropped suddenly with no recent deployments; debug instrumentation causes
Distinguishing real regressions from telemetry pipeline failures. Segment by device, channel, and geography to spot uniform loss signaling a tagging break; verify vendor delays and sampling; check for consent or ad-blocker shifts.
How do you measure data platform ROI and track it?
Cite adoption, time to insight, downtime cost, and cost per workload; then describe cost tags and usage telemetry. Linking platform spend to business value and team health.
How would you develop balanced KPIs for a two-sided marketplace?
Tests dual-sided metric design beyond B2C playbooks. Strong answers define buyer and seller liquidity separately, prioritize match rate over GMV, and monitor supply-demand balance granularly.
Pitfalls of using conversion rate as a checkout North Star?
Tests if you know over-optimizing conversion can degrade revenue quality or trust. Strong answers cite lower AOV or fraud risks, then list guardrails like refund rate, lifetime value, and checkout errors. Red flag: insisting conversion is the sole metric.
How do you translate increase user engagement into a technical measurement plan?
Align with PM to define engagement, map touchpoints for events, pick a north star and guardrails, then draft technical schema. turning vague goals into metrics.
MRR: The Subscription Heartbeat
MRR is the monthly pulse of a subscription business. SaaS teams use it to forecast growth and measure churn. Counting one-time fees or annual contracts without proration inflates the metric and misleads stakeholders.
Velocity is fluctuating wildly. How would you coach the team?
Tests whether you treat velocity as a diagnostic, not a target. A strong answer checks story sizing, unplanned work, definition of done, and team stability before changing process. Red flag: demanding higher estimates or comparing teams to normalize velocity.
How would you advocate for decentralizing a deployment approval dependency?
Propose a pilot with guardrails; track lead time, defect rate, rollbacks; define escalation paths. Separating strategic and local decisions while using data. Demanding autonomy without metrics or dismissing enterprise risk.
From an engineer's perspective, when does Cycle Time begin and end?
Tests if you set Cycle Time boundaries to expose wait states past coding. Strong answer: starts at In Progress, ends at Done or production, includes review/test, excludes backlog queues, and distinguishes from Lead Time. Red flag: starting at ticket creation.
Why is velocity as a primary KPI destructive, and what's better?
This tests if you see velocity as a planning gauge, not a performance metric. A strong answer notes points are subjective, cites Goldratt on gaming, and proposes team-driven improvement instead. A red flag is claiming velocity works if averaged over time.
Get Metrics bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.