Metrics
400 bites tagged Metrics — interview questions with model answers, and 60-second explainers.
Explain cohort analysis for user retention and write a pseudo-query
Tests your ability to use precise metrics. A good answer defines a cohort, explains why it isolates variables better than aggregate data, outlines the calculation, and provides a clear pseudo-query.
Prove API Latency Affects User Engagement
This tests your ability to design a controlled experiment for a backend attribute. A great answer outlines an A/B test that artificially adds latency for a treatment group, details the necessary logging with shared IDs, and explains how to join and analyze…
How would you measure P95 latency by geographic region?
Tests your ability to design a practical metrics pipeline, considering instrumentation, data types (metrics vs. logs), and aggregation. Instrument the API with a histogram metric and a `region` label, then query using `histogram_quantile`.
How would you measure a sales forecast model's accuracy?
This tests your ability to connect statistical metrics to business impact. A great answer defines MAE (linear error cost) and RMSE (penalizes large errors), explains the choice depends on business context, and stresses using a test set.
Describe EDA for a 3-year daily user sign-up dataset.
Tests your structured approach to time series EDA. A good answer identifies trend, seasonality, and anomalies before modeling. A red flag is jumping to forecasting models or only mentioning the overall average growth, ignoring cyclical patterns.
How does CUPED increase the statistical power of an experiment?
Tests your grasp of variance reduction. Explain CUPED as ANCOVA, using pre-experiment data (X) to remove predictable noise from the outcome (Y). Effectiveness depends on correlation (rho), reducing variance by (1-rho^2).
Primary vs. Guardrail Metrics in Experiments
This tests if you can balance improving a key metric with not harming the user experience. Define primary (the goal) and guardrail (don't harm) metrics. Give an example where a guardrail regression (e.g., latency) blocks a feature ship.
Why is stopping an A/B test when it hits significance problematic?
Tests your understanding of the 'peeking problem' in A/B testing. A great answer defines peeking, explains how it inflates the Type I error rate (false positives), and states the need for a predetermined sample size.
How do you determine A/B test sample size and duration?
This tests your ability to connect business goals to statistical parameters. A good answer defines the four power analysis inputs (baseline, MDE, alpha, power) and explains trade-offs, then converts sample size to duration using business cycles.
How would you A/B test a 'Buy Now' button color change?
This tests structured thinking. A good answer defines a hypothesis, selects primary and guardrail metrics, and outlines the experiment's duration and analysis plan. A red flag is focusing only on clicks without considering business impact.
Why not t-test p99 latency? Describe a valid alternative.
This tests your grasp of statistical test assumptions. A good answer explains why p99 violates t-test normality, then outlines a resampling method like bootstrapping to build a confidence interval on the *difference* of p99s.
How to Statistically Test a 10% DAU Drop?
Tests your knowledge of hypothesis testing. A good answer outlines the steps: state a null hypothesis, choose a test (e.g., Z-test), calculate a p-value, and compare it to a significance level (alpha).
Explain the difference between correlation and causation
Tests if you can avoid statistical fallacies. First, define correlation (association) and causation (cause-effect). Then, explain the difference via a confounding variable. A red flag is giving an example where one metric actually could cause the other.
p-value is 0.08, significance is 0.05. Ship it?
This tests your ability to translate statistical risk for business partners. Explain that p=0.08 means an 8% chance of a false positive, quantify the cost of a bad decision, and suggest extending the test to increase power.
What does a p-value of 0.03 mean in an A/B test?
Tests your practical statistical literacy. A good answer defines the p-value (3% chance of this result if the null is true), compares it to alpha (0.03 < 0.05) to reject the null, and decides to ship.
Mean vs. Median for API Response Times?
Tests if you know latency data has outliers. A good answer chooses median because it's robust to extremes. Explain that mean gets skewed by a few slow requests (e.g., GC pauses), hiding the typical user experience.
Generate a monthly cohort retention table from user events
This tests your ability to translate a business metric into a multi-step SQL query. A great answer defines cohorts by first activity, maps subsequent activity to period indices, counts distinct users, and pivots the result.
Average latency is up, but p99 is flat. Why?
This tests your grasp of latency distributions. Hypothesize that a large group of typical requests slowed, pulling up the average but not crossing the p99 threshold. Segment by endpoint or customer to find the cohort.
Is 20% higher retention from Feature X causal or correlational?
This tests your ability to distinguish correlation from causation. A great answer questions the data, identifies confounding variables (e.g., power users), and proposes a randomized A/B test as the gold standard to prove causality.
DAU dropped 10%. What user segments do you investigate first?
Tests your systematic problem-solving. First, clarify the metric and timeline. Then, segment by platform, geography, and user tenure (new vs. returning). A red flag is jumping to external causes before ruling out internal issues like a bad deployment.
Explain pre-attentive attributes in data visualization
Tests your grasp of visual psychology in data viz. Define pre-attentive attributes (instantly processed visuals), give examples (color, size, shape), and explain using one to highlight outliers in a dense plot.
How would you design a product management dashboard?
Tests your ability to structure data into a decision-making narrative. A good answer moves from a high-level summary (DAU) to trends (retention) and then actionable details (feature adoption). A red flag is simply listing charts without a narrative connection.
Bar Chart vs. Line Chart for Market Share Comparison?
Tests basic chart selection: comparing static categories vs. showing trends. A bar chart is correct for comparing discrete companies at one point in time. A line chart wrongly implies a time-series relationship. Red flag: choosing a line or pie chart.
When is a pie chart an appropriate choice for visualization?
Tests data viz principles for part-to-whole data. A good answer defines this use case, gives a clear example (market share), and lists pitfalls like too many slices or similar values. A red flag is defending them for complex data or time-series analysis.
Get Metrics bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.