Metrics
400 bites tagged Metrics — interview questions with model answers, and 60-second explainers.
Moving Average: Smoothing Out the Noise in Data
A moving average smooths out noisy data by replacing each point with the average of its neighbors, revealing the underlying trend. It's used to track stock prices or server metrics without overreacting to short spikes.
The Novelty Effect: When New Isn't Always Better
The Novelty Effect is a temporary metric spike from a feature's newness, not its inherent value. It often appears in A/B tests for high-frequency products, inflating short-term metrics. The footgun is mistaking this initial excitement for a long-term win.
Twyman's Law: Interesting Data is Usually Wrong
Twyman's Law states that any data point that looks interesting is probably wrong. Before celebrating a sudden 10x spike in user engagement, first suspect a bug in your analytics pipeline or a bot attack.
The Multiple Comparisons Problem
Running many statistical tests on one dataset is like buying many lottery tickets; your chance of a "winning" false positive increases with each test. This happens in A/B tests with many metrics.
Standard Error: Gauging Your Measurement's Precision
Standard error measures the precision of a sample statistic, like the mean. It answers: "If I ran this experiment again, how much would my result change?" It's key for building confidence intervals and A/B testing. Don't confuse it with standard deviation.
Type I vs. Type II Errors: False Alarms vs. Missed Detections
A Type I error is a false alarm (a smoke alarm with no fire), while a Type II error is a missed detection (a fire with no alarm). This trade-off is crucial in A/B testing and medical diagnostics.
Hypothesis Testing: Is Your Data Signal or Noise?
Hypothesis testing is a courtroom trial for your data: you assume a default 'null hypothesis' is true until your data provides enough evidence to reject it. It's used in A/B tests to validate changes.
Sampling: Estimating the Whole from a Small Part
Sampling lets you understand a large group by studying a small, representative piece. Think of it like tasting a spoonful of soup to season the whole pot. It's used in A/B tests and polls, but the main footgun is a biased sample.
Confidence Interval: Quantifying Uncertainty in Your Estimates
A confidence interval puts error bars around a measurement, showing the plausible range for a true value. It's used in A/B tests to report not just a winner, but the range of its likely impact.
Correlation Is Not Causation
Just because two metrics move together doesn't mean one causes the other. This is vital when analyzing user data, as a feature launch might correlate with higher signups when the real cause was a marketing campaign.
Lie Factor: Quantifying Visual Distortion in Graphs
The Lie Factor measures how much a graph's visuals distort the data's story. It's used to critique charts that exaggerate changes, like with a truncated y-axis.
Sankey Diagram: Visualizing Proportional Flow
A Sankey diagram visualizes flow, where the width of each path is proportional to the quantity moving through it. Use it to trace user journeys or track budget allocation.
Small Multiples: Comparing Data with a Grid of Charts
Small multiples are a comic strip for data, showing different dataset slices in a grid of charts with identical axes. They're used to compare trends across categories, like sales per region. The footgun is using inconsistent scales, which breaks comparison.
Chart Selection: Match Purpose, Not Looks
Start with the purpose, not the chart. The question you're asking—'how do these compare?' or 'what's the trend?'—determines the best visualization. A line chart shows trends; a bar chart compares categories.
Anscombe's Quartet: When Numbers Lie
Anscombe's Quartet shows how four datasets can share identical summary stats (mean, variance) but look completely different when plotted. It's a classic reminder to always visualize your data before trusting numerical summaries.
Exception Reporting: Focus on Signals, Not Noise
Exception reporting filters out the noise, showing only data that breaks predefined rules. It's used in financial reconciliation to flag mismatched transactions or to alert on system performance dips.
Benchmarking: Know Where You Stand in Your Industry
Benchmarking answers "Are we good?" by comparing your performance metrics against industry bests. It's used to set realistic goals for cost, quality, or time. The main footgun is comparing apples to oranges—using benchmarks from dissimilar companies.
Data Visualization: Turning Numbers into Insight
Data visualization turns raw data into pictures, revealing stories that numbers alone can't tell. It's used to spot trends, find outliers, and grasp complex relationships in datasets.
Cross-Tabulation: Finding Relationships in Your Data
Cross-tabulation reveals how two variables are related by counting their joint occurrences in a grid. It's key for survey analysis or A/B testing. The footgun is assuming correlation implies causation; the table shows a relationship, not its cause.
Drill-Down Analysis: From Summary to Specifics
Drill-down analysis moves from a high-level data summary to the granular details composing it. It's used in dashboards to investigate a metric's change, like clicking a monthly sales dip to see daily figures.
Data Aggregation: The Big Picture from Small Details
Data aggregation rolls up granular records into high-level summaries, like turning individual sales logs into a daily sales report. It's used to power dashboards and speed up warehouse queries.
Descriptive Statistics: What Your Data Looks Like
Descriptive statistics summarize the data you have, painting a picture of your sample without making guesses about the wider world. It's used for calculating things like average age or max response time.
User ID: The Key to Cross-Device Analytics
A User ID stitches together a person's journey across devices and sessions, moving beyond anonymous tracking. It's key for apps with logins to see the full customer lifecycle. The footgun: never use the User ID for a custom dimension; it will break reporting.
Event Tracking: Measuring What Users Do
Event tracking turns user actions like clicks and purchases into analyzable data. Analytics platforms use this data to report on engagement and conversions. The biggest footgun is inconsistent naming, which pollutes your data and breaks reports.
Get Metrics bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.