Statistics
196 bites tagged Statistics — interview questions with model answers, and 60-second explainers.
Comparing skewed task-time data without a t-test
T-test assumes normality and is mean-sensitive to outliers; use Mann-Whitney U on ranks; trade-off is it tests distribution shift, not means. Knowing test assumptions.
Checking if a 5% A/B lift is significant
State hypotheses, check predetermined sample and power, run the right test for the metric, read p-value and confidence interval. Hypothesis-testing fundamentals.
Designing an in-house A/B testing platform
Config service, deterministic bucketing SDK, metrics pipeline, analysis engine; fix peeking with fixed samples or sequential tests. System design plus statistical rigor.
Confidence intervals versus p-values, explained simply
Define a CI as a plausible range for the true effect with its precision, contrast it with a binary p-value, and read [-1%, 7%] as inconclusive because it spans zero. statistical literacy for A/B decisions.
Design automated canary analysis scoring
Track the golden-signal SLIs, compare canary to baseline statistically, weight and combine into a score with promote/rollback thresholds. Automated release gating.
Isolating the design system's velocity impact
Use comparison groups, difference-in-differences or before-after baselines, and control for confounders; cite mechanism data like component reuse. Whether you can argue causation, not correlation.
Logical vs physical query plans and the optimizer
Logical plan says what (relational algebra, no algorithms); physical plan says how (specific operators); cost-based optimizer enumerates physical options and picks the cheapest using statistics. query-planning stages.
Optimizer picks nested loop over hash join
Nested loop wins on few rows, so bad row estimates from stale stats or skewed data trick it; fix by refreshing statistics, adding histograms, rewriting predicates, or ensuring memory for hashing. optimizer reasoning.
Handling missing numerical values
Dropping rows is simple but loses data and can bias if missingness is non-random; mean or median imputation keeps rows but shrinks variance and ignores correlations; model-based imputation is… judgment about imputation trade-offs.
MLE versus MAP estimation and the role of priors
MLE maximizes likelihood alone; MAP maximizes likelihood times a prior, acting as regularization that shrinks toward prior beliefs; with abundant data they converge. Bayesian versus frequentist parameter estimation.
Manage localized copy across many A/B tests
A keyed versioned string store, a translation workflow with per-locale status, and pooling or hierarchical analysis for small languages. localization plus experimentation at scale.
The multiple comparisons problem in A/B testing
Many tests at alpha 0.05 inflate the chance of a false positive; mitigate with Bonferroni or FDR control plus pre-registered metrics. Statistical rigor at scale. Cherry-picking whichever metric crosses p<0.05.
Communicate experiment results and check guardrails
Explain the lift and confidence in plain business terms, report a confidence interval not just a point, and verify guardrails before shipping. translating statistics and protecting against harm.
Describe the architecture of a generic A/B testing framework
Hash-based user bucketing, config service, pre-registered metrics, and confidence intervals on dashboards. system design with statistical safety. request-level randomization or skipping power analysis.
Explain statistical power and respond to extending a null A/B test
This tests statistical power and p-hacking judgment. A strong answer defines power as detecting a true effect, rejects extending the test to chase significance, and requires pre-registered sample sizes. Agreeing to run until it hits significance is a red flag.
What is the multiple comparisons problem in UX research?
This tests whether you know running many tests inflates false positives. Strong answers define family-wise error, give a UX example like comparing twenty metrics in one A/B test, and name a correction like Bonferroni.
How many users do we need to survey?
Tests pragmatism over guesswork. A strong answer asks if this is a census or sample, population size, whether stratified or multi-group designs apply, and cost, time, and power constraints. Red flag: blurting a number without probing scope, design, or budget.
Are 1-5 satisfaction ratings continuous or discrete?
Tests knowledge of measurement scales. A strong answer calls 1-5 ratings discrete and ordinal, notes intervals may be unequal, and prefers medians or non-parametric tests over means unless equal spacing is defensible.
Explain what a p-value represents in an A/B test and interpret 0.03
This tests frequentist hypothesis testing literacy. It is the chance of seeing a result this extreme if the null of no effect were true; 0.03 means 3% chance of such data under the null, not 97% chance the variant wins. Red flag: saying 97% likely.
Mean, median, and mode: which measure for task completion times?
This tests whether you know task times are skewed and sample size drives the mean-versus-median choice. A strong answer cites the under-25 rule, contrasts mean outlier sensitivity with median robustness, and dismisses mode. Red flag: defaulting to the mean.
Effect Size: How Large Is the Difference?
Effect size measures how much a phenomenon actually matters, not just whether it exists. In UX research, it quantifies the real-world impact of a design change beyond hypothesis testing. Ignoring it leads to chasing tiny, meaningless wins.
T-test: Are Two Groups Actually Different?
A t-test asks whether the gap between two groups is a real signal or just sampling noise. UX researchers use it to compare task times or conversion rates between designs. The footgun is trusting results from tiny samples where the math gives false confidence.
Normal Distribution: Two Parameters, One Bell Curve
A normal distribution is a bell curve for real-valued variables, defined by mean and standard deviation. Use it when values cluster around a center. The footgun is forcing normality on skewed or bounded data, which corrupts the probability density.
Stevens' Four Levels of Measurement
Data is not just numbers; Stevens' levels classify the nature of information each variable holds. In UX research, this determines whether you can average feedback or only count it. Treating every rating scale as a ratio number wrecks your analysis.
Get Statistics bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.