Effect Size: How Large Is the Difference?

WHY IT EXISTS: Statistical hypothesis testing tells you if an effect exists, but not how big it is. A phenomenon can be real yet trivial. Effect size was developed to measure magnitude itself, giving researchers a way to judge practical importance separate from the binary logic of hypothesis testing.

THE MENTAL MODEL: Think of effect size as the volume knob to hypothesis testing's mute button. Statistical tests tell you whether sound is coming through the speakers; effect size tells you how loud it is. In UX research, a new checkout flow might produce a real detectable result with ten thousand users yet move conversion by only a fraction of a percent. The effect size reveals whether that signal is worth building.

HOW IT WORKS: An effect size can be a statistic calculated from sample data, a parameter for a hypothetical population, or the equation that converts those values into a single magnitude number. Common forms include the correlation between two variables, the regression coefficient in a model, the mean difference between groups, and the risk of a particular event occurring. Each operationalizes magnitude in units appropriate to the data. Collectively, the group of methods concerning effect sizes is referred to as estimation statistics.

WHEN TO USE IT: Use effect size when planning experiments to perform statistical power analyses and determine the sample size you will need. Use it when reporting results to show practical impact alongside the outcome of hypothesis tests. Use it during meta-analysis to combine and compare magnitude across multiple studies, such as aggregating usability findings from several independent product teams.

WHEN NOT TO USE IT: Do not use effect size alone to claim a finding is important if you have not also used hypothesis testing; the two are complementary. Avoid comparing effect sizes across studies that use fundamentally different metrics or populations without proper standardization, because a raw mean difference in task time is not directly comparable to a raw mean difference in satisfaction scores.

ONE CANONICAL EXAMPLE: A UX researcher runs an A/B test on a new onboarding flow. The hypothesis test detects an effect. However, the effect size for the mean difference in completion rate is tiny. The team realizes the new flow requires rebuilding the navigation stack for a gain that is real but practically negligible, and they deprioritize the project.