T-test: Are Two Groups Actually Different?

WHY IT EXISTS: When you compare two sets of numbers, the averages almost never match exactly even if the underlying groups are identical. The gap could be a real effect or merely random fluctuation from sampling. The t-test was invented to quantify that exact uncertainty so you do not launch a design change based on noise alone.

THE MENTAL MODEL: Think of the t-test as a calibrated skeptic. You hand it two groups of measurements and it answers one question: how surprised should I be if these groups actually came from the same population? If the observed gap is large relative to the variation inside each group, the test returns a small probability that the gap is mere luck. That probability becomes your evidence that the difference is genuine.

HOW IT WORKS: The test computes a statistic that measures the distance between the two group means in units of standard error. Under the null hypothesis, which assumes both groups share the same true mean, this statistic follows a Student's t-distribution rather than a normal curve because the scaling term, the standard deviation, is estimated from the sample data instead of being known in advance. As sample sizes grow large, that t-distribution converges toward the normal distribution used in a Z-test, which is why the two methods often agree on big datasets but can diverge on small ones.

WHEN TO USE IT: Use a t-test when you need to decide whether the mean response of one group differs from another and you do not know the true population standard deviation. In UX research this appears when comparing average task completion times, error rates, or Likert-scale ratings between a control design and a variant.

WHEN NOT TO USE IT: Do not use it when the scaling term, the standard deviation, is already known from the population rather than estimated from your sample, because a Z-test is the exact tool for that situation. You should also avoid treating the result as reliable when the sample is so small or the data so ill-behaved that the test statistic no longer satisfies the conditions required to follow a t-distribution.

ONE CANONICAL EXAMPLE: Imagine you measure task completion time for twenty users on your old checkout flow and twenty on a redesigned flow. The redesign averages twelve seconds faster. A t-test tells you whether a gap that large would be unlikely if both designs were actually equally fast. If the resulting p-value is very small, you have evidence that the speedup is not just sampling noise.