Student's t-test: Is This Difference Real or Just Noise?

WHY IT EXISTS: When you compare two groups, like in an A/B test, you need to know if an observed difference is a real effect or just random luck from the specific users you sampled. This is especially tricky with small sample sizes where randomness has a bigger impact. The t-test was created to solve this by quantifying the likelihood that the difference is just noise.

THE MENTAL MODEL: Think of a t-test as a signal-to-noise ratio calculator. The "signal" is the difference between the two group averages. The "noise" is the variability or spread of data within the groups. The t-test combines these into a single number, the t-statistic, that tells you how surprising your result is if there were actually no difference between the groups.

HOW IT WORKS: The test starts by assuming the "null hypothesis": that there's no real difference between the two groups. It then calculates a t-statistic based on three things: the difference between the two group means, the standard deviation of each group, and the size of each group. This t-statistic is then used to find a p-value, which is the probability of observing a difference at least this large if the null hypothesis were true. A small p-value (e.g., less than 0.05) suggests the difference is "statistically significant."

WHEN TO USE IT: Use a t-test when comparing the means of two groups to see if they are different. It is a standard tool for analyzing A/B test results, especially when sample sizes are not massive. For example, checking if a new checkout flow increased average order value.

WHEN NOT TO USE IT: Don't use a t-test for comparing more than two groups at once; use a technique like ANOVA instead. Also, as sample sizes become very large (thousands of data points), the t-test and the Z-test become practically identical. The t-test's main advantage is with smaller samples where the true population standard deviation is unknown.

ONE CANONICAL EXAMPLE: An e-commerce site tests a new product page (Variant B) against the old one (Control A). After 1000 users see each version, Control A has an average purchase value of 50 and Variant B has 55. A t-test is used to determine the probability that this $5 difference is a real improvement and not just random chance from the specific users who happened to see each version.