Chi-Squared Test: Are These Two Things Related?

WHY IT EXISTS We often need to know if two categorical things are related. For example, does changing the color of a 'Buy' button actually make more people click it? We need a formal way to distinguish a real effect from random noise in the data. The Chi-Squared test provides a statistical method to answer this for categorical data.

THE MENTAL MODEL Think of the Chi-Squared test as a formal way of measuring surprise. You start by assuming two variables are independent (the 'null hypothesis'). For example, you assume button color and click-through rate are unrelated. Then, you calculate what the click counts should look like in that 'unrelated' world. The test measures the squared difference between your actual, observed counts and these expected counts. A large difference means you're 'surprised', suggesting your initial assumption was wrong and the variables are likely related.

HOW IT WORKS The process uses a contingency table, which cross-tabulates your two categorical variables (e.g., rows for 'Button A', 'Button B'; columns for 'Clicked', 'Did Not Click'). First, you calculate the 'expected' frequency for each cell, assuming independence. Then, for each cell, you compute (Observed - Expected)^2 / Expected. The sum of all these values is the Chi-Squared statistic. You compare this statistic to a critical value from a Chi-Squared distribution (or just look at the p-value) to determine if the result is statistically significant.

WHEN TO USE IT Use it to compare two categorical variables when you have a large enough sample size. It's ideal for analyzing results from A/B tests where both the intervention and the outcome are categorical. Examples include testing different headlines on click-through rates, or seeing if user segment (e.g., 'new' vs. 'returning') affects feature adoption.

WHEN NOT TO USE IT Do not use it for continuous variables (e.g., comparing average session duration; use a t-test for that). It's also unreliable for very small sample sizes, specifically when any cell in your expected contingency table has a count less than 5. This is a common footgun. Also, it only shows association, not causation.

ONE CANONICAL EXAMPLE An e-commerce site tests two checkout button texts: 'Complete Purchase' and 'Buy Now'. They observe the number of users who click each button versus those who abandon the cart. They create a 2x2 contingency table. The Chi-Squared test is used to determine if the difference in click rates between 'Complete Purchase' and 'Buy Now' is statistically significant, or if it could have just happened by chance.