Analyzing skewed revenue-per-user experiments

WHAT THIS TESTS This assesses whether you recognize that a few extreme users break the assumptions behind standard tests and whether you know practical ways to recover statistical power without distorting the conclusion.

A GOOD ANSWER COVERS Heavy right skew from whale users inflates the variance of average revenue per user. Because the standard error scales with variance, high variance means low statistical power, so even a real treatment effect may not reach significance without an impractically large sample. The mean is also dominated by a handful of users, making it unstable across resamples. Mitigations include capping or winsorizing the metric at a high percentile to bound the influence of outliers, applying a log transform to compress the tail, and using variance-reduction methods such as CUPED, which subtracts variance explained by a pre-experiment covariate like prior spend. Nonparametric approaches like the Mann-Whitney U test or bootstrap confidence intervals avoid the normality assumption. Reporting a more robust statistic, such as a capped mean or a conversion-based metric, complements the analysis.

COMMON WRONG ANSWERS Applying a plain two-sample t-test and trusting the p-value despite extreme skew. Deleting whale users entirely, which biases the estimate of true average revenue. Log-transforming and then interpreting the result on the original scale without care. Ignoring variance reduction altogether.

LIKELY FOLLOW-UPS How does CUPED reduce variance and why does it stay unbiased. What is the downside of winsorization. Why might the median or a trimmed mean be preferable here.

ONE CONCRETE EXAMPLE Suppose ninety-nine percent of users spend under ten dollars but a few spend thousands. A raw t-test yields a wide confidence interval and a non-significant result. Winsorizing revenue at the ninety-ninth percentile and applying CUPED with each user's pre-experiment spend can cut variance substantially, tightening the interval enough to detect a true two-percent lift that the naive test missed, while keeping the estimate honest.