Comparing skewed task-time data without a t-test

WHAT THIS TESTS This probes whether you understand the assumptions behind a t-test and can choose an appropriate alternative when they break. Task completion times are a classic skewed, outlier-heavy metric, so it tests applied statistics judgment.

A GOOD ANSWER COVERS The t-test compares means and assumes the sampling distribution is approximately normal. Completion times are right-skewed: most users finish quickly, a long tail takes very long. The mean is dragged by that tail and by outliers, so a t-test can be misled and its p-values unreliable, especially with small usability-study samples where the central limit theorem does not rescue you. A robust alternative is the Mann-Whitney U test, which converts values to ranks and tests whether one group tends to produce larger values than the other, making no normality assumption. Trade-offs: it answers a different question, whether the distributions differ or one stochastically dominates, not whether means differ, so reporting is less intuitive; it can lose power if you only care about the mean; and ties need handling. Alternatives include log-transforming the times and reporting medians.

COMMON WRONG ANSWERS Claiming the t-test is robust enough for skewed data regardless of sample size. Using the mean as the headline metric for skewed times. Forgetting that small samples lack CLT protection. Not knowing any non-parametric test or confusing Mann-Whitney with a paired test.

LIKELY FOLLOW-UPS When would a log transform plus t-test be preferable to Mann-Whitney. What does Mann-Whitney actually test if not the mean. How does sample size change your choice.

ONE CONCRETE EXAMPLE Version A times cluster near twenty seconds but a few users take minutes; version B is similar. A t-test on means could be swayed by one outlier and report no difference or a spurious one. Running Mann-Whitney U on the ranks shows version B's times are systematically lower across the distribution with a significant result. You report the median completion time per version alongside the test, since the median resists the skew that distorts the mean.