Comparing skewed task-time data without a t-test
Knowing test assumptions.
t-test assumes normality and is mean-sensitive to outliers; use Mann-Whitney U on ranks; trade-off is it tests distribution shift, not means.
What's really being asked
This probes whether you understand the assumptions behind a t-test and can choose an appropriate alternative when they break. Task completion times are a classic skewed, outlier-heavy metric, so it tests applied statistics judgment.
The full answer
The t-test compares means and assumes the sampling distribution is approximately normal. Completion times are right-skewed: most users finish quickly, a long tail takes very long. The mean is dragged by that tail and by outliers, so a t-test can be misled and its p-values unreliable, especially with small usability-study samples where the central limit theorem does not rescue you. A robust alternative is the Mann-Whitney U test, which converts values to ranks and tests whether one group tends to produce larger values than the other, making no normality assumption. Trade-offs: it answers a different question, whether the distributions differ or one stochastically dominates, not whether means differ, so reporting is less intuitive; it can lose power if you only care about the mean; and ties need handling. Alternatives include log-transforming the times and reporting medians.
The mistakes people make
Claiming the t-test is robust enough for skewed data regardless of sample size. Using the mean as the headline metric for skewed times. Forgetting that small samples lack CLT protection. Not knowing any non-parametric test or confusing Mann-Whitney with a paired test.
What usually comes next
When would a log transform plus t-test be preferable to Mann-Whitney. What does Mann-Whitney actually test if not the mean. How does sample size change your choice.
A concrete example
Version A times cluster near twenty seconds but a few users take minutes; version B is similar. A t-test on means could be swayed by one outlier and report no difference or a spurious one. Running Mann-Whitney U on the ranks shows version B's times are systematically lower across the distribution with a significant result. You report the median completion time per version alongside the test, since the median resists the skew that distorts the mean.
Interview question
What does the Mann-Whitney U test actually compare, making it suitable for skewed completion times?
- a.The variance ratio between the two distributions
- b.Whether one group tends to produce larger ranked values than the other, without assuming normalityCorrect
- c.The difference in sample means between the two groups
- d.Whether both samples come from a normal distribution
Why? this is the answer
Mann-Whitney ranks values and tests stochastic dominance without a normality assumption; it does not directly compare means, which is exactly why it suits skewed data the t-test mishandles.
Just read this? Test yourself on what you have been reading.
- #statistics
- #non-parametric
- #t-test
- #mann-whitney
- #ux-research
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on statistics — each one lists the topics its interview covers.
See open roles