When should you keep outliers, not drop them?

What's really being asked

The interviewer wants to know whether you treat outliers reflexively or contextually. Strong candidates distinguish data-entry errors from rare-but-real observations, and connect that distinction to model and metric choices.

The full answer

Keep outliers whenever they are the phenomenon of interest. In fraud detection, intrusion detection, equipment failure, or rare-disease diagnosis, the outliers ARE the positive class, so deleting them destroys the signal. Keep them when the underlying process is genuinely heavy-tailed, such as income, insurance claims, or web traffic, where extreme values are expected and informative. When you keep them, prefer models that tolerate them: tree-based ensembles like random forests and gradient boosting split on thresholds and are insensitive to magnitude, unlike ordinary least squares. Adjust evaluation too, using MAE or median absolute error rather than RMSE, which squares and overweights large errors.

The mistakes people make

Saying you always remove points beyond three standard deviations or 1.5 times the interquartile range. This conflates statistical extremeness with invalidity and can silently delete the exact records the business cares about.

What usually comes next

How do you distinguish a real outlier from a data error? How would winsorizing or a log transform compare to deletion? Would your answer change for a deep neural network versus a linear model?

A concrete example

A payments team builds a model to catch fraudulent transactions. A handful of accounts show charges fifty times their normal volume. Removing them as outliers would erase most of the fraud labels. Instead the analyst keeps them, trains a gradient-boosted classifier robust to scale, and evaluates with precision and recall on the rare positive class rather than overall accuracy, which would look high even if every fraud case were missed.