How do you measure forecast accuracy and compare MAE to RMSE?

WHAT THIS TESTS: This question tests two distinct skills that separate junior analysts from senior forecasters. First, it checks whether you know that forecast accuracy must be measured on genuine out-of-sample forecasts rather than in-sample residuals. Second, it checks whether you understand the mathematical and business implications of choosing MAE versus RMSE as your accuracy metric.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, it insists on a train-test split where the model is fit on training data and evaluated on a test set that was not used during fitting, because only held-out data provides a reliable indication of how the model will perform on new data. Second, it defines MAE as the mean of absolute forecast errors, which is robust and expressed in the original units of the series. Third, it defines RMSE as the square root of the mean squared error, which is also in original units but penalizes large deviations more heavily because errors are squared before averaging. Fourth, it gives a concrete decision rule: prefer MAE when outliers are common and you want an interpretable, robust measure of typical error; prefer RMSE when large misses are disproportionately costly and you want to strongly penalize them.

COMMON WRONG ANSWERS: Common wrong answers include three red flags. One is reporting in-sample fit statistics like R-squared or standard error of the regression as if they were forecast accuracy, which ignores the fact that residuals on fitted data are not genuine forecasts. Another is claiming that RMSE is always better because it is more sensitive, without acknowledging that this same sensitivity makes it unstable in the presence of outliers. A third is failing to mention that both metrics are scale-dependent, so comparing them across products with very different sales volumes requires normalization or percentage errors.

LIKELY FOLLOW-UPS: Interviewers often follow up by asking how you would choose the size of the test set, in which case you should say it is typically about 20 percent of the total sample but depends on the length of the series and the forecast horizon. They may also ask about scale-independent alternatives like MAPE or MASE, or ask how you would handle a situation where the test set contains an anomalous month like a holiday spike.

ONE CONCRETE EXAMPLE: Suppose you are forecasting monthly sales for a retail SKU. You hold out the most recent six months as a test set. In month one the forecast is off by 10 units, in month two by 100 units, and the remaining months are within 5 units. The MAE would be roughly 25 units, treating the large miss as just another error. The RMSE would be roughly 42 units, pulling the average upward because the 100-unit miss is squared. If a 100-unit miss means stockouts and lost customers, RMSE correctly surfaces that pain; if the 100-unit miss was a one-time data entry error, MAE gives a truer picture of model performance.

Source: Forecasting: Principles and Practice (3rd ed), otexts.com