How would you measure a sales forecast model's accuracy?

What's really being asked

This question isn't a pop quiz on statistical formulas. It tests your ability to connect abstract metrics to concrete business outcomes. The interviewer wants to see if you can justify your choice of an error metric based on the business's tolerance for different types of mistakes. It also serves as a basic check for good machine learning hygiene, specifically the use of a hold-out test set.

The full answer

First, state that accuracy must be measured on a hold-out test set—data the model has not seen during training. For time series, this must be a recent block of time (e.g., train on 2022 data, test on Jan 2023).

Second, define Mean Absolute Error (MAE). It's the average of the absolute differences between the forecast and the actual values. It's easy to interpret because it's in the same units as the data (e.g., dollars of sales). It treats a 100 error and a 1,000,000 error linearly in its calculation.

Third, define Root Mean Squared Error (RMSE). It's the square root of the average of the squared differences. The key is the squaring step, which penalizes large errors much more heavily than small ones. An error of 100 contributes 10,000 to the sum, while an error of 10 contributes only 100.

Fourth, explain the trade-off. You prefer MAE when the cost of an error scales linearly. For example, if being off by 200 units is exactly twice as bad as being off by 100. You prefer RMSE when large errors are disproportionately costly. For example, a massive under-forecast causes a factory shutdown, which is exponentially more damaging than a small inventory discrepancy.

The mistakes people make

Reciting the mathematical formulas without explaining the business implications is a junior-level answer. Another red flag is evaluating the model on the training data; residuals are not a reliable indicator of future forecast performance. Stating that one metric is universally "better" is also incorrect. A subtle error is forgetting the "root" in RMSE and implying the units are squared; the square root returns the metric to the original data's units, making it comparable to MAE.

What usually comes next

"If your RMSE is 50,000 and your MAE is 10,000, what does that tell you?" This indicates your model is making a few very large errors, as RMSE is much more sensitive to outliers than MAE. "What about MAPE?" Mean Absolute Percentage Error is useful for comparing forecast accuracy on items with very different scales (e.g., forecasting sales for a 10 item vs. a 10,000 item).

A concrete example

Imagine forecasting staffing for a call center. If you are understaffed by 5 agents (a small error), wait times go up slightly. If you are understaffed by 50 agents (a large error), the phone system melts down, SLA targets are breached, and you face contractual penalties. The business cost is non-linear. Therefore, you would prefer RMSE as your primary metric because it would heavily penalize the model for making those large, catastrophic forecast errors.