How would you measure a sales forecast model's accuracy?
Tests if you can link statistical metrics to business outcomes. Define MAE (average error) and RMSE (penalizes large errors). Choose RMSE when large misses are costly (e.g., stock-outs), MAE otherwise. A red flag is reciting formulas without business context.
What's really being asked
This question assesses your practical understanding of forecast evaluation metrics. The interviewer wants to see if you can move beyond textbook definitions and connect the choice of a metric (MAE vs. RMSE) to real-world business consequences. It's a test of your ability to justify technical decisions based on business needs, a key senior-level skill. They're looking for nuance, not just a formula.
The full answer
A strong answer has four parts. First, state that accuracy must be measured on a hold-out test set, not the training data. Second, define Mean Absolute Error (MAE) as the average of the absolute errors, making it easy to interpret (e.g., "on average, our forecast is off by $X"). Third, explain Root Mean Squared Error (RMSE), noting that by squaring errors before averaging, it gives much higher weight to large errors. The final square root brings the unit back to the original scale. Fourth, and most importantly, state the decision criteria: prefer RMSE when large errors are disproportionately damaging (e.g., under-forecasting a key product leads to a stock-out). Prefer MAE when the business cost of an error is linear or when you need a metric less sensitive to a few large outliers.
The mistakes people make
A major red flag is simply stating the formulas for MAE and RMSE without explaining the practical implications of the squaring step in RMSE. Another weak answer is saying "RMSE is always better because it's more common" without justification. A critical omission is failing to mention that evaluation must happen on a test set; accuracy on training data (residuals) is not a reliable indicator of future performance. Finally, some candidates incorrectly state that the units of RMSE are squared (they are not, due to the final square root).
What usually comes next
Be prepared for "Can you give a specific business scenario where you'd absolutely choose one over the other?" or "What are the units of MAE and RMSE for our sales forecast?" (Answer: The same as the quantity being forecast, e.g., dollars or units sold). Another could be: "What other metrics might you look at besides these two?" (Answer: MAPE for percentage error, or bias to see if you're consistently over/under-forecasting).
A concrete example
Imagine forecasting sales for milk and a new gaming console. For milk, a stable product, being off by 1,000 units either way has a relatively linear cost (spoilage vs. minor stock-out). MAE is a good fit. For the console, under-forecasting by 1,000 units could mean missing a massive launch-day rush and losing customers to competitors. The cost of this large error is huge. RMSE is better here because it will heavily penalize the model for making such a large, costly mistake, guiding model selection to avoid these specific failures.
Interview question
When evaluating a sales forecast model where a large error is significantly more damaging to the business than a small one, which metric is most appropriate?
- a.Root Mean Squared Error (RMSE), because it disproportionately penalizes large, costly errors.Correct
- b.Bias, to determine if the model consistently over-forecasts or under-forecasts.
- c.Accuracy on the training set, to ensure the model has learned from all available historical data.
- d.Mean Absolute Error (MAE), because it provides the most direct and interpretable measure of average error.
Why? this is the answer
RMSE is correct because by squaring errors before averaging, it gives much more weight to large mistakes, aligning with the business need to avoid them. While MAE is more interpretable, it treats the business cost of errors as linear, which is not the case here.
Just read this? Test yourself on what you have been reading.
Read the original → otexts.com
- #analytics
- #metrics
- #forecasting
- #modeling
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on analytics — each one lists the topics its interview covers.
See open roles