How do you measure forecast accuracy and compare MAE to RMSE?
This tests out-of-sample validation and how MAE and RMSE weight errors. A strong answer demands a train-test split, defines both, and notes RMSE punishes outliers more while MAE is more robust. A red flag is citing in-sample fit instead of held-out error.
What's really being asked
This question tests two distinct skills that separate junior analysts from senior forecasters. First, it checks whether you know that forecast accuracy must be measured on genuine out-of-sample forecasts rather than in-sample residuals. Second, it checks whether you understand the mathematical and business implications of choosing MAE versus RMSE as your accuracy metric.
The full answer
A good answer hits four things in order. First, it insists on a train-test split where the model is fit on training data and evaluated on a test set that was not used during fitting, because only held-out data provides a reliable indication of how the model will perform on new data. Second, it defines MAE as the mean of absolute forecast errors, which is robust and expressed in the original units of the series. Third, it defines RMSE as the square root of the mean squared error, which is also in original units but penalizes large deviations more heavily because errors are squared before averaging. Fourth, it gives a concrete decision rule: prefer MAE when outliers are common and you want an interpretable, robust measure of typical error; prefer RMSE when large misses are disproportionately costly and you want to strongly penalize them.
The mistakes people make
Common wrong answers include three red flags. One is reporting in-sample fit statistics like R-squared or standard error of the regression as if they were forecast accuracy, which ignores the fact that residuals on fitted data are not genuine forecasts. Another is claiming that RMSE is always better because it is more sensitive, without acknowledging that this same sensitivity makes it unstable in the presence of outliers. A third is failing to mention that both metrics are scale-dependent, so comparing them across products with very different sales volumes requires normalization or percentage errors.
What usually comes next
Interviewers often follow up by asking how you would choose the size of the test set, in which case you should say it is typically about 20 percent of the total sample but depends on the length of the series and the forecast horizon. They may also ask about scale-independent alternatives like MAPE or MASE, or ask how you would handle a situation where the test set contains an anomalous month like a holiday spike.
A concrete example
Suppose you are forecasting monthly sales for a retail SKU. You hold out the most recent six months as a test set. In month one the forecast is off by 10 units, in month two by 100 units, and the remaining months are within 5 units. The MAE would be roughly 25 units, treating the large miss as just another error. The RMSE would be roughly 42 units, pulling the average upward because the 100-unit miss is squared. If a 100-unit miss means stockouts and lost customers, RMSE correctly surfaces that pain; if the 100-unit miss was a one-time data entry error, MAE gives a truer picture of model performance.
Interview question
In which scenario is RMSE preferred over MAE for measuring forecast accuracy?
- a.When large misses are disproportionately costly to the businessCorrect
- b.When evaluating a model on the same data used to train it
- c.When a single large forecast error is caused by a data entry mistake
- d.When comparing forecast accuracy across products with vastly different sales volumes
Why? this is the answer
The card recommends RMSE when large forecast misses are disproportionately costly because squaring errors heavily penalizes large deviations. Option C is tempting because it describes an outlier scenario, but the card explicitly pairs one-time anomalies with MAE since it is robust and does not let a single error dominate the metric.
Just read this? Test yourself on what you have been reading.
Read the original → otexts.com
- #forecasting
- #mae
- #rmse
- #model-evaluation
- #time-series
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on forecasting — each one lists the topics its interview covers.
See open roles