Training vs. Test Sets: Judging a Forecast's True Accuracy
Don't judge a forecast by how well it fits the data it was trained on. To measure true accuracy, split your data into a training set for model fitting and a test set for evaluation, using data the model has never seen before.
Why it exists
A model can get very good at explaining the data it was built with, but that doesn't mean it has learned the right patterns. We need a way to check if a model has learned a generalizable pattern or just memorized the noise in the training data, a problem known as overfitting.
The mental model
Think of it like studying for an exam. If you only study the exact questions from last year's test (the training data), you might ace those specific questions. But a real exam (the test data) will have new questions. Your performance on the new questions is the true measure of your understanding, not how well you memorized the old ones.
How it works
You partition your available historical data into two distinct, non-overlapping sets. The first, larger portion is the training set. You use this data to select a forecasting method and estimate its parameters. The second, smaller portion is the test set. You generate forecasts for the period covered by the test set (using only information available up to that point) and compare these forecasts to the actual values in the test set. The size of the test set is often around 20% of the total data.
When to use it
Always use a train/test split when developing and comparing forecasting models. It is the fundamental step for any form of model evaluation, from simple regressions to complex neural networks. It's how you gain confidence that your model will perform well on future, unseen data.
When not to use it
You might not use a simple train/test split if you have very little data, as holding some back could prevent fitting a meaningful model. In such cases, more advanced techniques like time series cross-validation are used, but the principle of evaluating on unseen data remains. Once a final model is chosen, it's common to retrain it on the full dataset (training + test) to make future forecasts.
One canonical example
You have 100 months of sales data. To evaluate a new forecasting model, you designate the first 80 months as the training set and fit your model using only this data. Then, you use the fitted model to forecast sales for months 81 through 100. Finally, you compare your forecasts for those 20 months against the actual sales data you held back in the test set. The errors in this comparison give a realistic estimate of the model's future performance.
Interview question
What is the primary reason for using a separate test set when evaluating a forecasting model?
- a.To prevent the model from becoming too complex and difficult to interpret.
- b.To allow for easier comparison between different forecasting methods during development.
- c.To ensure the model is trained on the most recent and relevant data.
- d.To measure how well the model generalizes to unseen data, avoiding over-optimistic performance estimates.Correct
Why? this is the answer
The core purpose of a test set is to evaluate a model's ability to generalize to data it has never seen, preventing over-optimistic performance estimates that arise from models merely memorizing the training data. While a test set facilitates comparison, its fundamental role is to provide a realistic assessment of future performance.
Just read this? Test yourself on what you have been reading.
Read the original → otexts.com
- #forecasting
- #machine learning
- #model evaluation
- #data science
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on forecasting — each one lists the topics its interview covers.
See open roles