Train-Test Split vs. Time-Series Cross-Validation
This tests your grasp of data leakage in temporal data. A good answer explains why random splits create lookahead bias, then details how rolling-origin validation respects time. A red flag is just describing methods without explaining *why* one is necessary.
What's really being asked
This question tests your understanding of a fundamental principle in machine learning: preventing data leakage. Specifically, it probes whether you recognize that time-series data has an inherent temporal order that random sampling violates. Answering correctly shows you can design a valid model evaluation framework that produces realistic performance metrics, which is critical for any production forecasting system.
The full answer
A strong answer has four parts. First, identify the core problem with random splits for time-series: they cause 'lookahead bias' by allowing the model to train on future data to predict the past. Second, briefly define a traditional random split where, for example, 80% of data points are randomly selected for training. Third, explain a time-series cross-validation method like 'evaluation on a rolling forecasting origin,' where the training set only contains observations that occurred prior to the test set. For example, the test set could be a single day, and the training set is all preceding days. Fourth, conclude that this temporal discipline is necessary to get a realistic estimate of the model's future performance.
The mistakes people make
The most common red flag is describing the 'what' but not the 'why.' A candidate might correctly define both methods but fail to explain that random splits produce deceptively high accuracy scores because of data leakage. Another mistake is confusing general k-fold cross-validation with time-series splits; they are not the same because standard k-fold still shuffles data randomly. Finally, weak answers avoid specific terms like 'temporal dependency' or 'lookahead bias,' suggesting a surface-level understanding.
What usually comes next
Expect questions about implementation trade-offs. For example: 'How do you decide the size of the initial training set in a rolling-origin approach?' (Answer: It's a trade-off; it must be large enough for a stable initial model, but not so large that you have too few validation splits). Or, 'What if you have thousands of independent time series, like for different products? Can you use a random split then?' (Answer: You can split randomly by product ID, but for each product's timeline, you must still respect the temporal order).
A concrete example
Imagine you're forecasting daily user signups for 2024 using data from 2021-2023. A random split might put data from a marketing campaign in November 2024 into your training set to predict signups in February 2024. The model learns from the future, giving an inflated and useless accuracy score. The correct approach, time-series CV, would be to train on data up to Jan 31, 2024, to predict Feb 1, 2024; then train on data up to Feb 1, 2024, to predict Feb 2, 2024, and so on. This rolling origin simulates how the model would actually be deployed.
Interview question
What is the primary risk of using a standard random train-test split for a time-series forecasting model instead of a method that respects temporal order?
- a.The model will overfit to the training data, resulting in poor performance on the randomly selected test data.
- b.The test set may not contain enough recent data, making the evaluation less relevant to current conditions.
- c.The training process becomes computationally inefficient as it cannot leverage sequential data patterns.
- d.The model's performance metrics will be unrealistically high because it learns from future data to predict past events.Correct
Why? this is the answer
A random split allows the model to train on data from the future to predict the past (lookahead bias), leading to inflated and unrealistic performance scores. Distractor A is tempting, but the issue isn't poor test performance; it's deceptively good test performance due to this data leakage.
Just read this? Test yourself on what you have been reading.
Read the original → otexts.com
- #time series
- #ml
- #model evaluation
- #cross-validation
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on time series — each one lists the topics its interview covers.
See open roles