Train-test split vs. time-series cross-validation?
Tests if you see why temporal data breaks random splits. Contrast random sampling with sequential 'walk-forward' validation, where you only use past data to predict the future.
What's really being asked
This question tests your grasp of a fundamental concept in modeling: data leakage. For time-series data, the interviewer is checking if you understand that the temporal order of data is non-negotiable. Using future data to train a model that predicts the 'future' is a critical error that invalidates the model's evaluation. They want to see if you can articulate why a standard train-test split is inappropriate and what the correct, temporally-aware alternative is.
The full answer
First, define a traditional random train-test split, noting it assumes data points are independent and is fine for many non-temporal problems. Second, define time-series cross-validation, often called 'evaluation on a rolling forecasting origin' or walk-forward validation. Explain that this involves creating a series of training/test sets where the training set always consists of observations that occurred before the test set. For example, train on data from periods 1-100 to test on 101, then train on 1-101 to test on 102. Third, explain the core problem: a random split on time-series data leaks information from the future into the training set. This violates the real-world constraint of only having past data to make a forecast, resulting in overly optimistic and useless performance metrics. Finally, conclude that time-series CV is necessary because it simulates this real-world constraint, providing a realistic estimate of model performance on unseen future data.
The mistakes people make
Suggesting standard k-fold cross-validation. This is a major red flag as it completely randomizes the data, ignoring the temporal structure and committing the exact error the interviewer is probing for. Another error is downplaying the issue, suggesting a random split is 'good enough' to start. For forecasting, it is fundamentally invalid. A less severe but still weak answer describes the 'how' (rolling windows) but fails to explain the 'why' (data leakage, causality, optimistic bias).
What usually comes next
How would you set the initial training set size and the step size for the rolling origin? What are the computational trade-offs of this method compared to a single split? Can you describe a situation where a simple, single time-based split (e.g., train on 2020-2022, test on 2023) might be sufficient?
A concrete example
Imagine you're forecasting daily sales for a retail store. A traditional 80/20 random split might put sales data from December into your training set to help predict a value in July. The model learns from future events, which is impossible in reality. A time-series approach, 'evaluation on a rolling forecasting origin', would be to train the model on Jan-June to predict July 1st, then train on Jan-June + July 1st to predict July 2nd, and so on. This ensures the model only ever uses past information, just as it would have to in a live production environment.
Interview question
What is the primary reason a standard random train-test split is unsuitable for evaluating time-series forecasting models?
- a.It often results in an uneven distribution of seasonal components between training and test sets.
- b.It significantly increases the computational cost of model training and evaluation compared to other methods.
- c.It can lead to data leakage by allowing future information into the training set, yielding an overly optimistic performance estimate.Correct
- d.It makes it difficult to compare model performance across different time periods due to varying data distributions.
Why? this is the answer
A random train-test split on time-series data can inadvertently place future observations into the training set, a phenomenon known as data leakage. This violates the real-world constraint of only having past data for forecasting, leading to an unrealistic and overly optimistic assessment of the model's performance on unseen future data.
Just read this? Test yourself on what you have been reading.
Read the original → otexts.com
- #machine learning
- #time series
- #model evaluation
- #data science
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on machine learning — each one lists the topics its interview covers.
See open roles