Random split vs walk-forward validation in forecasting

WHAT THIS TESTS: This question probes whether you understand that time-series data is not independent and identically distributed. In forecasting, the order of observations carries information; a standard random split severs that order and allows the model to peek at the future during training. Interviewers want to see that you treat temporal structure as a first-class constraint rather than an afterthought.

A GOOD ANSWER COVERS: First, the mechanism of a random split: it shuffles all observations and partitions them into train and test without regard to time, which means a model can learn patterns from 2024 and be tested on 2022. Second, the mechanism of walk-forward validation: you start with a minimum training set of early observations, forecast the very next period, record the error, then expand the training set by one observation and repeat. Third, the core principle of no future data in training: every training set must contain only observations that occurred prior to the test observation. Fourth, the practical output: you average forecast accuracy across all test periods to get a robust estimate of out-of-sample performance. Fifth, optional nuance about expanding versus sliding windows: an expanding window keeps all past data, while a sliding window uses a fixed history length, which matters for non-stationary series.

COMMON WRONG ANSWERS: Claiming that cross-validation always requires shuffling, so random splits are fine for time series. Saying that walk-forward validation is only about getting more test folds rather than preventing leakage. Asserting that tree-based models are immune to temporal leakage because they do not assume linearity. Suggesting that you can fix the problem by blocking contiguous chunks randomly without preserving the forward order.

LIKELY FOLLOW-UPS: How would you handle multiple forecast horizons, such as predicting one step ahead versus twelve steps ahead? When would you prefer a sliding window over an expanding window? How do you deal with seasonality when your minimum training set is shorter than one full seasonal cycle? What metrics do you use when errors are not symmetric, and how does that change your cross-validation setup?

ONE CONCRETE EXAMPLE: Imagine you are forecasting daily electricity demand. A random split might place a hot summer Tuesday in the training set and a cool autumn Tuesday in the test set. The model learns the summer peak and appears to predict the autumn day well because it has memorized similar day-of-week patterns from other seasons, but it has never been forced to forecast forward from autumn into winter. Walk-forward validation starting in January would require you to forecast February 1st using only January data, then February 2nd using January through February 1st, and so on through December. Your final accuracy reflects true sequential prediction rather than interpolation.

Source: otexts.com