Feature Backfilling: Populating Historical Data for ML
Feature backfilling computes a new feature's values for historical data. It's how you generate a complete training dataset after defining a new signal, like a user's 7-day purchase history. The footgun is using future data, causing data leakage.
WHY IT EXISTS ML models require consistency between the data used for training (historical) and serving (live). When you invent a new feature, it exists for future predictions but is missing from the historical data needed to train the model. Feature backfilling solves this by generating the feature's values for the past.
THE MENTAL MODEL Imagine adding a new column to a massive, historical spreadsheet, but the column is empty. You've just defined 'Column F' (e.g., 7-day average transaction size), but all rows from last year are blank. Backfilling is the process of running the calculation for 'Column F' on every single past row to make the dataset complete.
HOW IT WORKS A backfill is a large-scale data processing job. First, you define the feature's transformation logic, like a SQL query or Python function. Second, you specify the historical time range to populate. Third, you execute a job that reads the raw historical data, applies the transformation for each point in time, and writes the resulting feature values to your feature store or training dataset. Feature store platforms often automate and manage this process.
WHEN TO USE IT Use backfilling whenever you create a new feature or change an existing feature's logic. It is a mandatory step before retraining a model with an updated feature set. It's also how you populate a central feature store, allowing other teams to reuse a new, production-ready feature without re-computing it themselves.
WHEN NOT TO USE IT Avoid backfilling if the computational cost is prohibitive and a simple proxy value (like zero or a statistical mean) is sufficient for model training. You also don't need it if a new feature is only for real-time monitoring and has no impact on historical model training.
ONE CANONICAL EXAMPLE A credit card company adds a new feature to its fraud model: is_tx_in_new_city_last_24h. To train the model, they need this feature's value for millions of past transactions. They run a backfill job that iterates through their transaction history. For each transaction, it checks the user's transaction history from the preceding 24 hours to calculate the feature's true or false value. This populates the feature store, ready for model training.
Read the original → feast.dev
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.