Unit Testing ML: Beyond Standard Code Checks

Why it exists

Traditional software testing focuses on deterministic code paths. ML systems are different; they combine code with data and trained models, all of which can fail. Unit testing for ML was created to isolate and verify these individual components—data processing, feature engineering, and model inference—before they are integrated.

The mental model

Think of an ML pipeline as a factory assembly line. Unit testing is not about checking the final product. It's about inspecting each individual station in isolation. Does the station that cleans raw materials (data validation) handle rust and dirt (nulls, outliers)? Does the part-fitting station (model inference) accept the right components and produce a sub-assembly with the correct dimensions (output shape)?

How it works

ML unit tests fall into a few categories. First, data tests check schemas, ranges, and statistical properties of datasets (e.g., assert column 'age' > 0). Second, feature logic tests verify transformation functions with sample inputs (e.g., test_one_hot_encoder_handles_new_category). Third, model tests check the mechanics of prediction, not its accuracy. This involves loading a model and ensuring it produces outputs of the correct shape and type for a dummy input tensor, and that it doesn't crash on edge cases.

When to use it

Apply unit tests to any component that can be tested in isolation. This is perfect for data validation scripts, feature engineering functions, data transformation steps (like normalization or encoding), and the model serving class that loads and runs inference. The goal is to catch errors early, before running a full, expensive training or integration test.

When not to use it

Unit tests are not for evaluating model performance (like accuracy, precision, or recall). That's the job of evaluation tests run on a holdout dataset. They also can't detect complex system-level failures or data drift over time; those require integration tests and monitoring.

One canonical example

A unit test for a feature engineering function might check that a log_transform function correctly handles a zero or negative input. Instead of failing silently or producing a NaN that breaks the entire pipeline downstream, the test would assert that the function either raises a specific error or returns a pre-defined, safe value. This prevents a single bad data point from causing a catastrophic failure.