What automated tests belong in CI before deploying a classification model?

WHAT THIS TESTS: This question checks whether you treat a machine learning model as a first-class software asset that requires its own validation layer in CI. The interviewer wants to see that you understand the distinction between testing the code that trains a model and testing the model artifact itself. MLOps maturity depends on automating validation across data, model, and code pipelines, not just running unit tests on Python functions.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, data validation tests such as schema checks, distribution drift detection, and null rate assertions on the training and holdout sets. Second, model performance regression tests that compare the new model against a baseline or champion on accuracy, precision, recall, or AUC-ROC with thresholds that fail the build if the new model underperforms by more than one or two percent. Third, model-specific behavior tests including fairness or bias audits across demographic slices, confusion matrix validation per class, and out-of-vocabulary or adversarial input checks for the classification task. Fourth, artifact and reproducibility checks such as verifying model file hashes, ensuring the training container is reproducible, and confirming that the serialized model loads correctly in the inference environment.

COMMON WRONG ANSWERS: The biggest red flag is listing only API or integration tests like HTTP status codes and response latency while ignoring model behavior. Another weak pattern is mentioning manual notebooks or human-in-the-loop review as a CI gate. Saying you would test the training code but not the resulting artifact is also a miss. Finally, vague answers like just run end-to-end tests without specifying what properties of the model are being asserted show shallow MLOps experience.

LIKELY FOLLOW-UPS: The interviewer may ask how you would automate a champion-challenger rollout in the CD stage after CI passes. They might probe how you handle data drift detection when labels are delayed, or ask you to define the exact metrics and thresholds you would enforce for a binary classifier in a regulated domain like finance or healthcare.

ONE CONCRETE EXAMPLE: For a fraud detection classifier, the CI pipeline would first assert that the transaction feature schema has not changed and that the fraud rate in the training batch is between 0.1 and 5 percent. Next, it would run the trained model on a holdout set and require that precision at the default threshold is no more than 0.5 percent below the current production model. Then it would run a slice analysis to ensure recall for high-value accounts does not drop below 92 percent. Finally, it would verify the exported ONNX artifact checksum and run a smoke test loading the model in the serving container to confirm inference completes in under 50 milliseconds.

Source: ml-ops.org