Explain ML pipelines and typical CI/CD/CT components

WHAT THIS TESTS: The interviewer wants to know if you see ML systems as composite artifacts where code, data, and models all change at different speeds. They are checking whether you can separate the software delivery loop from the model training loop and explain the handoffs between them. Senior candidates should show awareness of reproducibility, promotion gates, and production monitoring that triggers retraining.

A GOOD ANSWER COVERS: First, define the classic CI/CD spine: source control for code and configs, automated unit and integration tests, artifact builds such as containers or packages, and deployment automation to staging or production. Second, add the CT layer: data extraction and validation to catch schema drift or quality issues, reproducible training pipelines with versioned datasets and code, offline evaluation against a baseline using held-out data or pre-defined metrics, a model registry that stores blessed artifacts with metadata, and a staged rollout like canary or shadow deployment. Third, mention the feedback loop: production monitoring for prediction drift or performance decay, and an automated or human-in-the-loop trigger that starts a new training run. Fourth, note governance and rollback: the ability to roll back a model independently of the serving code because model artifacts are separate deployable units.

COMMON WRONG ANSWERS: A major red flag is describing an ML pipeline as if it were a standard web service CI/CD pipeline with no mention of data versioning or model registries. Another mistake is conflating training and deployment into a single step without evaluation gates. Candidates sometimes forget monitoring entirely or suggest retraining on a fixed schedule without linking it to business metrics or data drift. Saying that you simply retrain manually when accuracy drops signals immature operations.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle data schema changes that break training pipelines, how you version datasets alongside code, or what metrics you use to decide whether a new model should replace the old one in production. They might also probe rollback strategies when a deployed model degrades, or ask how you test ML code when model behavior is probabilistic rather than deterministic.

ONE CONCRETE EXAMPLE: Imagine a recommendation system. The CI/CD pipeline builds the serving container and feature engineering library from a Git commit, runs unit tests, and deploys the service. Separately, the CT pipeline ingests daily user interaction logs, validates that click-through rates and feature distributions look normal, trains a new ranking model with the latest data, evaluates it against last weeks champion model using A/B test metrics, and if lift exceeds two percent with no fairness regressions, promotes it to the model registry. The production service pulls the new artifact, runs a canary on five percent of traffic, and monitors conversion rate for an hour before full rollout. If conversion drops, the system rolls back to the previous model artifact without redeploying code.