What infrastructure is needed for a Continuous Training pipeline?

WHAT THIS TESTS:

This question tests whether you can distinguish Continuous Training from generic CI/CD and design an event-driven ML system where data changes initiate retraining. Interviewers want to see that you know ML-specific infrastructure components and how they interact to automate retraining and validation safely.

A GOOD ANSWER COVERS:

A good answer hits five components and their interactions in order. First, event triggers such as data drift detection, schedule-based cron jobs, or monitoring alerts that signal when retraining is necessary. Second, a pipeline orchestrator like Agent Platform Pipelines or TensorFlow Extended to manage the workflow graph of extraction, transformation, training, and evaluation. Third, a feature management layer to ensure training-serving consistency and point-in-time correctness across automated runs. Fourth, a model registry to version candidate artifacts and store evaluation baselines so new models can be compared against production. Fifth, automated validation gates that run offline evaluation, fairness checks, or shadow tests before any promotion occurs. The interaction flow should be explicit: a trigger invokes the orchestrator, which executes training, evaluates the candidate against the baseline in the registry, and either promotes the model or halts and alerts the team.

COMMON WRONG ANSWERS:

Common wrong answers treat CT as standard software CI/CD by only discussing Git webhooks and Cloud Build for code deployment while ignoring data-driven triggers. Another red flag is describing a single monolithic script without orchestration, which prevents step-level retries, caching, and rollback. Candidates also err by omitting validation gates, implying every retrained model should automatically reach production.

LIKELY FOLLOW-UPS:

Interviewers may ask how you prevent training-serving skew, how you rollback a promoted model that degrades in production, or how you control compute costs when automating large model retraining. They might also ask how you version datasets alongside models for reproducibility.

ONE CONCRETE EXAMPLE:

A monitoring alert on incoming data triggers a Cloud Build webhook or an orchestrated pipeline run. Using TensorFlow Extended on Agent Platform Pipelines, the workflow ingests data, transforms features consistently, trains a new model, and evaluates it against the current production baseline stored in a model registry. If evaluation metrics exceed predefined thresholds, the pipeline updates the serving artifact; if not, it aborts and notifies engineers. Cloud Build can manage the deployment step, ensuring the validated model reaches serving infrastructure only after passing all gates.