Describe securing an automated ML pipeline and CI/CD integration points

WHAT THIS TESTS: The interviewer wants to know if you treat an ML pipeline as a software supply chain that requires layered controls, not just traditional application security. They are looking for awareness of OWASP ML06 AI Supply Chain Attacks and the understanding that ML artifacts such as serialized models, training containers, and data ingestion paths are as critical as production code. The question also checks whether you know where to inject security gates without destroying iteration speed.

A GOOD ANSWER COVERS: A good answer maps specific scans to CI/CD stages. At the code and commit stage, run SCA on dependency manifests, secrets scanning, and SAST on training code and notebooks. During the build stage, scan container base images for CVEs, lint Infrastructure-as-Code for misconfigurations, and validate notebook outputs. Before model registry promotion, scan serialized artifacts for unsafe deserialization such as pickle payloads, enforce model signing, and verify data provenance and lineage to mitigate data poisoning. At deployment and runtime, implement input validation and adversarial detection to guard against input manipulation, plus runtime container scanning. Explicit targets to name include base images, requirements files, conda environments, serialized model files, training data buckets, and feature store connectors.

COMMON WRONG ANSWERS: A red flag is confining security to a single post-training penetration test or manual review. Another mistake is treating the model registry as a trusted zone without artifact scanning, which ignores model poisoning and supply chain risks. Saying you would only scan application code while ignoring notebooks and data pipelines is also weak, since Jupyter notebooks and training data are common attack vectors. Finally, suggesting that pickle or joblib files are safe without additional sandboxing or signing shows inexperience with ML-specific deserialization risks.

LIKELY FOLLOW-UPS: Expect the interviewer to ask how you would handle a vulnerable upstream base image when no patched version exists, or how you would verify data provenance in a multi-tenant environment. They may also probe whether you would block a deployment based on a critical CVE in a training dependency, or how you balance scanning latency against researcher iteration speed in an automated pipeline.

ONE CONCRETE EXAMPLE: Suppose a team uses Kubeflow on Kubernetes. In CI, Trivy scans the training container image at build time and fails on critical CVEs. SCA scans requirements.txt for known malicious packages. Before the model hits the registry, a pipeline step validates that the artifact is a signed ONNX file rather than an unsigned pickle, and checks data lineage metadata in the feature store. At runtime, an API gateway validates input schema and rate limits requests to mitigate adversarial probing. If a container scan finds a critical vulnerability in a CUDA base image with no immediate patch, the pipeline quarantines the image and falls back to a hardened CPU training node until the fix is available.