CI/CD for microservice-based ML systems

WHAT THIS TESTS Whether you understand that decomposing an ML system into services shifts CI/CD from one big pipeline to many coordinated ones, with contracts as the linchpin.

HOW IT DIFFERS FROM A MONOLITH In a monolith, one pipeline builds, tests, and deploys everything together, simple but coupled. With microservices each component, feature engineering, training, and inference, has its own repository, pipeline, version, and deploy cadence. This enables faster, isolated releases and independent scaling, but introduces integration risk because services evolve separately.

MANAGING DEPENDENCIES Version every interface explicitly: API schemas, feature definitions, and model artifact formats. Pin which model version and feature schema the inference service expects. Because ML services share data contracts, not just function signatures, a feature schema change is as dangerous as an API change and must be versioned and backward compatible during transition.

CONTRACT TESTING Use consumer-driven contract tests so the inference service's expectations of the feature service are verified in the feature service's pipeline; a breaking change fails that pipeline before it reaches production. Add schema validation on data passed between services. This replaces relying on a single end-to-end test.

DEPLOYMENTS Deploy services incrementally with canary or blue-green and keep APIs versioned so old and new can coexist during rollout. Roll back a single service without touching the rest.

COMMON WRONG ANSWERS Treating it as a monolith with one all-or-nothing deploy, ignoring data and model contracts, or relying solely on end-to-end tests that are slow and flaky.

ONE CONCRETE EXAMPLE The feature service adds a column and renames another. Its pipeline runs the inference service's consumer contract test, which fails because the renamed field breaks the expected schema. The team ships a backward-compatible version first, canary-deploys it, migrates the inference service, then removes the old field, all without redeploying training.