Describe a robust strategy for GitOps database schema migrations

WHAT THIS TESTS: This question tests whether you understand the impedance mismatch between imperative stateful operations like database schema migrations and declarative stateless GitOps workflows. Interviewers want to see that you recognize why tools like Argo CD or Flux cannot naively apply schema changes the way they apply Kubernetes manifests, and that you can design a pipeline where Git remains the source of truth without pretending that a database schema is just another YAML file.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, version control: migration files live in the same Git repository as application code, using tools like Flyway, Liquibase, or golang-migrate, with sequential or timestamped filenames so ordering is explicit. Second, execution mechanics: migrations run as idempotent jobs triggered before the application deployment, such as Argo CD pre-sync hooks, Kubernetes Jobs, or init containers, ensuring the schema is ready before new app code starts. Third, safety and rollback: the strategy includes backward-compatible migrations, expand-contract patterns for zero-downtime deploys, and a clear rollback path that does not rely on Git revert alone because you cannot un-apply a destructive ALTER TABLE. Fourth, separation of concerns: infrastructure tools like Terraform manage the database instance and credentials, while schema tools manage the logical state inside it; never let Terraform execute live schema changes.

COMMON WRONG ANSWERS: Common wrong answers include suggesting Terraform should manage schema state via SQL providers, which couples infrastructure and data lifecycle and risks partial failures. Another red flag is proposing that the application itself runs migrations on startup in a distributed system, which creates race conditions when multiple replicas start simultaneously. Saying you will just have a manual step for schema changes also undermines the GitOps goal of full automation and auditability.

LIKELY FOLLOW-UPS: Interviewers often follow up by asking how you handle long-running migrations on large tables without locking, how you manage secrets for database credentials in a GitOps pipeline, or how you would recover if a migration fails halfway through and leaves the database in a dirty state. They may also ask how to keep schema changes backward compatible across multiple microservices sharing a database.

ONE CONCRETE EXAMPLE: Imagine a team using Argo CD and Flyway. The Git repo contains a migrations directory with V001__add_index.sql. The Argo CD Application manifest defines a pre-sync hook that runs a Kubernetes Job using the Flyway container against the database. The job must complete successfully before the app Deployment sync proceeds. If the migration fails, the sync aborts, the app does not roll out, and an alert fires. For a large table, V001 uses an online DDL tool like pt-online-schema-change or native ONLINE ALTER to avoid locking, and the application code is deployed only after the schema change succeeds.

Source: thenewstack.io