Change Data Capture (CDC): Turn Your Database Into a Stream

THE MENTAL MODEL: Think of your database not as a static place to store data, but as a source of truth that emits a continuous stream of events. CDC is the mechanism that captures every insert, update, and delete as a discrete, ordered event. Instead of asking "what is the current state of the entire user table?", you listen for "user 123's email was just updated." This shifts your architecture from batch-oriented polling to real-time, event-driven flows. The result is a delta-driven dataset, where you only process what has changed.

HOW IT WORKS: While there are several patterns, a common and robust method involves tapping into the database's internal transaction log (also known as a write-ahead log or WAL). This log is an ordered record of every change the database is about to commit. CDC tools read this log, interpret the low-level database operations, and transform them into a structured, logical stream of events. For example, an event might look like: "Row with ID 456 in the 'orders' table was updated. Old value: {status: 'pending'}, New value: {status: 'shipped'}". This is more reliable and less intrusive than using triggers or polling a last_updated_at column.

WHEN TO USE IT: CDC shines when you need to propagate data changes from a system of record to other systems with low latency. Three key use cases are: first, data replication to secondary databases or read replicas; second, feeding data warehouses and analytics platforms in near real-time, avoiding slow batch ETL jobs; and third, keeping external systems like search indexes or caches perfectly in sync with your primary database.

WHEN NOT TO USE IT: CDC is overkill for simple, infrequent data syncs where a nightly batch job is sufficient. It also introduces operational complexity: you now have a critical streaming pipeline to monitor and manage. If your application can tolerate stale data for periods, the cost of implementing and maintaining a CDC pipeline may not be justified. It is not a replacement for APIs when you need complex business logic or validation before an action is taken.

ONE CANONICAL EXAMPLE: An e-commerce site uses a database for its orders. They need to update both their inventory system and a customer-facing dashboard immediately when an order's status changes. When an order is updated from 'processing' to 'shipped' in the orders table, CDC captures this change from the database log. It produces an event that is published to a message queue. The inventory service and the dashboard service both subscribe to this queue, see the event, and update their own state accordingly, all without putting extra load on the primary database.