Stream-Table Duality: Two Views of One Dataset

Why it exists

For decades, databases and message queues lived in separate worlds. You queried a table for current state, then published events to a bus for downstream systems. This meant every application had to write to both, or run fragile ETL jobs to keep them in sync. The result was duplicated effort, inconsistent data, and the hidden cost of maintaining two pipelines for the same facts.

The mental model

Think of a table as the current frame of a movie, and a stream as the full reel of film. The frame is just a projection of every past event played forward. If you have the reel, you can always reconstruct the frame. If you have the frame and a camera running, you can produce a new reel. They are not different data; they are different temporal views of one reality.

How it works

A table stores the latest value for each key. A stream stores an ordered log of every insert, update, and delete. In systems like Kafka with ksqlDB or Flink, a stream of changes can be materialized into a table by replaying events and keeping the newest state per key. Conversely, a table can be turned into a stream by emitting a change event whenever a row is modified, often through change data capture on the transaction log. The schema and keys must align so that the mapping is deterministic.

When to use it

Use this model when you need both real-time processing and historical lookup. Event sourcing is the classic pattern: your stream is the system of record, and tables are ephemeral queries built on top. It also appears in lambda or kappa architectures where you want to serve low-latency dashboards from a table while feeding the same facts into stream analytics. If you already have a database and want to unlock stream processing without rewriting producers, CDC plus a stream processor gives you duality for free.

When not to use it

Do not force duality onto systems that only need one view. A pure lookup cache with no downstream consumers does not need a changelog. Likewise, a fire-and-forget telemetry stream that is never queried by key should probably stay a stream; materializing it into a table wastes storage and adds operational complexity. Also avoid it when your event schema lacks primary keys or ordering guarantees, because rebuilding state becomes nondeterministic.

One canonical example

Imagine a ride-sharing app tracking driver locations. The database table holds each driver's latest latitude and longitude. The CDC stream emits a record every time a driver moves. A pricing service consumes the stream to detect surge in real time, while a support dashboard queries the table to see where a driver is right now. Both services read from the same logical dataset: the dashboard sees the table view, the pricing algorithm sees the stream view. If the team had built a separate event bus from application code instead of tapping the database log, they would face dual writes and drift.