Kappa Architecture Unifies Batch and Stream

WHY IT EXISTS: Lambda architecture split big data systems into a batch layer for correctness and a speed layer for low latency. Engineers maintained two codebases, two schedulers, and complex merge logic to reconcile slightly different results. The pain of duplication often exceeded the value. Kappa architecture was created to answer a simpler question: what if you only needed one processing layer, and batch was merely a special case of streaming?

THE MENTAL MODEL: Imagine your entire dataset as a single VHS tape that you never erase. Real-time processing is watching the tape as it records. Batch processing is popping the tape out and watching from the beginning. If you keep every frame forever, you do not need a separate player for old footage; you rewind the same player. That is Kappa: one immutable log, one engine, two speeds of reading it.

HOW IT WORKS: Every event appends to a distributed commit log such as Apache Kafka or Apache Pulsar. A stream processing job consumes the log, maintains local state, and publishes materialized views to a database or cache. When business logic changes, you deploy a new job version, reset its offset to the start of the log, and let it replay history to rebuild the view. Once it catches up to the tail, it continues in real time. There is no nightly batch script because a batch job is semantically just a stream job that has finished reading a closed log.

WHEN TO USE IT: Kappa fits event-centric domains where history must be reprocessed whenever logic changes. It works well for event-sourced microservices, real-time feature stores, and recommendation engines that rebuild user profiles from raw clickstreams. Choose it when operational simplicity and a single codebase matter more than squeezing every dollar from storage.

WHEN NOT TO USE IT: Avoid Kappa for ad-hoc analytical queries over petabyte-scale history, where dedicated batch engines like Spark or Presto outperform stream replay. It also falters with massive shuffle-heavy joins that streaming state backends handle poorly. If your organization cannot afford indefinite log retention or if replaying years of data would take weeks, the unified model becomes a liability.

ONE CANONICAL EXAMPLE: LinkedIn pioneered this approach for activity data. Each user click, search, and share appends to Kafka. A stream job continuously updates a real-time profile view. When the product team redefines an influence score, engineers launch a new consumer group, replay months of events from the log, and backfill a fresh serving table without maintaining a separate batch pipeline.