Apache Kafka: A Distributed Log for Data Streams

THE MENTAL MODEL: Apache Kafka is best understood as a distributed, append-only log file that services can write to and read from. Unlike a traditional message queue where messages are deleted after being consumed, Kafka retains data for a configurable period, allowing multiple independent consumers to process the same event stream at their own pace.

HOW IT WORKS: Kafka achieves high throughput by optimizing for how computers actually work. It uses an efficient binary TCP protocol and groups messages into sets. This strategy turns a bursty stream of random message writes into large, sequential writes to the disk and contiguous memory blocks. Sequential disk I/O is orders of magnitude faster than random I/O, which is the key to Kafka's low-latency performance even with massive data volumes.

WHEN TO USE IT: Use Kafka as the backbone for systems that handle real-time data feeds. Three common use cases are: first, building real-time analytics pipelines that ingest user activity or IoT sensor data; second, aggregating logs and metrics from hundreds or thousands of services into a central location; third, as an event store for event-driven microservice architectures. Its ecosystem includes Kafka Connect for integrating with external systems like databases.

WHEN NOT TO USE IT: Kafka is overkill for simple, low-volume task queues where a lighter message broker would suffice. Its distributed nature adds operational complexity, making it a poor choice for simple point-to-point messaging. If you need complex server-side message filtering or routing logic, a traditional message broker like RabbitMQ might be a better fit.

ONE CANONICAL EXAMPLE: An e-commerce site tracks every user click as an event. Web servers act as producers, writing each click event to a 'clicks' topic in Kafka. Multiple consumer applications can then read from this topic independently: a real-time fraud detection system scans for suspicious patterns, a dashboarding service aggregates clicks for business metrics, and a batch process archives the data to a data warehouse nightly. All consumers operate on the same unified data stream without interfering with each other.