Apache Kafka: A Distributed Log for Data Streams
Think of Kafka as a durable, append-only log for events, not just a temporary message queue. It excels at handling high-throughput, real-time data feeds for analytics or log aggregation. The footgun is treating it like a simple broker, leading to data loss.
The mental model
Apache Kafka is best understood as a distributed, append-only log file that services can write to and read from. Unlike a traditional message queue where messages are deleted after being consumed, Kafka retains data for a configurable period, allowing multiple independent consumers to process the same event stream at their own pace.
How it works
Kafka achieves high throughput by optimizing for how computers actually work. It uses an efficient binary TCP protocol and groups messages into sets. This strategy turns a bursty stream of random message writes into large, sequential writes to the disk and contiguous memory blocks. Sequential disk I/O is orders of magnitude faster than random I/O, which is the key to Kafka's low-latency performance even with massive data volumes.
When to use it
Use Kafka as the backbone for systems that handle real-time data feeds. Three common use cases are: first, building real-time analytics pipelines that ingest user activity or IoT sensor data; second, aggregating logs and metrics from hundreds or thousands of services into a central location; third, as an event store for event-driven microservice architectures. Its ecosystem includes Kafka Connect for integrating with external systems like databases.
When not to use it
Kafka is overkill for simple, low-volume task queues where a lighter message broker would suffice. Its distributed nature adds operational complexity, making it a poor choice for simple point-to-point messaging. If you need complex server-side message filtering or routing logic, a traditional message broker like RabbitMQ might be a better fit.
One canonical example
An e-commerce site tracks every user click as an event. Web servers act as producers, writing each click event to a 'clicks' topic in Kafka. Multiple consumer applications can then read from this topic independently: a real-time fraud detection system scans for suspicious patterns, a dashboarding service aggregates clicks for business metrics, and a batch process archives the data to a data warehouse nightly. All consumers operate on the same unified data stream without interfering with each other.
Interview question
Which characteristic best highlights Kafka's fundamental difference and advantage over a traditional message queue for data processing?
- a.Its ability to automatically delete messages once they are consumed by any single subscriber.
- b.Its built-in complex server-side routing and filtering capabilities for messages.
- c.Its design as an append-only log that retains data for a period, enabling multiple independent consumers to process the same events.Correct
- d.Its primary optimization for random disk I/O operations to achieve high throughput.
Why? this is the answer
The card emphasizes Kafka's design as an append-only log that retains data, allowing multiple independent consumers to process the same event stream without interfering with each other. This contrasts with traditional message queues, which often delete messages after a single consumer processes them, making option A incorrect.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #kafka
- #stream processing
- #event sourcing
- #distributed systems
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles