Design a scalable, fault-tolerant real-time IoT data ingestion system
This tests separation of edge connectivity, buffering, and processing. A strong answer names an edge gateway, Kafka as the backplane, stream processing, and cold storage, plus backpressure and partitioning.
What's really being asked
Your ability to separate the concerns of ingestion, buffering, processing, and serving when dealing with high fan-in from thousands of unreliable clients. Interviewers want to see that you understand backpressure, partitioning for parallelism, fault domains, and the difference between a control plane and a data plane.
The full answer
First, an edge or gateway layer that handles protocol adaptation, device authentication, and rate limiting so that thousands of devices are not talking directly to the core pipeline. Second, a distributed streaming backplane such as Apache Kafka acting as the durable buffer with replication and topic partitioning to absorb traffic spikes and decouple producers from consumers. Third, a stream processing layer for filtering, enrichment, deduplication, or windowed aggregations before data lands in long-term storage. Fourth, a governance and operations layer that provides visibility, schema enforcement, and access control, analogous to a Kafka control plane and proxy layer that can route, protect, and transform traffic. Fifth, cold storage or a data lake for historical analytics and model training, plus a serving layer for real-time dashboards.
The mistakes people make
Proposing that devices connect directly to a relational database or a single API node without an intermediate buffer. Ignoring backpressure and assuming devices will self-throttle or that the network is always reliable. Failing to mention partitioning or replication and thus presenting a single point of failure. Overlooking schema governance and letting downstream consumers break when device firmware changes. Confusing the data path with the control path and omitting operational tooling.
What usually comes next
How would you handle a partition imbalance if one device model sends ten times more data than expected? What is your strategy for late-arriving or out-of-order events? How do you upgrade the schema without breaking existing consumers? How do you cost-optimize storage after ninety days while keeping query performance acceptable?
A concrete example
Imagine ten thousand smart meters reporting every second across a metropolitan area. A fleet of lightweight edge gateways accepts MQTT over TLS, validates JWT tokens, applies local rate limiting, and forwards compressed batches into a regional Kafka cluster configured with three replicas per partition and acks equals all. A Kafka proxy layer enforces encryption, routing rules, and traffic shaping while a schema registry ensures every message adheres to a versioned Avro contract. A stream processing job computes fifteen-minute rolling averages and writes raw telemetry to object storage. A console provides operators with consumer lag alerts, partition health, and automated failover visibility. When a rack fails, Kafka promotes replicas and the meters retry without data loss.
Interview question
Which design decision most directly prevents a surge from thousands of IoT devices from overwhelming downstream stream processors?
- a.Increasing topic partition count to match the number of connected devices
- b.Writing raw telemetry directly to cold storage instead of the streaming buffer
- c.Configuring Kafka with three replicas per partition and producer acks set to all
- d.Placing local rate limiting and batching at the edge gateway before the Kafka backplaneCorrect
Why? this is the answer
The edge gateway isolates the core pipeline by applying local rate limiting and batching before Kafka. While three replicas with acks=all ensures durability against rack failures, it does not throttle an incoming flood from thousands of devices.
Just read this? Test yourself on what you have been reading.
Read the original → conduktor.io
- #iot
- #kafka
- #system design
- #streaming
- #fault tolerance
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles