Design a near real-time pipeline to monitor orders per minute
Tests stream architecture and batch trade-offs. Outline: Kafka or Kinesis ingestion, Flink with tumbling windows, Druid or Pinot storage, Grafana alerts. Contrast batch on latency, exactly-once semantics, and cost. Red flag: calling cron SQL real-time.
Tests whether you can architect low-latency distributed pipelines and reason about operational complexity versus batch. A strong answer covers: ingestion via Kafka or Kinesis to decouple producers; stream processing with Flink or Spark Streaming using event-time windows and watermarking; OLAP storage like Druid or ClickHouse for sub-second queries; and Grafana for visualization. Contrast batch on latency, exactly-once semantics, backpressure, and cost.
Read the original → evermethod.com
- #real-time analytics
- #stream processing
- #system design
- #data engineering
- #kafka
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.