Streaming
35 bites tagged Streaming — interview questions with model answers, and 60-second explainers.
Design a Near Real-Time Analytics Pipeline
Tests your ability to design a low-latency data system and articulate trade-offs. A good answer covers ingestion (Kafka), processing (Flink), storage (Druid), and visualization (Grafana), contrasting the architecture's low latency with a batch setup.
How do you handle duplicate events in an analytics pipeline?
Tests your grasp of data integrity and idempotent processing. Explain how duplicates inflate COUNT(*), then propose adding a unique event_id and using a stateful stream processor to track seen IDs. Mention query-time COUNT(DISTINCT event_id) as an alternative.
Handling Late-Arriving Data in a Streaming Pipeline
Tests your grasp of event time vs. processing time. A great answer defines watermarks to track completeness, uses event-time windowing to group data, and sets triggers with allowed lateness to correctly incorporate out-of-order events.
Design a Real-Time Analytics Pipeline for Mobile Events
Tests your grasp of low-latency, high-throughput design. A strong answer outlines ingestion (Kafka), stream processing (Flink), and a real-time OLAP database (Druid/ClickHouse). A red flag is proposing a slow, batch-only architecture.
Design a near real-time analytics pipeline for a critical metric
This tests your grasp of stream processing trade-offs (latency, cost, correctness). Outline a 4-stage pipeline (ingest, process, store, visualize) with specific tech choices, contrasting its low-latency, high-cost nature with batch.
How do you handle late-arriving data in a streaming pipeline?
Tests understanding of event time vs. processing time and the mechanisms (watermarks, triggers, allowed lateness) to ensure correctness. Answer by distinguishing time types, using event-time windowing, defining watermarks, and configuring allowed lateness.
Media Source Extensions: The Engine for Web Streaming
MSE lets you build streaming video players in JavaScript by feeding media chunks to a `<video>` element, instead of a single file URL. It's the foundation for adaptive streaming like DASH/HLS.
FastAPI's StreamingResponse: Send Data in Chunks
StreamingResponse sends data piece by piece, like a live broadcast, instead of sending a complete file all at once. This keeps your server's memory low for huge responses like file downloads, video streams, or live data from AI models.
DDM: Detecting Drift with Error Rate Statistics
DDM acts as a statistical alarm on your model's error rate, watching for spikes that signal the underlying data has changed. Use it for online binary classification with immediate feedback, like spam filtering.
Streaming Ingestion: Catching Data as It Happens
Streaming ingestion is a conveyor belt for data, catching events as they happen instead of in batches. It's used for real-time fraud detection and IoT monitoring. The footgun is confusing ingestion (getting data in) with processing (acting on it).
Lambda Architecture: Batch and Stream for Big Data
Lambda Architecture splits data into two paths: a slow, comprehensive batch layer and a fast, real-time stream layer. It's used in big data systems needing both historical accuracy and live views.
Get Streaming bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.