Streaming
35 bites tagged Streaming — interview questions with model answers, and 60-second explainers.
Reading the full response body in middleware
Responses stream as multiple body messages and headers go first, so you cannot add a header after seeing the body; you must buffer all chunks, compute the hash, set the header, then resend. Understanding ASGI's streaming send model.
Async iterators and for await...of for streaming
Async iterators yield values lazily over time; for await...of consumes them sequentially with backpressure, keeping memory bounded. streaming vs buffering everything.
Backpressure
Backpressure is a mechanism by which a slow consumer signals an upstream producer to slow down or stop, preventing unbounded queues and resource exhaustion. It keeps systems stable under overload by propagating capacity limits backward through a pipeline.
Exactly-once semantics in stream processing
Exactly-once means each event affects state once despite retries, it is hard because of failures between processing and committing, and you achieve it via idempotency or atomic… understanding delivery guarantees and effects.
Clickstream architecture for real-time and batch
Ingest events into a log like Kafka, fan out to a real-time path for dashboards and a batch path to a lake for ad-hoc analysis. designing a dual-path streaming pipeline. a single path that can't serve both latency profiles.
Design a personalized ad copy pipeline
Identify sources, choose batch versus streaming, generate and deliver copy via APIs. end-to-end data pipeline design for personalization. jumping to a model with no ingestion, privacy, or delivery plan.
Handling late data in streaming windows
Use event-time windows with watermarks to bound lateness, allow a grace period before finalizing, and route data later than that to a side output. event-time stream processing.
High-throughput serverless stream processing
Partition by key for per-user ordering, use a sharded log with batched consumers for backpressure, and tune batch size and shards for cost. stream design at scale. a global FIFO queue or per-event invocation at 100k/sec.
How do you implement response streaming in a Next.js Route Handler?
This tests Web Streams API mastery in Next.js. Strong answer: create a ReadableStream with TextEncoder, return it in a Response, and cite AI chat or large JSON as use cases. Red flag: suggesting Node res.write instead of standard Web Streams.
Explain streaming with Server Components, Suspense, and loading.js
Tests progressive HTML streaming through Suspense boundaries. Strong answers state loading.js auto-wraps a route segment in Suspense so the server can stream the shell instantly and defer slow data without blocking first paint.
How do loading.js and error.js integrate with Suspense and Error Boundaries?
Tests grasp of App Router conventions wrapping React primitives. loading.js suspends its segment for instant navigation feedback; error.js adds a client Error Boundary for child segment crashes. Wrong: assuming they are server-only or replace manual usage.
Design ingestion for clickstream and batch product metadata
Use a data lake for raw data, a feature store for joins, columnar formats for batch, and row formats for events. Marrying streaming clicks and batch metadata into feature pipelines.
Design a near real-time user interaction tracking and analytics system
Tests decoupling ingestion from querying with justified tech choices. Outline: client → Kafka → Flink → ClickHouse → API; budget sub-30s latency and backpressure per stage. Red flag: one monolithic RDBMS or batch ETL handling both writes and reads.
Compare efficient line-by-line file reading in Go and Rust
Go uses bufio.Scanner with ScanLines/Scan(); Rust uses BufReader with lines() or read_line(). Memory-efficient streaming I/O idioms. Loading the whole file with ioutil.ReadFile or fs::read_to_string.
Backpressure: Slow the Producer or Crash
Backpressure is a feedback signal telling upstream to slow down when downstream cannot keep up. You see it in stream processors like Flink or Kafka where a slow consumer risks memory exhaustion. Ignore it and queues grow until the service crashes.
When is streaming better than batch, and what are its infrastructure challenges?
Tests if you separate low-latency decisions from analytics and know ops pain. Good answer: fraud detection vs batch; cite exactly-once, state recovery, backpressure, schema drift. Red flag: calling streaming faster batch while ignoring backpressure or state.
Design a scalable, fault-tolerant real-time IoT data ingestion system
This tests separation of edge connectivity, buffering, and processing. A strong answer names an edge gateway, Kafka as the backplane, stream processing, and cold storage, plus backpressure and partitioning.
Process a 50GB CSV with only 16GB RAM
Chunk with read_csv chunksize, filter columns via usecols, downcast int64 to int32/int16, skip rows. Streaming aggregation under memory constraints. Loading everything into one DataFrame or using default dtypes.
Spark Structured Streaming: Unify Batch and Stream
Spark Structured Streaming treats a live stream as an unbounded DataFrame. It unifies batch and streaming ETL on Kafka, but the footgun is confusing event time with processing time without watermarks, which silently drops late data.
Design a real-time top-10 dashboard for a global news site
Stream ingestion, windowed aggregation, Redis top-N cache, and TTL eviction. separating hot-path reads from cold-path analytics at scale. scanning raw events or running global SQL GROUP BY per request.
Design a system to detect sudden add-to-cart drops in real time
This tests streaming pipeline design and seasonality-aware anomaly detection. Outline Kafka or Kinesis ingestion, windowed aggregations, and ML baselines tuned to hourly and weekly trends. Red flag: static thresholds that ignore daily patterns.
How do duplicate events bias COUNT(*) and daily login reports?
Tests idempotency in streaming analytics. COUNT(*) overcounts; fix with unique event ID dedup via idempotent writes or COUNT(DISTINCT id), plus daily partition reconciliation. Red flag: SELECT DISTINCT * without a stable key or no reporting safeguard.
How would you handle late-arriving data in a streaming analytics pipeline?
Tests understanding of event-time processing, watermarks, and windowing for correctness. A strong answer covers watermarks with allowed lateness, event-time triggers, and idempotent updates to sinks.
Design a Real-Time Analytics Pipeline for Mobile Events
This tests your grasp of low-latency streaming architectures. A good answer outlines ingestion (SDK to Kafka/Kinesis), real-time processing (Flink/Spark), and sinking to a fast OLAP database (Druid/ClickHouse). A red flag is proposing a batch-based ETL design.
Get Streaming bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.