Data Engineering
46 bites tagged Data Engineering: interview questions with model answers, and 60-second explainers.
Test yourself: Top 30 Data Engineering interview questions →Multiple choice, with the correct answer and why it is correct on every question. Free, no sign-in.Problems the Lakehouse architecture solves
Lakehouse adds ACID transactions, schema enforcement, and time travel on cheap object storage.
Exactly-once semantics in stream processing
Exactly-once means each event affects state once despite retries, it is hard because of failures between processing and committing, and you achieve it via idempotency or atomic…
Clickstream architecture for real-time and batch
Ingest events into a log like Kafka, fan out to a real-time path for dashboards and a batch path to a lake for ad-hoc analysis.
Schema-on-read in data lakes
Structure is applied at query time not ingest, enabling flexible raw storage and ML, but costing query-time validation and risking data swamps.
Data warehouse vs data lake
Warehouses store structured, schema-on-write data for BI; lakes store raw multi-format data with schema-on-read for exploration and ML.
What is the difference between ETL and ELT?
ETL transforms before loading, on a separate engine; ELT loads raw then transforms inside a scalable warehouse. Choose ELT with cloud warehouses and large raw or schema-on-read data.
Design a SQL upsert from a staging table
Define a stable key, use MERGE or INSERT ON CONFLICT, dedupe the staging set first, run in a transaction.
Schema evolution without rewriting history
Use a table format with metadata-level evolution, add a new column rather than mutating the old, and reconcile types at read time; avoid rewriting petabytes.
Idempotency in data ingestion pipelines
Idempotency means re-running a step yields the same result with no duplicates; it matters because retries and at-least-once delivery are inevitable; achieve it with deduplication keys or upserts.
Handling late data in streaming windows
Use event-time windows with watermarks to bound lateness, allow a grace period before finalizing, and route data later than that to a side output.
Diagnosing and fixing data skew in Spark
This is data skew, caused by uneven key distribution concentrating rows on few partitions; mitigate with salting, broadcast joins, repartitioning, or adaptive execution.
ETL versus ELT in cloud data platforms
ETL transforms before loading into the target; ELT loads raw first then transforms in the warehouse, leveraging cheap storage and elastic compute.
Data lake versus data warehouse
Lakes store raw, schema-on-read data of any type cheaply; warehouses store curated, schema-on-write structured data for fast SQL; choose a lake for varied raw data and ML.
Trace an event from click to analysis
Client SDK captures and batches, a collection endpoint ingests, a stream and ETL enrich and load into a warehouse for analysis.
Trade-offs between pre-aggregated and raw event data for dashboards
Pre-aggregations trade freshness for speed; raw queries preserve flexibility but spike cost and latency under load.
How do you fulfill a GDPR erasure request across data stores?
This tests cross-system deletion under GDPR's 30-day SLA. A strong answer maps PII lineage across S3, databases, and analytics; uses soft deletes for backups; and handles dashboards via reprocessing.
How do you technically implement an A/B test for onboarding flows?
Tests experiment pipeline design: deterministic user bucketing, an exposure event before rendering, and an event schema linking actions to variant_id. Red flag: re-randomizing per session or skipping exposure logs.
Design a near real-time cost visibility system for ML teams
Tests cost attribution across shared ML infrastructure and streaming pipeline design. Strong answers combine billing exports with resource labels, sub-hour aggregation, and anomaly detection for training spikes.
Argue for declarative or imperative feature platforms with trade-offs
This tests whether you weigh control flow against data flow. A strong answer argues from org maturity: declarative systems abstract DAG topology, while imperative ones offer Spark control at the cost of manual idempotency. Red flag: ignoring org culture.
Design system ensuring point-in-time correctness for training data joins
Tests temporal join design to prevent data leakage from slowly changing dimensions. Strong answers use an AS OF join on entity ID and timestamp, materialize features as of label time, and handle late arrivals. Joining on user_id alone is a red flag.
How would you implement versioning for feature definitions in a feature store?
Tests separation of metadata schema versions from data snapshots for reproducible training. Strong answers cover: immutable schema versions on breaking changes, safe appends without backfill, and time-travel data reads.
Backfill a complex feature for millions of users without impacting production
Reuse the live pipeline on historical partitions, run bounded batches on separate compute, stage results, and validate before promotion.
Design a sub-50ms real-time bidding feature pipeline
Tests merging batch historical and streaming data under sub-50ms latency. Strong answers use dual paths: batch backfills a KV store, streaming writes to an in-memory cache, serving merges both at request time. Red flag: one database without hot-cold split.
Design ingestion for clickstream and batch product metadata
Use a data lake for raw data, a feature store for joins, columnar formats for batch, and row formats for events.
Get Data Engineering bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.