Data Engineering
46 bites tagged Data Engineering — interview questions with model answers, and 60-second explainers.
Problems the Lakehouse architecture solves
Lakehouse adds ACID transactions, schema enforcement, and time travel on cheap object storage. knowing the gaps in raw data lakes. describing it as merely a faster query engine rather than a table format.
Exactly-once semantics in stream processing
Exactly-once means each event affects state once despite retries, it is hard because of failures between processing and committing, and you achieve it via idempotency or atomic… understanding delivery guarantees and effects.
Clickstream architecture for real-time and batch
Ingest events into a log like Kafka, fan out to a real-time path for dashboards and a batch path to a lake for ad-hoc analysis. designing a dual-path streaming pipeline. a single path that can't serve both latency profiles.
Schema-on-read in data lakes
Structure is applied at query time not ingest, enabling flexible raw storage and ML, but costing query-time validation and risking data swamps. understanding deferred schema application.
Data warehouse vs data lake
Warehouses store structured, schema-on-write data for BI; lakes store raw multi-format data with schema-on-read for exploration and ML. distinguishing two storage paradigms.
What is the difference between ETL and ELT?
ETL transforms before loading, on a separate engine; ELT loads raw then transforms inside a scalable warehouse. Choose ELT with cloud warehouses and large raw or schema-on-read data. knowing where transformation runs in a pipeline.
Design a SQL upsert from a staging table
Define a stable key, use MERGE or INSERT ON CONFLICT, dedupe the staging set first, run in a transaction. knowledge of idempotent loads. a naive INSERT that duplicates or a delete-then-insert race.
Schema evolution without rewriting history
Use a table format with metadata-level evolution, add a new column rather than mutating the old, and reconcile types at read time; avoid rewriting petabytes. schema evolution strategy.
Idempotency in data ingestion pipelines
Idempotency means re-running a step yields the same result with no duplicates; it matters because retries and at-least-once delivery are inevitable; achieve it with deduplication keys or upserts. reliability under retries.
Handling late data in streaming windows
Use event-time windows with watermarks to bound lateness, allow a grace period before finalizing, and route data later than that to a side output. event-time stream processing.
Diagnosing and fixing data skew in Spark
This is data skew, caused by uneven key distribution concentrating rows on few partitions; mitigate with salting, broadcast joins, repartitioning, or adaptive execution. distributed processing skew. just adding more executors.
ETL versus ELT in cloud data platforms
ETL transforms before loading into the target; ELT loads raw first then transforms in the warehouse, leveraging cheap storage and elastic compute. data pipeline pattern fluency.
Data lake versus data warehouse
Lakes store raw, schema-on-read data of any type cheaply; warehouses store curated, schema-on-write structured data for fast SQL; choose a lake for varied raw data and ML. storage architecture fundamentals.
Trace an event from click to analysis
Client SDK captures and batches, a collection endpoint ingests, a stream and ETL enrich and load into a warehouse for analysis. end-to-end understanding of an analytics event pipeline.
Trade-offs between pre-aggregated and raw event data for dashboards
Pre-aggregations trade freshness for speed; raw queries preserve flexibility but spike cost and latency under load. Balancing latency, cost, and freshness in analytics.
How do you fulfill a GDPR erasure request across data stores?
This tests cross-system deletion under GDPR's 30-day SLA. A strong answer maps PII lineage across S3, databases, and analytics; uses soft deletes for backups; and handles dashboards via reprocessing.
How do you technically implement an A/B test for onboarding flows?
Tests experiment pipeline design: deterministic user bucketing, an exposure event before rendering, and an event schema linking actions to variant_id. Red flag: re-randomizing per session or skipping exposure logs.
Design a near real-time cost visibility system for ML teams
Tests cost attribution across shared ML infrastructure and streaming pipeline design. Strong answers combine billing exports with resource labels, sub-hour aggregation, and anomaly detection for training spikes.
Argue for declarative or imperative feature platforms with trade-offs
This tests whether you weigh control flow against data flow. A strong answer argues from org maturity: declarative systems abstract DAG topology, while imperative ones offer Spark control at the cost of manual idempotency. Red flag: ignoring org culture.
Design system ensuring point-in-time correctness for training data joins
Tests temporal join design to prevent data leakage from slowly changing dimensions. Strong answers use an AS OF join on entity ID and timestamp, materialize features as of label time, and handle late arrivals. Joining on user_id alone is a red flag.
How would you implement versioning for feature definitions in a feature store?
Tests separation of metadata schema versions from data snapshots for reproducible training. Strong answers cover: immutable schema versions on breaking changes, safe appends without backfill, and time-travel data reads.
Backfill a complex feature for millions of users without impacting production
Reuse the live pipeline on historical partitions, run bounded batches on separate compute, stage results, and validate before promotion. Isolating large-scale backfill from production while ensuring correctness.
Design a sub-50ms real-time bidding feature pipeline
Tests merging batch historical and streaming data under sub-50ms latency. Strong answers use dual paths: batch backfills a KV store, streaming writes to an in-memory cache, serving merges both at request time. Red flag: one database without hot-cold split.
Design ingestion for clickstream and batch product metadata
Use a data lake for raw data, a feature store for joins, columnar formats for batch, and row formats for events. Marrying streaming clicks and batch metadata into feature pipelines.
Get Data Engineering bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.