Diagnosing Out-of-Memory Errors in a Spark Job

WHAT THIS TESTS: This question evaluates your ability to apply a systematic, data-driven debugging methodology to a complex distributed system problem. The interviewer is looking for more than just a list of Spark configurations. They want to see if you can logically deduce the root cause of an OOM error by moving from symptoms to diagnosis to solution, distinguishing between data issues, code issues, and configuration issues. It tests your practical experience with distributed data processing principles like partitioning, shuffling, and data skew.

A GOOD ANSWER COVERS: A good answer hits four points in order. First, diagnosis: start by analyzing the Spark UI and executor logs to pinpoint the exact stage and tasks that are failing. Note whether it's one executor (likely skew) or all (likely insufficient memory or inefficient code). Second, data skew: check the Spark UI's stage details. If a few tasks process significantly more data or run much longer than others, it's skew. The solution involves repartitioning, often by salting the skewed keys. Third, code optimization: review the DAG for wide transformations. Look for anti-patterns like groupByKey (use reduceByKey or aggregateByKey instead), collect() on large datasets, or joins where one side is small enough to be broadcast but isn't. Fourth, configuration tuning: as a last resort, adjust memory and partition settings. Mention specific parameters like increasing spark.sql.shuffle.partitions (default is 200) to create smaller post-shuffle partitions, or adjusting spark.executor.memory.

COMMON WRONG ANSWERS: The most common red flag is jumping straight to "I'd increase spark.executor.memory." This is a brute-force, expensive fix that ignores the root cause. It signals a lack of diagnostic skill. Another weak answer is being vague, like "I'd optimize the query," without providing concrete examples of anti-patterns (groupByKey) or specific optimization techniques (salting, broadcast joins). Finally, failing to mention the Spark UI or logs as the primary tool for investigation is a major omission.

LIKELY FOLLOW-UPS: Be prepared for "What if the OOM is on the driver, not an executor?" (The cause is likely a large collect() call, an overly large broadcast variable, or accumulating state in the driver process). Another is "Explain how Adaptive Query Execution (AQE) helps." (AQE can dynamically coalesce partitions, convert sort-merge joins to broadcast joins, and handle skew joins at runtime, reducing manual tuning).

ONE CONCRETE EXAMPLE: A job joining a 1TB event table to a 100MB dimension table is failing with OOMs during the join. The Spark UI shows most tasks in the join stage finish in 2 minutes, but 2 out of 500 tasks are running for 30+ minutes and eventually fail. This indicates severe data skew on the join key. The fix is to enable AQE's skew join optimization (spark.sql.adaptive.skewJoin.enabled) or manually salt the key. For example, if the skew is on a user_id = -1 for guest users, you would change the key to concat(user_id, '_', floor(rand() * 10)) on the large table and explode the small table to match on the salted keys.