How do you debug out-of-memory errors in a Spark job?

WHAT THIS TESTS: This question tests your debugging methodology for complex, distributed systems, not just your knowledge of Spark configurations. The interviewer wants to see a structured approach: diagnose first, then form a hypothesis, then test a solution. They are evaluating if you can distinguish between a symptom (the OOM error) and the root cause (data skew, inefficient code, resource starvation). It shows if you can reason about data distribution and its impact on memory across a cluster.

A GOOD ANSWER COVERS: A strong answer outlines a clear, multi-step process. First, DIAGNOSE: check driver and executor logs for the exact OOM error and the stage where it occurred. Use the Spark UI to inspect the DAG, looking for stages with high shuffle writes or long-running tasks. Second, INVESTIGATE DATA SKEW: check task metrics in the Spark UI's stage detail page. If a few tasks take much longer or process significantly more data (e.g., max vs. median), you have skew. This is a primary cause of OOMs on specific executors. Third, AUDIT THE CODE: look for expensive, wide transformations. Are you using groupByKey instead of a more efficient alternative like reduceByKey or aggregateByKey? Is there a large join that could be optimized with a broadcast hint? Are inefficient User-Defined Functions (UDFs) holding large objects in memory? Fourth, TUNE RESOURCES METHODICALLY: only after investigating the above, consider adjusting configurations. This might involve increasing spark.executor.memory, but a more targeted fix is often better. For example, increasing spark.sql.shuffle.partitions (default is 200) can create more, smaller post-shuffle tasks, reducing memory pressure on each executor.

COMMON WRONG ANSWERS: The biggest red flag is immediately suggesting, "I'd increase the executor memory." This is a brute-force, expensive approach that doesn't solve the underlying problem and indicates a lack of diagnostic skill. Another weak answer is listing Spark configurations without explaining why you'd change them. For example, saying "I'd tune spark.sql.shuffle.partitions" is weak. A strong answer says, "If the UI shows a large shuffle write before the OOM stage, I'd increase spark.sql.shuffle.partitions to reduce the size of each post-shuffle partition, lowering the memory requirement for downstream tasks."

LIKELY FOLLOW-UPS: "Let's say you've identified data skew on a join key. What are two specific techniques to mitigate it?" (Answer: Salting the key, or enabling Spark 3's Adaptive Query Execution which can handle skew in joins automatically). "What's the difference between an OOM on the driver vs. an OOM on an executor?" (Answer: Driver OOM is often from collect()-ing too much data to the driver or trying to broadcast a massive object. Executor OOM is from data processing tasks, like a skewed partition or a large shuffle block).

ONE CONCRETE EXAMPLE: A job aggregating user activity by user_id is failing. The Spark UI shows one executor consistently failing during a shuffle stage. The task metrics reveal that 90% of the data is being processed by a single task associated with a null or default user_id. The fix is not more memory. The fix is to first, filter out null keys if they are not needed. Second, to handle the remaining skew, apply a "salting" technique where the skewed key is appended with a random integer (e.g., power_user_id_1, power_user_id_2, etc.) to distribute its data across multiple partitions and tasks. This spreads the load instead of concentrating it on one executor.