Diagnosing Spark executor OutOfMemoryError

WHAT THIS TESTS This evaluates a structured debugging mindset for distributed memory issues, distinguishing engineers who reason from telemetry from those who randomly raise settings.

A GOOD ANSWER COVERS Start by reproducing and locating the failure in the Spark UI: which stage and task throw the error, how partitions are sized, and whether the event timeline shows skew, where one task processes far more than others, or heavy disk spill. Next examine executor memory configuration, spark.executor.memory and the off-heap overhead spark.executor.memoryOverhead, and recall that unified memory splits between execution memory for shuffles and sorts and storage memory for cached data. Then audit code patterns that commonly cause out-of-memory: calling collect or take on a large dataset pulls everything to one place, groupByKey buffers all values per key in memory versus reduceByKey which aggregates incrementally, skewed join or group keys overload a single executor, exploding joins multiply rows, and caching datasets too large for storage memory. Remedies follow the diagnosis: increase the number of partitions to shrink each task's footprint, repartition or salt to fix skew, replace groupByKey with reduceByKey or aggregateByKey, avoid driver-side collection, unpersist unused cached data, and only then raise executor memory or overhead.

COMMON WRONG ANSWERS Immediately raising executor memory without finding the cause. Ignoring data skew as a source. Confusing driver out-of-memory, often from collect, with executor out-of-memory. Forgetting memoryOverhead for off-heap and Python or shuffle buffers.

LIKELY FOLLOW-UPS Why does groupByKey risk out-of-memory more than reduceByKey. How does the unified memory model split execution and storage. When is the driver, not the executor, the one running out.

ONE CONCRETE EXAMPLE A job groups events by user with groupByKey and one power user has millions of events. That single key's values must all fit in one executor's memory, triggering the OutOfMemoryError. The Spark UI shows one straggler task with huge input. Switching to reduceByKey so values aggregate incrementally, plus increasing partitions and salting the hot key, eliminates the error without simply throwing more memory at every executor.