Diagnosing and fixing data skew in Spark

What's really being asked

This checks whether you can identify the classic distributed-computing pathology of uneven load and apply the right targeted fixes rather than throwing hardware at it.

The full answer

The symptom of a few tasks running far longer than the rest is data skew. In a shuffle, such as a join or group-by, rows are partitioned by key, and if some keys hold a hugely disproportionate share of rows, the tasks handling those keys process far more data and become stragglers that dominate the stage's wall-clock time while other tasks finish and sit idle. Common causes include hot keys like a default or null value, a dominant entity such as one giant customer, or naturally Zipfian distributions. Mitigations depend on the operation. For joins where one side is small, a broadcast join avoids the shuffle entirely. For skewed shuffle keys, salting adds a random suffix to the hot key to spread it across partitions, with the other side replicated accordingly. You can isolate and process skewed keys separately, then union the result. Repartitioning or increasing shuffle partitions helps mild cases. Modern Spark's Adaptive Query Execution can detect and split skewed partitions automatically. Filtering or pre-aggregating before the shuffle also reduces the volume on hot keys.

The mistakes people make

Simply adding more executors or memory, which does not rebalance a skewed key still pinned to one task. Increasing partitions blindly without addressing the dominant key. Confusing skew with a small-files or memory problem. Forgetting that nulls often collapse onto a single partition.

What usually comes next

How exactly does salting work. When is a broadcast join unsafe. How does AQE detect skew.

A concrete example

A join on customer_id stalls because one wholesale account owns most rows. Salting that key into several variants spreads its rows across many tasks, the straggler disappears, and the stage finishes in a fraction of the time.