Data skew in Spark and salting

WHAT THIS TESTS This checks whether you understand that distributed performance is bounded by the slowest task and that uneven key distribution, not total cluster size, is often the culprit.

A GOOD ANSWER COVERS Data skew occurs when the values of a grouping or join key are unevenly distributed, so a small number of keys account for a large fraction of the rows. When Spark shuffles by that key, all rows for a hot key go to a single partition handled by one task. That task processes far more data than the others, becomes a straggler, and the whole stage cannot finish until it does, while other executors sit idle. Adding more executors does not help because the bottleneck is one oversized partition. Salting mitigates skew by adding a random salt, an integer suffix, to the hot key so its rows are split across many partitions. For a join, the larger skewed table gets a random salt per row, and the smaller table is replicated across all salt values so matches are preserved; the join then runs on the composite salted key, spreading the load, after which you strip the salt and aggregate. Spark's adaptive query execution can also handle skew automatically by splitting large partitions.

COMMON WRONG ANSWERS Thinking more executors or memory alone fixes skew. Forgetting to replicate the other side of a salted join, which drops matches. Confusing skew with general data volume. Salting every key uniformly instead of targeting hot keys.

LIKELY FOLLOW-UPS How does adaptive query execution mitigate skew without manual salting. How do you identify the skewed keys. What is the downside of salting all keys.

ONE CONCRETE EXAMPLE Imagine joining transactions on customer_id where one wholesale account holds eighty percent of all transactions. After the shuffle, one partition holds eighty percent of the rows and its task runs for hours while others finish in minutes. By salting that customer_id with a random value from zero to nine, its rows scatter across ten partitions, and replicating the matching dimension row across those ten salts preserves correctness, turning one multi-hour straggler into ten balanced tasks.