Spark broadcast join versus shuffle join

WHAT THIS TESTS This assesses practical Spark performance knowledge: recognizing when network shuffle dominates cost and how broadcasting eliminates it, plus knowing the safety boundary.

A GOOD ANSWER COVERS In a standard join, Spark's default for large tables is the shuffle sort-merge join. It hash-partitions both DataFrames on the join key and shuffles their rows across the network so that rows with matching keys land on the same executor, then sorts and merges them. Shuffling moves a large volume of data over the network and writes intermediate data to disk, making it the dominant cost. A broadcast join, also called a broadcast hash join, instead sends a full copy of the small DataFrame to every executor. Each executor then joins its local partitions of the large DataFrame against the in-memory broadcast copy, with no shuffle of the large table at all. This is far more efficient when one side is small, because it trades a small one-time broadcast for eliminating a massive shuffle. Spark can trigger it automatically when a table is below the autoBroadcastJoinThreshold, or you can force it with a broadcast hint.

COMMON WRONG ANSWERS Broadcasting a table that is too large, exhausting executor memory and causing failures. Thinking a broadcast join shuffles the large table too. Believing it always beats sort-merge regardless of the small table's size. Forgetting the autoBroadcastJoinThreshold setting.

LIKELY FOLLOW-UPS What is the default broadcast threshold and how do you tune it. How do you force a broadcast with a hint. What goes wrong if you broadcast a table that is too big.

ONE CONCRETE EXAMPLE Joining a billion-row clickstream DataFrame with a small country-code lookup table of two hundred rows is the ideal case. Without broadcasting, Spark would shuffle the entire billion-row table across the cluster by key, an enormous cost. With a broadcast join, the tiny lookup table is copied to every executor and each clickstream partition is enriched locally, so the huge table never moves over the network, often cutting job time dramatically.