Spark broadcast join versus shuffle join
Spark join optimization.
a broadcast join sends the small table to every executor so the large table joins locally with no shuffle of its rows; the default sort-merge join shuffles both tables across the network, which is costly.
What's really being asked
This assesses practical Spark performance knowledge: recognizing when network shuffle dominates cost and how broadcasting eliminates it, plus knowing the safety boundary.
The full answer
In a standard join, Spark's default for large tables is the shuffle sort-merge join. It hash-partitions both DataFrames on the join key and shuffles their rows across the network so that rows with matching keys land on the same executor, then sorts and merges them. Shuffling moves a large volume of data over the network and writes intermediate data to disk, making it the dominant cost. A broadcast join, also called a broadcast hash join, instead sends a full copy of the small DataFrame to every executor. Each executor then joins its local partitions of the large DataFrame against the in-memory broadcast copy, with no shuffle of the large table at all. This is far more efficient when one side is small, because it trades a small one-time broadcast for eliminating a massive shuffle. Spark can trigger it automatically when a table is below the autoBroadcastJoinThreshold, or you can force it with a broadcast hint.
The mistakes people make
Broadcasting a table that is too large, exhausting executor memory and causing failures. Thinking a broadcast join shuffles the large table too. Believing it always beats sort-merge regardless of the small table's size. Forgetting the autoBroadcastJoinThreshold setting.
What usually comes next
What is the default broadcast threshold and how do you tune it. How do you force a broadcast with a hint. What goes wrong if you broadcast a table that is too big.
A concrete example
Joining a billion-row clickstream DataFrame with a small country-code lookup table of two hundred rows is the ideal case. Without broadcasting, Spark would shuffle the entire billion-row table across the cluster by key, an enormous cost. With a broadcast join, the tiny lookup table is copied to every executor and each clickstream partition is enriched locally, so the huge table never moves over the network, often cutting job time dramatically.
Interview question
Why is a broadcast join more efficient than a sort-merge join when one table is tiny?
- a.It shuffles only the small table while keeping the large one in memory
- b.It skips the join condition for unmatched rows
- c.It sends the small table to every executor, avoiding a shuffle of the large tableCorrect
- d.It compresses both tables before joining them
Why? this is the answer
Broadcasting copies the small table to all executors so the large table is joined locally with no network shuffle of its rows. A sort-merge join shuffles both tables, which is the expensive part.
Just read this? Test yourself on what you have been reading.
Read the original → spark.apache.org
- #spark
- #broadcast-join
- #shuffle
- #join-optimization
- #big-data
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles