The MapReduce paradigm explained
distributed batch processing basics.
map applies a function to each input record emitting key-value pairs in parallel; a shuffle groups values by key; reduce aggregates each key's values into a result.
What's really being asked
This confirms you understand the foundational distributed batch-processing model that inspired much of big data, including the often-overlooked shuffle stage between map and reduce.
The full answer
MapReduce processes large datasets in parallel across a cluster using two user-defined functions. In the map phase, the input is split into chunks, and a mapper applies a function to each record independently and in parallel, emitting intermediate key-value pairs. Between the phases the framework performs a shuffle and sort, grouping all values that share a key and routing them to the appropriate reducer; this is handled by the system, not the user. In the reduce phase, each reducer receives a key together with the list of all values for that key and aggregates them into the final output, such as a sum or count. The framework manages data partitioning, scheduling, parallelism, and fault tolerance by re-running failed tasks. The power comes from expressing a job as map plus reduce and letting the system scale it horizontally.
The mistakes people make
Omitting the shuffle and sort stage entirely. Thinking reducers begin before all mappers finish for a given key partition. Believing the developer writes the shuffle logic. Confusing map output, key-value pairs, with the final result.
What usually comes next
What is a combiner and why use one. Why is the shuffle often the performance bottleneck. How does MapReduce achieve fault tolerance. How does Spark improve on this model.
A concrete example
Word count is the canonical job. Each mapper reads lines of text and emits a pair for every word, the word as key and the number one as value. The shuffle gathers all pairs with the same word to one reducer. Each reducer then sums the ones for its word and emits the word with its total count. Across a huge corpus this runs in parallel on many machines, with the framework handling distribution and any task failures.
Interview question
What happens between the map and reduce phases in a MapReduce job?
- a.The reducers run first and feed results back into the mappers
- b.A shuffle and sort groups all values by key for the reducersCorrect
- c.The framework discards duplicate keys before reducing
- d.The map output is sent unchanged directly to a single reducer
Why? this is the answer
The shuffle and sort stage groups every value sharing a key and routes it to the right reducer, which is essential for aggregation. Reducers run after mappers, not before, and duplicate keys are grouped, not discarded.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #mapreduce
- #hadoop
- #distributed-computing
- #big-data
- #parallel-processing
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles