The MapReduce paradigm explained

WHAT THIS TESTS This confirms you understand the foundational distributed batch-processing model that inspired much of big data, including the often-overlooked shuffle stage between map and reduce.

A GOOD ANSWER COVERS MapReduce processes large datasets in parallel across a cluster using two user-defined functions. In the map phase, the input is split into chunks, and a mapper applies a function to each record independently and in parallel, emitting intermediate key-value pairs. Between the phases the framework performs a shuffle and sort, grouping all values that share a key and routing them to the appropriate reducer; this is handled by the system, not the user. In the reduce phase, each reducer receives a key together with the list of all values for that key and aggregates them into the final output, such as a sum or count. The framework manages data partitioning, scheduling, parallelism, and fault tolerance by re-running failed tasks. The power comes from expressing a job as map plus reduce and letting the system scale it horizontally.

COMMON WRONG ANSWERS Omitting the shuffle and sort stage entirely. Thinking reducers begin before all mappers finish for a given key partition. Believing the developer writes the shuffle logic. Confusing map output, key-value pairs, with the final result.

LIKELY FOLLOW-UPS What is a combiner and why use one. Why is the shuffle often the performance bottleneck. How does MapReduce achieve fault tolerance. How does Spark improve on this model.

ONE CONCRETE EXAMPLE Word count is the canonical job. Each mapper reads lines of text and emits a pair for every word, the word as key and the number one as value. The shuffle gathers all pairs with the same word to one reducer. Each reducer then sums the ones for its word and emits the word with its total count. Across a huge corpus this runs in parallel on many machines, with the framework handling distribution and any task failures.