repartition() versus coalesce() in Spark

What's really being asked

Whether you grasp how Spark physically reorganizes data across partitions and the cost difference between a full shuffle and a narrow merge, which directly affects job runtime.

The full answer

repartition performs a full shuffle, redistributing rows across the cluster, and can either increase or decrease the number of partitions while producing roughly even partition sizes. That balance is valuable but the shuffle is expensive. coalesce instead merges existing partitions on the same executors to reduce the count, avoiding a full shuffle by combining adjacent partitions; it is much cheaper but can only decrease the partition count and may leave partitions unevenly sized. The decision hinges on direction and balance. If you need more partitions, or need to fix skew so work is evenly spread before a costly join or aggregation, use repartition despite the shuffle. If you simply need fewer partitions, for example to avoid writing thousands of tiny output files at the end of a job, use coalesce because it gets you there without the network cost.

The mistakes people make

Claiming coalesce can increase partitions, or that it is always preferable. Forgetting that coalesce can create skewed partitions that hurt a downstream join.

What usually comes next

What happens if you coalesce to one partition on a huge dataset? How does partition count relate to executor cores? When does repartitioning by a key column help a join?

A concrete example

A pipeline finishes with two thousand small partitions and writes two thousand tiny files to S3. The engineer calls coalesce to ten before writing, merging partitions without a shuffle for a cheap fix. Earlier, before a large skewed join, the same engineer uses repartition on the join key to spread data evenly and avoid stragglers.