Spark transformations versus actions

What's really being asked

This confirms you understand Spark's defining feature, lazy evaluation, and why deferring work until an action enables whole-job optimization.

The full answer

A transformation creates a new dataset from an existing one and is lazy: calling it does not run any computation, it only records the operation in Spark's lineage graph, the directed acyclic graph of dependencies. Examples include map, filter, flatMap, groupByKey, and join, each returning a new RDD or DataFrame. An action triggers actual execution of the accumulated transformations and either returns a value to the driver program or writes data out. Examples include count, collect, take, first, reduce, and saveAsTextFile. The distinction is fundamental because laziness lets Spark see the entire chain of transformations before executing, so the Catalyst optimizer and the DAG scheduler can pipeline narrow transformations into stages, push down filters, and avoid materializing intermediate results, all of which improve performance and reduce data movement.

The mistakes people make

Believing a map or filter computes results immediately. Calling collect a transformation. Thinking lineage is only for fault tolerance and not optimization. Assuming every line of Spark code launches a job; only actions do.

What usually comes next

What is the difference between a narrow and a wide transformation. How does lineage provide fault tolerance. Why can calling collect on a huge dataset crash the driver.

A concrete example

Given an RDD of log lines, calling filter to keep error lines and then map to extract timestamps does nothing observable; Spark just extends the DAG. Only when you call count does Spark launch a job, read the data, apply the filter and map in a pipelined stage, and return the number of error lines. If you never call an action, the data is never read at all, which surprises engineers who expect eager execution.