Catalyst Optimizer and Project Tungsten in Spark

What's really being asked

Whether you understand that Spark SQL does not execute queries literally but compiles them through an optimizer, and whether you can separate plan-level optimization (Catalyst) from execution-level optimization (Tungsten).

The full answer

Catalyst is Spark SQL's extensible query optimizer. It first parses a query or DataFrame into an unresolved logical plan, resolves column and table references against the catalog to produce an analyzed logical plan, then applies a library of rule-based transformations such as predicate pushdown, projection pruning, constant folding, and Boolean simplification to produce an optimized logical plan. From that it generates one or more physical plans and uses cost-based optimization to choose among them, for example selecting a broadcast hash join over a sort-merge join when one side is small. Project Tungsten then makes the chosen plan run fast at the execution layer. It manages memory off-heap in compact binary format to cut garbage-collection pressure, exploits cache-aware data structures, and uses whole-stage code generation to fuse multiple operators into a single compiled JVM function, avoiding virtual-call overhead.

The mistakes people make

Saying Catalyst executes the query, or that Tungsten optimizes the logical plan. Tungsten is about CPU and memory efficiency, not plan rewriting.

What usually comes next

What is the difference between rule-based and cost-based optimization? How does whole-stage codegen reduce overhead? When does Adaptive Query Execution re-optimize at runtime?

A concrete example

A join filters one table by date. Catalyst pushes the date predicate down so the scan reads fewer rows, prunes unused columns, and picks a broadcast join because the dimension table is tiny. Tungsten then code-generates the filter and join into one tight loop operating on off-heap binary rows, so the query runs far faster than an interpreted plan would.