Spark RDDs, DataFrames, and Datasets

WHAT THIS TESTS This confirms you know Spark's three core abstractions and, crucially, why the structured APIs let Spark optimize work that raw RDDs cannot.

A GOOD ANSWER COVERS RDDs, resilient distributed datasets, are the original low-level abstraction: an immutable distributed collection of arbitrary objects. They give fine-grained control and work with unstructured data, but they have no schema, so Spark treats them as opaque and applies no query optimization, and they incur Java object serialization overhead. DataFrames organize data into named, typed columns like a relational table with a known schema. That schema lets the Catalyst query optimizer reorder and prune operations, push down predicates, and lets the Tungsten engine use compact off-heap memory and code generation, so DataFrames are typically faster and more concise. Datasets, available in Scala and Java, combine the optimization of DataFrames with compile-time type safety using JVM objects, catching type errors before runtime; in Python a DataFrame is effectively the untyped Dataset. New applications prefer DataFrames and Datasets because the optimizer and memory management deliver better performance with less code, while RDDs are reserved for low-level control or unstructured data.

COMMON WRONG ANSWERS Claiming RDDs are generally faster. Saying DataFrames lack optimization. Forgetting that Catalyst and Tungsten apply only to the structured APIs. Believing Datasets give type safety in Python, which lacks compile-time typing.

LIKELY FOLLOW-UPS What is the Catalyst optimizer and what does Tungsten do. When would you still drop down to RDDs. Why are Datasets unavailable in Python.

ONE CONCRETE EXAMPLE Filter records where amount exceeds one hundred and then sum by category. With an RDD you write explicit map and reduceByKey calls that Spark runs literally as written, with no reordering. With a DataFrame you express the same as a filter and groupBy, and Catalyst can push the filter before the shuffle and generate optimized bytecode, often running noticeably faster while the code is shorter and more readable.