Spark RDDs, DataFrames, and Datasets
knowledge of Spark's APIs and the optimizer.
RDDs are low-level typed object collections with no built-in optimization; DataFrames are named columns optimized by Catalyst and Tungsten; Datasets add compile-time type safety in…
What's really being asked
This confirms you know Spark's three core abstractions and, crucially, why the structured APIs let Spark optimize work that raw RDDs cannot.
The full answer
RDDs, resilient distributed datasets, are the original low-level abstraction: an immutable distributed collection of arbitrary objects. They give fine-grained control and work with unstructured data, but they have no schema, so Spark treats them as opaque and applies no query optimization, and they incur Java object serialization overhead. DataFrames organize data into named, typed columns like a relational table with a known schema. That schema lets the Catalyst query optimizer reorder and prune operations, push down predicates, and lets the Tungsten engine use compact off-heap memory and code generation, so DataFrames are typically faster and more concise. Datasets, available in Scala and Java, combine the optimization of DataFrames with compile-time type safety using JVM objects, catching type errors before runtime; in Python a DataFrame is effectively the untyped Dataset. New applications prefer DataFrames and Datasets because the optimizer and memory management deliver better performance with less code, while RDDs are reserved for low-level control or unstructured data.
The mistakes people make
Claiming RDDs are generally faster. Saying DataFrames lack optimization. Forgetting that Catalyst and Tungsten apply only to the structured APIs. Believing Datasets give type safety in Python, which lacks compile-time typing.
What usually comes next
What is the Catalyst optimizer and what does Tungsten do. When would you still drop down to RDDs. Why are Datasets unavailable in Python.
A concrete example
Filter records where amount exceeds one hundred and then sum by category. With an RDD you write explicit map and reduceByKey calls that Spark runs literally as written, with no reordering. With a DataFrame you express the same as a filter and groupBy, and Catalyst can push the filter before the shuffle and generate optimized bytecode, often running noticeably faster while the code is shorter and more readable.
Interview question
Why do DataFrames generally outperform RDDs in Spark for the same logic?
- a.Their schema lets the Catalyst optimizer and Tungsten engine optimize executionCorrect
- b.DataFrames avoid distributing data across the cluster
- c.DataFrames skip the shuffle stage entirely
- d.RDDs cannot be cached in memory
Why? this is the answer
A known schema enables Catalyst query optimization and Tungsten's efficient memory and code generation, which opaque RDDs cannot benefit from. Both are distributed and both can shuffle and cache.
Just read this? Test yourself on what you have been reading.
Read the original → spark.apache.org
- #spark
- #rdd
- #dataframe
- #dataset
- #catalyst-optimizer
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles