How would you scale 1TB Pandas feature computation across machines?

WHAT THIS TESTS: This question probes your ability to recognize when an in-memory tool like pandas hits hard physical limits and how you systematically select a replacement. The interviewer wants to see that you understand pandas is built for in-memory analytics, that intermediate copies during operations can balloon memory well beyond the base dataset size, and that scaling requires evaluating trade-offs between code rewrite cost, execution latency, and infrastructure complexity.

A GOOD ANSWER COVERS: A good answer hits four things in order. First, acknowledge that pandas keeps data in RAM and that a one terabyte dataset will fail or thrash because many operations create intermediate copies, so even a machine with one terabyte of RAM is risky. Second, mention single-machine mitigation tactics that come directly from pandas best practices: load only the columns you need using the columns parameter in read_parquet or usecols in read_csv, and downcast numeric types to efficient dtypes to shrink memory footprint by up to fifty percent or more. Third, contrast two distributed frameworks. Dask DataFrames offer a familiar pandas-like API and run on a Python cluster with minimal code changes, but they can struggle with complex shuffles and require thoughtful partition sizes, typically one hundred megabyte to one gigabyte chunks. Apache Spark with PySpark is more resilient at terabyte scale due to its optimized query planner and Catalyst optimizer, though it demands a heavier rewrite away from pandas syntax and introduces JVM operational overhead. Fourth, discuss operational realities: data locality, the cost of serializing and moving data across the network, partition skew, and whether the feature computation is embarrassingly parallel or requires global aggregations that trigger expensive shuffle stages.

COMMON WRONG ANSWERS: Common wrong answers include suggesting vertical scaling to a single massive VM without mentioning that pandas intermediate copies can still exhaust RAM, or recommending chunking via read_csv iterator loops without explaining how to handle stateful features that cross chunk boundaries. Another red flag is naming distributed tools but ignoring data movement costs, such as claiming Dask or Spark will automatically solve the problem without considering partition count, memory per worker, or whether the algorithm itself is parallelizable.

LIKELY FOLLOW-UPS: Interviewers often push deeper by asking how you would handle stateful features like rolling window aggregates if the window spans partition boundaries. They may ask you to estimate cluster size and memory per worker for one terabyte of input, or to explain how you would diagnose an out-of-memory error in a Dask or Spark job. Another common follow-up is asking when you would prefer SQL-based engines like BigQuery or Snowflake over a general-purpose distributed dataframe.

ONE CONCRETE EXAMPLE: Suppose you have one terabyte of Parquet files storing user events and you need to compute per-user session counts and average dwell time. In pandas you would read the whole dataset and group by user, but that is impossible on a single node. Using Dask, you would read the Parquet directory lazily, repartition so each partition holds roughly five hundred megabytes, perform the groupby aggregation, and let Dask spill to disk if needed. Using Spark, you would read with spark.read.parquet, ensure a salting key if user cardinality causes skew, and rely on Spark SQL or RDD operations to compute the aggregates, writing results back to Parquet. The Spark approach typically wins at terabyte scale if the pipeline already runs on a shared cluster, while Dask wins if the team wants to stay in the Python ecosystem and reuse existing scipy or scikit-learn logic.