How would you standardize a 500GB dataset that does not fit in RAM?

WHAT THIS TESTS: This question evaluates your ability to design distributed algorithms for numerical preprocessing when data exceeds single-machine RAM. The interviewer cares about your understanding of two-pass statistical aggregation, numerical stability, and familiarity with out-of-core frameworks like Dask or PySpark. Specifically, they want to see that you recognize Z-score standardization requires global statistics that cannot be computed from isolated subsets without proper aggregation.

A GOOD ANSWER COVERS: First, state that standardization requires two passes over the data because you need the global mean and standard deviation before you can transform any row. Second, describe the first pass: aggregate sufficient statistics per partition, specifically the count, sum, and sum of squared differences or sum of squares. Third, combine these partition-level statistics into global values using weighted formulas. For the mean, this is the weighted average of local means using counts as weights. For the variance, use the parallel variance formula that combines sum of squares and counts, or prefer Welford's online algorithm for better numerical stability. Fourth, describe the second pass: read the data again and apply z = (x - mean) / std to each column. Fifth, mention that Dask-ML StandardScaler and PySpark StandardScaler implement exactly this two-pass logic internally, handling partitioning and aggregation automatically. If building manually with chunked pandas, you must maintain running aggregates rather than storing intermediate arrays.

COMMON WRONG ANSWERS: A red flag is suggesting that you can compute local means on chunks and then average those means equally, which ignores differing partition sizes and produces a biased global mean. Another red flag is trying to fit the entire dataset into memory by buying more RAM or using swap, which fails at the 500GB scale. Some candidates also propose single-pass approximate methods without acknowledging the loss of exactness, or they ignore numerical stability issues with the naive sum of squares formula, which can suffer from catastrophic cancellation on large datasets.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle streaming data where a second full pass is expensive, which leads to incremental or online standardization. They might also ask about sparse matrices, where centering destroys sparsity and you should set with_mean to False. Another follow-up is how to treat null values or mixed types during aggregation, or how you would validate that the transformed data actually has zero mean and unit variance without loading it all into memory.

ONE CONCRETE EXAMPLE: Suppose you have a CSV dataset with a numeric column price. In Dask, you would read the CSV into a Dask DataFrame, then use dask_ml.preprocessing.StandardScaler, which under the hood computes the column mean and standard deviation across all partitions in the first pass and transforms the blocks in the second pass. If doing this manually in Python with pandas chunks of 1GB each, you would iterate through the file with read_csv chunksize, accumulate total count, total sum, and total sum of squares, then compute global mean and population standard deviation, and finally iterate again to write standardized chunks to disk.