Process a 50GB CSV with only 16GB RAM

WHAT THIS TESTS: This question tests whether you understand how Pandas materializes CSV data in memory and whether you can design a streaming aggregation pipeline when working sets exceed available RAM. A senior candidate should demonstrate that default pd.read_csv behavior is dangerous for large files and should articulate a sequence of memory-reduction tactics before suggesting distributed frameworks.

A GOOD ANSWER COVERS four things in order. First, explain why the naive approach fails: a 10GB CSV can expand past 30GB in memory because Pandas stores string columns as Python objects, uses int64 for integers by default, and adds index overhead. Second, reduce the data footprint before reading by using the usecols parameter to load only necessary columns, which shrinks the dataset dramatically when you need just a few fields from a wide file. Third, optimize data types explicitly by passing a dtype dictionary to downcast integers from int64 to int32 or int16 where the range allows, which the reference notes can cut memory usage by 50 to 90 percent. Fourth, process the file as a stream rather than a monolithic load by using chunksize in read_csv to iterate over chunks, aggregating each chunk and discarding it before reading the next, which keeps memory bounded by the chunk size rather than the full 50GB. The candidate should also mention using nrows to sample first and inspect dtypes, and skiprows to omit unnecessary records.

COMMON WRONG ANSWERS: Proposing to load the entire file into a single DataFrame and hope the OS swap handles it, suggesting to buy more RAM instead of engineering a solution, or recommending distributed tools like Dask or Spark without first explaining why native Pandas chunking and dtype optimization are the immediate levers. Another red flag is ignoring the object dtype bloat from strings and simply accepting Pandas defaults.

LIKELY FOLLOW-UPS: The interviewer may ask how you would aggregate statistics across chunks without holding all intermediate data, how you would handle groupby operations when groups span multiple chunks, or what you would do if even a single chunk with all columns still exceeds RAM. They might also ask you to estimate memory usage with df.memory_usage(deep=True).

ONE CONCRETE EXAMPLE: Suppose you need to compute the total count from a 50GB CSV with 100 columns on a 16GB machine. A strong answer is to first read 1000 rows with nrows to inspect the data, then define a dtype spec that casts IDs to int32 and category codes to int16. Next, open the file with read_csv(chunksize=100000, usecols=['id', 'count'], dtype=dtype_spec), iterate through each chunk, sum the count column, and accumulate a running total in a plain Python variable. This keeps peak memory under a few hundred megabytes.

Source: oneuptime.com