Process a 50GB CSV with only 16GB RAM
Streaming aggregation under memory constraints.
Chunk with read_csv chunksize, filter columns via usecols, downcast int64 to int32/int16, skip rows.
Loading everything into one DataFrame or using default dtypes.
What's really being asked
This question tests whether you understand how Pandas materializes CSV data in memory and whether you can design a streaming aggregation pipeline when working sets exceed available RAM. A senior candidate should demonstrate that default pd.read_csv behavior is dangerous for large files and should articulate a sequence of memory-reduction tactics before suggesting distributed frameworks.
A GOOD ANSWER COVERS four things in order. First, explain why the naive approach fails: a 10GB CSV can expand past 30GB in memory because Pandas stores string columns as Python objects, uses int64 for integers by default, and adds index overhead. Second, reduce the data footprint before reading by using the usecols parameter to load only necessary columns, which shrinks the dataset dramatically when you need just a few fields from a wide file. Third, optimize data types explicitly by passing a dtype dictionary to downcast integers from int64 to int32 or int16 where the range allows, which the reference notes can cut memory usage by 50 to 90 percent. Fourth, process the file as a stream rather than a monolithic load by using chunksize in read_csv to iterate over chunks, aggregating each chunk and discarding it before reading the next, which keeps memory bounded by the chunk size rather than the full 50GB. The candidate should also mention using nrows to sample first and inspect dtypes, and skiprows to omit unnecessary records.
The mistakes people make
Proposing to load the entire file into a single DataFrame and hope the OS swap handles it, suggesting to buy more RAM instead of engineering a solution, or recommending distributed tools like Dask or Spark without first explaining why native Pandas chunking and dtype optimization are the immediate levers. Another red flag is ignoring the object dtype bloat from strings and simply accepting Pandas defaults.
What usually comes next
The interviewer may ask how you would aggregate statistics across chunks without holding all intermediate data, how you would handle groupby operations when groups span multiple chunks, or what you would do if even a single chunk with all columns still exceeds RAM. They might also ask you to estimate memory usage with df.memory_usage(deep=True).
A concrete example
Suppose you need to compute the total count from a 50GB CSV with 100 columns on a 16GB machine. A strong answer is to first read 1000 rows with nrows to inspect the data, then define a dtype spec that casts IDs to int32 and category codes to int16. Next, open the file with read_csv(chunksize=100000, usecols=['id', 'count'], dtype=dtype_spec), iterate through each chunk, sum the count column, and accumulate a running total in a plain Python variable. This keeps peak memory under a few hundred megabytes.
Interview question
When aggregating a 50GB CSV on a 16GB machine, which strategy keeps peak memory usage proportional to a small fragment rather than the entire file?
- a.Use read_csv with usecols and dtype downcasting, then load everything into one DataFrame
- b.Sample 1000 rows with nrows to inspect dtypes, then process the full file with default parameters
- c.Iterate with read_csv(chunksize=...), aggregating each fragment and discarding it before reading the nextCorrect
- d.Load the full file with optimized dtypes but rely on OS swap for overflow
Why? this is the answer
Streaming with chunksize processes only one fragment at a time, keeping memory bounded by that fragment instead of the full 50GB. Option A is tempting because filtering columns and downcasting dtypes are valid optimizations, but materializing the entire file in a single DataFrame still exhausts RAM.
Just read this? Test yourself on what you have been reading.
Read the original → oneuptime.com
- #python
- #pandas
- #csv
- #memory-optimization
- #streaming
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on python — each one lists the topics its interview covers.
See open roles