How do you analyze and reduce large pandas DataFrame memory usage?

WHAT THIS TESTS: This question probes whether you treat memory optimization as a measurement-driven process or a cargo-cult checklist. Senior candidates should demonstrate awareness that pandas is an in-memory analytics tool, so footprint directly dictates which operations can complete without spilling to disk or crashing. The interviewer wants to see that you understand dtypes are not just semantic labels but physical storage decisions with arithmetic and interoperability consequences.

A GOOD ANSWER COVERS: First, measurement. Mention df.info with memory_usage set to deep because the default shallow mode fails to capture object-dtype overhead accurately. Second, column selection at load time using usecols in read_csv or columns in read_parquet, which avoids materializing data you will immediately drop. Third, numeric downcasting with pd.to_numeric and downcast set to integer or float to move from int64 to int32 or int8 when ranges permit, and using nullable extension arrays like Int8 instead of float64 for missing integers. Fourth, categorical conversion for object columns with low cardinality, explaining that the trade-off is higher memory if cardinality exceeds roughly fifty percent of row count because the category codes plus the unique values list outweigh a simple object array. Fifth, the string dtype migration in pandas 3.0, which replaces object with a dedicated backed string array that is often more memory-efficient.

COMMON WRONG ANSWERS: Claiming df.shape tells you memory usage. Recommending category dtype universally without mentioning cardinality thresholds. Suggesting pd.to_numeric without specifying downcast parameters. Ignoring the difference between shallow and deep memory reporting. Proposing chunking as a memory-reduction technique when the question specifically asks about DataFrame memory, not out-of-core processing.

LIKELY FOLLOW-UPS: How would you handle a DataFrame that still does not fit in memory after optimization? When is category dtype slower than object despite using less RAM? How does the new string dtype in pandas 3.0 change your approach? What is the memory cost of a MultiIndex versus a flat integer index?

ONE CONCRETE EXAMPLE: Suppose you load a CSV with a user_id column stored as int64 but containing values only up to one million. Downcasting from int64 to int32 halves the memory from eight bytes per value to four bytes per value. On ten million rows, that is eighty megabytes reduced to forty megabytes. If the same file contains a country column with two hundred unique strings among ten million rows, converting to category stores ten million integer codes at one byte each plus the two hundred unique strings, using roughly ten megabytes instead of hundreds of megabytes of object pointers and scattered strings.