Describe strategies for handling missing values in pandas DataFrames

What's really being asked

This question probes whether you treat missing data as a modeling decision rather than a code snippet. Interviewers want to see that you understand deletion loses information, imputation introduces bias, and the right choice depends on the missingness mechanism and the downstream task.

The full answer

First, deletion strategies such as dropna on rows or columns, which are appropriate when data is missing completely at random and the remaining sample is still large enough to power the analysis, but which risk discarding useful signal if missingness is correlated with the target. Second, single-value imputation using fillna with the mean or median for numerical columns, which preserves row count and is fast, but artificially reduces variance and ignores inter-column relationships; or forward fill and backward fill for time-ordered data, which respects sequence but can propagate stale values. Third, model-based imputation such as KNN or iterative imputers, which captures correlations but is computationally expensive and can leak target information if done carelessly. Fourth, the importance of inspecting missingness patterns with isna and notna before acting, and the fact that pandas uses different sentinels such as np.nan for floats, NaT for datetimes, and pd.NA for nullable dtypes like Int64 or boolean, which affect how operations propagate.

The mistakes people make

Prescribing mean imputation for every column without checking distributions or missingness rates. Filling NaNs with zeros as a default, which creates misleading values for features like income or temperature. Dropping all rows containing any NaN without considering how much data is lost. Ignoring that np.nan does not equal itself, so equality checks fail and you must use isna instead. Overlooking that nullable extension dtypes such as Int64 preserve integer type whereas standard numpy int64 coerces missing values to float.

What usually comes next

How would you handle missingness in a time-series index versus a cross-sectional table? What would you do if a single column is ninety percent missing? How does imputation change the variance of a feature and why does that matter for a linear model? When would you use a mask or indicator column instead of filling the value?

A concrete example

Suppose you have a DataFrame of one million retail transactions with columns for customer_age, purchase_amount, and transaction_timestamp, and roughly five percent of customer_age values are missing. If age is missing completely at random, dropping those rows leaves nine hundred fifty thousand records and is safe for a quick aggregate report. If you are training a churn model where age is predictive, dropping rows loses labeled outcomes and reduces model performance; instead you might fill with the median age of that customer segment, or add an is_age_missing indicator column so the model learns whether missingness itself carries signal. For the timestamp column, a missing value would be NaT, and you could either drop the row if time is essential or infer it from the order of a sorted index.