Why avoid one-hot encoding for high cardinality and what are alternatives?

What's really being asked

This question checks whether you recognize the curse of dimensionality in sparse feature spaces and whether you can move beyond textbook one-hot encoding to practical strategies for high-cardinality nominal variables. Interviewers want to see that you understand memory, compute, and overfitting trade-offs when a single categorical feature can explode into hundreds or thousands of binary columns.

The full answer

First, explain the core problem: one-hot encoding creates a new binary column for every unique category, so a feature like City with hundreds of values becomes hundreds of sparse columns. This increases the feature vector, memory use, and training cost; it can also increase overfitting risk in some models and datasets. Second, name two alternatives and briefly describe how they avoid the dimensionality trap. Target encoding replaces each category with a target statistic, often a smoothed mean, collapsing the feature into one numeric column. Compute it out-of-fold for training data and from training data only for validation or test data to avoid target leakage. Feature hashing uses a hash function to map categories into a fixed, smaller number of columns, guaranteeing bounded dimensionality regardless of cardinality. Either pair is acceptable, but you should show you know why they work.

The mistakes people make

A major red flag is proposing label encoding or ordinal encoding for nominal data like City, because that imposes an artificial order on unordered categories and can mislead linear or distance-based models. Another weak answer is suggesting dropping the feature entirely without discussing information loss. Simply saying use fewer categories without a concrete strategy such as grouping rare levels is also vague. Finally, recommending one-hot encoding anyway without acknowledging the computational cost signals a lack of production awareness.

What usually comes next

The interviewer may ask how you prevent data leakage with target encoding, so you should mention cross-fold target encoding or smoothing to avoid overfitting on rare categories. They might also ask when feature hashing collisions matter, or how embeddings differ from the other methods, especially in deep learning contexts. You could also be asked to compare memory footprints: one-hot scales with cardinality, while hashing scales with the chosen output dimension.

A concrete example

Imagine a dataset with a City column containing thousands of values. One-hot encoding creates thousands of sparse columns. A smoothed, out-of-fold target encoding can reduce this to one numeric feature, while feature hashing maps cities into a chosen fixed number of columns. Each approach trades simplicity, leakage risk, collisions, and interpretability differently.