Why avoid one-hot encoding for high cardinality and what are alternatives?

WHAT THIS TESTS: This question checks whether you recognize the curse of dimensionality in sparse feature spaces and whether you can move beyond textbook one-hot encoding to practical strategies for high-cardinality nominal variables. Interviewers want to see that you understand memory, compute, and overfitting trade-offs when a single categorical feature can explode into hundreds or thousands of binary columns.

A GOOD ANSWER COVERS: First, explain the core problem: one-hot encoding creates a new binary column for every unique category, so a feature like City with hundreds of values becomes hundreds of sparse columns. This blows up the feature vector, increases memory usage, slows training, and makes overfitting more likely because the model must search a much larger, sparser solution space. Second, name two alternatives and briefly describe how they avoid the dimensionality trap. Target encoding replaces each category with the mean of the target variable for that category, collapsing the feature into one numeric column while preserving predictive signal. Feature hashing uses a hash function to map categories into a fixed, smaller number of columns, guaranteeing bounded dimensionality regardless of cardinality. Either pair is acceptable, but you should show you know why they work.

COMMON WRONG ANSWERS: A major red flag is proposing label encoding or ordinal encoding for nominal data like City, because that imposes an artificial order on unordered categories and can mislead linear or distance-based models. Another weak answer is suggesting dropping the feature entirely without discussing information loss. Simply saying use fewer categories without a concrete strategy such as grouping rare levels is also vague. Finally, recommending one-hot encoding anyway without acknowledging the computational cost signals a lack of production awareness.

LIKELY FOLLOW-UPS: The interviewer may ask how you prevent data leakage with target encoding, so you should mention cross-fold target encoding or smoothing to avoid overfitting on rare categories. They might also ask when feature hashing collisions matter, or how embeddings differ from the other methods, especially in deep learning contexts. You could also be asked to compare memory footprints: one-hot scales with cardinality, while hashing scales with the chosen output dimension.

ONE CONCRETE EXAMPLE: Consider the Criteo Display Advertising dataset, which has 26 categorical variables with a total cardinality of 241,338. Applying one-hot encoding would expand the feature space from 39 dimensions to over 241,000 dimensions, producing a massive sparse matrix that is computationally expensive to store and train. Instead, target encoding could replace each site ID with its historical click-through rate, and feature hashing could compress advertiser IDs into a fixed 100-column vector, keeping the model tractable without discarding the predictive signal.

Source: Towards Data Science