How would feature engineering for categoricals differ for logistic regression versus LightGBM?

WHAT THIS TESTS: This question probes whether you understand that preprocessing is not model-agnostic. Specifically it checks if you know why linear models like logistic regression are sensitive to the numeric representation of categories while tree-based models like LightGBM are not. The interviewer wants to see that you connect algorithm mechanics to feature engineering decisions rather than applying a single recipe everywhere.

A GOOD ANSWER COVERS: First state that logistic regression is a linear model that assumes a continuous input space and learns weights implying direction and magnitude. Therefore it requires one-hot encoding for nominal variables so the model does not impose a false order or distance between categories. For ordinal variables integer encoding is acceptable if the order is real and roughly linear. Second explain that tree-based models split on thresholded feature values and do not compute dot products. This means they can handle ordinally encoded nominal features without assuming linear distance because the tree simply finds optimal cut points. Third mention that one-hot encoding for tree-based models can hurt performance by creating sparse features and deeper trees. Fourth note that high-cardinality features may push you toward target encoding or embeddings for either model type but the default split remains one-hot for logistic regression and ordinal for LightGBM.

COMMON WRONG ANSWERS: A major red flag is saying both models should use one-hot encoding. This reveals a lack of understanding about how trees handle ordinality and ignores the sparsity and memory costs one-hot imposes on tree learners. Another red flag is claiming that LightGBM always requires label encoding because it handles categories automatically. You should still discuss whether the category order is meaningful and whether native categorical support or ordinal encoding is being used. Saying that encoding never matters because the model will learn anyway is also incorrect especially for logistic regression where ordinality directly corrupts the weight interpretation.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle a categorical feature with ten thousand unique values. They might also ask what you would do if the categorical variable has a true hierarchy but nonlinear effects or how you would validate that your encoding choice actually improved performance. Be ready to discuss target encoding embeddings or hash encoding as alternatives when dimensionality explodes.

ONE CONCRETE EXAMPLE: Imagine a customer churn dataset with a nominal feature called neighborhood containing fifty distinct values. For logistic regression you would one-hot encode this into fifty binary columns so the model can learn independent weights for each area without assuming neighborhood A is closer to neighborhood B. For LightGBM you could ordinally encode neighborhood into integers one through fifty. The tree would evaluate splits like neighborhood less than or equal to twelve versus greater than twelve grouping similar churn rates without ever assuming that twelve is numerically farther from thirteen than from eleven. If you one-hot encoded for LightGBM you would create fifty sparse columns forcing the tree to evaluate many shallow splits and increasing training time without accuracy gains.

Source: MachineLearningMastery.com