Stemming versus lemmatization in text preprocessing

WHAT THIS TESTS This evaluates whether you understand classic text normalization and, more importantly, when modern models make it unnecessary. It rewards matching preprocessing to the downstream representation.

A GOOD ANSWER COVERS Stemming reduces words to a root by chopping affixes using rule-based heuristics, for example the Porter stemmer. It is extremely fast and shrinks vocabulary, but the output is often not a real word and it can over-stem, conflating unrelated words, or under-stem, missing related ones. Lemmatization maps a word to its dictionary base form, the lemma, using a vocabulary and part-of-speech tagging, so better becomes good and running becomes run correctly. It is more accurate but slower and needs linguistic resources. The deeper insight is that with subword tokenization, as used by BERT and other transformers, or with learned word embeddings, you should usually skip both because the model already captures morphology and casing.

COMMON WRONG ANSWERS Saying stemming and lemmatization produce the same output. Always lemmatizing even when feeding a transformer, which can hurt by mismatching the pretraining distribution. Ignoring runtime cost on a large corpus. Claiming stemming always beats lemmatization or vice versa without naming the use case.

LIKELY FOLLOW-UPS Why does lemmatization need part-of-speech information. Give an example where stemming over-stems. How do subword tokenizers like Byte Pair Encoding change the calculus.

ONE CONCRETE EXAMPLE For the words organize, organizes, and organizing, a Porter stemmer returns organ for some inputs, an over-stem that collides with the unrelated word organ. A lemmatizer returns organize for all three, preserving meaning. In a bag-of-words logistic regression for a high-throughput search index, stemming's speed may be worth the noise; in a careful legal-document classifier, lemmatization's precision wins; in a fine-tuned transformer, you feed raw text and let WordPiece handle it.