Stemming versus lemmatization in text preprocessing
NLP normalization trade-offs.
stemming chops affixes fast but crudely, yielding non-words; lemmatization maps to real dictionary base forms using POS, slower but accurate; skip both for embedding or transformer models.
What's really being asked
This evaluates whether you understand classic text normalization and, more importantly, when modern models make it unnecessary. It rewards matching preprocessing to the downstream representation.
The full answer
Stemming reduces words to a root by chopping affixes using rule-based heuristics, for example the Porter stemmer. It is extremely fast and shrinks vocabulary, but the output is often not a real word and it can over-stem, conflating unrelated words, or under-stem, missing related ones. Lemmatization maps a word to its dictionary base form, the lemma, using a vocabulary and part-of-speech tagging, so better becomes good and running becomes run correctly. It is more accurate but slower and needs linguistic resources. The deeper insight is that with subword tokenization, as used by BERT and other transformers, or with learned word embeddings, you should usually skip both because the model already captures morphology and casing.
The mistakes people make
Saying stemming and lemmatization produce the same output. Always lemmatizing even when feeding a transformer, which can hurt by mismatching the pretraining distribution. Ignoring runtime cost on a large corpus. Claiming stemming always beats lemmatization or vice versa without naming the use case.
What usually comes next
Why does lemmatization need part-of-speech information. Give an example where stemming over-stems. How do subword tokenizers like Byte Pair Encoding change the calculus.
A concrete example
For the words organize, organizes, and organizing, a Porter stemmer returns organ for some inputs, an over-stem that collides with the unrelated word organ. A lemmatizer returns organize for all three, preserving meaning. In a bag-of-words logistic regression for a high-throughput search index, stemming's speed may be worth the noise; in a careful legal-document classifier, lemmatization's precision wins; in a fine-tuned transformer, you feed raw text and let WordPiece handle it.
Interview question
When is it usually best to apply neither stemming nor lemmatization to your text?
- a.When using a bag-of-words logistic regression model
- b.When stop words have already been removed
- c.When feeding a subword-tokenized transformer like BERTCorrect
- d.When the corpus is very small
Why? this is the answer
Transformers use subword tokenization and pretraining that already handle morphology and casing, so normalizing can mismatch their expected input. Bag-of-words models, by contrast, benefit from vocabulary reduction via stemming or lemmatization.
Just read this? Test yourself on what you have been reading.
Read the original → nlp.stanford.edu
- #nlp
- #stemming
- #lemmatization
- #text-preprocessing
- #tokenization
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on nlp — each one lists the topics its interview covers.
See open roles