The analysis phase: tokenizers and token filters
understanding text analysis in search indexing.
analysis turns raw text into index terms via a tokenizer that splits text into tokens then token filters that transform them, like lowercasing or stemming.
What's really being asked
This evaluates whether you understand the pipeline that converts raw text into the normalized terms stored in the inverted index, and can cleanly separate the roles of a tokenizer and a token filter.
The full answer
Analysis is performed by an analyzer, a pipeline with three stages. First, optional character filters preprocess the raw character stream, for example stripping HTML tags or mapping characters. Second, exactly one tokenizer breaks the character stream into a sequence of tokens, the candidate terms, also recording offsets and positions. Third, a chain of zero or more token filters operates on that token stream, adding, removing, or rewriting tokens. The same analyzer typically runs at both index time and query time so terms match. The output terms populate the inverted index.
Tokenizer versus token filter
A tokenizer decides where token boundaries are. The standard tokenizer splits on word boundaries and punctuation; a whitespace tokenizer splits only on spaces. A token filter transforms the already-split tokens. A lowercase filter normalizes case so Dog and dog match; a stop filter drops common words like the and is; a stemmer reduces running, runs, and ran toward run. The key distinction: tokenizers split, filters transform.
Why it matters
The analysis configuration directly controls recall and precision. Aggressive stemming and synonym filters boost recall; minimal analysis preserves exact matching. Mismatched index-time and query-time analysis silently breaks searches.
What usually comes next
Why must index and query analyzers agree. What does a stemmer versus a lemmatizer do. How do synonym filters work. What are character filters for.
A concrete example
Analyzing The Quick Foxes with a standard tokenizer yields tokens The, Quick, Foxes. A lowercase filter makes them the, quick, foxes, a stop filter removes the, and a stemmer reduces foxes to fox, so the indexed terms are quick and fox.
Interview question
In a search analyzer, what is the precise division of labor between a tokenizer and a token filter?
- a.The token filter splits text and the tokenizer applies stemming
- b.The tokenizer splits text into tokens; token filters transform, add, or remove those tokensCorrect
- c.Both split text identically but the filter runs only at query time
- d.The tokenizer lowercases text and the token filter splits it into words
Why? this is the answer
A tokenizer determines token boundaries (splitting), while token filters like lowercase, stop, and stemmer transform the resulting tokens. Options A and D reverse these roles.
Just read this? Test yourself on what you have been reading.
Read the original → lucene.apache.org
- #text-analysis
- #tokenizer
- #token-filter
- #elasticsearch
- #lucene
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles