The analysis phase: tokenizers and token filters

WHAT THIS TESTS This evaluates whether you understand the pipeline that converts raw text into the normalized terms stored in the inverted index, and can cleanly separate the roles of a tokenizer and a token filter.

A GOOD ANSWER COVERS Analysis is performed by an analyzer, a pipeline with three stages. First, optional character filters preprocess the raw character stream, for example stripping HTML tags or mapping characters. Second, exactly one tokenizer breaks the character stream into a sequence of tokens, the candidate terms, also recording offsets and positions. Third, a chain of zero or more token filters operates on that token stream, adding, removing, or rewriting tokens. The same analyzer typically runs at both index time and query time so terms match. The output terms populate the inverted index.

TOKENIZER VERSUS TOKEN FILTER A tokenizer decides where token boundaries are. The standard tokenizer splits on word boundaries and punctuation; a whitespace tokenizer splits only on spaces. A token filter transforms the already-split tokens. A lowercase filter normalizes case so Dog and dog match; a stop filter drops common words like the and is; a stemmer reduces running, runs, and ran toward run. The key distinction: tokenizers split, filters transform.

WHY IT MATTERS The analysis configuration directly controls recall and precision. Aggressive stemming and synonym filters boost recall; minimal analysis preserves exact matching. Mismatched index-time and query-time analysis silently breaks searches.

LIKELY FOLLOW-UPS Why must index and query analyzers agree. What does a stemmer versus a lemmatizer do. How do synonym filters work. What are character filters for.

ONE CONCRETE EXAMPLE Analyzing The Quick Foxes with a standard tokenizer yields tokens The, Quick, Foxes. A lowercase filter makes them the, quick, foxes, a stop filter removes the, and a stemmer reduces foxes to fox, so the indexed terms are quick and fox.