Full-Text Search: Beyond Simple String Matching

Why it exists

Standard database queries like WHERE description LIKE '%term%' are slow and unintelligent for searching large bodies of text. They can't handle typos, understand that "running" and "ran" are related, or rank results by relevance. We need a way to search unstructured text like a human would, finding meaning, not just character sequences.

The mental model

Think of full-text search as creating a specialized index for human language, not just data values. Instead of indexing raw text, you index processed 'tokens'. This index is like one in the back of a textbook, but smarter. It knows that 'quick', 'quicker', and 'quickly' are related and can point you to all the right documents, then rank them by how well they match your intent.

How it works

Text is passed through an analyzer pipeline. This process first breaks text into tokens (usually words). Then, it might perform several steps: converting to lowercase, removing common stopwords ('the', 'a', 'is'), and stemming or lemmatizing words to their root form (e.g., 'running' becomes 'run'). These final tokens are stored in an 'inverted index', which maps each token back to the documents containing it. When you search, your query goes through the same analysis, and the engine uses the inverted index to find matching documents instantly, calculating a relevance score for each.

When to use it

Use it when you need to search through large volumes of unstructured text and relevance matters. This is ideal for e-commerce search bars, log analysis platforms (searching for error messages), content management systems, and any application where users need to find documents by searching their actual content.

When not to use it

Don't use it for searching structured, exact-match data. If you're looking up a user by their unique email or an order by its ID, a standard database index (like a B-tree) is far more efficient. Full-text search is overkill and less performant for simple key-value lookups.

One canonical example

A user searches for "running shoe" on an e-commerce site. A simple database LIKE query would miss products described as "shoes for running." A full-text search engine, however, would have stemmed "running" to "run" during indexing. It would find documents containing "run," "running," or even synonyms if configured. It would then rank the results, showing products with "running shoe" in the title or with high sales for that query first.