Embedding Models: The 'Retrieval' in RAG

WHY IT EXISTS Large Language Models have a knowledge cut-off and no access to your private documents. To answer questions about proprietary data or recent events, a system must first find relevant information and provide it to the LLM. Embedding models solve this "finding" problem efficiently.

THE MENTAL MODEL Think of an embedding model as a universal translator that converts any piece of text into a fixed-size list of numbers, called a vector. This vector is like a coordinate in a giant "meaning space." The model is trained so that texts with similar meanings (e.g., "dog" and "canine") have vectors that are close to each other in this space. This allows a computer to "understand" and compare semantic relationships mathematically.

HOW IT WORKS In a Retrieval-Augmented Generation (RAG) system, the process is two-fold. First, you perform offline indexing: every document in your knowledge base is passed through an embedding model to generate a vector, which is then stored in a specialized vector database. Second, at query time, the user's question is passed through the exact same embedding model to create a query vector. The system then searches the database for the document vectors that are mathematically closest to the query vector. These top-matching documents are retrieved and passed to the LLM as context along with the original question.

WHEN TO USE IT The primary use case is the retrieval step in RAG for question-answering systems. They are also fundamental to any application requiring semantic understanding, such as semantic search engines, document clustering for topic modeling, and finding paraphrases or duplicate content.

WHEN NOT TO USE IT Embedding models are not ideal for pure keyword matching. If you need to find every document containing the exact term "GDPR compliance," a traditional inverted index is faster and more accurate. Also, while fast "bi-encoder" models are great for initial retrieval from millions of documents, they are less accurate than "cross-encoder" models, which are better suited for re-ranking a smaller set of top candidates.

ONE CANONICAL EXAMPLE The Retrieve & Re-Rank pattern is a common, powerful setup. A user asks a question. First, a fast bi-encoder model generates an embedding for the query and quickly retrieves the top 100 potentially relevant documents from a massive database. This is the "retrieve" step. Then, a slower but more precise cross-encoder model examines the query paired with each of the 100 documents individually to produce a more accurate relevance score. This is the "re-rank" step. The top 3-5 documents from the re-ranker are then sent to the LLM. This balances speed at scale with high-quality results.