HyDE: Find Documents by Embedding a Fake Answer

WHY IT EXISTS Standard dense retrieval struggles in zero-shot scenarios where no labeled data is available for fine-tuning. A short query's embedding vector can be semantically distant from a long, relevant document's vector, leading to poor search results. HyDE was created to bridge this gap without needing any relevance labels.

THE MENTAL MODEL Imagine you need a specific screw from a hardware store. Instead of describing it vaguely ('a small metal thing'), you show the clerk a perfect 3D-printed model of the screw. The model isn't a real screw, but it's a perfect representation that helps the clerk find the real one in the bins. HyDE does this for search: it 'prints' a hypothetical document to find the real ones.

HOW IT WORKS The process has two steps. First, given a query, an instruction-following LLM generates a 'hypothetical document' that answers it. This document is fictional but captures the expected content and terminology of a correct answer. Second, this hypothetical document is passed to an unsupervised, contrastively learned encoder (like Contriever) to create an embedding vector. This vector, representing an ideal answer, is then used to perform a similarity search against the pre-computed embeddings of the actual document corpus, retrieving the closest real documents.

WHEN TO USE IT Use HyDE for zero-shot dense retrieval, especially when you have no labeled data to fine-tune your retriever model. It excels in tasks like question answering, fact verification, and web search, where it closes the semantic gap between a short query and a long document. It's effective across many languages.

WHEN NOT TO USE IT Do not use HyDE if you need the generated document itself to be factual. The hypothetical document is a tool for creating a better search vector and is expected to contain fabrications. It also adds an extra LLM call, which introduces latency and cost compared to directly embedding a query. If you have a large, high-quality set of labeled query-document pairs, a fine-tuned retriever might be more efficient.

ONE CANONICAL EXAMPLE For a query like 'What is the function of a transformer's attention mechanism?', HyDE first asks an LLM to generate a document explaining it. The fake document might say, 'The attention mechanism in a transformer model allows it to weigh the importance of different words...' This text, though generated, is then embedded. The resulting vector is used to find real documents in the corpus that also explain attention in similar terms, grounding the search in reality.