Cross-Encoder Re-ranking: Accuracy Over Speed

Why it exists

Fast search methods often struggle with nuance, matching keywords but missing context. To solve this, we need a way to deeply understand the relationship between a query and a potential answer, not just their surface-level similarity. Cross-encoders provide this deep, contextual understanding at the cost of speed.

The mental model

Think of it as an expert reviewer versus a quick skimmer. A fast "dual-encoder" skims the query and a library of document summaries separately, then finds the closest match. A cross-encoder is an expert who reads the query and each shortlisted document side-by-side, word-for-word, to give a definitive relevance score. It's slow but extremely thorough.

How it works

A cross-encoder concatenates the query and a candidate document into a single input sequence, separated by special tokens (e.g., [CLS] query [SEP] document [SEP]). This combined text is fed through a transformer. Because the entire sequence is processed at once, the model's self-attention mechanism can weigh the importance of every query token against every document token. The final output from the special [CLS] token is then used to produce a single relevance score for that query-document pair.

When to use it

Use it as a second stage in a multi-stage retrieval pipeline. First, use a fast but less accurate retriever (like a dual-encoder or BM25) to narrow millions of documents down to a small shortlist of 50-100 candidates. Then, apply the computationally expensive cross-encoder to re-rank only this small set to get the final, high-quality ordering. This is ideal for semantic search, question answering, and entity linking.

When not to use it

Never use a cross-encoder as the first-stage retriever over a large corpus. The cost is prohibitive because you would need to run a full transformer forward pass for every single document in your database for every single query. This would be incredibly slow and expensive. The model is designed for re-ranking a small set, not initial retrieval from a large one.

One canonical example

To disambiguate "Washington" in a news article, a fast retriever might pull up documents about the state, the city, and the president. A cross-encoder would then take the query ("Washington") and its surrounding sentence ("Washington crossed the Delaware") and process it jointly with each candidate. It would identify the strong contextual link between "crossed the Delaware" and "George Washington," scoring that candidate highest.