Retrieve-and-Rerank

Problem

Cheap retrieval is either precise or broad, never both. A vector search for “What was ACME Corp’s revenue growth in Q2 2023?” finds chunks about revenue growth in general, from any company and any quarter. A keyword search finds “ACME” and “Q2 2023” but misses a chunk that never names them. Worse, a chunk cut from a filing may say only “revenue grew by 3% over the previous quarter”, which matches neither search well because it has lost its context. And if you raise the top-k to catch more, the generator drowns in near-misses.

How it works

The pattern has two halves.

At index time, a contextualizer reads each chunk of the documents together with its whole document and writes a one-line prefix: “This chunk is from ACME Corp’s Q2 2023 10-Q; the previous quarter’s revenue was $314M.” The prefixed chunk goes into a BM25 keyword index and a vector index. Now the orphaned “grew by 3%” chunk carries the company and the quarter, so both searches can find it. Anthropic calls this contextual retrieval.

At query time, hybrid search runs the question against both indexes at once, casting a wide net: about 150 candidates from each, which it fuses into one list. Then the reranker does the expensive part. It reads every candidate alongside the question, with a cross-encoder or an LLM, and keeps only the best 20. Fast retrieval maximizes recall, and the slow reranker restores precision. The generator answers from those 20, and the answer cites them.

In Anthropic’s tests, contextual embeddings cut top-20 retrieval failures by 35%, adding contextual BM25 brought that to 49%, and adding reranking brought it to 67%.

When to use

• The corpus is too large to put in the prompt. Anthropic suggests about 200,000 tokens as the threshold below which you can skip retrieval entirely.
• Plain vector search misses exact terms such as product names, error codes or ticker symbols.
• Precision in the top few results matters, because the generator trusts whatever it is given.

When to avoid

• The corpus fits in the context window. Put it all in the prompt and use prompt caching instead.
• The latency budget cannot absorb a reranker scoring 150 candidates.
• The corpus changes constantly, and re-running the contextualizer over every new chunk is too expensive.

Trade-offs

The contextualizer costs one model call per chunk, but only once, at index time, and prompt caching on the shared document makes it cheap. Keeping two indexes in sync costs operational effort. The reranker adds latency to every query, though a cross-encoder scoring 150 short chunks is usually far cheaper than a generation call. In return, you get the recall of broad search with the precision of a careful judge.

Failure modes

• Bad prefixes: the contextualizer invents context the document does not support, and chunks get indexed under the wrong company or date. Keep prefixes short, and check a sample.
• Narrow net: the first stage returns too few candidates, so the right chunk never reaches the reranker. Recall is set by the first stage; the reranker can only reorder what it is given.
• Index drift: a chunk is updated in one index but not the other. Write to both under one chunk ID, in one job.
• Reranker as single point of failure: when it times out, the request fails. Fall back to the fused order.

Implement the Retrieve-and-Rerank pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer questions over a large corpus by indexing context-prefixed chunks in both a keyword index and a vector index, retrieving a wide candidate set from both, reranking it, and answering from the top few.

## How it works

1. At index time, split each document into chunks. For each chunk, call the contextualizer model with the Chunk context prompt, passing the whole document and the chunk, and prepend the short context it returns.
2. Write every context-prefixed chunk to a BM25 keyword index and, embedded, to a vector index, under the same chunk ID.
3. At query time, search both indexes with the question in parallel, taking the top 150 candidates from each.
4. Fuse the two lists by chunk ID with Reciprocal Rank Fusion and keep the top 150 unique chunks.
5. Score each candidate against the question with a reranker (a cross-encoder model, or an LLM with the LLM reranker prompt) and keep the 20 highest-scoring chunks.
6. Call the generator model with the Grounded answer prompt, passing the question and the 20 chunks, and return its answer.

## Prompts

### Chunk context prompt

```markdown
<document>
{{whole_document}}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{{chunk}}
</chunk>

Give a short, succinct context to situate this chunk within the overall
document, for the purpose of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.
```

### LLM reranker prompt

```markdown
Rate how well the passage answers the question.

<question>{{question}}</question>
<passage>{{passage}}</passage>

Reply with a single integer from 0 to 10, where 10 means the passage
directly contains the answer and 0 means it is unrelated. Judge only
relevance to this question, not the passage's general quality.
```

### Grounded answer prompt

```markdown
Answer the question using only the passages below, which are ranked
from most to least relevant.

<question>{{question}}</question>
<passages>{{passages}}</passages>

Cite the passage ID after each claim. If the passages do not answer the
question, say so.
```

## Stop rule

Stop after one retrieval, one rerank and one generation. If the best reranker score is below the relevance floor, say the corpus has no answer instead of generating.

## Requirements

- Make the candidate count (default 150), the kept count (default 20) and the relevance floor configurable.
- Cache the contextualizer calls, for example with prompt caching on the whole document, since every chunk of a document repeats it.
- Index the prefix and chunk text together in both indexes, and pass the prefixed chunk to the generator so it knows the source.
- If the reranker fails or times out, fall back to the fused order instead of failing the request, and log that it did. If the LLM reranker's reply for a candidate is not an integer from 0 to 10, score that candidate 0 and log it.
- Add a test with fake BM25 and vector indexes and a stub reranker that scores one chunk highest although it ranks 140th after fusion; assert that chunk is first in the 20 passed to the generator, and exactly 20 are passed.

<document>
{{whole_document}}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{{chunk}}
</chunk>

Give a short, succinct context to situate this chunk within the overall
document, for the purpose of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.

Rate how well the passage answers the question.

<question>{{question}}</question>
<passage>{{passage}}</passage>

Reply with a single integer from 0 to 10, where 10 means the passage
directly contains the answer and 0 means it is unrelated. Judge only
relevance to this question, not the passage's general quality.

Answer the question using only the passages below, which are ranked
from most to least relevant.

<question>{{question}}</question>
<passages>{{passages}}</passages>

Cite the passage ID after each claim. If the passages do not answer the
question, say so.