HyDE

Problem

Questions and the documents that answer them are written differently. A user types “why does my sourdough collapse in the oven?”, while the article that answers it talks about overproofing, gluten structure and the poke test. Embed the short question, and its vector can sit far from the vector of the article it needs. Dense retrievers close that gap with training on labelled query-document pairs, but a new corpus rarely has any.

How it works

HyDE answers first and searches second. The question goes to a writer model, which drafts a short passage as if it were a document that answers the question: “Sourdough that collapses in the oven is usually overproofed. Long fermentation weakens the gluten network…” The passage may get details wrong. What matters is that it uses the vocabulary and shape of a real answer.

The embedder turns that passage into a vector, and the vector index returns the real documents nearest to it. An answer-shaped vector lands among answer-shaped documents, so the guide on overproofing comes back even though the question never used the word.

Then the hypothetical passage is thrown away. The generator gets the original question and the real documents only, and writes the answer with citations. The fake passage steered the search; it is never evidence. The second scenario shows why this matters: the writer guesses the wrong baking temperature, the vector still finds the baking guides, and the user gets the guides’ figures.

The paper also samples several hypothetical passages and averages their vectors with the question’s own vector, which smooths out any single odd guess.

When to use

• Queries are short or worded very differently from the documents.
• You have no labelled data to fine-tune the retriever, as with a new corpus or a new language.
• The model knows the domain well enough to write a plausible answer, even if its details are unreliable.

When to avoid

• The model knows nothing about the domain. Its passage points the search in the wrong direction.
• Exact matches matter, such as part numbers, error codes or names. Use keyword search or hybrid search instead (see retrieve-and-rerank).
• Latency is tight. The writer call runs before retrieval can start.

Trade-offs

HyDE adds one generation before every search: a short passage, so the cost is small, but it sits on the critical path. In return it lifts zero-shot retrieval without training anything, and you can still use a plain off-the-shelf embedding model. It also works as a drop-in step in front of an existing RAG pipeline, since it only changes the query vector.

Failure modes

• Confident wrong direction: the passage commits to a wrong premise and retrieves documents about that. Average several passages with the question’s vector.
• Leaked hallucination: the hypothetical passage reaches the generator and its invented facts end up in the answer. Never pass it on.
• Encoder mismatch: the passage is embedded with a different model from the index, and the nearest neighbours are noise. Use one embedding model throughout.
• Lost specifics: the passage paraphrases away an ID or a rare term the user typed. Combine HyDE with a keyword search for such queries.

Implement the HyDE pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer a question from a document index by searching with the embedding of a model-written hypothetical answer instead of the embedding of the question, then answering only from the real documents found.

## How it works

1. Call the writer model with the Hypothetical passage prompt, filling in the question. It returns one short passage written as if it came from a document that answers the question.
2. Embed the passage with the same embedding model that built the index. Optionally write N passages, embed each, and average their vectors together with the question's vector.
3. Run a nearest-neighbour search on the index with that vector and keep the top k documents (default 3).
4. Discard the hypothetical passage. It only steered the search, and none of its claims may reach the answer.
5. Call the generator model with the Grounded answer prompt, passing the original question and the k real documents, and return its answer.

## Prompts

### Hypothetical passage prompt

```markdown
Write a short passage that answers the question below, in the style of
{{document_type}}.

<question>{{question}}</question>

Write 3 to 5 sentences of plain prose with the vocabulary such a document
would use. Do not hedge or say you are unsure: this passage is only used to
search for real documents, and it is never shown to the user.
```

### Grounded answer prompt

```markdown
Answer the question using only the documents below.

<question>{{question}}</question>
<documents>{{documents}}</documents>

Cite the document ID after each claim, like [doc-12]. If the documents
do not answer the question, say so instead of guessing.
```

## Stop rule

Stop after one retrieval and one generation. If no document clears the similarity floor, say the index has no answer instead of answering from the hypothetical passage.

## Requirements

- Use one embedding model for both the index and the passage vectors; a mismatched encoder makes the search meaningless.
- Never pass the hypothetical passage to the generator; it may contain invented facts. If the writer call fails or times out, embed the question itself and continue, logging the fallback.
- Make k, the number of hypothetical passages and the similarity floor configurable, with defaults 3, 1 and 0.3.
- Log the hypothetical passage next to the IDs and scores of the documents it retrieved, so bad retrievals can be traced to the passage.
- Add a test with a stub LLM that returns a fixed passage and a fake embedder that records its inputs; assert the embedder received the passage, not the question, and the generator prompt holds the question and exactly 3 documents but not the passage.

Write a short passage that answers the question below, in the style of
{{document_type}}.

<question>{{question}}</question>

Write 3 to 5 sentences of plain prose with the vocabulary such a document
would use. Do not hedge or say you are unsure: this passage is only used to
search for real documents, and it is never shown to the user.

Answer the question using only the documents below.

<question>{{question}}</question>
<documents>{{documents}}</documents>

Cite the document ID after each claim, like [doc-12]. If the documents
do not answer the question, say so instead of guessing.