Multi-Query RAG

RAG-Fusion

Implement the Multi-Query RAG pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer a question from a document index by searching with several model-written rewordings of it in parallel, fusing the ranked results with Reciprocal Rank Fusion, and answering from the fused top documents.

## How it works

1. Call the rewriter model with the Query rewriter prompt, filling in the question. Parse its reply into a list of N queries (default 3), drop any identical to the original, and add the original question as one more query.
2. Run every query against the retriever in parallel, keeping each ranked list of document IDs (default top 10 per query).
3. Fuse the lists with Reciprocal Rank Fusion: each document scores the sum of 1 / (60 + rank) over every list it appears in, with ranks starting at 1.
4. Sort by fused score, drop duplicates, and keep the top k documents (default 5).
5. Call the generator model with the Grounded answer prompt, passing the original question, never a rewrite, and the k fused documents, and return its answer.

## Prompts

### Query rewriter prompt

```markdown
You write search queries for a document index.

<question>{{question}}</question>

Write {{n}} different search queries that together cover what someone
would need to read to answer the question. Vary the wording and the angle:
use the terms a document on the subject would use, not the question's own.
Return one query per line, with no numbering and no commentary.
```

### Grounded answer prompt

```markdown
Answer the question using only the documents below. They were retrieved
with several rewordings of the question, so some may cover only part of it.

<question>{{question}}</question>
<documents>{{documents}}</documents>

Cite the document ID after each claim, like [D7]. If a part of the
question is not covered by the documents, say which part.
```

## Stop rule

Stop after one fan-out, one fusion and one generation. If every retrieval returns nothing, say the index has no answer instead of generating.

## Requirements

- Make the number of rewrites, the per-query depth, k and the RRF constant (default 60) configurable.
- Run the retrievals concurrently with a per-query timeout; a failed or timed-out query drops out of the fusion instead of failing the request.
- If the rewriter's reply cannot be parsed, fall back to searching with the original question alone.
- Log each query with its ranked IDs and the fused scores, so a surprising answer can be traced to the query that pulled its document in.
- Add a test with a stub LLM returning 3 queries and a fake retriever keyed by query: D7 ranks 2, 1 and 3 in their lists, and the original question gets an empty list. Assert 4 retrieval calls and D7 first with score exactly 1/62 + 1/61 + 1/63.

Sub-questions

Implement the Multi-Query RAG pattern (Sub-questions variant) in this codebase, using the LLM client and language we already use.

## Goal

Answer a multi-part question by splitting it into independent sub-questions, answering each from its own retrieval, and combining the sub-answers into one final answer.

## How it works

1. Call the decomposer model with the Decomposer prompt, filling in the question. Parse its reply into a list of 2 to 4 independent sub-questions.
2. For each sub-question, in parallel, retrieve the top k documents (default 4).
3. For each sub-question, call the answer model with the Sub-answer prompt, passing that sub-question and only its own documents.
4. Call the combiner model with the Combiner prompt, passing the original question and every sub-question with its sub-answer.
5. Return the combined answer, with the sub-answers attached so a reader can check each part.

## Prompts

### Decomposer prompt

```markdown
Break the question into the smallest set of independent sub-questions
that together answer it.

<question>{{question}}</question>

Each sub-question must make sense on its own and must not depend on the
answer to another one. Write 2 to 4 sub-questions, one per line, with no
numbering and no commentary.
```

### Sub-answer prompt

```markdown
Answer the sub-question using only the documents below.

<sub_question>{{sub_question}}</sub_question>
<documents>{{documents}}</documents>

Give a short, factual answer and cite the document ID after each claim.
If the documents do not answer it, reply exactly: not found
```

### Combiner prompt

```markdown
Answer the original question from the sub-answers below.

<question>{{question}}</question>
<sub_answers>{{sub_questions_and_answers}}</sub_answers>

Keep every citation. If a sub-answer is "not found", say which part of
the question could not be answered.
```

## Stop rule

Stop once the combiner has answered. A sub-question with no documents gets the sub-answer 'not found', and the combiner must say which part it could not answer.

## Requirements

- Cap the sub-questions (default 4); if the decomposer returns more, keep the first four and log the rest.
- Run the retrieve-and-answer branches concurrently, each with its own timeout.
- Give each sub-answer call only its own documents, so one branch's sources cannot leak into another's answer.
- Reject sub-questions that depend on another's answer (that needs a loop, as in agentic RAG). If the decomposer's reply cannot be parsed or yields fewer than 2 sub-questions, answer the original question with a single retrieval.
- Add a test with a stub LLM that splits a question into 3 sub-questions; assert 3 retrievals, 3 sub-answer calls each holding only its own documents, and one combiner call holding all 3 sub-answers.

Step-back query

Implement the Multi-Query RAG pattern (Step-back query variant) in this codebase, using the LLM client and language we already use.

## Goal

Answer a specific question by retrieving for it and for a more general step-back question in parallel, then answering from both sets of documents.

## How it works

1. Call the step-back model with the Step-back prompt, filling in the question. It returns one more general question about the principles or background behind it.
2. Retrieve the top k documents (default 4) for the original question and, in parallel, for the step-back question.
3. Call the generator model with the Answer with background prompt, passing the original question, the step-back question and the two document sets, labelled separately.
4. Return the answer.

## Prompts

### Step-back prompt

```markdown
You are an expert at stepping back from a specific question to the
general principle or background it depends on.

<question>{{question}}</question>

Write one more general question whose answer would give the background
needed to answer the original. Return only that question.
```

### Answer with background prompt

```markdown
Answer the question. Use the background documents for principles and
the specific documents for facts.

<question>{{question}}</question>
<step_back_question>{{step_back_question}}</step_back_question>
<background_documents>{{background_docs}}</background_documents>
<specific_documents>{{specific_docs}}</specific_documents>

Cite the document ID after each claim.
```

## Stop rule

Stop after one step-back question, two retrievals and one generation. If both retrievals come back empty, say the index has no answer.

## Requirements

- Keep the original-question retrieval even when the step-back retrieval succeeds; the specifics come from it.
- Label the two document sets separately in the generator prompt, so the model knows which documents are background.
- If the step-back call fails, fall back to the original retrieval alone instead of failing the request.
- Log the step-back question next to both result lists.
- Add a test with a stub LLM returning a fixed step-back question and a fake retriever; assert the retriever was called twice, once with each question, and the generator prompt holds the step-back documents inside <background_documents> and the original-question documents inside <specific_documents>.