Agentic RAG

Source router

Implement the Agentic RAG pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer a question by letting a model decide, hop by hop, whether to search, which source to search and with what rewritten query, until the context it has gathered is enough for a cited answer.

## How it works

1. Describe each source (vector index, web search, SQL) in one line, and keep a table from each source name to the function that searches it.
2. Call the model with the Retrieval agent prompt, filling in the question, the context gathered so far (empty at first) and the source descriptions, and parse its JSON step.
3. If action is answer, return the answer with its citations.
4. If action is search, look up the named source in the table, run the rewritten query, and append the top results to the context, each tagged with its source.
5. If the JSON does not parse or names an unknown source, retry the call once with the error appended; if it fails again, answer from the context so far.
6. If a search fails or times out, append a note saying that source failed, so the model can choose another, and loop.

## Prompts

### Retrieval agent prompt

```markdown
You answer the user's question. You may search before answering.

<question>{{question}}</question>
<context_so_far>{{context}}</context_so_far>

Sources you can search: {{source_descriptions}}

Decide the next step. Respond in JSON:
{"action": "search" | "answer",
 "source": "<one of the sources>",
 "query": "a standalone search query, rewritten for that source",
 "answer": "only when action is answer; cite the context you used"}
Answer as soon as the context is enough. Never invent facts that are not in the context.
```

## Stop rule

Stop when the model answers, or after 4 search hops. At the cap, call the model once more with search removed from the allowed actions and return that answer, flagged as having hit the hop limit.

## Requirements

- Make the hop cap (default 4) and the results per search (default 3 chunks or rows) parameters.
- Route with plain code, never a model call: the model names the source and the table dispatches the query.
- Give the SQL source a read-only connection, a timeout and a row limit.
- Log every hop: the chosen source, the rewritten query and the ids of the results.
- Add a test with a stub LLM that searches vectordb, then web, then answers; assert one call to each of those two backends, none to SQL, and that the answer cites the web result.

Interleaved (IRCoT)

Implement the Agentic RAG pattern (Interleaved (IRCoT) variant) in this codebase, using the LLM client and language we already use.

## Goal

Answer a multi-hop question by alternating one sentence of reasoning with one retrieval, using each new sentence as the next search query.

## How it works

1. Search the index with the question and put the top 4 paragraphs in a paragraph pool.
2. Call the model with the Step-by-step reasoner prompt, filling in the question, the pool and the reasoning so far; keep only the first sentence of its reply.
3. If the sentence contains 'So the answer is:', return the text after it, with the reasoning and the ids of the paragraphs used.
4. Otherwise search the index with that sentence, add any new paragraphs to the pool, append the sentence to the reasoning, and call the model again.
5. If a search fails, keep the pool as it is and continue; the next sentence becomes the next query.

## Prompts

### Step-by-step reasoner prompt

```markdown
Answer the question by reasoning one step at a time, using the paragraphs.

<paragraphs>{{paragraphs}}</paragraphs>
<question>{{question}}</question>
<reasoning_so_far>{{reasoning}}</reasoning_so_far>

Write exactly one next sentence of reasoning.
When you know the answer, write that sentence as: So the answer is: <answer>
```

## Stop rule

Stop when a sentence contains the answer marker, or after 6 sentences. At the cap, ask the model for the answer directly from the pool and return it, flagged as having hit the step limit.

## Requirements

- Make the sentence cap (default 6), the paragraphs per search (default 4) and the pool cap (default 15 paragraphs) parameters.
- De-duplicate paragraphs by id before adding them, and drop the oldest when the pool is full.
- Cut the model's reply to its first sentence, so a reply that runs ahead cannot skip retrievals.
- Log each sentence with the paragraph ids its search returned.
- Add a test with a stub LLM that writes 'X was directed by Y.' and then 'So the answer is: Z'; assert two searches, the second with the query 'X was directed by Y.', and the answer 'Z'.

Adaptive by complexity

Implement the Agentic RAG pattern (Adaptive by complexity variant) in this codebase, using the LLM client and language we already use.

## Goal

Spend retrieval only where it pays off: classify each question as needing no retrieval, one retrieval or iterative multi-step retrieval, then run the matching strategy.

## How it works

1. Send the question to a small classifier that returns one label: A (no retrieval), B (single-step) or C (multi-step).
2. For A, call the model with the question alone and return its answer.
3. For B, search the index once with the question, then call the model with the question and the top 5 documents.
4. For C, loop: search, call the model with the Retrieval agent prompt (with the index as the only source) and every document so far, and either return its answer or search again with the follow-up query it writes.
5. If the classifier fails or returns an unknown label, treat the question as B. If a search fails, answer from the documents so far (for B, from the question alone) and flag the answer.

## Prompts

### Retrieval agent prompt

```markdown
You answer the user's question. You may search before answering.

<question>{{question}}</question>
<context_so_far>{{context}}</context_so_far>

Sources you can search: {{source_descriptions}}

Decide the next step. Respond in JSON:
{"action": "search" | "answer",
 "source": "<one of the sources>",
 "query": "a standalone search query, rewritten for that source",
 "answer": "only when action is answer; cite the context you used"}
Answer as soon as the context is enough. Never invent facts that are not in the context.
```

## Stop rule

A and B end after one model call. C ends when the model answers, or after 4 searches; at the cap, answer from the documents gathered and flag the answer as having hit the limit.

## Requirements

- Make the search cap for C (default 4) and the documents per search (default 5) parameters.
- Train or prompt the classifier on labelled examples of all three classes, and track how often each label is chosen.
- Log the label, the strategy run, its search count and its latency for every question.
- Add a test with a stub classifier that returns A and a stub retriever; assert the retriever is never called and the model is called exactly once.