Chain-of-Verification

Factored

Implement the Chain-of-Verification pattern in this codebase, using the LLM client and language we already use.

## Goal

Reduce made-up facts in an answer by checking each of its claims with a separate question the model answers without seeing the answer, then rewriting the answer to agree with those checks.

## How it works

1. Call the model with the Draft prompt to write a baseline answer to the user's question.
2. Call the model with the Plan verification prompt, passing the question and the draft, and parse one verification question per line.
3. Answer every verification question in its own call with the Verification prompt. Pass only that question: never the draft, the user's question or the other checks. Run these calls in parallel.
4. Call the model with the Revise prompt, passing the question, the draft and every verification question with its answer.
5. Return the revised answer, and keep the verification questions and answers alongside it for logs and review.

## Prompts

### Draft prompt

```markdown
Answer the question.

<question>{{question}}</question>
```

### Plan verification prompt

```markdown
Below is a question and a draft answer. Write questions that check each
factual claim in the draft. Each question must make sense on its own,
without the draft, and must be answerable with a short fact.

<question>{{question}}</question>
<draft>{{draft}}</draft>

Write one verification question per line, and nothing else.
```

### Verification prompt

```markdown
Answer the question with a short, factual answer. If you are not sure,
say "unsure".

<question>{{verification_question}}</question>
```

### Revise prompt

```markdown
Below is a question, a draft answer, and checks of the draft's claims that
were answered independently. Rewrite the answer so it agrees with the checks.
Remove any claim a check contradicts, and add nothing new. A check answered
"unsure" neither confirms nor contradicts; keep that claim unless another
check contradicts it.

<question>{{question}}</question>
<draft>{{draft}}</draft>
<checks>
{{#each checks}}
Q: {{question}}
A: {{answer}}
{{/each}}
</checks>

Return only the revised answer.
```

## Stop rule

Stop after the revise call: the pattern makes one pass and never loops. If the plan produces no questions, return the draft unchanged with a status saying nothing was checked.

## Requirements

- Never include the draft in a verify call; build those prompts from the verification question alone, so a hallucination cannot be copied into its own check.
- Cap the verification questions (default 10) and run the verify calls concurrently with a configurable limit.
- Keep each verification question and answer paired, in plan order, when passing them to the revise prompt.
- If a verify call fails, retry it once, then record its answer as 'unsure' so the claim is neither confirmed nor removed by it.
- Log the draft, the plan, every check and the revised answer, so a removed claim can be traced to the check that removed it.
- Add a test with a stub LLM: draft 'Hillary Clinton, Donald Trump, Theodore Roosevelt', checks answer Chicago, Queens, Manhattan; assert three verify calls, none containing the draft, and that the revise prompt contains 'Chicago'.

Joint

Implement the Chain-of-Verification pattern (Joint variant) in this codebase, using the LLM client and language we already use.

## Goal

Check a draft's claims with the fewest calls: one call writes the verification questions and answers them, then a second rewrites the draft.

## How it works

1. Call the model with the Draft prompt to write a baseline answer.
2. Call the model with the Plan and answer prompt, passing the question and the draft. It writes each verification question followed by its answer.
3. Parse the reply into question and answer pairs.
4. Call the model with the Revise prompt, passing the question, the draft and the pairs, and return the revised answer.

## Prompts

### Draft prompt

```markdown
Answer the question.

<question>{{question}}</question>
```

### Plan and answer prompt

```markdown
Below is a question and a draft answer. For each factual claim in the
draft, write a question that checks it, then answer that question.

<question>{{question}}</question>
<draft>{{draft}}</draft>

Use this format, one pair per claim:
Q: <verification question>
A: <short factual answer>
```

### Revise prompt

```markdown
Below is a question, a draft answer, and checks of the draft's claims.
Rewrite the answer so it agrees with the checks. Remove any claim a
check contradicts, and add nothing the checks do not support.

<question>{{question}}</question>
<draft>{{draft}}</draft>
<checks>{{checks}}</checks>

Return only the revised answer.
```

## Stop rule

Stop after the revise call; the pattern makes one pass. If no pairs can be parsed, return the draft with a status saying nothing was checked.

## Requirements

- Parse pairs strictly as alternating 'Q:' and 'A:' lines, and skip a question with no answer.
- Log the joint reply in full; it is the only record of what was checked.
- Measure how often the checks agree with the draft; if they almost always do, switch to the factored form.
- Add a test with a stub LLM whose joint reply has three Q and A pairs; assert exactly three model calls and that the revise prompt contains all three answers.

Two-step

Implement the Chain-of-Verification pattern (Two-step variant) in this codebase, using the LLM client and language we already use.

## Goal

Check a draft's claims in two calls, keeping the draft out of the call that answers the checks, then rewrite the draft to match.

## How it works

1. Call the model with the Draft prompt to write a baseline answer.
2. Call the model with the Plan verification prompt, passing the question and the draft, and parse one verification question per line.
3. Call the model once with the Answer all checks prompt, passing only the verification questions as objects with number (counting from 1) and text. It must not see the draft.
4. Parse the numbered answers and pair each with its question.
5. Call the model with the Revise prompt, passing the question, the draft and the pairs, and return the revised answer.

## Prompts

### Draft prompt

```markdown
Answer the question.

<question>{{question}}</question>
```

### Plan verification prompt

```markdown
Below is a question and a draft answer. Write questions that check each
factual claim in the draft. Each question must make sense on its own,
without the draft, and must be answerable with a short fact.

<question>{{question}}</question>
<draft>{{draft}}</draft>

Write one verification question per line, and nothing else.
```

### Answer all checks prompt

```markdown
Answer each question with a short, factual answer. If you are not sure,
say "unsure". Number your answers to match the questions.

{{#each questions}}
{{number}}. {{text}}
{{/each}}
```

### Revise prompt

```markdown
Below is a question, a draft answer, and checks of the draft's claims that
were answered without seeing the draft. Rewrite the answer so it agrees
with the checks. Remove any claim a check contradicts.

<question>{{question}}</question>
<draft>{{draft}}</draft>
<checks>{{checks}}</checks>

Return only the revised answer.
```

## Stop rule

Stop after the revise call; the pattern makes one pass. If the plan produces no questions, return the draft with a status saying nothing was checked.

## Requirements

- Never include the draft or the user's question in the verify-all call.
- Number the verification questions from 1 and require answers with the same numbers, so each answer can be matched to its question.
- Cap the verification questions (default 10).
- Add a test with a stub LLM: draft three names, plan three questions, answers 'Chicago', 'Queens', 'Manhattan'; assert four model calls and that the verify-all prompt does not contain the draft.

Factor + revise

Implement the Chain-of-Verification pattern (Factor + revise variant) in this codebase, using the LLM client and language we already use.

## Goal

Check each claim of a draft with an independent question, mark each claim consistent, inconsistent or partly consistent with its check, then rebuild the answer from the consistent parts.

## How it works

1. Call the model with the Draft prompt to write a baseline answer.
2. Call the model with the Plan claims and checks prompt, passing the question and the draft, and parse its reply into pairs: each claim, quoted from the draft, with the question that checks it.
3. Answer every question in its own call with the Verification prompt, passing only that question, in parallel.
4. For each pair, call the model with the Cross-check prompt, passing the claim and its check's question and answer. Parse CONSISTENT, INCONSISTENT, or PARTIALLY CONSISTENT with the consistent part.
5. Call the model with the Revise prompt, passing the question and every claim with its verdict line (for a partial one, 'PARTIALLY CONSISTENT: ' plus the consistent part), and return the answer it splices together from the consistent parts.

## Prompts

### Draft prompt

```markdown
Answer the question.

<question>{{question}}</question>
```

### Plan claims and checks prompt

```markdown
Below is a question and a draft answer. List each factual claim in the
draft, quoted or closely restated, and a question that checks it. Each
question must make sense on its own, without the draft, and must be
answerable with a short fact.

<question>{{question}}</question>
<draft>{{draft}}</draft>

Use this format, one pair per claim:
Claim: <the claim>
Question: <the verification question>
```

### Verification prompt

```markdown
Answer the question with a short, factual answer.

<question>{{verification_question}}</question>
```

### Cross-check prompt

```markdown
Is the claim consistent with the evidence?

<claim>{{claim}}</claim>
<evidence>
Q: {{verification_question}}
A: {{verification_answer}}
</evidence>

Reply with one of:
CONSISTENT
INCONSISTENT
PARTIALLY CONSISTENT: <the part of the claim the evidence supports>
```

### Revise prompt

```markdown
Below is a question and the verdict on each claim of a draft answer.
Write the final answer from the consistent material only: keep CONSISTENT
claims, keep only the supported part of PARTIALLY CONSISTENT claims, and
leave out INCONSISTENT claims. Add nothing new.

<question>{{question}}</question>
<verdicts>{{claims_with_verdicts}}</verdicts>

Return only the final answer.
```

## Stop rule

Stop after the revise call; the pattern makes one pass. If the plan produces no claim and question pairs, return the draft with a status saying nothing was checked.

## Requirements

- Never include the draft in a verify call; the cross-check is the only step that sees a claim next to its check.
- Give each cross-check exactly one claim, so verdicts cannot bleed between claims.
- Treat a cross-check reply that does not start with one of the three verdicts as INCONSISTENT, and log it.
- Cap the claims (default 10), and run the verify and cross-check calls concurrently with a limit.
- Add a test with a stub LLM: claim 'Donald Trump was born in Queens in 1948' gets PARTIALLY CONSISTENT, consistent part 'Donald Trump was born in Queens'; assert the revise prompt pairs that claim with 'PARTIALLY CONSISTENT: Donald Trump was born in Queens'.