CRITIC

Search and code

Implement the CRITIC pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer a question, then have the same model verify the answer's claims with tools and revise it from the evidence, until a verification round finds nothing wrong.

## How it works

1. Call the model with the question to get a first answer.
2. Call the same model with the Verification prompt, the question, the answer and the tool definitions. It lists the claims a tool can check and requests the tool calls.
3. Run each requested call (a search, a code execution, a calculation) and return the results, or their errors, to the model.
4. Parse the model's verdict: either OK, or a critique that names each wrong claim and quotes the evidence.
5. If the verdict is OK, return the answer. Otherwise call the model with the Revision prompt, passing the question, the answer and the critique, and verify the revised answer from the start.

## Prompts

### Verification prompt

```markdown
Check the answer below before it is shown to anyone. You have tools.

<question>{{question}}</question>
<answer>{{answer}}</answer>

1. List every claim in the answer that a tool could check: dates, numbers,
   names, quotations, calculations.
2. Use the tools to check each one. Do not rely on memory.
3. Then reply with exactly one of:
   OK
   or
   WRONG: one line per wrong claim, with the tool evidence that contradicts it.
```

### Revision prompt

```markdown
Your answer contained claims that the evidence contradicts.

<question>{{question}}</question>
<answer>{{answer}}</answer>
<critique>{{critique}}</critique>

Rewrite the answer so that every claim agrees with the evidence.
Keep everything that was not criticised. Return only the revised answer.
```

## Stop rule

Stop when a verification round finds no wrong claim, or after 3 revisions. On the cap, return the last answer with its unresolved critique and a status saying it was not verified.

## Requirements

- Use one model for answering, verifying and revising; only the prompt changes.
- Give the verifier read-only tools with timeouts, and cap tool calls per round (default 5) as well as revisions (default 3).
- Accept a critique only if it quotes a tool result; a critique without evidence counts as OK, so the model cannot argue itself out of a correct answer.
- Log every claim checked, the tool call that checked it, the result and the verdict.
- Add a test with a stub LLM that first answers 1936 and a fake search that returns 1937; assert the critique quotes 1937, one revision happens, and the returned answer says 1937.

Code and tests (Self-Debug)

Implement the CRITIC pattern (Code and tests (Self-Debug) variant) in this codebase, using the LLM client and language we already use.

## Goal

Generate code for a task, run it against the available tests, and have the same model explain its code against the failures and then fix it, until the tests pass.

## How it works

1. Call the model with the task and the visible tests to write the first version of the code.
2. Run the code against the tests in a sandbox and capture, for each failing test, the input, the expected output, the actual output or the error.
3. If every test passes, return the code.
4. Otherwise call the same model with the Explanation prompt, passing the code and the failing test results. It explains the code line by line and says where its behaviour differs from what the tests expect, without writing code.
5. Call the model with the Fix prompt, passing the code, the failures and that explanation, to write a complete fixed version.
6. Run the fixed version against the tests and repeat.

## Prompts

### Explanation prompt

```markdown
Your code fails some tests.

<task>{{task}}</task>
<code>{{code}}</code>
<failures>{{failing_tests_with_expected_and_actual}}</failures>

Explain what the code does, line by line, as if to a colleague.
Then trace one failing test through it and say exactly where its
behaviour differs from what the test expects. Do not write code.
```

### Fix prompt

```markdown
<task>{{task}}</task>
<code>{{code}}</code>
<failures>{{failing_tests_with_expected_and_actual}}</failures>
<explanation>{{explanation}}</explanation>

Using the explanation, fix the code so every failing test passes
without breaking the ones that pass. Return the complete fixed code
in one code block, with no commentary.
```

## Stop rule

Stop when every visible test passes, or after 3 debugging rounds. On the cap, return the version that passed the most tests, with the failures that remain.

## Requirements

- Run code in a sandbox with time and memory limits; treat a timeout as a failing test with the message 'timed out'.
- Pass the actual output and the expected output for every failing test, not only a pass/fail count.
- Keep every version, its test results and its explanation, and return the best version if the cap is hit.
- Do not show the model hidden tests; check the final code against them separately, if you have them.
- Add a test with a stub LLM whose first program fails one test and whose second passes all; assert one explain call, one fix call, two test runs, and that the second program is returned.