Evaluator-Optimizer

Separate judge

Implement the Evaluator-Optimizer pattern in this codebase, using the LLM client and language we already use.

## Goal

Turn a task plus a list of pass criteria into an output that a separate evaluator model has checked against every criterion, revising until it passes or a round budget runs out.

## How it works

1. Take the task and its acceptance criteria as inputs. The generator sees only the task; the criteria go to the evaluator.
2. Call the generator model with the Generator prompt, filling in the task and, after the first round, the latest critique.
3. Send the draft and the criteria to the evaluator model with the Evaluator prompt, and parse its JSON reply into passed and critique.
4. If passed is true, return the draft.
5. Otherwise append the critique to a running list and call the generator again with every critique so far, so a later fix does not undo an earlier one.

## Prompts

### Generator prompt

```markdown
You are writing {{artifact}} for the task below.

<task>{{task}}</task>

{{#if feedback}}
A reviewer rejected your previous draft. Address every point:
<feedback>{{feedback}}</feedback>
{{/if}}

Return only the {{artifact}}, with no commentary.
```

### Evaluator prompt

```markdown
You are a strict reviewer. Judge the draft against each criterion.

<criteria>{{criteria}}</criteria>
<draft>{{draft}}</draft>

Respond in JSON:
{"passed": true|false, "critique": "specific, actionable fixes; empty if passed"}
Only pass the draft if every criterion is fully met.
```

## Stop rule

Stop as soon as the evaluator passes a draft, or after 3 rounds. If the budget runs out, return the last draft together with its unresolved critique and a status saying it did not pass.

## Requirements

- Make the round cap a parameter (default 3); the loop must never run without one.
- If the evaluator's reply is not valid JSON, retry that call once, then count the round as failed with the raw reply as the critique.
- Let the generator and the evaluator use different models and temperatures, configured separately (a strict, low-temperature evaluator works best).
- Log each round: the draft, the verdict, the critique and the token counts.
- Add a test with a stub LLM whose evaluator rejects the first two drafts and passes the third; assert three generator calls and that the third draft is returned.

Same model

Implement the Evaluator-Optimizer pattern (Same model variant) in this codebase, using the LLM client and language we already use.

## Goal

Have one model improve its own output: draft, critique the draft against the criteria, and rewrite from the critique, until its own critique finds nothing left to fix.

## How it works

1. Call the model with the Draft prompt to write the first draft from the task.
2. Call the same model with the Self-critique prompt, passing the draft and the criteria. It lists concrete problems, or replies with the single word STOP when there are none.
3. If the feedback is STOP, return the current draft.
4. Otherwise append the draft and its feedback to a history list.
5. Call the model with the Refine prompt, passing the task and the whole history, to write the next draft; then go back to the feedback step.

## Prompts

### Draft prompt

```markdown
You are writing {{artifact}} for the task below.

<task>{{task}}</task>

Return only the {{artifact}}, with no commentary.
```

### Self-critique prompt

```markdown
You wrote the draft below. Review it as a strict editor, against each criterion.

<criteria>{{criteria}}</criteria>
<draft>{{draft}}</draft>

List every concrete problem, one per line, each with the fix it needs.
If every criterion is fully met, reply with exactly: STOP
```

### Refine prompt

```markdown
Improve your {{artifact}} using the feedback on each earlier version.

<task>{{task}}</task>
<history>{{drafts_and_feedback}}</history>

Fix every listed problem without bringing back an earlier one.
Return only the revised {{artifact}}, with no commentary.
```

## Stop rule

Stop when the feedback is STOP, or after 3 refinements. If the cap is hit, return the latest draft with its last feedback and a status saying the model was still finding problems.

## Requirements

- Use one model client for all three prompts; only the prompt changes between the roles.
- Cap refinements with a configurable limit (default 3).
- Pass every earlier draft and its feedback to the refine prompt, oldest first, so a fix does not undo an earlier one.
- Treat the feedback as STOP only on an exact match after trimming whitespace, so a critique that merely mentions the word does not end the loop.
- Add a test with a stub LLM whose first feedback lists one problem and whose second is STOP; assert two drafts were written and the second is returned.

Tool-checked

Implement the Evaluator-Optimizer pattern (Tool-checked variant) in this codebase, using the LLM client and language we already use.

## Goal

Revise an output until an evaluator that verifies its checkable claims with tools passes it, so that every critique quotes evidence from a tool result.

## How it works

1. Call the generator model with the Generator prompt, filling in the task and, after the first round, the latest critique.
2. Send the draft and the criteria to the evaluator model with the Tool-checking evaluator prompt and the tool definitions; it picks the claims a tool can check and requests tool calls.
3. Run each requested tool call (a database query, a test run, a search) and return every result, or its error, to the evaluator.
4. Parse the evaluator's final JSON reply: passed, plus a critique that quotes the evidence for every failed claim.
5. If passed is true, return the draft; otherwise send the critique to the generator and run another round.

## Prompts

### Generator prompt

```markdown
You are writing {{artifact}} for the task below.

<task>{{task}}</task>

{{#if feedback}}
A reviewer checked your previous draft against the data and rejected it.
Fix every point, using the evidence they quote:
<feedback>{{feedback}}</feedback>
{{/if}}

Return only the {{artifact}}, with no commentary.
```

### Tool-checking evaluator prompt

```markdown
You are a strict reviewer with tools. Before judging, use the tools to check
every number, name and date in the draft that they can confirm.

<criteria>{{criteria}}</criteria>
<draft>{{draft}}</draft>

When you have checked the claims, respond in JSON:
{"passed": true|false, "critique": "each failed claim, the tool evidence, and the fix; empty if passed"}
Fail the draft if any checked claim disagrees with the evidence.
```

## Stop rule

Stop when the evaluator passes a draft, or after 3 rounds. On the cap, return the last draft with its critique and the tool evidence, and a status saying it did not pass.

## Requirements

- Give the evaluator read-only tools only, each with a timeout; report a failed or timed-out call to the evaluator as an error instead of retrying it silently.
- Cap tool calls per round (default 5) as well as rounds (default 3).
- Reject a failing verdict whose critique quotes no tool result, and retry the evaluator once.
- Log every tool call with its arguments and result, next to the verdict it supported.
- Add a test with a stub LLM and a fake tool that returns a different figure from the first draft's; assert the critique quotes the tool's figure and the second draft passes.