LLM-as-Judge

Pointwise

Implement the LLM-as-Judge pattern in this codebase, using the LLM client and language we already use.

## Goal

Score a system's outputs on a fixed test set with a separate judge model and a written rubric, and block a change from shipping when the aggregate score falls below a threshold.

## How it works

1. Assemble a test set of inputs that represent real traffic, and write a rubric with a 1-5 scale per criterion and an example of each score.
2. Run the system under test on every input and keep each output alongside its input.
3. For each pair, call the judge model with the Pointwise judge prompt, filling in the input, the output and the rubric. Parse its JSON reply into a reasoning string and an integer score.
4. Aggregate the scores into a report: the mean per criterion, the score histogram, and the lowest-scoring cases with the judge's reasons.
5. Compare the mean with the threshold and with the last released version. Pass if it clears both; otherwise block the release and attach the report.
6. Every so often, have a person score a sample of cases, and report how often the judge agrees with them.

## Prompts

### Pointwise judge prompt

```markdown
You are grading one reply from a customer-support assistant. Judge it only
against the rubric. Length and confident tone are not quality.

<rubric>{{rubric}}</rubric>
<customer_message>{{input}}</customer_message>
<assistant_reply>{{output}}</assistant_reply>

First explain in two or three sentences how the reply meets or misses each
criterion. Then give one integer score from 1 to 5 as defined in the rubric.
Respond in JSON: {"reasoning": "...", "score": 1-5}
```

## Stop rule

Stop after every case is scored once. If more than 5% of judge replies fail to parse after one retry each, stop and report the run as invalid; an invalid run blocks the release.

## Requirements

- Use a judge from a different model family than the system under test, at temperature 0; if you cannot, say so in the report.
- Make the judge write its reasoning before its score, and validate that the score is an integer on the rubric's scale.
- Keep the test set, rubric and judge prompt under version control, and record their versions in every report.
- Report the judge's agreement with the latest human-scored sample; if agreement drops below a set level, block releases until the judge is recalibrated.
- Add a test with a stub judge that returns scores 4, 5, 2 and 5 for four cases, a threshold of 3.5 and a previous release mean of 3.9; assert the report's mean is exactly 4.0 and the gate passes.

Pairwise

Implement the LLM-as-Judge pattern (Pairwise variant) in this codebase, using the LLM client and language we already use.

## Goal

Decide which of two system versions is better by having a judge model compare their replies case by case, in both orders, and report the win rate.

## How it works

1. Run both versions, A and B, on every input in the test set.
2. For each input, call the judge with the Pairwise judge prompt, showing A's reply first and B's second. Parse the verdict: A, B or tie.
3. Call the judge again on the same input with the replies swapped, B first and A second, and map its verdict back to A or B.
4. If the two verdicts agree, count a win for that version; if they disagree, count a tie, because the judge was following the order, not the content.
5. Report the win, loss and tie counts, the win rate excluding ties, and the cases where the verdicts disagreed.

## Prompts

### Pairwise judge prompt

```markdown
Two assistants answered the same customer message. Decide which reply is
better against the rubric. The order of the replies means nothing, and
a longer reply is not a better one.

<rubric>{{rubric}}</rubric>
<customer_message>{{input}}</customer_message>
<reply_1>{{first}}</reply_1>
<reply_2>{{second}}</reply_2>

Explain your comparison in two or three sentences, then decide.
Respond in JSON: {"reasoning": "...", "winner": "1" | "2" | "tie"}
```

## Stop rule

Stop after every input has been judged in both orders. If more than a quarter of cases end in disagreement, report the comparison as inconclusive instead of naming a winner.

## Requirements

- Always judge each pair in both orders; never count a single-order verdict.
- Hide which version is which from the judge: label the replies only 'Reply 1' and 'Reply 2'.
- Keep a fixed random seed for any sampling of the test set, so the two versions see identical inputs.
- Report the tie rate as a measure of how much the judge was guided by order.
- Add a test with a stub judge that always picks Reply 1; assert every case is counted as a tie and the result is reported as inconclusive.

Jury

Implement the LLM-as-Judge pattern (Jury variant) in this codebase, using the LLM client and language we already use.

## Goal

Score each output with a panel of smaller judges from different model families and pool their verdicts, so no single model's bias decides the result.

## How it works

1. Pick three or more judge models from different model families. Prefer small, cheap models over one large one.
2. For each output, call every judge in parallel with the same Juror prompt, and parse each reply into a verdict.
3. Pool the verdicts: a majority vote for pass or fail, or the mean for numeric scores.
4. Record each judge's verdict next to the pooled one, so a judge that keeps disagreeing with the panel stands out.
5. Aggregate the pooled verdicts across the test set into a report, as for a single judge.

## Prompts

### Juror prompt

```markdown
You are one of several independent graders. Judge the reply only against
the rubric.

<rubric>{{rubric}}</rubric>
<customer_message>{{input}}</customer_message>
<assistant_reply>{{output}}</assistant_reply>

Explain your judgement in one or two sentences, then decide whether the
reply passes. Respond in JSON: {"reasoning": "...", "verdict": "pass" | "fail"}
```

## Stop rule

Stop after every output has a pooled verdict. If a judge fails to answer after one retry, pool the rest; if fewer than three answered, or the remaining votes tie, mark that case as unjudged.

## Requirements

- Draw the judges from different model families, and never include the family of the system under test.
- Give every judge the identical prompt and rubric.
- Use an odd number of judges for pass or fail votes, so there are no ties.
- Report per-judge agreement with the pooled verdict, and review any judge that falls far below the others.
- Add a test with three stub judges returning pass, pass and fail; assert the pooled verdict is exactly pass and the dissent is recorded for the third judge.

Agent-as-a-Judge

Implement the LLM-as-Judge pattern (Agent-as-a-Judge variant) in this codebase, using the LLM client and language we already use.

## Goal

Evaluate an agent's run by giving a judge agent tools to read its workspace and trace, so each requirement gets a met or not-met verdict backed by evidence.

## How it works

1. Write the task's requirements as a numbered checklist, each one specific enough to check, such as 'saves the plot to out/accuracy.png'.
2. Run the agent under test on the task, and keep its final workspace (the files it wrote) and its trace of steps and tool calls.
3. Give the judge agent read-only tools over the workspace and trace: list and read files, search, and run a file in a sandbox.
4. Run the judge with the Agent judge prompt, one requirement at a time. It gathers evidence with its tools, then returns met or not met with the evidence it relied on.
5. Collect the verdicts into a report: the share of requirements met, and for each failure, the evidence.

## Prompts

### Agent judge prompt

```markdown
You are checking whether another agent's work meets one requirement. You can
inspect its workspace and its trace with your tools; you cannot change them.

<task>{{task}}</task>
<requirement>{{requirement}}</requirement>

Gather evidence before deciding: read the relevant files, search the trace,
and run code if that is the only way to know. Then respond in JSON:
{"verdict": "met" | "not_met", "evidence": "file paths, commands and output you relied on"}
```

## Stop rule

Stop after every requirement has a verdict, or the judge has used 15 tool calls on one requirement; then record that requirement as unverified rather than guessing.

## Requirements

- Give the judge read-only access; a judge that can edit the workspace can change what it is judging.
- Run any code the judge executes in a sandbox with a timeout.
- Require every verdict to cite evidence: a file path and line, a command and its output, or a step in the trace.
- Judge each requirement in its own call, so one failure does not color the rest.
- Add a test with a stub judge and a fake workspace missing out/accuracy.png; assert that requirement's verdict is not met and its evidence names the missing path.