Model Cascade

Sequential cascade

Implement the Model Cascade pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer each query with the cheapest model whose answer passes a reliability score, escalating up an ordered list of models only when the score falls below that rung's threshold.

## How it works

1. Configure an ordered list of models from cheapest to most capable, each with a score threshold; the last rung has no threshold.
2. Call the cheapest model with the Answer prompt (same at every rung).
3. Score the answer from 0 to 1. FrugalGPT uses a small trained scorer; until you have labelled data, use the Scorer prompt on a cheap model, and calibrate its self-reported scores before trusting a threshold.
4. If the score meets the rung's threshold, return the answer. Otherwise call the next model up with the same prompt and score again.
5. Return the last rung's answer without scoring it, and record which rung answered and the total cost of the request.

## Prompts

### Answer prompt (same at every rung)

```markdown
You are a support assistant for {{product}}. Answer the customer's question
accurately and concisely, using the documentation below.

<docs>{{docs}}</docs>
<question>{{question}}</question>

If the documentation does not cover it, say what is missing rather than guessing.
```

### Scorer prompt

```markdown
Rate how likely the answer below is to be correct and complete for the question.
Judge only from the question, the documentation and the answer.

<docs>{{docs}}</docs>
<question>{{question}}</question>
<answer>{{answer}}</answer>

Reply with a single number between 0 and 1, and nothing else.
```

## Stop rule

Stop at the first answer that meets its threshold, or after the top model answers. If a scorer call fails, treat the score as 0 and escalate, so a broken scorer costs money rather than quality.

## Requirements

- Keep the models and thresholds in configuration, and tune the thresholds on a labelled sample so that the cascade matches the top model's accuracy at the lowest cost.
- Parse the scorer's reply as a number; if it is not one, treat it as 0.
- Log per request: each rung tried, its score, its token cost, and the rung that answered.
- Report the share of traffic answered at each rung; if most traffic reaches the top, the cascade is costing more than calling the top model directly.
- Add a test with stub models where the small model's answer scores 0.41 against a threshold of 0.8; assert exactly one large-model call and that the large model's answer is returned. A second case scoring 0.93 must make zero large-model calls.

Predictive router

Implement the Model Cascade pattern (Predictive router variant) in this codebase, using the LLM client and language we already use.

## Goal

Before any answer is generated, predict whether the cheap model will answer a query well, and send it to exactly one model: the cheap one or the strong one.

## How it works

1. Collect labelled examples of queries where the cheap model's answer was as good as the strong model's, and where it was not (from preference data, evaluations or past cascade logs).
2. Train a small classifier on them that outputs the probability the strong model is needed. Until you have data, use a cheap model with the Difficulty router prompt as the router.
3. For each query, get the router's probability and compare it with a threshold set from your cost budget.
4. Send the query to the strong model if the probability exceeds the threshold, otherwise to the cheap model, using the Answer prompt for both.
5. Log the decision and sample routed traffic for offline grading, to retrain the router as the models change.

## Prompts

### Difficulty router prompt

```markdown
Decide whether a small, fast model can answer this support question well,
or whether it needs a stronger model.

<question>{{question}}</question>

Needs the strong model: multi-step troubleshooting, several interacting
conditions, ambiguous or unusual setups.
Small model is fine: single-fact lookups, standard how-to steps.

Reply with a single number between 0 and 1: the probability the strong model is needed.
```

### Answer prompt

```markdown
You are a support assistant for {{product}}. Answer the customer's question
accurately and concisely, using the documentation below.

<docs>{{docs}}</docs>
<question>{{question}}</question>

If the documentation does not cover it, say what is missing rather than guessing.
```

## Stop rule

Each query makes exactly one answer call. If the router fails or times out, send the query to the strong model.

## Requirements

- Make the threshold a parameter, and report the share of traffic it sends to the strong model.
- Keep the router far cheaper than the cheap model: a small classifier, or a short prompt on the smallest model.
- Re-evaluate the router whenever either model is upgraded; its training data describes the old pair.
- Add a test with a stub router returning 0.9 and 0.1 for two queries and a threshold of 0.5; assert the first reaches only the strong model and the second only the cheap one.

Draft and verify

Implement the Model Cascade pattern (Draft and verify variant) in this codebase, using the LLM client and language we already use.

## Goal

Have a cheap model draft every answer and a strong model verify it in one call, accepting the draft unchanged when it is right and rewriting it only when it is not.

## How it works

1. Call the small model with the Draft prompt to draft an answer.
2. Call the large model with the Verify prompt, passing the question, the documents and the draft.
3. Parse its JSON reply. If verdict is accept, return the draft; if it is rewrite, return the large model's answer instead.
4. Record the verdict, so you can track how often drafts are accepted.

## Prompts

### Draft prompt

```markdown
You are a support assistant for {{product}}. Answer the customer's question
accurately and concisely, using the documentation below.

<docs>{{docs}}</docs>
<question>{{question}}</question>
```

### Verify prompt

```markdown
A junior assistant drafted an answer. Check it against the documentation.

<docs>{{docs}}</docs>
<question>{{question}}</question>
<draft>{{draft}}</draft>

If the draft is correct and complete, reply exactly: {"verdict": "accept"}
Otherwise reply: {"verdict": "rewrite", "answer": "<your corrected answer>"}
```

## Stop rule

Every query makes two calls (three if the verifier's reply is retried). If the verifier's reply cannot be parsed, retry once, then return the draft flagged as unverified.

## Requirements

- Keep the verifier's output short when it accepts: accepting must cost far fewer output tokens than rewriting, or the pattern saves nothing.
- Track the acceptance rate; below about half, call the large model directly instead.
- Log both calls' token counts per query.
- Add a test with a stub verifier that accepts the first draft and rewrites the second; assert the first returned answer equals the draft and the second equals the verifier's rewrite.