Routing

LLM classifier

Implement the Routing pattern in this codebase, using the LLM client and language we already use.

## Goal

Send each customer message to exactly one specialist handler, chosen by a cheap LLM classifier, with a general handler as the fallback for anything unclear.

## How it works

1. Define the route table in code: each label (billing, tech, general) maps to one handler with its own prompt, tools and model.
2. Call a small, fast model with the Classifier prompt, filling in the message, and parse the JSON reply into a label.
3. If the reply is not valid JSON, or the label is not in the route table, use the general route.
4. Call the chosen handler with the original message, not the classifier's output; for the tech label that is the Tech-support handler prompt with its knowledge-base and log tools.
5. Return the handler's reply together with the label that was used.

## Prompts

### Classifier prompt

```markdown
Classify the customer message into exactly one category:

- billing: charges, invoices, refunds, plan changes
- tech: errors, crashes, bugs, how-to questions about the product
- general: anything else

<message>{{request}}</message>

Respond in JSON: {"label": "billing" | "tech" | "general"}
If the message fits no category clearly, answer "general".
```

### Tech-support handler prompt

```markdown
You are a support engineer for {{product}}. You can search the
knowledge base and read the customer's recent error logs.

<message>{{request}}</message>

Diagnose the problem, give numbered steps to fix it, and say what
to send us if the steps do not work.
```

## Stop rule

Each message makes one classifier call and one handler call, then stops. Nothing re-routes: if the handler cannot help, it says so in its reply.

## Requirements

- Keep the labels in one list that both the classifier prompt and the route table are built from, so they cannot drift apart.
- Never throw on a bad classification: a parse failure or an unknown label goes to the general handler, and a classifier timeout (default 5 s) does too.
- Run the classifier at temperature 0 with its own model setting, separate from the handlers' models.
- Log the message id, the raw classifier reply, the label used and whether it fell back, and sample the logs into a labelled test set.
- Add a test with a stub LLM whose classifier returns {"label": "refunds"}; assert the general handler was called once and the billing and tech handlers were not called.

Embedding router

Implement the Routing pattern (Embedding router variant) in this codebase, using the LLM client and language we already use.

## Goal

Route each message to a specialist handler by comparing its embedding with stored example utterances for every route, with no LLM call in the routing step and a fallback when nothing is close.

## How it works

1. For each route (billing, tech), write 10 to 30 example utterances taken from real traffic.
2. At startup, embed every example once and store each vector with its route name.
3. For each incoming message, embed it with the same embedding model and compute its cosine similarity to every stored example.
4. Take the top 5 matches, group them by route, score each route by the sum of its matches' similarities, and pick the highest-scoring route.
5. If the chosen route's own best similarity is below the threshold, use the general handler; otherwise call the chosen route's handler with the original message, such as the Tech-support handler prompt for tech.

## Prompts

### Classifier prompt

```markdown
Classify the customer message into exactly one category:

- billing: charges, invoices, refunds, plan changes
- tech: errors, crashes, bugs, how-to questions about the product
- general: anything else

<message>{{request}}</message>

Respond in JSON: {"label": "billing" | "tech" | "general"}
If the message fits no category clearly, answer "general".
```

### Tech-support handler prompt

```markdown
You are a support engineer for {{product}}. You can search the
knowledge base and read the customer's recent error logs.

<message>{{request}}</message>

Diagnose the problem, give numbered steps to fix it, and say what
to send us if the steps do not work.
```

## Stop rule

Each message makes one embedding call and one handler call. There is no retry: a weak match goes to the general handler rather than being routed again.

## Requirements

- Make the similarity threshold (default 0.75) and the number of neighbours (default 5) parameters, and tune the threshold on labelled messages.
- Embed the examples and the messages with the same model; re-embed every example when that model changes.
- If the embedding call errors or times out, send the message to the general handler instead of failing.
- Log the chosen route with its score and best similarity, and the runner-up route with its score, so near-ties can be found and new examples added.
- Test with a stub embedder and fixed scores: a best match of 0.40 goes to general; tech matches 0.92 + 4 × 0.50 go to tech; billing 0.93 plus tech 3 × 0.60 goes to general (tech wins 1.80 to 0.93, but its best is under 0.75).

Rules first

Implement the Routing pattern (Rules first variant) in this codebase, using the LLM client and language we already use.

## Goal

Route each message to one specialist handler, letting cheap deterministic rules decide the clear cases and calling an LLM classifier only for messages no rule matches.

## How it works

1. Keep an ordered list of rules as data, each a pattern (a keyword, a regular expression or a metadata field) and the route it selects, such as 'charged twice' to billing.
2. Run the rules against each message in order; the first rule that matches decides the route.
3. If no rule matches, call the Classifier prompt and parse its JSON reply into a label; an invalid reply, an unknown label, or a classifier error or timeout (default 5 s) means general.
4. Call the chosen handler with the original message, such as the Tech-support handler prompt for tech, and return its reply with the route and whether a rule or the classifier chose it.

## Prompts

### Classifier prompt

```markdown
Classify the customer message into exactly one category:

- billing: charges, invoices, refunds, plan changes
- tech: errors, crashes, bugs, how-to questions about the product
- general: anything else

<message>{{request}}</message>

Respond in JSON: {"label": "billing" | "tech" | "general"}
If the message fits no category clearly, answer "general".
```

### Tech-support handler prompt

```markdown
You are a support engineer for {{product}}. You can search the
knowledge base and read the customer's recent error logs.

<message>{{request}}</message>

Diagnose the problem, give numbered steps to fix it, and say what
to send us if the steps do not work.
```

## Stop rule

Each message is decided by at most one rule match or one classifier call, then handled once. Nothing re-routes.

## Requirements

- Keep the rules as data, in order, each with a name, so a rule can be added or removed without a code change and every decision can name its rule.
- Only add a rule when it is precise on labelled traffic (default bar: 98% of its matches are correct); vague keywords belong to the classifier.
- A rule that throws (such as a bad regular expression) is skipped and logged, never fatal.
- Log which rule fired, or that the classifier was used, so the share of free decisions can be tracked.
- Add a test with a stub LLM: assert "I was charged twice" reaches billing with zero classifier calls, and "The app crashes on login" makes exactly one classifier call.