Batch & Throttled Fan-out

Throttled fan-out

Implement the Batch & Throttled Fan-out pattern in this codebase, using the LLM client and language we already use.

## Goal

Process a large list of independent items through a model as fast as the account's rate limits allow, retrying transient failures, and collect one result per item.

## How it works

1. Load the items into a queue, each with a stable id, and build each request from the Per-item classification prompt.
2. Estimate each request's token count, and run a limiter with two token buckets: requests per minute and tokens per minute, set just below your account's limits.
3. Start a fixed pool of concurrent workers. Each takes the next item, waits until the limiter has capacity for it, and calls the model.
4. On a 429 or 5xx, put the item back in the queue with exponential backoff and jitter, and pause the whole pool briefly so the other workers do not hit the same limit.
5. Parse each reply and write it to the results file under the item's id as soon as it arrives; record items that fail permanently in a separate errors file.

## Prompts

### Per-item classification prompt

```markdown
Classify the support ticket into exactly one category.

<categories>
{{categories}}
</categories>
<ticket id="{{ticket_id}}">
{{ticket_text}}
</ticket>

Reply in JSON: {"id": "{{ticket_id}}", "category": "<one category>", "confidence": 0.0-1.0}
```

## Stop rule

Stop when the queue is empty and no worker is busy. Items that fail 5 times go to the errors file with their last error; the run reports counts of succeeded and failed items.

## Requirements

- Make the request and token limits, the concurrency and the retry cap parameters.
- Write results incrementally, keyed by item id, so a crashed run can skip items that are already done when restarted.
- Never let results depend on completion order: match each reply to its item by id.
- Log throughput (items and tokens per minute), the 429 rate and the total cost as the run progresses.
- Add a test with a stub model that returns 429 on the first call for item 7 and succeeds otherwise; for 20 items, assert 20 results, that item 7 was called twice, and that every result carries the right id.

Provider batch

Implement the Batch & Throttled Fan-out pattern (Provider batch variant) in this codebase, using the LLM client and language we already use.

## Goal

Process a large list of independent items through the provider's asynchronous batch API at the batch discount, and match every result back to its item.

## How it works

1. Build one request per item from the Per-item classification prompt, each tagged with a custom id equal to the item's id.
2. Split the requests into batches under the provider's limits on count and size, and submit each batch.
3. Poll each batch's status at a modest interval (every few minutes) until it has ended.
4. Download the results and match each one to its item by custom id; never by position.
5. Collect the requests that errored or expired and submit them again as a new, smaller batch.

## Prompts

### Per-item classification prompt

```markdown
Classify the support ticket into exactly one category.

<categories>
{{categories}}
</categories>
<ticket id="{{ticket_id}}">
{{ticket_text}}
</ticket>

Reply in JSON: {"id": "{{ticket_id}}", "category": "<one category>", "confidence": 0.0-1.0}
```

## Stop rule

Stop when every item has a succeeded result, or after 2 resubmissions. Items still failing go to an errors file with their last error.

## Requirements

- Persist the batch ids as soon as they are created, so a restarted script polls existing batches instead of submitting duplicates.
- Treat a batch that is still running after the provider's deadline as expired and resubmit only its unfinished items.
- Log each batch's id, request count, status changes, and the counts of succeeded, errored and expired results.
- Add a test with a stub batch client that returns results out of order with one errored item; assert every result is matched to the right item and exactly one item is resubmitted.