Retries & Model Fallback

Problem

Every model API fails some of the time. Providers rate-limit you at peak hours, return 503s during incidents, time out on long generations, and reject inputs that exceed the context window. If your support chat calls one model directly, each of those failures becomes an error message in front of a customer. A single outage at one provider can take your whole product down even though other models could have answered just as well.

How it works

Every model call goes through a gateway: a thin layer of plain code with an ordered list of models. It sends the customer’s message to the primary model first. When the call fails with an error that might clear on its own, such as a 429 rate limit, a 503 or a timeout, the gateway waits and tries again, doubling the wait each time and honouring any Retry-After header the provider sends, as long as it fits the request’s overall time budget.

If the primary keeps failing after its retry budget, the gateway puts it in cooldown for a minute and sends the same request to the fallback, ideally a model from a different provider so one incident cannot take out both. The cooldown is shared, so the next customer’s request skips the broken model at once instead of paying for its own retries.

Some failures are predictable. If the input is bigger than the normal models’ context window, the gateway sends it straight to a long-context model rather than waiting for an overflow error. And some failures are not worth retrying at all. A malformed request will fail on every model, so the gateway raises it immediately. A model that declines a request on content grounds has answered, not failed, so the refusal goes back to the caller as is: shopping it around other providers until one complies is not reliability engineering.

Whichever model answers, the reply looks the same to the customer, and the logs record which model served it.

When to use

• The feature has an availability target, and a single provider’s outage would breach it.
• You already use, or can use, more than one model or provider.
• Some inputs occasionally exceed a model’s context window and need a bigger one.

When to avoid

• Outputs must come from one exact model, for reproducibility, compliance or an evaluation you cannot rerun.
• The task depends on features only one provider has, such as a specific tool or output format, so a fallback would answer differently.
• The call is in an offline batch job that can simply be rerun later.

Trade-offs

Fallback trades consistency for availability. The fallback model writes in a different style, follows instructions differently and may score worse on your evaluations, so you need one prompt that works on all of them and tests that run against each. Retries add latency, sometimes many seconds, which is acceptable for a chat reply but not for an autocomplete. Every retry also costs money if the provider bills partial work.

Failure modes

• Retry storm: every request retries a provider that is already overloaded, making the outage worse. Use backoff with jitter, cap the retries and share cooldown state.
• Silent degradation: the fallback serves most traffic for days and nobody notices the quality drop. Alert when the fallback rate rises.
• Retrying the unfixable: a 400 error, a bad prompt or a refusal is retried on every model. Classify errors and return fatal ones at once.
• Unbounded waits: a provider sends Retry-After: 120 and the chat reply stalls for two minutes. Cap the total time and fail over when the wait does not fit.
• Prompt drift: the prompt is tuned for the primary and the fallback ignores half of it. Keep one portable prompt and evaluate it on every model in the list.

Implement the Retries & Model Fallback pattern in this codebase, using the LLM client and language we already use.

## Goal

Wrap every model call so that transient errors are retried with backoff, persistent failures fail over to the next model in an ordered list, and a model that keeps failing is skipped for a cooldown period.

## How it works

1. Configure an ordered list of deployments (provider plus model), each with its context-window size, plus one long-context deployment for oversized inputs. Send every deployment the same Portable system prompt.
2. Before a call, count the input tokens. If they exceed the first deployment's window, skip straight to the long-context deployment.
3. Call the first deployment that is not cooling down. On a retryable error (rate limit, 5xx, timeout), wait with exponential backoff and jitter, honouring any Retry-After header that fits the remaining time budget, and try the same deployment again.
4. After the retry budget for a deployment is spent, put it in cooldown and move to the next deployment in the list.
5. On a fatal error, stop and return it to the caller at once, without failing over or cooling anything down: a malformed request fails on every model, and a model's refusal is an answer, not an outage.
6. Return the response together with the deployment that served it and the number of attempts.

## Prompts

### Portable system prompt

```markdown
You are the support assistant for {{product}}. Answer the customer's
question using only the help-centre articles provided.

<articles>{{articles}}</articles>

Rules:
- Reply in plain text, at most 120 words, in the customer's language.
- If the articles do not cover the question, say so and offer to open a ticket.
- Never mention which AI model or provider you are.

Write the prompt so that every model in the fallback list can follow it:
no provider-specific tags, tools or formatting features.
```

## Stop rule

Stop at the first successful response. If every deployment fails or is cooling down, raise one error that lists each deployment and its last error, so the caller can show a graceful message.

## Requirements

- Make retries per deployment (default 2), base backoff (default 1 s) and cooldown length (default 60 s) parameters.
- Classify errors in one function with a table of status codes, so retryable, fail-over and fatal errors are tested in one place; content refusals, detected from the response's stop reason or refusal field, are fatal and logged.
- Cap the total time across all attempts (default 30 s): if Retry-After exceeds the remaining budget, fail over instead of waiting. Never retry a streamed response once tokens have reached the user.
- Keep cooldown state shared across requests in the process, so one outage does not cost every request its own retries. Log every attempt with its deployment, error class and wait.
- Add a test with stub models where the primary returns 429 three times; assert three primary calls, one fallback call, the fallback's text returned, and that a second request goes straight to the fallback.

You are the support assistant for {{product}}. Answer the customer's
question using only the help-centre articles provided.

<articles>{{articles}}</articles>

Rules:
- Reply in plain text, at most 120 words, in the customer's language.
- If the articles do not cover the question, say so and offer to open a ticket.
- Never mention which AI model or provider you are.

Write the prompt so that every model in the fallback list can follow it:
no provider-specific tags, tools or formatting features.