Prefix cache (default)

All steps

Parts of this pattern

Stable prefix Memory: The part of the prompt that is identical on every call: tool definitions, the system prompt, and a 100-page product manual (about 80,000 tokens).
Question Input / output: The part that changes: one customer's question, a few dozen tokens placed after the prefix.
Model LLM call: Processes the prompt. On a cache hit it skips the prefix and only processes the new question, so it starts answering sooner.
Prefix cache Memory: The provider's store of already-processed prefixes. Writing costs a little extra once; reading costs a fraction of normal input price.
Answer Input / output: The model's answer. It is the same whether or not the prefix came from the cache; only cost and speed differ.
Stable prefix → Modelprefix: Every request starts with the same prefix, up to the cache breakpoint.[tools: lookup_order, open_ticket] [system: You are the support assistant for Routerly...] [manual v12: 80,214 tokens] <cache breakpoint>
Question → Model: The question is appended after the breakpoint and always processed in full.How do I set up a guest network?
Model → Prefix cachewrite 1.25×: On a miss, the processed prefix is written to the cache, at a small premium on those input tokens.usage: cache_creation_input_tokens=80,214, cache_read_input_tokens=0, input_tokens=14
Prefix cache → Modelread 0.1×: On a hit, the stored prefix is reused at about a tenth of the normal input price on standard models, and the answer starts sooner.usage: cache_creation_input_tokens=0, cache_read_input_tokens=80,214, input_tokens=17
Model → Answer: The answer is returned the same way on a hit or a miss.Open the app, go to Wi-Fi > Guest network (manual section 4.3), switch it on and set a password.

Exact response cache

Hash the whole request; if the same request was answered before, return the stored answer without calling the model.

All steps

Parts of this pattern

Request Input / output: A full model request: model, prompt version, parameters and messages.
Hash lookup Router: Plain code: normalizes the request and hashes it, so identical requests always produce the same key.
Answer store Memory: A key-value store mapping request hashes to finished answers, each with an expiry time.
Model LLM call: Called only on a miss, when no identical request has been answered recently.
Answer Input / output: The answer, either from the store in milliseconds or freshly generated.
Request → Hash lookup: The request is normalized and hashed before anything else happens.sha256(model=small-v3, prompt=faq-v7, temperature=0, 'What are your opening hours?') = 9f3c...
Hash lookup → Answer store: The lookup asks the store for an answer under that exact hash.GET answer:9f3c...
Answer store → Answerhit: On a hit, the stored answer is returned and the model is never called.found (stored 08:12, expires in 19 h): 'We are open 9:00 to 17:30, Monday to Friday.' (3 ms)
Answer store → Modelmiss: On a miss, the request goes to the model as usual.No entry for 4a71...; calling the model.
Model → Answer storeput: The fresh answer is stored under the request's hash for the next identical request.SET answer:4a71... 'Returns are free within 30 days...' EX 86400
Model → Answer: The freshly generated answer is returned to the caller.Returns are free within 30 days; print the label from your order page.

Semantic cache

Embed the question and return a stored answer when an earlier question was close enough in meaning, even if worded differently.

All steps

Reworded question hits

New question misses

Parts of this pattern

Question Input / output: A customer question, possibly a rewording of one asked before.
Embedder Tool: Turns the question into a vector so that questions with the same meaning land close together.
Vector cache Memory: Earlier questions as vectors, each stored with the answer the model gave.
Similarity Evaluator: Plain code: compares the new question with its nearest stored neighbour and decides hit or miss by a threshold.
Model LLM call: Called only on a miss; its answer is stored for future near-duplicates.
Answer Input / output: Either a stored answer to a near-identical question, or a fresh one.
Question → Embedder: The question is embedded.how can i turn on a wifi network for visitors
Embedder → Vector cache: The vector is used to find the most similar earlier question.nearest(k=1, vector=[0.12, -0.48, ...])
Vector cache → Similarity: The nearest stored question and its similarity score go to the threshold check.'How do I set up a guest network?' similarity 0.95
Similarity → Answerhit: Above the threshold, the stored answer is returned without calling the model.Open the app, go to Wi-Fi > Guest network (manual section 4.3), switch it on and set a password.
Similarity → Modelmiss: Below the threshold, the question goes to the model.Nearest is 'How do I change my Wi-Fi password?' at 0.81 < 0.92; calling the model.
Model → Vector cache: The new question's vector and the model's answer are added to the cache.insert(vector=[...], question='Can I limit guest bandwidth?', answer='Yes: Wi-Fi > Guest network > Limits...')
Model → Answer: The fresh answer is returned.Yes: go to Wi-Fi > Guest network > Limits and set a maximum speed per device.

← → neighboursesc atlas

Efficiency & Cost

Caching & Reuse

Also known as Prompt caching · Prefix caching · KV-cache reuse · Semantic cache · Response cache

Avoid recomputing. Reuse the processed prefix of a stable prompt, or return a stored answer when a request matches an earlier one.

Prefix cache: No single inventor · see source

No single inventor. Prompt Cache (MLSys 2024) was among the first papers dedicated to reusing attention states across prompts; vLLM's PagedAttention (2023) had already shared KV cache across requests with a common prefix. Anthropic popularized API prompt caching in 2024.

Exact response cache: No single inventor · see source

Memoization long predates LLMs. GPTCache packaged exact-match and semantic caches for LLM calls; the year is when its GitHub repository was created (March 2023).

Semantic cache: No single inventor · see source

GPTCache describes itself as a 'semantic cache for LLMs': an embedding generator, a vector store and a similarity evaluator. The year is when its GitHub repository was created (March 2023).

Measured over one run

Prefix cache

Nodes: 5. Boxes in the diagram: models, tools, stores, people and plain code.
LLM calls: 2. Model invocations in the first scenario: each time a model or agent node starts thinking, plus each branch it decides.
Tool calls: 0. Messages sent to a tool node in the first scenario.
Steps: 3. Steps in the first scenario, with repeated steps counted every time they run.
Scenarios: 1. Distinct example runs you can switch between on the diagram.
Loop: no. Whether the flow cycles back on itself until a condition is met.
Parallel: yes. Whether two or more messages are in flight at the same time.
Branch: no. Whether the flow takes one path out of several, chosen at run time.
Human in the loop: no. Whether a person appears as a node, to approve, correct or answer.
Autonomy: low. How much the model decides the control flow itself, from fixed code paths (low) to open-ended (high).
Cost: low. Relative token spend per request.
Latency: low. Relative wall-clock time per request.

Exact response cache

Nodes: 5. Boxes in the diagram: models, tools, stores, people and plain code.
LLM calls: 1. Model invocations in the first scenario: each time a model or agent node starts thinking, plus each branch it decides.
Tool calls: 0. Messages sent to a tool node in the first scenario.
Steps: 2. Steps in the first scenario, with repeated steps counted every time they run.
Scenarios: 1. Distinct example runs you can switch between on the diagram.
Loop: yes. Whether the flow cycles back on itself until a condition is met.
Parallel: yes. Whether two or more messages are in flight at the same time.
Branch: yes. Whether the flow takes one path out of several, chosen at run time.
Human in the loop: no. Whether a person appears as a node, to approve, correct or answer.
Autonomy: low. How much the model decides the control flow itself, from fixed code paths (low) to open-ended (high).
Cost: low. Relative token spend per request.
Latency: low. Relative wall-clock time per request.

Semantic cache

Nodes: 6. Boxes in the diagram: models, tools, stores, people and plain code.
LLM calls: 0. Model invocations in the first scenario: each time a model or agent node starts thinking, plus each branch it decides.
Tool calls: 1. Messages sent to a tool node in the first scenario.
Steps: 2. Steps in the first scenario, with repeated steps counted every time they run.
Scenarios: 2. Distinct example runs you can switch between on the diagram.
Loop: yes. Whether the flow cycles back on itself until a condition is met.
Parallel: no. Whether two or more messages are in flight at the same time.
Branch: yes. Whether the flow takes one path out of several, chosen at run time.
Human in the loop: no. Whether a person appears as a node, to approve, correct or answer.
Autonomy: low. How much the model decides the control flow itself, from fixed code paths (low) to open-ended (high).
Cost: low. Relative token spend per request.
Latency: low. Relative wall-clock time per request.

Counted from the first scenario's timeline when the site is built.

Problem

Many LLM calls repeat work. A support bot grounded in a 100-page product manual sends the same 80,000 tokens of manual, system prompt and tool definitions with every question, and the model processes them from scratch each time, so most of the bill and most of the wait go on text that never changes. Meanwhile, hundreds of customers ask the same few questions in slightly different words, and each gets a freshly generated answer that is nearly identical to the last.

How it works

Caching avoids recomputing what has already been computed. It comes in two forms: reusing part of the work inside a call, or skipping the call entirely.

The default tab shows prefix caching, which most model providers now offer. The prompt is split in two. The stable prefix (tools, system prompt and manual) is identical on every call and goes first. The question, which changes every time, goes last.

On the first call the model processes the whole prompt and the provider writes the processed prefix to a prefix cache, at a small premium (1.25× the input price, in one provider’s pricing). When a second customer asks something a minute later, the provider recognizes the identical prefix and reads it from the cache at about a tenth of the normal input price. Only the new question is processed, so the answer also starts sooner. Anthropic reports cost reductions of up to 90% and latency reductions of up to 85% for long prompts.

The rule that makes it work is strict: the cache matches on exact prefixes. One changed byte early in the prompt, such as a timestamp in the system prompt, invalidates everything after it.

Variants

Exact response cache: hash the whole request and return the stored answer if the identical request was answered before. No model call at all, but only exact repeats hit.
Semantic cache: embed the question and return a stored answer if an earlier question was close enough in meaning. It catches rewordings, at the risk of serving an answer to a question that only looked similar.

When to use

A large, unchanging prefix (system prompt, tool definitions, documents, few-shot examples) repeats across many calls.
Calls arrive often enough to keep the cache warm within its lifetime.
Many users ask near-identical questions whose answers do not depend on who is asking (response caches).

When to avoid

The prefix changes on every call. Fix that first: move anything variable to the end.
Answers must be fresh, personal or deliberately varied. A response cache would serve stale or someone else’s answers.
Traffic is so sparse that the cache expires between calls; you would pay the write premium without ever reading.

Trade-offs

Prefix caching is close to free: the answer is the same either way, and the only cost is the write premium on misses and the discipline of keeping the prefix stable. Response caches save more, a whole call per hit, but trade correctness for it. An exact cache is safe but rarely hits on free-text input; a semantic cache hits often but will sometimes return the answer to a different question. The similarity threshold sets that balance.

Failure modes

Silent cache busting: a timestamp, a random id or a reordered tool list in the prefix makes every call a miss. Watch the cache-read share and alert when it drops.
Wrong semantic hit: “How do I cancel my order?” matches “How do I cancel my subscription?” and gets the wrong answer. Tune the threshold on labelled pairs and sample hits for review.
Leaking across users: a cached answer containing one customer’s details is served to another. Never response-cache personal content.
Replayed actions: in an agent, a cached turn that called a tool replays an old decision against new state. Cache only final text answers, never turns with tool calls or results from side-effecting tools.
Stale answers: the manual changes but the cache still serves old answers. Put the document version in the cache key and purge on release.

Build it with your AI

Prefix cache

Paste this into Claude Code, Cursor, ChatGPT or any AI coding assistant.

Implement the Caching & Reuse pattern in this codebase, using the LLM client and language we already use.

## Goal

Cut the cost and latency of repeated calls by putting everything that stays the same at the front of the prompt and marking it cacheable, so later calls reuse the processed prefix and pay only for the new part.

## How it works

1. Split the prompt into a stable prefix (tool definitions, the Cache-friendly system prompt, reference documents) and a variable suffix (the user's turn and anything that changes per call).
2. Order the request so the stable parts come first and the variable parts last; move timestamps, user names and request ids out of the prefix.
3. Mark the end of the prefix with your provider's cache breakpoint, or rely on its automatic prefix caching, and choose a cache lifetime that covers the gap between calls.
4. Send requests as usual. Read the usage fields on each response to see how many input tokens were written to the cache and how many were read from it.
5. Keep the prefix byte-for-byte identical between calls: same tool order, same whitespace, same document version.

## Prompts

### Cache-friendly system prompt

```markdown
You are the support assistant for {{product}}. Answer only from the manual below.
Quote the section number for every instruction you give.
If the manual does not cover the question, say so and suggest contacting support.

<manual version="{{manual_version}}">
{{manual_text}}
</manual>

(Everything above this line is identical for every user and is cached.
The customer's question follows in the next message.)
```

## Stop rule

There is no loop: each call either hits the cache or rebuilds it. If the cache-read share drops below a set level (say 80% of eligible calls), alert, because something has started changing the prefix.

## Requirements

- Build the prefix in one function that is deterministic given the configuration, so two calls cannot serialize it differently.
- Keep anything per-user or per-request out of the prefix, and test that two users' requests share the same prefix bytes.
- Log cache-write and cache-read token counts per call, and chart the hit rate.
- Make sure the prefix is above the provider's minimum cacheable length; below it, caching silently does nothing.
- Add a test with a stub client that records requests; send two questions and assert the two requests' prefixes are byte-identical and differ only after the cache breakpoint.

Exact response cache

Paste this into Claude Code, Cursor, ChatGPT or any AI coding assistant.

Implement the Caching & Reuse pattern (Exact response cache variant) in this codebase, using the LLM client and language we already use.

## Goal

Return a stored answer instantly when a request is byte-for-byte identical to one already answered, and call the model only on a miss.

## How it works

1. Normalize the request (model, system prompt from the Cache-friendly system prompt, messages, temperature) into a canonical form and hash it.
2. Look the hash up in a key-value store. On a hit that has not expired, return the stored answer.
3. On a miss, call the model and return its answer; store it under the hash with an expiry time only if it is a final text answer with no tool use.
4. Bypass the cache for requests that must be fresh or random, such as those with temperature above 0 or tools that read live data.

## Prompts

### Cache-friendly system prompt

```markdown
You are the support assistant for {{product}}. Answer only from the manual below.
Quote the section number for every instruction you give.
If the manual does not cover the question, say so and suggest contacting support.

<manual version="{{manual_version}}">
{{manual_text}}
</manual>

(Everything above this line is identical for every user and is cached.
The customer's question follows in the next message.)
```

## Stop rule

Each request is either a hit (no model call) or a miss (one call, then a write). If the store is unreachable, call the model and skip the write.

## Requirements

- Include everything that changes the answer in the key: model id, prompt version, parameters and documents.
- Set an expiry (default 24 hours) and a way to purge all entries for a prompt version.
- Log hits and misses with the key, and report the hit rate.
- Cache only final text answers from calls with no tool use. Never cache a response that contains a tool call or was produced from live tool results, and never cache the result of a tool with side effects.
- Add a test with a stub model and an in-memory store: send the same request twice and assert one model call, and two identical answers.

Semantic cache

Paste this into Claude Code, Cursor, ChatGPT or any AI coding assistant.

Implement the Caching & Reuse pattern (Semantic cache variant) in this codebase, using the LLM client and language we already use.

## Goal

Answer near-duplicate questions from a store of earlier answers, matched by embedding similarity, and call the model only when no stored question is close enough.

## How it works

1. Embed the incoming question with an embedding model.
2. Search a vector store of earlier questions for the nearest neighbour, and get its similarity score.
3. If the similarity is above a threshold (start around 0.92 and tune it), return the stored answer.
4. Otherwise call the model with the Cache-friendly system prompt and the question and return its answer; store the question's embedding with it only if it is a final text answer with no tool use.
5. Scope the cache: keep separate stores, or a filter, per prompt version, language and anything else that changes the right answer.
6. Bypass the cache entirely for requests that must be fresh, that are personal, or whose answer depends on tools or live data.

## Prompts

### Cache-friendly system prompt

```markdown
You are the support assistant for {{product}}. Answer only from the manual below.
Quote the section number for every instruction you give.
If the manual does not cover the question, say so and suggest contacting support.

<manual version="{{manual_version}}">
{{manual_text}}
</manual>

(Everything above this line is identical for every user and is cached.
The customer's question follows in the next message.)
```

## Stop rule

Each request is either a hit (no model call) or a miss (one call, then a write). Entries expire after a set time so answers do not go stale.

## Requirements

- Make the similarity threshold configurable, and tune it on labelled pairs of questions that should and should not share an answer.
- Never serve cached answers across users for anything personal; exclude questions that mention account details.
- Log every hit with both questions and the score, and sample hits for review.
- Cache only final text answers from calls with no tool use. Never cache a response that contains a tool call or was produced from live tool results, and never cache the result of a tool with side effects.
- Add a test with a stub embedder returning fixed vectors: a question at similarity 0.95 to a stored one must return the stored answer with zero model calls; one at 0.80 must call the model once.

Prompt templates

Prefix cache · Exact response cache · Semantic cache

You are the support assistant for {{product}}. Answer only from the manual below.
Quote the section number for every instruction you give.
If the manual does not cover the question, say so and suggest contacting support.

<manual version="{{manual_version}}">
{{manual_text}}
</manual>

(Everything above this line is identical for every user and is cached.
The customer's question follows in the next message.)

Related patterns

Composes well with:

Sources

From the Agentic Atlas by Joran Kikke · Edit on GitHub · Suggest a correction