Memory Stream

Problem

A long-lived agent sees far more than it can keep in its prompt. A simulated townsperson that has lived for two days has hundreds of memories: breakfast, a walk, a chat about the weather, and one invitation to a party on Valentine’s Day. When someone asks “what are you doing tonight?”, the agent needs the invitation, not breakfast. Keeping the latest memories forgets the invitation; similarity search alone may surface a chat about evening plans from last week. And none of the raw memories says what they add up to.

How it works

Every observation goes to a rater model that scores how important it is, from 1 (brushing teeth) to 10 (a break-up). The observation is appended to the memory stream, a single flat log with a timestamp, the score and an embedding. Isabella’s invitation to her party scores 7; breakfast scores 1.

When the agent needs to act, the current situation becomes a query, and the retriever scores every memory on three things. Recency decays exponentially with the hours since the memory was last used. Importance is the stored score. Relevance is the similarity between the memory and the query. The three are normalized and added, and the top few go to the agent. The invitation is older than breakfast, but it wins easily on importance and relevance, so Maria’s reply is “I’m going to Isabella’s party at 5. Want to come?”

The third part is reflection. The agent keeps a running total of the importance of new memories, and when it passes a threshold (150 in the paper), a reflector reads the 100 most recent memories and asks three questions about them, such as “how does Maria feel about Klaus?”. Each question goes through the retriever as a query over the whole stream, so the evidence can include old memories and earlier reflections. From those memories the reflector writes insights, such as “Maria wants to spend more time with Klaus”, back into the stream with pointers to the memories behind them. (The paper’s own example is Klaus concluding that he is dedicated to his research on gentrification.) Reflections are retrieved like any other memory, and later reflections can build on earlier ones, so the agent forms a picture of its world instead of a pile of events.

When to use

• Long-lived agents or characters need believable continuity across days or weeks of events.
• What matters is uneven: most events are trivial, a few are decisive, and the agent must tell them apart.
• You want the agent to form opinions and relationships from experience, not only recall facts.

When to avoid

• You need exact, structured recall, such as account numbers or the user’s stated preferences. Use a fact store or hierarchical memory.
• The agent lives for one task or one conversation; the context window is enough.
• Every model call matters for cost. Each observation costs a rating call, and reflections add more.

Trade-offs

Each observation costs one small rating call and an embedding, and retrieval scores the whole stream, which stays cheap until the stream grows very large. Reflection is the expensive part, but it runs only when enough important things have happened. In exchange, the agent’s behaviour stays consistent with what matters in its past, which the paper’s ablations show depends on all three parts: observation, planning and reflection.

Failure modes

• Miscalibrated importance: the rater scores everything 5, and the retriever falls back to recency and relevance alone. Give the rater anchored examples at both ends of the scale.
• Runaway reflection: reflections about reflections drift away from what happened. Keep evidence pointers and weight insights by the observations they cite.
• Stale favourites: retrieving a memory refreshes its recency, so a few memories keep winning. Decay from the original timestamp as well, or cap how often one memory is returned.
• Fabricated insight: the reflector infers something the memories do not support. Require citations and drop insights whose evidence is thin.

Implement the Memory Stream pattern in this codebase, using the LLM client and language we already use.

## Goal

Give a long-lived agent a memory that logs every observation with an importance score, retrieves the memories that best combine recency, importance and relevance, and periodically distils them into reflections.

## How it works

1. When the agent observes something, call the rater model with the Importance rating prompt to score its importance from 1 to 10, and append it to the stream with its text, timestamp, last-access time, score and embedding.
2. To retrieve for a query, score every memory as the sum of recency (exponential decay since last access, factor 0.995 per hour), importance and relevance (cosine similarity to the query embedding), each normalized to 0..1.
3. Return the top k memories (default 5), update their last-access time, and pass them to the agent model with the current situation.
4. Keep a running total of importance since the last reflection. When it exceeds the threshold (default 150), reflect.
5. To reflect, call the reflector model with the Reflection questions prompt on the 100 most recent memories. Run each of the 3 questions it returns through the retriever as a query over the whole stream, then call the reflector with the Reflection insights prompt on the retrieved memories, numbered, and append each insight to the stream with pointers to the memories it cites.

## Prompts

### Importance rating prompt

```markdown
On a scale of 1 to 10, where 1 is purely mundane (such as brushing teeth
or making the bed) and 10 is extremely poignant (such as a break-up or a
college acceptance), rate the likely poignancy of the following memory
for {{agent_name}}.

<memory>{{observation}}</memory>

Reply with a single integer.
```

### Reflection questions prompt

```markdown
Here are {{agent_name}}'s most recent memories:

<memories>{{recent_memories}}</memories>

Given only the information above, what are the 3 most salient high-level
questions we can answer about the subjects in the statements?
Return one question per line, with no numbering.
```

### Reflection insights prompt

```markdown
Statements about {{agent_name}}, retrieved for the questions
{{questions}}:

<statements>{{numbered_retrieved_memories}}</statements>

What 5 high-level insights can you infer from the above statements?
Cite the statements each one rests on, in the form:
insight (because of 1, 5, 3)
```

### Agent prompt with retrieved memories

```markdown
You are {{agent_name}}. {{agent_summary}}

It is {{time}}. {{situation}}

Relevant memories, most relevant first:
<memories>{{retrieved_memories}}</memories>

What do you do or say next? Stay consistent with your memories.
```

## Stop rule

Retrieval and reflection never end the agent's run; each retrieval stops at k memories, and each reflection at one questions call, one retrieval per question and one insights call. The importance total resets to 0 after every reflection.

## Requirements

- Store reflections in the same stream as observations, so later reflections can build on earlier ones.
- Make the three weights (default 1 each), the decay factor, k and the reflection threshold configurable.
- If the rater's reply is not an integer from 1 to 10, retry once, then store the memory with importance 5 and log it.
- Keep every reflection's evidence pointers, so an insight can be traced to the observations behind it.
- Add a test with a fake clock, a fake embedder and stub importance scores in which an old but important, relevant memory beats a recent trivial one; assert it ranks first, and that crossing the threshold triggers one questions call, exactly 3 retriever calls (one per question) and one insights call.

On a scale of 1 to 10, where 1 is purely mundane (such as brushing teeth
or making the bed) and 10 is extremely poignant (such as a break-up or a
college acceptance), rate the likely poignancy of the following memory
for {{agent_name}}.

<memory>{{observation}}</memory>

Reply with a single integer.

Here are {{agent_name}}'s most recent memories:

<memories>{{recent_memories}}</memories>

Given only the information above, what are the 3 most salient high-level
questions we can answer about the subjects in the statements?
Return one question per line, with no numbering.

Statements about {{agent_name}}, retrieved for the questions
{{questions}}:

<statements>{{numbered_retrieved_memories}}</statements>

What 5 high-level insights can you infer from the above statements?
Cite the statements each one rests on, in the form:
insight (because of 1, 5, 3)

You are {{agent_name}}. {{agent_summary}}

It is {{time}}. {{situation}}

Relevant memories, most relevant first:
<memories>{{retrieved_memories}}</memories>

What do you do or say next? Stay consistent with your memories.