Checkpoint & Resume

Problem

Agents increasingly run for minutes, hours or days: migrating a database in batches, working through a research plan, or waiting for someone to approve a risky step. Processes do not live that long reliably. Deploys restart them, machines are replaced, and a pause for a human’s answer can outlast any request timeout. If the agent’s state lives only in memory, a crash at step 23 of 40 means starting again at step 1, paying for every model call twice and, worse, repeating side effects such as copying rows or sending emails.

How it works

The job starts a run with a stable run id. Worker A executes it one step at a time: the model picks the next action, the worker calls the database, and the result is added to the run’s state. After every step, before the next one begins, the whole state (messages, tool results and the step counter) is written to a durable checkpoint store.

In the example, worker A is killed during step 23. Nothing is lost except that step. Worker B, a fresh process, loads the step-22 checkpoint and continues from step 23 with the full history, as if nothing had happened. Because step 23 may have half-run before the crash, every side-effecting call carries an idempotency key built from the run id and step number, so redoing it cannot copy the same rows twice. The run finishes and reports to done.

The same mechanism handles deliberate pauses. When the run needs a person’s sign-off, it saves a checkpoint marked as waiting and the process simply exits. Days later, the approval loads the checkpoint on whichever worker is free.

Workflow engines such as Temporal take this further by recording every step’s result in an event history and replaying the code against it; agent frameworks such as LangGraph save a snapshot of the graph state after each step. Both give the same guarantee: completed work is never redone.

When to use

• Runs are long, from minutes to days, and will outlive a deploy or a machine.
• Steps are expensive or have side effects that must not repeat.
• The run waits on people or external events: approvals, replies, scheduled times.

When to avoid

• Calls are short and stateless. If the whole run takes ten seconds, retrying from scratch is simpler and cheaper.
• Nothing the agent does has side effects, and recomputing is cheap.
• You cannot make tool calls idempotent. Checkpoints alone will not stop a half-finished step from repeating its effect.

Trade-offs

Every step pays for a write to the store, and the state has to be serializable, which rules out holding open connections or in-memory objects across steps. In exchange, failures cost one step instead of the whole run, pauses become free, and the history of checkpoints doubles as an audit log you can inspect or replay. The real work is designing steps so they can safely run twice.

Failure modes

• Duplicate side effects: a step that ran just before the crash runs again on resume. Give every external write an idempotency key, or check before writing.
• Two workers, one run: a slow worker comes back after another has taken over, and both advance the run. Version checkpoints and reject stale writes.
• State bloat: the full message history is saved every step and grows without bound. Trim or summarize old messages, and store large tool outputs by reference.
• Code drift: a run paused for days resumes under a new code version that reads the old state differently. Version the state format and migrate it on load.

Implement the Checkpoint & Resume pattern in this codebase, using the LLM client and language we already use.

## Goal

Run a long, multi-step agent so that its full state is saved after every step and any process can resume the run from the last checkpoint, without repeating steps that already had side effects.

## How it works

1. Give every run a stable run id. Model the run as numbered steps, and keep its state (messages, tool results, current step, status) in one serializable object.
2. On start, load the latest checkpoint for the run id from a durable store (a database table, not process memory). If there is none, create the initial state.
3. For each step, call the model with the Step prompt, filling in the goal, the completed steps from the state and the latest tool result, and run the tool call it asks for.
4. Pass every side-effecting tool call an idempotency key built from the run id and step number, so a step that is retried after a crash does not apply twice.
5. After the step finishes, append its result to the state and write the new checkpoint (state plus step counter) atomically, with a version check, before starting the next step.
6. When the run must wait for a person or an event, save a checkpoint with status waiting and exit the process; a later request with the run id loads it and continues.

## Prompts

### Step prompt

```markdown
You are carrying out a long task one step at a time. The run may have been
paused or restarted; the completed steps below are the source of truth.

<goal>{{goal}}</goal>
<completed_steps>
{{completed_steps}}
</completed_steps>
<last_result>{{last_tool_result}}</last_result>

Decide the single next step and call one tool to do it.
Never redo a step listed as completed.
If the goal is fully done, reply with DONE and a one-paragraph summary.
```

## Stop rule

Stop when the model reports the goal is complete, or after a step cap (default 60). On a crash, nothing is returned: the run stays in the store and a supervisor restarts it on another worker. If a step fails 3 times, mark the run failed with its last checkpoint.

## Requirements

- Write checkpoints to durable storage with a version number per run, and reject a write whose version is stale so two workers cannot advance the same run.
- Make every external write idempotent with the run-and-step key, or check before writing; never rely on a step running exactly once.
- Keep old checkpoints for a retention window so a run can be inspected or replayed from any earlier step.
- Log each checkpoint write with the run id, step number, worker id and size.
- Add a test with a stub LLM and a fake tool where the process raises after step 2's checkpoint; restart with the same run id and assert the fake tool received steps 1 and 2 exactly once each and the run completes from step 3.
- Add a second test where the process raises after step 3's tool call but before its checkpoint; restart and assert the fake tool received key 'run/step-3' twice but applied it once (its row count is unchanged), and the run completes.

You are carrying out a long task one step at a time. The run may have been
paused or restarted; the completed steps below are the source of truth.

<goal>{{goal}}</goal>
<completed_steps>
{{completed_steps}}
</completed_steps>
<last_result>{{last_tool_result}}</last_result>

Decide the single next step and call one tool to do it.
Never redo a step listed as completed.
If the goal is fully done, reply with DONE and a one-paragraph summary.