CodeAct

Problem

“Which 3 of our 40 SaaS vendors raised prices the most this year?” needs the vendor list, then two invoice lookups per vendor, then a comparison. With JSON tool calls, that is up to 81 model turns, and every invoice row passes through the context, whether or not the model needs it. The work is a loop with arithmetic, which is exactly what code is good at and what a turn-by-turn tool loop is bad at.

How it works

CodeAct, from Wang et al., changes what an action is. Instead of emitting one JSON tool call, the model writes a snippet of Python. The snippet runs in an interpreter inside a sandbox, where each tool is an ordinary function. The model can call several tools, loop over results, filter and sort, all in one action.

In the example, the model writes one script. It calls list_vendors() once, loops over the 40 vendors, calls get_invoices() for this year and last, computes each rise and prints the top three. Those 81 tool calls happen inside the sandbox. Only what the code prints, three names and percentages, returns to the model as the observation, and it writes the answer.

Errors are observations too. If the script reads a field the invoice rows do not have, the KeyError and its traceback come back to the model. The model reads it, patches the snippet and runs it again. Because the interpreter keeps its variables between steps, the patched snippet can reuse the vendor list it already fetched.

This is still an agent loop, act and observe, like ReAct. What changed is the size of one action: a program instead of a single call.

When to use

• One step needs several tool calls composed with logic: loops, filters, joins, arithmetic.
• The data is large and should stay out of the context: let code reduce it and print a summary.
• The model is good at code in your domain, and your tools can be wrapped as functions.

When to avoid

• You cannot sandbox code execution. Running model-written code with real credentials and no limits is a security hole.
• The tool set is tiny and calls are single-shot. Plain JSON tool calls are simpler to log, review and restrict.
• Every action needs approval or a precise audit. A script that makes 80 calls is harder to review than 80 separate, typed calls.

Trade-offs

CodeAct trades auditability for leverage. One turn does the work of dozens, so cost and latency drop and the context stays small, because intermediate data lives in variables. In exchange you must run a sandbox, with resource limits and no stray network or file access, and permissions move from “which tool may be called” to “what this code may do”. Logs need to capture both the code and the tool calls made from inside it, or you lose track of what the agent actually did.

Failure modes

• Unsandboxed execution: the code reaches the file system, network or credentials it should not. Expose tools only as functions and block everything else.
• Printing everything: the model prints a whole dataset and floods its own context. Truncate observations and tell the model to print summaries.
• Silent wrong logic: the code runs but computes the wrong thing, such as comparing the wrong months. Ask the model to print a small sample of intermediate values before the final result.
• Fix loops: the model keeps patching the same error. Cap executions and detect repeated tracebacks.

Implement the CodeAct pattern in this codebase, using the LLM client and language we already use.

## Goal

Let a model answer a task by writing code that runs in a sandboxed interpreter, where the available tools are plain functions, looping on the printed output or error until it gives a final answer.

## How it works

1. Wrap each tool as a function inside the sandbox, with a docstring giving its arguments and return value, and list those signatures in the System prompt.
2. Call the model with the system prompt, the task and the transcript so far.
3. If the reply contains a code block, run it in the sandbox. Keep one interpreter session per task, so variables defined in one step are still there in the next.
4. Capture stdout and any error with its traceback, truncate it to a size limit, and append it to the transcript in the format of the Observation format.
5. Call the model again with the longer transcript; it can fix an error, keep working with the variables it made, or finish.
6. If the reply contains a final answer and no code block, return the answer.

## Prompts

### System prompt

```markdown
You solve tasks by writing Python, which runs in a sandbox where these
functions are already defined:

{{tool_signatures}}

To act, reply with one fenced python code block. Use print() for anything
you need to see: only printed output comes back to you, truncated to
{{max_chars}} characters. Variables persist between your code blocks.

Keep large data in variables and print only summaries.
If your code raises an error, read the traceback and fix the code.

When you have the answer, reply with "Final answer:" followed by it, and no code block.
```

### Observation format

```markdown
<execution>
<stdout>{{stdout}}</stdout>
{{#if error}}<error>{{traceback}}</error>{{/if}}
{{#if truncated}}(output truncated to {{max_chars}} characters){{/if}}
</execution>
```

## Stop rule

Stop when the model gives a final answer, or after 10 code executions. On the cap, return the last output with a status saying the agent did not finish.

## Requirements

- Run the code in a sandbox with no file system or network access except through the tool functions, and with CPU, memory and wall-clock limits on each execution.
- Truncate each observation (default 2,000 characters) and tell the model it was truncated, so large data stays in variables and out of the context.
- Return errors to the model as observations, with the traceback, rather than ending the run.
- Log every code block, its output, and the tool calls made from inside it.
- Add a test with a stub LLM whose first code block raises a KeyError and whose second prints the fixed result, then answers; assert two executions, that the second observation holds the printed result, and that the answer is returned.

You solve tasks by writing Python, which runs in a sandbox where these
functions are already defined:

{{tool_signatures}}

To act, reply with one fenced python code block. Use print() for anything
you need to see: only printed output comes back to you, truncated to
{{max_chars}} characters. Variables persist between your code blocks.

Keep large data in variables and print only summaries.
If your code raises an error, read the traceback and fix the code.

When you have the answer, reply with "Final answer:" followed by it, and no code block.

<execution>
<stdout>{{stdout}}</stdout>
{{#if error}}<error>{{traceback}}</error>{{/if}}
{{#if truncated}}(output truncated to {{max_chars}} characters){{/if}}
</execution>