LLMCompiler

Joiner with replan

Implement the LLMCompiler pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer a question by having a planner emit a dependency graph of tool calls, running every call whose inputs are ready in parallel, and letting a joiner answer or ask for a replan.

## How it works

1. Call the planner model with the Planner prompt, passing the question, the tool descriptions and, on a replan, the joiner's reason.
2. Parse each numbered line into a task (id, tool, arguments) and collect the $N references in its arguments as its dependencies; stop parsing at end_of_plan.
3. Run a scheduler in plain code: start every task whose dependencies all have results, concurrently, with each $N replaced by task N's result.
4. As each task finishes, store its result or its error under its id and start any task that has just become ready, until every task has finished or been skipped.
5. Call the joiner model with the Joiner prompt, passing the question and every task result, and parse its JSON reply.
6. If the reply has answer, return it. If it has replan, go back to the planner with the reason and the results so far, so it plans only what is missing.

## Prompts

### Planner prompt

```markdown
Break the question into tool calls and list them as a dependency graph.
Give every task a number. If a task needs the output of an earlier task,
write its number as $1, $2 and so on in the arguments. Tasks that do not
reference each other will run at the same time, so do not add a
dependency unless it is really needed.

Tools:
{{tool_descriptions}}

Format:
1. tool_name(arguments)
2. tool_name(arguments)
3. tool_name(arguments using $1)
end_of_plan

{{#if context}}
An earlier plan did not answer the question:
{{context}}
Plan only what is still missing.
{{/if}}

Question: {{question}}
```

### Joiner prompt

```markdown
Decide whether the results below answer the question.

Question: {{question}}
Results: {{results}}

Respond in JSON, either
{"answer": "..."}
or, if something is missing or a task failed,
{"replan": true, "reason": "what is missing and why"}
```

## Stop rule

Stop when the joiner returns an answer, or after 2 replans. On the cap, return the best partial results with the joiner's last reason and a status saying the question was not fully answered.

## Requirements

- Optionally parse the planner's output as it streams, and start each task as soon as its line is complete and its dependencies are met.
- Make the replan cap (default 2), the tasks per plan (default 10) and the concurrent tool calls (default 5) parameters, and give every task a timeout (default 30 seconds).
- Reject a plan in which a $N points to a missing or later task, or the dependencies form a cycle; retry the planner once with the error, then stop with a failed status and the validation error.
- When a task errors, store the error as its result, skip every task that depends on it, and let the joiner see both.
- If the joiner's reply is not valid JSON, retry it once, then treat it as a replan with the raw reply as the reason.
- Add a test with a stub LLM whose planner returns 3 tasks where task 3 uses $1 and whose joiner returns an answer, and fake 100 ms tools; assert 1 planner call, 1 joiner call, tasks 1 and 2 overlap, and task 3 starts after task 1 ends with its result.

Native parallel calls

Implement the LLMCompiler pattern (Native parallel calls variant) in this codebase, using the LLM client and language we already use.

## Goal

Answer a question with one tool-using model that may request several tool calls per turn, running each turn's calls concurrently and feeding all their results back in the next turn.

## How it works

1. Send the question to the model with the System prompt and the tool definitions, with multiple tool calls per turn enabled.
2. If the reply contains tool calls, run all of them concurrently, each with a timeout, and wait for every one to finish.
3. Append the model's turn and one result message per call, matched to its call id, to the conversation; send an error as that call's result instead of dropping it.
4. Call the model again with the whole conversation; it may request further calls that need the earlier results.
5. When a reply contains no tool calls, return its text as the answer.

## Prompts

### System prompt

```markdown
You can call tools. When you need several pieces of information that
do not depend on each other, request all of those tool calls in the
same turn so they run together. Wait for a result only when a later
call needs it as an argument.

When you have everything you need, answer without calling a tool.
```

## Stop rule

Stop when the model replies without tool calls, or after 5 turns. On the cap, return the last reply and the tool results gathered so far, with a status saying the turn budget ran out.

## Requirements

- Make the turn cap (default 5), the calls per turn (default 8) and the per-call timeout (default 30 seconds) parameters; beyond 8 calls, run the first 8 and return a call-limit-exceeded error as each extra call's result.
- Run one turn's calls concurrently, but never start a call from the next turn until every result of the current turn has been returned to the model.
- If a tool call's arguments are not valid JSON for its schema, return the validation error as that call's result and let the model correct it.
- Log each turn with its calls, their start and end times, and their results.
- Add a test with a stub model that requests flights and weather in turn 1, hotel in turn 2, then answers; assert 3 model calls and that flights and weather overlapped.