Least-to-Most Prompting

Problem

Chain-of-thought works well on problems like its examples, and less well on problems that are harder than anything it was shown. “Amy takes 4 minutes to climb the slide and 1 minute to slide down. The slide closes in 15 minutes. How many times can she slide?” is easy for a person because they first work out how long one trip takes. A model reasoning in one pass can divide 15 by 4, or forget the slide down, because nothing makes it settle the easier question first.

How it works

Least-to-most splits the work into two stages. In the first, the question goes to the decompose call, which lists simpler subquestions in the order they have to be answered, ending with the original question. For the slide: “Q1: how long does each trip take? Q2: how many times can she slide before it closes?” The list is kept as the subquestions.

In the second stage, the solve call takes the subquestions one at a time. Each call sees the original problem, every earlier question with its answer, and the current subquestion. Q1 gets “4 + 1 = 5 minutes”, which is written in beside it. Q2 is then asked with that answer above it, and 15 ÷ 5 = 3 is easy. Because the last subquestion is the original one, its answer is the answer.

The name describes the order: least difficult first, most difficult last, with each answer making the next question easier. The model chooses the subquestions, not your code, but there are no tools and no replanning. If a decomposition is wrong, the solving stage follows it anyway. Successive prompting (Dua et al., 2022) is a close cousin that interleaves the two stages instead: it asks one simple question, answers it, then decides the next.

When to use

• The problems are harder or longer than your worked examples, such as longer word problems or longer instruction sequences.
• The problem splits into parts that build on each other, where a later part needs an earlier answer.
• Plain chain-of-thought gets the parts right but loses track of how they combine.

When to avoid

• The parts are independent of each other. Solve them in parallel instead, as in sectioning.
• The steps need facts from outside the model or need to act on something. Use self-ask with search, or plan-and-execute with tools.
• The problem takes one or two steps. The extra decompose call costs more than it saves.

Trade-offs

You make one more call than the number of subquestions, in sequence, so latency grows with the length of the decomposition. Each solve call also repeats the problem and all earlier answers, so prompts get longer as you go. In return, each call faces an easier question than the original, and the intermediate answers are explicit, so you can see exactly where a wrong result came from.

Failure modes

• A bad decomposition: the subquestions miss a step or are in the wrong order, and every later answer inherits the mistake. Give decompose examples close to your real problems, and log the list.
• An early wrong answer: a mistake in Q1 carries forward, because later calls trust the answers above them. Check intermediate answers that can be checked cheaply.
• Runaway lists: the model splits a simple problem into many tiny steps. Cap the number of subquestions.
• A missing final question: the list ends before the original question is asked. Append it if it is missing.

Implement the Least-to-Most Prompting pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer a hard question by having the model list easier subquestions first, then solve them in order, with every earlier question and answer in the prompt for the next.

## How it works

1. Call the model with the Decompose prompt and the question. Parse its reply into an ordered list of subquestions, the last of which is the original question.
2. Start an empty list of solved pairs.
3. For each subquestion in order, call the model with the Solve prompt, passing the original problem, every solved question and answer so far, and the current subquestion.
4. Parse the answer from the reply's 'Answer:' line and append the subquestion with its answer to the solved list.
5. When the last subquestion is answered, return its answer as the final answer, together with the solved list as the working.

## Prompts

### Decompose prompt

```markdown
Break the problem into simpler subquestions that, answered in order,
lead to the solution. Each subquestion may rely on the answers before it.
Make the last subquestion the original question.

{{#each examples}}
Problem: {{problem}}
Subquestions:
{{subquestions}}

{{/each}}
Problem: {{problem}}
Subquestions:
```

### Solve prompt

```markdown
{{problem}}

{{#each solved}}
Q: {{question}}
A: {{answer}}
{{/each}}
Q: {{subquestion}}

Use the answers above. Reason briefly, then end with one line
that starts with "Answer:".
```

## Stop rule

Stop when every subquestion is answered. If the decomposition, including the appended original question, has more than 6 subquestions, return no answer before solving, with status 'decomposition too long'. If an answer cannot be parsed, return no answer with the solved list so far.

## Requirements

- Make the subquestion cap a parameter (default 6) and check it before solving, not after.
- If the decomposition does not end with the original question, append the original question as the last subquestion.
- Pass the solved pairs to each solve call oldest first, in the same question and answer format the examples use.
- Log the decomposition and every solve call with its prompt, reply and token counts.
- Add a test with a stub LLM that decomposes into two subquestions, answers '5 minutes' then '3'; assert exactly three model calls and that the final answer is exactly '3'.

Break the problem into simpler subquestions that, answered in order,
lead to the solution. Each subquestion may rely on the answers before it.
Make the last subquestion the original question.

{{#each examples}}
Problem: {{problem}}
Subquestions:
{{subquestions}}

{{/each}}
Problem: {{problem}}
Subquestions:

{{problem}}

{{#each solved}}
Q: {{question}}
A: {{answer}}
{{/each}}
Q: {{subquestion}}

Use the answers above. Reason briefly, then end with one line
that starts with "Answer:".