Agentic Atlas

With search (default)

Self-Ask diagramQuestion, Model, Search, Answer. Step captions follow the diagram.QuestionModelFollow up / AnswerSearchAnswerSelf-Ask diagramQuestion, Model, Search, Answer. Step captions follow the diagram.QuestionModelFollow up/ AnswerSearchAnswer

All steps
Parts of this pattern
Question Input / output
A question that joins two facts: 'Who was US president when the Eiffel Tower was completed?'
Model LLM call
Writes one 'Follow up:' question at a time, reads the intermediate answer, and decides whether to ask another.
Search Tool
A search engine. It answers each follow-up question with a short snippet, the 'Intermediate answer'.
Answer Input / output
The text after 'So the final answer is:', which your code returns: 'Benjamin Harrison'.
Question → Model
The question goes to the model inside the self-ask scaffold, with one worked example.Question: Who was US president when the Eiffel Tower was completed? Are follow up questions needed here:
Model → SearchFollow up
Generation stops at 'Intermediate answer:', and your code sends the follow-up question to search.Yes. Follow up: When was the Eiffel Tower completed?
Search → ModelIntermediate
The search result is added to the transcript as the intermediate answer, and the model is called again.Intermediate answer: The Eiffel Tower was completed in March 1889.
Model → Answerfinal answer
Once it has the facts it needs, the model writes the final answer line instead of another follow-up.So the final answer is: Benjamin Harrison

← → neighboursesc atlas

Reasoning & Prompting

Self-Ask

Also known as Self-ask with search · Follow-up questioning

The model asks itself follow-up questions, answers each one (optionally with search), then composes the final answer.

Introduced by Press et al. (2022) · University of Washington, MosaicML and others

Published in Findings of EMNLP 2023. The paper named the 'compositionality gap': models that know two facts often fail to combine them, and asking the sub-questions explicitly narrows it.

Measured over one run

With search

Nodes
4. Boxes in the diagram: models, tools, stores, people and plain code.
LLM calls
6. Model invocations in the first scenario: each time a model or agent node starts thinking, plus each branch it decides.
Tool calls
2. Messages sent to a tool node in the first scenario.
Steps
4. Steps in the first scenario, with repeated steps counted every time they run.
Scenarios
1. Distinct example runs you can switch between on the diagram.
Loop
yes. Whether the flow cycles back on itself until a condition is met.
Parallel
no. Whether two or more messages are in flight at the same time.
Branch
yes. Whether the flow takes one path out of several, chosen at run time.
Human in the loop
no. Whether a person appears as a node, to approve, correct or answer.
Autonomy
medium. How much the model decides the control flow itself, from fixed code paths (low) to open-ended (high).
Cost
medium. Relative token spend per request.
Latency
medium. Relative wall-clock time per request.

Without search

Nodes
4. Boxes in the diagram: models, tools, stores, people and plain code.
LLM calls
5. Model invocations in the first scenario: each time a model or agent node starts thinking, plus each branch it decides.
Tool calls
0. Messages sent to a tool node in the first scenario.
Steps
4. Steps in the first scenario, with repeated steps counted every time they run.
Scenarios
1. Distinct example runs you can switch between on the diagram.
Loop
yes. Whether the flow cycles back on itself until a condition is met.
Parallel
no. Whether two or more messages are in flight at the same time.
Branch
yes. Whether the flow takes one path out of several, chosen at run time.
Human in the loop
no. Whether a person appears as a node, to approve, correct or answer.
Autonomy
medium. How much the model decides the control flow itself, from fixed code paths (low) to open-ended (high).
Cost
medium. Relative token spend per request.
Latency
medium. Relative wall-clock time per request.

Counted from the first scenario's timeline when the site is built.

Problem

“Who was US president when the Eiffel Tower was completed?” A model may well know that the tower was finished in 1889, and that Benjamin Harrison took office that year, and still answer wrongly. The question asks it to combine two facts it has never seen together, and it tends to guess at the combination. Press et al. called this the compositionality gap: models get better at single facts faster than they get better at joining them.

How it works

Self-ask makes the joining explicit. The question goes to the model inside a fixed scaffold, shown with one worked example:

Question: ...
Are follow up questions needed here: Yes.
Follow up: ...
Intermediate answer: ...
So the final answer is: ...

The model starts by writing a follow-up question: “When was the Eiffel Tower completed?” Your code stops generation as soon as the model reaches “Intermediate answer:”, sends the follow-up to search, and pastes the result (“1889”) into the transcript as the intermediate answer. Then it calls the model again.

Now the model can see 1889, so its next follow-up is “Who was US president in 1889?” Search returns Benjamin Harrison. With both facts written down, the model writes “So the final answer is: Benjamin Harrison” instead of another follow-up, and your code returns that as the answer.

Each follow-up is decided after the previous answer is in, which is the difference from least-to-most, where all the subquestions are written first. And there is exactly one tool, used in exactly one way, which is the difference from ReAct.

The search engine is optional. In the “Without search” tab the model answers its own follow-ups from memory, in a single call. The scaffold alone already narrows the gap; search narrows it further and lets the answers be newer than the model’s training data.

When to use

  • Multi-hop factual questions: a date, then who held a role at that date; a company, then its founder’s birthplace.
  • The individual facts are easy to look up, but the combination is not written down anywhere.
  • You want each hop visible, so a wrong answer can be traced to the follow-up that went wrong.

When to avoid

  • The task needs several kinds of tool or real actions. Use ReAct.
  • The question is a single fact. One search, or one call, is enough.
  • The parts are known in advance and do not depend on each other. Ask them in parallel instead.

Trade-offs

With search, each hop adds a model call and a search call, in sequence, so latency grows with the number of hops. The fixed scaffold is easy to parse and debug, but it also limits the pattern: it can only ask questions, it cannot compute, filter or act. Without search, self-ask costs one call, like chain-of-thought, and inherits the model’s own mistakes about facts.

Failure modes

  • Self-written answers: the model writes its own intermediate answer instead of waiting for search. Use “Intermediate answer:” as a stop sequence.
  • Bad snippets: search returns a related but wrong fact, and the model builds on it. Keep the snippets short and log every query with its result.
  • Endless follow-ups: the model keeps asking, or asks the same question again. Cap the hops and stop on a repeat.
  • Skipping the scaffold: the model answers “No” to the follow-up question and guesses. Include an example where follow-ups are needed, close to your real questions.

Build it with your AI

With search

Paste this into Claude Code, Cursor, ChatGPT or any AI coding assistant.

Implement the Self-Ask pattern in this codebase, using the LLM client and language we already use.

## Goal

Answer a multi-hop factual question by letting the model ask one follow-up question at a time, answering each with a search call, until it writes its final answer.

## How it works

1. Start a transcript with the Self-ask prompt and the user's question, ending with 'Are follow up questions needed here:'.
2. Call the model with the transcript, and stop generation at the text 'Intermediate answer:'.
3. If the new text contains 'So the final answer is:', parse what follows it and return that as the answer.
4. Otherwise take the question after the last 'Follow up:', send it to the search tool, and keep the top answer snippet.
5. Append the model's text, then 'Intermediate answer: ' and the search result, to the transcript, and call the model again.

## Prompts

### Self-ask prompt

```markdown
Answer the question. If it needs facts you should look up first, ask
follow-up questions, one per line, and wait for each intermediate answer.

Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali

Question: {{question}}
Are follow up questions needed here:
```

## Stop rule

Stop when the model writes 'So the final answer is:', or after 4 follow-up questions. On the cap, return no answer with the transcript so far and a status saying the follow-up limit was hit.

## Requirements

- Use the stop sequence 'Intermediate answer:' so the model never writes its own answer to a follow-up.
- Cap follow-ups with a parameter (default 4), and stop early if the same follow-up question is asked twice.
- Trim each search result to one short snippet. If the search returns nothing, or fails or times out after one retry, use 'No answer found' as the intermediate answer and log the error.
- Log every follow-up question with its search query and the snippet that came back.
- Add a test with a stub LLM and a fake search: two follow-ups get '1889' and 'Benjamin Harrison'; assert exactly two searches and the answer exactly 'Benjamin Harrison'.

Without search

Paste this into Claude Code, Cursor, ChatGPT or any AI coding assistant.

Implement the Self-Ask pattern (Without search variant) in this codebase, using the LLM client and language we already use.

## Goal

Answer a multi-hop question in one model call by having the model write each follow-up question and its own intermediate answer before the final answer.

## How it works

1. Fill in the Self-ask prompt with the user's question; the worked example shows the follow-up and intermediate-answer lines.
2. Call the model once, with no stop sequence, so it writes the follow-ups and their answers itself.
3. Parse the follow-up questions and intermediate answers from the reply into a list, for logging and display.
4. Parse the text after 'So the final answer is:' and return it as the answer.

## Prompts

### Self-ask prompt

```markdown
Answer the question. If it needs facts you should look up first, ask
follow-up questions, one per line, and wait for each intermediate answer.

Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali

Question: {{question}}
Are follow up questions needed here:
```

## Stop rule

Stop after the one call. If the reply has no 'So the final answer is:' line, return no answer with the reply and a status saying it could not be parsed.

## Requirements

- Keep the scaffold identical to the search version, so you can switch search on later without changing the prompt.
- Parse the final answer from the last 'So the final answer is:' line only.
- Log each follow-up with the intermediate answer the model gave itself, so wrong facts can be traced to a hop.
- Add a test with a stub LLM whose reply has two follow-ups and ends 'So the final answer is: Benjamin Harrison'; assert one call and the answer exactly 'Benjamin Harrison'.

Prompt templates

With search · Without search

Self-ask prompt
Answer the question. If it needs facts you should look up first, ask
follow-up questions, one per line, and wait for each intermediate answer.


Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali


Question: {{question}}
Are follow up questions needed here:

Related patterns

Composes well with:

  • Chain-of-Thought The model writes out its intermediate reasoning steps before the final answer, all inside one call.
  • Agentic RAG The agent decides whether to retrieve, from which source, and with what rewritten query, and repeats until it has enough context.

Sources

  1. Measuring and Narrowing the Compositionality Gap in Language Models (Press et al., 2022)

From the Agentic Atlas by Joran Kikke · Edit on GitHub · Suggest a correction