Module 6 of 11 · Expert · 10 min read

Tree of Thoughts: branching and searching

Explore several lines of reasoning, evaluate them, and pursue the promising ones, in a single prompt or with orchestration.

Key takeaways

1Tree of Thoughts (Yao et al., 2023) treats reasoning as a search over branching partial solutions.
2At each step: propose candidates, evaluate them, keep the best, and backtrack when stuck.
3Use it for problems where early choices matter and dead ends are common.
4A single-prompt version captures some of the benefit; real search needs orchestration.

01From a chain to a tree

Chain-of-thought follows one line of reasoning from start to finish; if an early step is wrong, everything after it is too. Tree of Thoughts (Yao et al., 2023) instead lets the model consider several possible next steps, called thoughts, at each point, evaluate how promising each one is, and explore the best, backtracking from dead ends using search strategies such as breadth-first or depth-first search. In the paper this clearly outperformed standard chain-of-thought on tasks that need planning or search, such as the Game of 24 arithmetic puzzle, constrained creative writing and mini crosswords.

02The three operations

The evaluator is the key component, just as the critic is in a refinement loop.

Propose: generate several distinct candidate next steps or partial solutions.
Evaluate: score each candidate, for example sure / maybe / impossible, or 1 to 10 against criteria.
Search: keep the top few (a beam), expand them further, and drop or backtrack from weak branches.

03Orchestrated version

In code or a workflow tool, each operation is its own prompt and a small controller runs the search.

Example

Propose prompt: Here is the problem and the partial solution so far: <problem>{{problem}}</problem> <so_far>{{path}}</so_far>. Propose 3 distinct next steps. Make them genuinely different approaches, not rewordings. Evaluate prompt: For the partial solution in <candidate>, judge whether it can still lead to a valid full solution. Answer SURE, MAYBE or IMPOSSIBLE with one sentence of justification. Controller: keep the best 2 candidates at each depth (beam width 2), expand to depth 4, and return the best complete solution.

04Single-prompt approximation

You can capture part of the benefit in one prompt by asking the model to lay out several approaches, evaluate them against criteria, and then develop only the strongest. It is not true search, because the model commits within one generation, but it is cheap and often good enough for planning and design decisions.

Example

We need to {{goal}} under these constraints: {{constraints}}. 1. Propose three fundamentally different approaches, one paragraph each. 2. Evaluate each against cost, time to first result, risk, and fit with the constraints, in a table. 3. Pick the strongest, explain why the others lost, then develop the winner into a step-by-step plan. 4. Name the earliest point at which we would know the chosen approach is failing, and which alternative we would switch to.

05When it is worth it

Strong reasoning models already explore alternatives internally on many problems, so test whether explicit branching adds value before building it.

Good fit: puzzles and constraint problems, multi-step planning, architecture and design choices, strategy with real trade-offs.
Poor fit: tasks with an obvious method, simple retrieval, or anything a single careful pass already gets right.
Cost: orchestrated search multiplies calls by branching factor and depth. Keep both small.