Module 7 of 11 · Advanced · 8 min read

Self-consistency and multiple samples

Ask more than once: majority voting for answers, best-of-n for open-ended work, and disagreement as a warning light.

Key takeaways

1Self-consistency (Wang et al., 2022) samples several reasoning paths and keeps the most common answer.
2It works best when there is a single checkable answer.
3For open-ended work, generate several candidates and pick with a rubric.
4When samples disagree, treat it as a signal to verify or escalate.

01The idea

Language models are probabilistic: ask the same hard question several times and you may get different reasoning and different answers. Self-consistency, introduced by Wang et al. (2022), exploits this. Sample several independent chain-of-thought solutions, then take the answer that most of them reach. Errors tend to scatter while correct reasoning tends to converge.

02How to run it

A handful of samples is the usual starting point. Increase only if tests show it helps, because cost scales with every sample.

Send the same prompt several times with some randomness (a non-zero temperature, where your API allows it).
Extract each final answer in a consistent, ideally structured, format.
Take the majority answer, or pass the candidates to a judge prompt if answers are not directly comparable.
Record how strongly they agreed: unanimous, majority or split.

03Best-of-n for open-ended work

Headlines, designs, slogans and plans have no single right answer to vote on. Instead, generate several candidates and choose with explicit criteria, either yourself or with a rubric-driven judge prompt. Asking for variety explicitly ('make each option take a different angle') produces a more useful spread.

Example

Write five subject lines for the email in <email>. Make each take a different angle: curiosity, benefit, urgency, social proof and plain description. Under 50 characters each. Then score each from 1 to 5 against: clear benefit, specific, honest (no clickbait). Show the scores in a table and recommend one.

04Disagreement is information

If five samples give three different answers, the model is uncertain, even if each answer sounds confident. In an automated system you can use agreement as a confidence signal: act automatically when samples agree, and route to a person or a stronger model when they split.

Insight

Agreement is not proof. Models can be consistently wrong, especially about facts they never learned. Self-consistency reduces random errors, not systematic ones.

05When not to bother

Simple tasks, deterministic extraction and anything where one careful run is good enough. Reasoning models with high effort settings often explore alternatives internally, so test whether extra samples add anything before paying for them.