Chain-of-Thought Prompting: Making LLMs Reason

Ask a person a three-step arithmetic problem and insist on an instant answer, and they will often get it wrong. Give them a sheet of paper and they will work through it. A language model is in the same position, for a mechanical reason: it produces one token at a time, and each token can only build on what is already written.

Chain-of-thought prompting gives the model its sheet of paper. You ask it to write out the steps before the answer, so that the answer is produced from the working instead of from nothing.

Why writing the steps helps

A model generating a direct answer has to get from the question to the final number in a single leap. There is nowhere to hold an intermediate result. See how LLMs work.

When the model writes "she has 16 minus 3 minus 4, which is 9 eggs left", that 9 is now text in its context. The next step can use it. Each step is small, and small steps are what these models do well.

The technique was named in a 2022 paper, which showed that giving a model a few worked examples with their reasoning written out significantly improved results on arithmetic, commonsense and symbolic reasoning tasks (Wei et al.: Chain-of-Thought Prompting). A second paper found that the examples were not even needed. Adding the sentence "Let's think step by step" raised accuracy on one arithmetic benchmark from 17.7% to 78.7%, and on the GSM8K word problems from 10.4% to 40.7%, with a large model of that time (Kojima et al.: Zero-Shot Reasoners).

A test

The script takes the first 20 problems from the test set of GSM8K, a collection of school-level maths word problems that each need a few steps (Cobbe et al.: GSM8K), published under the MIT licence (OpenAI: grade-school-math). A small local model answers each one twice: once required to reply with the number alone, and once told to work step by step and then state the answer.

import json
import re
import urllib.request

# The first 20 problems of the GSM8K test set: school word problems with one numeric answer.
problems = [json.loads(line) for line in open("../../genai/gsm8k/test.jsonl", encoding="utf-8")][:20]

DIRECT = "{question}\nGive only the final number."
STEPS = ("{question}\nWork through this step by step. "
         "Then give the final answer on a last line in the form: Answer: <number>")
ONE_NUMBER = {"type": "object", "required": ["answer"], "properties": {"answer": {"type": "number"}}}


def ask(prompt, limit, schema=None):
    body = {"model": "llama3.2:3b", "prompt": prompt, "stream": False,
            "options": {"temperature": 0, "seed": 42, "num_predict": limit}}
    if schema:
        body["format"] = schema                   # the reply must be {"answer": <number>} and nothing else
    request = urllib.request.Request("http://localhost:11434/api/generate", data=json.dumps(body).encode(),
                                     headers={"Content-Type": "application/json"})
    reply = json.load(urllib.request.urlopen(request, timeout=1800))
    return reply["response"], reply["eval_count"]


def number_in(text):
    """The number after 'Answer:' if there is one, otherwise the last number in the text."""
    tail = text.split("Answer:")[-1]
    found = re.findall(r"-?\d[\d,]*(?:\.\d+)?", tail) or re.findall(r"-?\d[\d,]*(?:\.\d+)?", text)
    return float(found[-1].replace(",", "")) if found else None


right = {"direct": 0, "steps": 0}
tokens = {"direct": 0, "steps": 0}
rows = []
for problem in problems:
    expected = float(problem["answer"].split("####")[1].replace(",", ""))
    row = []
    for name, template, limit, schema in (("direct", DIRECT, 30, ONE_NUMBER), ("steps", STEPS, 400, None)):
        reply, used = ask(template.format(question=problem["question"]), limit, schema)
        ok = number_in(reply) == expected
        right[name] += ok
        tokens[name] += used
        row.append("right" if ok else "wrong")
    rows.append(row)

print(f"{len(problems)} problems")
print("prompt            correct   output tokens per problem")
print(f"answer only       {right['direct']:>7}   {tokens['direct'] / len(problems):>25.0f}")
print(f"step by step      {right['steps']:>7}   {tokens['steps'] / len(problems):>25.0f}")
print()
both = sum(r == ["right", "right"] for r in rows)
print(f"right both ways: {both}   only with steps: {sum(r == ['wrong', 'right'] for r in rows)}   "
      f"only without: {sum(r == ['right', 'wrong'] for r in rows)}   wrong both ways: {sum(r == ['wrong', 'wrong'] for r in rows)}")

20 problems
prompt            correct   output tokens per problem
answer only             1                          11
step by step           14                         212

right both ways: 1   only with steps: 13   only without: 0   wrong both ways: 6

One detail of the method matters. Told to give only the final number, this model usually started writing its working anyway. So the test enforced the format: in the first condition the reply had to be a JSON object holding one number, leaving no room for working. That measures a genuinely direct answer.

• Answering directly, the model solved 1 problem in 20. It had to produce the result in a single leap, and almost never landed it.
• Working step by step, it solved 14. The same model, the same problems, the same settings.
• Nothing was lost by thinking. No problem was solved directly and then missed with working. 13 were solved only with it.
• The working cost 19 times the output tokens: 212 per problem against 11. Output tokens are the slow and expensive ones.
• Six problems were missed both ways. Writing out steps makes reasoning possible. It does not make it reliable, and a chain can contain a slip in any line.

The size of the gap is partly about the model. A 3-billion-parameter model has very little capacity to do several steps silently, so it gains the most from writing them down. Larger models answer more often directly, and the gain shrinks, but the direction of the effect is the one the original papers reported.

Variants

Zero-shot. Add an instruction to reason step by step. This is what the test used.

Few-shot. Include worked examples with their reasoning, in the format you want. This steers how the working is laid out as well as triggering it. See few-shot vs zero-shot prompting.

Self-consistency. Run the same step-by-step prompt several times with some randomness and take the most common final answer. Different routes that reach the same number are evidence that it is right. The paper that proposed it reported a gain of 17.9 points on GSM8K over a single chain (Wang et al.: Self-Consistency). It also multiplies the cost by the number of runs.

Separate the working from the answer. Ask for the reasoning first and the final answer on a marked last line, as the test did, so that code can pick out the answer without parsing prose.

Reasoning models changed the advice

Since those papers, model makers have built the step-by-step process into the models themselves. A reasoning model works through a problem internally before it replies, and you are billed for that hidden working.

For such models the instruction is redundant. OpenAI's guidance says so directly: because these models reason internally, telling them to think step by step is unnecessary, may not improve results and can sometimes hinder them. It recommends short, direct prompts instead (OpenAI: reasoning best practices). Anthropic's current models decide for themselves when and how much to think, and its guidance notes that thinking adds latency and is worth it mainly for multistep problems (Anthropic: prompting best practices).

So the rule today depends on the model:

When not to use it

• Simple lookups and classification. There are no steps to write, only extra tokens.
• Tight latency budgets. The working has to be generated before the answer appears.
• When exactness matters more than explanation. For real arithmetic, give the model a calculator. A written chain can contain a slip in any line. See function calling and tool use.
• As proof. The written reasoning is text the model generated. It is not a guaranteed record of how the answer was reached, and a fluent chain can end in a wrong answer.

Common mistakes

• Adding "think step by step" to a reasoning model and paying twice.
• Parsing the answer out of free prose instead of asking for a marked final line.
• Truncating the output. If the token limit cuts the working short, there is no answer at all.
• Trusting the chain because it reads well.
• Using it everywhere. Most requests do not need it.

For the broader set of prompt techniques, see prompt engineering techniques. For what the extra tokens do to a bill, see cost optimization for LLM applications.

Chain-of-thought prompting is part of the Prompt Engineering module of the Generative AI course.

Sources

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv) — Wei et al.
• Large Language Models are Zero-Shot Reasoners (arXiv) — Kojima et al.
• Self-Consistency Improves Chain of Thought Reasoning in Language Models (arXiv) — Wang et al.
• Training Verifiers to Solve Math Word Problems (arXiv) — Cobbe et al.
• grade-school-math: the GSM8K dataset (GitHub, OpenAI) — OpenAI
• Reasoning best practices (OpenAI API documentation) — OpenAI
• Prompting best practices (Claude Platform Docs) — Anthropic