Chain-of-Thought Prompting: When It Helps and When It Wastes Money

Chain-of-thought prompting means asking a model to work through its reasoning before giving an answer. In its simplest form it is four words appended to a question: "think step by step."

The effect is real and well documented. On multi-step reasoning tasks it substantially improves accuracy. But it is not free, and it does not help everywhere.

Why it works

When you ask for an answer directly, the model must produce it in a single pass. When you ask it to reason first, it generates intermediate steps, and each step conditions on the previous ones.

Two things follow. The model gets more computation to apply to the problem, and errors become more visible — both to the model as it proceeds and to you reading the output.

The analogy is a person asked to multiply two three-digit numbers in their head versus on paper. Same capability, very different accuracy.

Where it helps most

Arithmetic and quantitative comparison. Anything requiring multiple operations. "Which has better cost per year" with two prices and two lifespans is exactly the shape of problem that fails when answered immediately.

Logic with dead ends. Problems where the first plausible path is often wrong.

Debugging. Asking for possible root causes, evaluating each against the evidence, then concluding, produces far better diagnoses than asking "what's wrong with this code."

Multi-criteria decisions. When several factors trade off, forcing the model to address each in turn prevents it from latching onto one and ignoring the rest.

Where it is wasted

Simple factual lookup. "What is the capital of France" does not benefit from deliberation. You pay for tokens and latency to receive the same answer.

Short creative tasks. Asking a model to reason step by step before writing a headline produces analysis of headline-writing followed by a mediocre headline.

Classification at volume. For high-throughput tagging, the accuracy gain rarely justifies multiplying token cost across millions of calls. Test whether it actually helps on your data before assuming.

How to use it well

The bare instruction works, but structured versions work better because they specify what kind of reasoning you want.

Instead of:

Think step by step. Which product is better value?

Try:

1. Calculate cost per year for each product.
2. Note any factor besides cost that affects the decision.
3. State which is better value and why, in one sentence.

This is prompt chaining compressed into a single request. You get the accuracy benefit plus control over the reasoning shape.

For hard problems where you can verify the answer, self-consistency goes further: run the same reasoning prompt several times at non-zero temperature and take the most common answer. Wrong reasoning fails in varied ways; correct reasoning converges. It costs N times more, so reserve it for decisions that matter.

Reasoning models change the calculus

Newer reasoning models generate extended internal deliberation automatically. With those, explicitly asking for step-by-step reasoning is largely redundant — it is already happening, and you are already paying for it.

The practical routing rule:

• Reasoning model → hard analytical problems. Do not add chain-of-thought instructions.
• Standard model + chain-of-thought → moderately hard problems where you want better accuracy without reasoning-model cost.
• Standard model, no chain-of-thought → writing, summarisation, extraction, conversation.

One caveat worth knowing

The reasoning a model shows you is not a transparent window into its computation. It is generated text that describes plausible reasoning. Models can produce sound-looking reasoning that leads to a wrong answer, and occasionally reach a right answer via visibly flawed reasoning.

So treat the reasoning trace as useful for spotting errors, not as proof of correctness. It is a debugging aid, not a guarantee.

Our Prompt Engineering Fundamentals course covers this with runnable examples, and Loop & Agentic Engineering extends it into multi-step chains.