- CoT changes the format of the answer: reasoning first, conclusion second.
- It works because LLMs generate one token at a time, so earlier tokens condition later ones.
- Two flavors: zero-shot ("think step by step") and few-shot (show worked examples).
- A plausible chain can still lead to a wrong answer, so score correctness and groundedness, not format.
- 2025–2026 reasoning models perform CoT internally; explicit prompts still help, but the gains shrink.
What is chain-of-thought prompting?
Explicitly reasoning through the problem makes the model measurably more accurate on math, logic, planning, and multi-step questions, and makes its answers easier to audit.
Chain-of-thought prompting was popularized by Wei et al. (Google Brain, 2022), who showed that giving a large language model worked examples with their reasoning written out, instead of just the answers, substantially improved performance on arithmetic, symbolic reasoning, and multi-hop question answering. Kojima et al. (2022) then showed that a plain cue, "Let's think step by step," gets much of the same effect with no examples.
The idea is simple. An LLM generates text token by token; each token conditions the next. If the model outputs its reasoning first, the final answer is generated given that reasoning as context. That extra context often leads to correct answers the model couldn't reach in one leap.
Chain-of-thought is one of several prompt engineering patterns, alongside few-shot and zero-shot prompting, and one of the building blocks for agentic AI systems that need to plan.
How does it work?
CoT works by exploiting the autoregressive structure of language models. Instead of a single guess, the answer format becomes reasoning, then answer. A typical CoT prompt has three parts, and the answer improves because each generated token conditions the next.
The three components of a CoT prompt:
- Question. The task the model needs to complete.
- Instruction. A cue like "Let's think step by step" or a template that requires the model to enumerate its reasoning. See the Enterprise h2oGPTe prompt catalog for reusable CoT templates.
- Answer format. Where the final answer should go (often after a marker like
Answer:so it can be parsed). Pairs well with structured output.
The model then writes reasoning, and because that reasoning becomes part of the running context, the final answer is grounded in it.
Zero-shot vs. few-shot CoT
Zero-shot CoT
Few-shot CoT
Prompt style
Add "Let's think step by step"
Include 2–8 worked examples showing reasoning
Setup cost
None
Craft examples, test them
Accuracy
Good for strong models
Higher on hard tasks
Cost per query
Low
Higher (examples eat context)
Best for
Quick wins, new tasks
Domain-specific reasoning, high-stakes tasks
When to use chain-of-thought
CoT helps most when the answer requires multiple steps a person would work through on paper: arithmetic, unit conversions, logic puzzles, planning, code debugging, multi-hop questions over documents, agent planning.
CoT usually doesn't help, and sometimes hurts, on tasks that are one-shot classifications, well-formed factual lookups, or short answers where the model already knows the answer. When in doubt, evaluate both patterns.
Limits and pitfalls
Reasoning can look right and be wrong. A confident, plausible chain-of-thought that leads to a wrong answer is a common hallucination failure mode. Score groundedness, not just answer format.
Cost. CoT answers are longer, so cost more per query. For high-volume workloads, use it selectively (e.g., only when the model reports low confidence).
Latency. Longer answers mean longer waits. Streaming helps; caching helps more.
Prompt leakage. Chain-of-thought can expose sensitive intermediate steps in logs. Redact before storing.
Modern reasoning models. 2025–2026 "thinking" models perform CoT internally by default; explicit CoT prompts can still help but the gains shrink. Practitioners now distinguish between external CoT (visible in the response) and internal CoT (hidden reasoning tokens the model uses but does not surface). Both consume compute; only external CoT is auditable, so regulated deployments often prefer models that expose the chain even when it costs more.
Consistency and self-verification. A single CoT run is one sample. Better production patterns run the chain multiple times ("self-consistency": majority vote over N chains) or ask the model to critique its own reasoning ("self-verification"). Both raise accuracy on hard tasks at the cost of more tokens. H2O Eval Studio scoring can compare single-run vs. self-consistency variants side by side to help teams tune the cost/accuracy trade-off on their own workloads.
Chain-of-thought prompting in H2O AI
Enterprise h2oGPTe supports chain-of-thought and other prompt engineering patterns through its Prompt Catalog, so teams can standardize the patterns that work for their tasks. H2O Eval Studio scores whether CoT-produced answers are actually correct and grounded, an important guardrail because CoT makes answers sound more authoritative even when they're wrong. When agents need multi-step reasoning, the same patterns run in agentic AI workflows on the H2O AI Super Agent.