Why chain-of-thought helps large models but not small ones

WHAT THIS TESTS Whether you understand that prompting techniques interact with model capability and can tailor strategy by tier rather than applying one recipe everywhere.

A GOOD ANSWER COVERS WHY THE GAP. Chain-of-thought only pays off if the model can actually carry out coherent intermediate reasoning. Larger models trained on enough data acquire reliable multi-step reasoning and instruction-following, so prompting them to reason step by step unlocks capability that was latent. Smaller models often cannot keep the chain coherent; their intermediate steps are noisy, so spelling out steps adds more chances to make and propagate errors, sometimes lowering accuracy. This is the emergent ability pattern, where a behavior is near-absent below some scale and appears more sharply above it, though the sharpness can partly depend on the metric. STRATEGY BY TIER. For capable models, exploit CoT, self-consistency, and lightly structured reasoning prompts. For smaller models, prefer direct answers, invest in high-quality few-shot exemplars, and move the reasoning burden outside the model by decomposing the task with an orchestrator that makes each step simple. You can also fine-tune the small model on the target task or distill reasoning traces from a larger teacher so the small model learns to produce useful steps. Always measure per tier rather than assume transfer.

COMMON WRONG ANSWERS Assuming CoT helps every model. Saying small models are just slower, missing the capability threshold. Claiming emergence is purely magical with no relation to scale or data or metric. Recommending the same prompt for all tiers.

LIKELY FOLLOW-UPS Why can CoT hurt a weak model? Is emergence partly a metric artifact? How would distillation transfer reasoning to a small model? How do you decide a model is capable enough for CoT?

ONE CONCRETE EXAMPLE On a multi-step arithmetic benchmark, the large model jumps from 60 to 85 percent with CoT because it executes each step reliably. The small model stays around 30 percent or dips, since its flawed intermediate steps cascade. So in production you enable CoT and self-consistency on the large tier, but for the small tier you decompose the problem externally and fine-tune it on worked examples instead of relying on the model to reason in one prompt.