Self-Consistency: Majority Rules for LLM Reasoning

WHY IT EXISTS Standard chain-of-thought prompting often uses a "greedy" approach, picking the single most likely next word at each step. This can lock the model into a flawed reasoning path early on. Self-consistency was created to overcome this limitation by exploring a wider range of possibilities to find a more robust answer.

THE MENTAL MODEL Think of it as a committee of experts solving a problem. Instead of trusting the first expert who speaks up (greedy decoding), you ask several experts to work through the problem independently. Even if they take different routes, they should ideally arrive at the same correct answer. You then tally their final answers and trust the majority opinion.

HOW IT WORKS Self-consistency is a decoding strategy, not a new type of model. First, using a technique like chain-of-thought prompting, it samples a diverse set of reasoning paths from the language model instead of just the single most probable one. This is achieved by introducing some randomness during generation. Second, it lets each reasoning path run to its conclusion to produce a final answer. Finally, it aggregates all the final answers and selects the one that appears most frequently. This final step marginalizes out the reasoning paths to find the most consistent result.

WHEN TO USE IT Use self-consistency when accuracy on complex reasoning tasks is critical and you can afford the extra compute. It is particularly effective for arithmetic (like the GSM8K benchmark), commonsense reasoning (StrategyQA), and other tasks where a problem has a unique correct answer but multiple valid ways to derive it. It provides a significant performance boost over naive greedy decoding.

WHEN NOT TO USE IT Avoid it in latency-sensitive applications where the cost of generating many response paths is too high. It is also less useful for open-ended or creative generation tasks where there is no single "correct" answer to vote on. The core assumption is that a consensus answer is the correct one, which may not hold if the model has a strong, systematic bias that leads it to a consistent but incorrect conclusion.

ONE CANONICAL EXAMPLE For a math word problem, a standard chain-of-thought prompt might generate one step-by-step solution. With self-consistency, the model would generate, say, ten different step-by-step solutions. Path 1 might solve it with algebra, Path 2 by breaking it down arithmetically, and Path 3 might make a calculation error. If seven paths conclude the answer is "42", one says "45", and two have errors, the system chooses "42" as the final, most consistent answer.