Self-consistency over chain-of-thought
Sampling-based reasoning improvement and its cost.
Sample multiple CoT paths at nonzero temperature and majority-vote the final answer; cost scales with the number of samples.
What's really being asked
Whether you understand the mechanism behind self-consistency, the vote over answers not text, and can quantify its compute and latency footprint.
The full answer
MECHANISM. Standard chain-of-thought greedily decodes one reasoning trace, so a single early misstep dooms the answer. Self-consistency instead samples multiple diverse reasoning paths using nonzero temperature, then extracts the final answer from each path and selects the most frequent answer by majority vote. The intuition is that there are many valid reasoning routes to a correct answer, so correct answers recur across samples while different errors disagree with each other, making the mode a robust estimate. Note the vote is over final answers, not over the prose. IMPLEMENTATION. Issue N generations of the same CoT prompt at a temperature like 0.5 to 0.7, run them in parallel rather than sequentially, parse a normalized answer from each, and aggregate by count, optionally weighting. Pick N to balance accuracy gains, which have diminishing returns, against cost. COSTS. Token and dollar cost scale roughly linearly with N since you run N full generations. If you parallelize, end-to-end latency stays close to a single call, but you consume N times the compute and reduce system throughput; sequential execution would multiply latency too.
The mistakes people make
Saying it averages or concatenates the reasoning text. Picking the longest or most confident chain instead of voting on answers. Assuming it is free or low cost. Claiming accuracy scales without limit in N. Forgetting that answer extraction and normalization are needed.
What usually comes next
Why does temperature matter for diversity? How do you set N given diminishing returns? How is this different from beam search? When is the extra cost not worth it for production latency budgets?
A concrete example
A math word problem is sampled five times. Three paths reach 42 via different arithmetic routes, one reaches 36, one reaches 40. Majority vote returns 42, overriding the two divergent errors that a single greedy chain might have produced. Running the five samples in parallel keeps latency near one call while costing about five times the tokens.
Interview question
How does self-consistency select a final answer from multiple chain-of-thought samples?
- a.It averages the text of all reasoning chains into one chain
- b.It picks the chain with the lowest temperature
- c.It takes the majority vote over the extracted final answers across the sampled reasoning pathsCorrect
- d.It always chooses the longest reasoning chain
Why? this is the answer
Self-consistency samples diverse paths and votes on the final answers, since correct answers recur while errors scatter. It does not average text or rank by chain length, and it costs roughly N times the compute.
Just read this? Test yourself on what you have been reading.
- #llm
- #self-consistency
- #chain-of-thought
- #prompting
- #inference-cost
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles