Speculative Decoding: Faster LLM Inference, Same Results

Speculative decoding accelerates LLM inference by using a small, fast "draft" model to propose a sequence of future tokens. The large, powerful model then validates this entire sequence in a single parallel pass, rather than generating one token at a time sequentially. If the draft model's predictions match what the large model would have generated, they are accepted. The first mismatch discards the rest of the draft, and the large model generates the next token normally. This is used to achieve 2-3x speedups on off-the-shelf models without retraining. A common footgun is confusing this with lossy optimizations; speculative decoding is guaranteed to produce bit-for-bit identical output to the original model, just faster.