Speculative Decoding: A Small LLM Speeds Up a Big One

WHY IT EXISTS Autoregressive LLM inference is slow because it generates tokens one by one; you can't compute the tenth word until you have the ninth. This sequential dependency creates a latency bottleneck. Speculative decoding was created to parallelize parts of this process to get answers faster.

THE MENTAL MODEL Think of a senior engineer (the large target model) and a junior engineer (the small draft model). Instead of the senior writing all the code line-by-line, the junior quickly drafts a few lines. The senior then reviews the entire block at once, accepting the correct lines and rewriting only the first one that's wrong. This is much faster than the senior writing everything from scratch.

HOW IT WORKS Instead of the large target model generating one token, a smaller, faster draft model generates a sequence of candidate tokens. The large model then takes this entire sequence and, in a single forward pass, checks which tokens it would have generated itself. It accepts the matching prefix of tokens and discards the rest. If any were rejected, the large model generates one correct token itself, and the process repeats. This verification scheme guarantees the final output is identical to what the target model would have produced alone.

WHEN TO USE IT Use this to reduce inference latency for autoregressive models in production, especially for user-facing applications where response time is critical. It provides a 2-3x speedup without sacrificing any output quality, making it a pure optimization. It is most effective when you have a much faster draft model that is still reasonably accurate.

WHEN NOT TO USE IT This technique adds complexity. If latency is not a primary concern, or if the overhead of maintaining and running a second draft model outweighs the performance gains (e.g., for very short text generations), the added complexity may not be worthwhile.

ONE CANONICAL EXAMPLE The technique is named by analogy to speculative execution in CPUs. A processor doesn't wait to confirm a conditional branch; it predicts the outcome and executes instructions down that path. If correct, it keeps the results and has saved time. If wrong, it discards the work and takes the correct path. Speculative decoding makes a similar bet on the draft model's output, gaining speed when it's right and correcting when it's wrong.