Speculative Decoding: Faster LLM Inference, Same Results
Speculative decoding accelerates LLM inference by using a small, fast "draft" model to predict a sequence of tokens. The large, accurate model then validates this entire sequence in a single parallel pass, instead of generating one token at a time. This is used to get 2-3x speedups on production models without retraining. The common misconception is that it's a lossy approximation; in reality, it produces bit-for-bit identical output to the original model.
### The mental model Think of a fast assistant drafting a response for a slow, meticulous expert. Instead of the expert writing word-by-word, they review the assistant's entire drafted sentence at once. They approve the correct parts and only step in to fix the first mistake, saving significant time.
### How it works Speculative decoding uses two models: a small, fast "draft" model and the large, powerful "target" model you want to accelerate.
1. The draft model autoregressively generates a short block of candidate tokens (e.g., 4-5 tokens). 2. The target model then takes the original prompt plus this entire block of draft tokens and performs a *single* forward pass. 3. This one pass allows the target model to verify all the draft tokens in parallel. It checks if the draft model's chosen token at each step is the same one it would have chosen. 4. All matching tokens from the start of the block are accepted. If a mismatch occurs at token `i`, the first `i-1` tokens are kept, the target model's corrected token is used for position `i`, and the rest of the draft is discarded. 5. The process then repeats from the new state.
### When to use it * To reduce inference latency for large autoregressive models in user-facing applications like chatbots. * When you need a speedup but cannot tolerate *any* change in the model's output distribution (unlike quantization or pruning). * To accelerate existing, off-the-shelf models without needing to retrain or modify their architecture.
### When NOT to use it * If the task is highly creative or unpredictable. The draft model's guesses will be wrong more often, diminishing the speedup as fewer tokens are accepted per verification step. * When memory, not latency, is the main constraint. This technique requires holding both models in memory.
### One canonical example The original paper demonstrated this on Google's T5-XXL model. By using a smaller T5 model as the drafter, they achieved a 2x-3x reduction in inference time compared to the standard implementation. Crucially, the output text generated was identical to the output from the original, slower T5-XXL model, proving it's a lossless optimization.
Read the original → arxiv.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.