Speculative Decoding: Faster LLM Inference, Same Results
Speculative decoding accelerates LLM inference by using a small, fast "draft" model to predict a sequence of tokens. The large, accurate model then validates this entire sequence in a single parallel pass, instead of generating one token at a time. This is used to get 2-3x speedups on production models without retraining. The common misconception is that it's a lossy approximation; in reality, it produces bit-for-bit identical output to the original model.
The mental model
Think of a fast assistant drafting a response for a slow, meticulous expert. Instead of the expert writing word-by-word, they review the assistant's entire drafted sentence at once. They approve the correct parts and only step in to fix the first mistake, saving significant time.
How it works
Speculative decoding uses two models: a small, fast "draft" model and the large, powerful "target" model you want to accelerate.
1. The draft model autoregressively generates a short block of candidate tokens (e.g., 4-5 tokens).
2. The target model then takes the original prompt plus this entire block of draft tokens and performs a single forward pass.
3. This one pass allows the target model to verify all the draft tokens in parallel. It checks if the draft model's chosen token at each step is the same one it would have chosen.
4. All matching tokens from the start of the block are accepted. If a mismatch occurs at token i, the first i-1 tokens are kept, the target model's corrected token is used for position i, and the rest of the draft is discarded.
5. The process then repeats from the new state.
When to use it
• To reduce inference latency for large autoregressive models in user-facing applications like chatbots. When you need a speedup but cannot tolerate any* change in the model's output distribution (unlike quantization or pruning). • To accelerate existing, off-the-shelf models without needing to retrain or modify their architecture.
When not to use it
• If the task is highly creative or unpredictable. The draft model's guesses will be wrong more often, diminishing the speedup as fewer tokens are accepted per verification step. • When memory, not latency, is the main constraint. This technique requires holding both models in memory.
One canonical example
The original paper demonstrated this on Google's T5-XXL model. By using a smaller T5 model as the drafter, they achieved a 2x-3x reduction in inference time compared to the standard implementation. Crucially, the output text generated was identical to the output from the original, slower T5-XXL model, proving it's a lossless optimization.
Interview question
How does speculative decoding ensure bit-for-bit identical output to the original target model while accelerating inference?
- a.A. The draft model is trained to perfectly mimic the target model's token probabilities, making its output inherently identical.
- b.B. The target model performs a single parallel pass to validate a block of draft tokens, accepting only those it would have generated and correcting any mismatches.Correct
- c.C. It employs a post-processing step that compares the draft model's full output to the target model's expected output and adjusts accordingly.
- d.D. The target model's internal states are directly copied from the draft model, ensuring consistency across generations.
Why? this is the answer
The correct answer (B) highlights that the target model's validation process, where it accepts only tokens it would have generated and corrects any mismatches, is what guarantees identical output. Option (A) is a common misconception because while the draft model aims to be accurate, the final output's integrity relies on the target model's explicit verification, not the draft model's inherent perfection.
Just read this? Test yourself on what you have been reading.
Read the original → arxiv.org
- #llm
- #inference
- #optimization
- #transformer
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles