Speculative Decoding
Large language models often generate text through a step-by-step loop. After producing a token, the model conditions on the updated sequence and predicts the next token. Because each prediction depends on the previous one, generation tends to be sequential.
Even when modern GPUs offer strong parallel compute, decoding can still run into memory bandwidth limits. The repeated movement of data through the model can become a significant source of latency.
Introducing Speculative Decoding
Speculative decoding is an inference technique that aims to increase generation throughput while keeping the target model’s output distribution intact.
Its central idea is to let a smaller, faster model propose tokens and then let a larger, more capable model verify those proposals.
The Core Observations
Two observations support the method:
- Compute asymmetry: Smaller draft models can often run much faster than larger target models.
- Predictability: Smaller models may capture common language patterns, grammar, and continuations well.
The Typical Workflow
- Drafting: The draft model generates several candidate tokens.
- Verification: The target model evaluates the candidates in parallel.
- Acceptance and rejection: Token-level probability checks determine whether a candidate should be accepted.
If a candidate fails verification, the target model can issue a correction. If all candidates pass, the target model may generate an additional token as part of the verification process.
Preserving Output Quality
A key challenge is maintaining consistency with the target model’s sampling behavior.
Speculative decoding uses a rejection-based approach. For a candidate token, the acceptance probability is calculated using both the draft and target distributions.
This allows the system to accept likely tokens quickly while still ensuring that the final sequence remains faithful to the target model’s distribution.
Benefits and Tradeoffs
The technique can deliver:
- Faster token generation.
- Better hardware utilization.
- Reduced memory access pressure.
- Small additional memory overhead for the draft model.
The draft model does not need to be extremely large. For example, a smaller model might be used to accelerate a much larger one, or a specialized decoding head might be employed.
Practical Considerations
The performance of speculative decoding depends on the draft model, the number of candidate tokens, and the target model. Teams often tune these settings to balance speed and quality.
Conclusion
Speculative decoding offers a practical way to improve large language model inference. By combining draft generation with target verification, it can reduce the cost of producing each token while aiming to preserve the target model’s behavior.