Speculative decoding is one of the most important tricks behind fast LLM serving.
The basic idea is simple:
Use a small fast model to guess several future tokens, then use the original large model to verify those guesses in one pass.
If the guesses are right, the large model moves forward by multiple tokens instead of one. If a guess is wrong, the large model corrects it and the system continues.
The important part is that speculative decoding can be lossless. When the verification and sampling rule is implemented correctly, the final output follows the same distribution as the large target model would have produced by itself. We are changing the schedule of computation, not asking a weaker model to replace the stronger model.
In this post, I explain speculative decoding from first principles: why normal decoding is slow, what the draft model does, why target verification is cheap, how acceptance works, what the latency equation looks like, and why this technique is powerful but not automatic magic.
Why Normal LLM Decoding Is Slow
A decoder-only LLM generates text autoregressively.
That means it predicts the next token, appends that token to the context, then predicts the next token again:
prompt -> token 1
prompt + token 1 -> token 2
prompt + token 1 + token 2 -> token 3
...
If the model needs to generate 200 output tokens, the decode phase needs about 200 target-model steps.
The KV cache makes each step much cheaper than recomputing the full sequence from scratch. I covered that separately in KV Cache Explained. But the model is still fundamentally generating one new token per decode step.
That creates two practical problems.
First, latency grows with answer length. A longer answer means more decode steps.
Second, decode is often memory-bound. The GPU is not always doing a huge dense matrix multiplication over thousands of tokens. It is often doing a tiny single-token step while repeatedly touching large model weights and a growing KV cache. That is a bad shape for modern GPUs, which like large parallel work.
So the bottleneck is not only that the model is “thinking”. A lot of the time, the system is mechanically stepping through tokens one at a time.
The Core Trick
Speculative decoding adds a second model:
- the target model: the large model whose output we want to preserve
- the draft model: a smaller or cheaper model that proposes likely future tokens
One decoding round looks like this:
- The draft model proposes a block of tokens, for example 4 or 8 tokens.
- The target model scores those proposed tokens in one parallel verification pass.
- The system accepts the longest prefix that is consistent with the target model.
- At the first wrong token, the target model corrects the stream.
- The next speculative round starts from the corrected context.
For a simple greedy example, suppose the target model would generate:
The request is valid
The draft model proposes:
The request is invalid
The verifier accepts:
The request is
and rejects:
invalid
Then the target model supplies:
valid
The system did not trust the small model blindly. It only used the small model as a proposal generator.
Why Verification Is Cheap
This is the part that makes speculative decoding worth understanding.
If the target model normally generates four tokens, it usually needs four decode steps:
target pass -> token 1
target pass -> token 2
target pass -> token 3
target pass -> token 4
With speculative decoding, the draft model proposes those four tokens first. Then the target model receives the proposed block and computes the logits needed to verify all positions together.
That verification pass is not free. It still runs the big model. But it is much cheaper than doing four independent target-model decode passes because:
- the target model processes a short block in parallel instead of running separate single-token steps
- GPU work is denser and has better parallelism
- fixed runtime overheads are amortized over multiple candidate positions
- target-model KV/state handling can be reused for the verified block rather than advancing one token at a time
Another way to say it:
Verification is cheap because the target model can score many proposed positions in one forward pass, while normal decoding asks it to discover those positions one by one.
There is a caveat. Verification is cheap per candidate token, but it still consumes target-model capacity. In a production server with many concurrent users, verifying a long low-quality draft block can waste batch slots that could have served other requests.
This caveat matters a lot in systems like DeepSeek’s DSpark, which I cover in the next post.
A Tiny Walkthrough
Assume the current context is:
The database query
The draft model proposes four tokens:
uses an index scan
Now the target model verifies those four positions.
It checks:
The database query -> uses
The database query uses -> an
The database query uses an -> index
The database query uses an index -> scan
But it does this as one parallel target-model pass over the proposed block, not as four separate decode passes.
If all four tokens are accepted, the system advances four tokens in one round.
If only the first two are accepted, the system keeps:
uses an
Then the target model supplies the corrected next token, perhaps:
efficient
The new context becomes:
The database query uses an efficient
Then the next round starts.
Greedy Decoding vs Sampling
The easiest way to explain speculative decoding is with greedy decoding:
- draft proposes a token
- target computes its own best token
- if they match, accept
- if they differ, reject and use the target token
But real LLM serving often uses sampling: temperature, top-p, top-k, and similar controls.
For sampling, the verification rule has to preserve the target distribution, not just match the target argmax.
The classic speculative sampling rule works like this for each draft token:
accept token x with probability min(1, target_probability(x) / draft_probability(x))
If the token is rejected, the replacement token is sampled from the residual distribution:
max(0, target_distribution - draft_distribution)
That sounds abstract, but the intuition is straightforward.
If the draft model proposes a token that the target model likes at least as much as the draft did, the token is safe to accept.
If the draft model over-promoted a token compared with the target model, the verifier sometimes rejects it and samples a correction from the probability mass that the target wanted but the draft underrepresented.
This is why speculative decoding can preserve the target model’s distribution exactly. The draft model proposes. The target model remains the authority.
The Bonus Token
Many descriptions of speculative decoding mention a “bonus token”.
After the verifier accepts the whole drafted block, the target model already has the logits for the next position. So the system can often sample one extra token from the target model at the end of the successful block.
That means a round can produce:
accepted draft tokens + one target bonus token
This is why papers sometimes report accepted length in a way that includes the target-generated bonus token. When comparing numbers across papers or libraries, I always check whether the bonus token is included.
The Latency Equation
The practical speed of speculative decoding depends on three terms:
time per accepted token = (draft time + verify time) / accepted tokens per round
That gives us three levers.
1. Draft Faster
The draft model must be much cheaper than the target model.
If the draft model is only slightly faster than the target model, speculative decoding can be slower. You pay draft time and verification time, but you do not save enough target-model steps.
This is why good speculative systems use:
- small draft models
- shallow draft heads
- draft layers attached to the target
- n-gram or prompt-lookup drafters for easy text
- parallel block drafters
- hardware-aware serving kernels
The draft model does not need to be as smart as the target model. It needs to be cheap and aligned enough with the target to guess useful prefixes.
2. Accept More Tokens Per Round
Speedup depends heavily on acceptance length.
If the draft proposes eight tokens and the target accepts seven, the round is excellent.
If the draft proposes eight tokens and the first token is wrong, the whole suffix is wasted. Speculative decoding accepts a prefix. Once the verifier rejects position 1, positions 2 through 8 cannot be used, even if some of them would have been good under a different prefix.
That prefix property makes early tokens extremely important.
Good drafters are trained not only to predict plausible text, but to match the target model’s next-token distribution. A fluent draft is not enough. The draft must be fluent in the same way the target model is fluent.
3. Verify Smarter
The target verifier should not blindly score low-value tokens forever.
If the draft block gets worse toward the tail, verifying the entire block may waste GPU capacity.
This is especially true under high concurrency. When a server is lightly loaded, extra verification tokens may be almost free because the GPU has idle capacity. When the server is busy, every extra verification token competes with real user requests.
Modern systems therefore try to adapt the verification length:
- verify longer when the draft is confident
- verify shorter when the draft is uncertain
- verify longer when the server has spare capacity
- prune low-confidence suffixes when the server is saturated
This is one of the main ideas behind DSpark.
Why the Draft Model Can Be Wrong and Still Useful
At first, it feels strange to add a weaker model to speed up a stronger one.
But the draft model does not need to be perfect. It only needs to be right often enough on prefixes.
Many next tokens are easy:
Thank you for
The next token is probably:
your
Many code continuations are also structured:
for i in range(
The next few tokens are constrained by syntax and local context.
In these regions, a small model can guess several tokens correctly. The target model then verifies them quickly.
Open-ended chat is harder. There may be many valid continuations, and the target model’s exact distribution may differ from the draft model’s distribution. That lowers acceptance.
This is why speculative decoding tends to work better on structured tasks like code, math formatting, boilerplate, and predictable continuations than on high-entropy creative chat.
What Happens to Quality?
If speculative decoding is implemented correctly, quality should match the target model because the target model verifies the output distribution.
But there are several ways to accidentally break that guarantee:
- accepting draft tokens without the proper probability correction
- using an approximate verifier that does not match the target model
- applying different sampling settings to the draft and target in incompatible ways
- pruning verification based on future token information
- changing tokenization or chat formatting between draft and target
- using kernels that score the wrong positions because of masking or KV-cache bugs
This is why I think of speculative decoding as a systems technique, not only a modeling technique. The math can be clean, but the serving implementation has to be careful.
When Speculative Decoding Works Best
Speculative decoding is a strong fit when:
- the target model is large and expensive
- the draft model is much faster than the target
- the draft model’s distribution is close to the target’s distribution
- output tokens are somewhat predictable
- the serving engine can verify blocks efficiently
- batching and KV-cache handling are implemented well
It is weaker when:
- the draft model is too slow
- acceptance is low
- requests are very open-ended
- the server is already saturated and full-block verification wastes capacity
- implementation complexity outweighs the speedup
On a laptop, speculative decoding can easily disappoint if the draft model is not much faster than the target model or if the runtime cannot exploit parallel verification well. In production, with large models and carefully tuned kernels, the same idea can be very powerful.
Speculative Decoding Is Not Quantization
It is useful to separate speculative decoding from other inference optimizations.
Quantization changes the numeric representation of weights or activations.
KV-cache optimization changes how attention state is stored and reused.
Batching changes how multiple requests share GPU work.
Speculative decoding changes the decode schedule:
one target token at a time
becomes:
many draft tokens, one target verification pass
These techniques can be combined. A production system may use quantized weights, paged KV cache, continuous batching, optimized attention kernels, and speculative decoding at the same time.
A Practical Mental Model
The simplest mental model is:
Speculative decoding turns some sequential target-model steps into cheap guesses plus one parallel check.
The draft model is allowed to speculate because the target model still verifies.
The target model verification is cheap because it scores a block in parallel instead of discovering every token with separate decode passes.
The speedup appears only when enough draft tokens are accepted to pay for the extra draft and verification work.
That is the whole game.