How to Use Speculative Decoding for Faster LLM Inference

Running large language models in production is notoriously slow and expensive. When you deploy a model like Llama 3 70B, every single character it outputs requires the GPU to process billions of parameters. This autoregressive nature of text generation means the system is constantly waiting on memory transfers, leaving powerful processors sitting idle. If you are running large language models in production, learning how to use speculative decoding for faster LLM inference is one of the most effective ways to slash latency.

For more on this topic, you might also want to check out How to Master Graph RAG for Deep Knowledge Retrieval and How to Use Sparse Autoencoders for LLM Interpretability and Mastering AI Agents with Memory: Beyond Short-Term Recall and Ultimate Guide: Mastering AI Agents with Multi-Modal Planning.

This technique does not require pruning your models or running aggressive quantization that ruins accuracy. Instead, it alters the math of token generation by pairing your massive model with a smaller, faster draft assistant. This guide breaks down exactly how to use speculative decoding for faster LLM inference without losing model accuracy.

The Memory Bandwidth Bottleneck in LLM Generation

To understand why language models are slow, we have to look at the hardware wall. LLM generation has two distinct phases. First is the prefill phase, where the model processes your input prompt. This phase is compute-bound, meaning the GPU uses its cores to analyze the entire input text simultaneously. The second phase is the decoding phase, where the model generates output tokens one by one. This phase is entirely memory-bound.

To generate a single token, the GPU must load every single weight of the neural network from its High Bandwidth Memory into its local cache. For a 70-billion parameter model stored in 16-bit precision, this means reading 140 gigabytes of data just to output a single word. It stalls the chip. The tensor cores spend most of their time waiting for memory buses to deliver the weights rather than doing actual math.

When exploring how to use speculative decoding for faster LLM inference, we first have to look at this hardware wall. Speculative decoding bypasses this memory bottleneck by changing how often we load the large model’s weights. Instead of loading those 140 gigabytes of parameters for every single token, we load them once to verify multiple tokens at the same time.

Under the Hood of Speculative Decoding

The core concept relies on a simple observation: a small model can often predict the next few tokens just as well as a large model, especially for simple words, punctuation, and predictable phrasing. Speculative decoding pairs a large target model with a small, highly efficient draft model. The process follows a clear, repeating sequence.

• The Draft Phase: The small draft model runs autoregressively for a set number of steps, usually four or five. Because it has far fewer parameters, it completes these runs incredibly fast. It generates a short sequence of candidate tokens.
• The Verification Phase: The target model receives the original prompt plus the candidate tokens. It runs a single forward pass over the entire sequence. Because this is a single pass, the GPU only loads the target model’s massive weights once to evaluate all the candidate tokens simultaneously.
• The Acceptance Phase: The system compares the probability distributions of both models. If the target model agrees with the draft model’s choices, those tokens are accepted. If the target model disagrees at token three, we keep the first two tokens, accept the target model’s corrected third token, and discard the rest of the draft.

This method is mathematically lossless. The final output distribution remains identical to what the large model would have produced on its own. When setting up your pipeline, knowing how to use speculative decoding for faster LLM inference requires a grasp of the draft-then-verify loop.

If you want to verify that learning how to use speculative decoding for faster LLM inference does not alter your outputs, you can test it directly by comparing speculative outputs against baseline outputs using the Text Diff Checker to ensure they match down to the exact character. This simple test proves that the acceleration does not compromise the quality of your generation.

Implementing Speculative Decoding in Production

Setting up speculative decoding manually from scratch is complex, but modern inference engines have built-in support for it. The most popular open-source engine for this is vLLM, which handles the memory management and scheduling automatically.

Let’s look at how to use speculative decoding for faster LLM inference using vLLM, which is currently the easiest open-source path. You will need a GPU with enough memory to hold both the target model and the draft model simultaneously. For example, you can pair Llama 3 70B as your target model with Llama 3 8B as your draft model.

You can launch a vLLM server with speculative decoding enabled using the following terminal command:

python -m vllm.entrypoints.openai.api_server 
    --model meta-llama/Meta-Llama-3-70B-Instruct 
    --tensor-parallel-size 4 
    --speculative-model meta-llama/Meta-Llama-3-8B-Instruct 
    --num-speculative-tokens 5

This command configures the server to load the 70B model across four GPUs. It sets the 8B model as the draft assistant and tells the engine to speculate five tokens ahead on each iteration. If the draft model is highly accurate, you can get up to a 2.5x speedup in token generation.

Many enterprise systems show how to use speculative decoding for faster LLM inference to achieve up to a 2.5x speedup on heavy models like Llama 3 70B. However, there is a catch. Speculative decoding increases the memory usage because both models must reside in GPU memory. It also uses more Key-Value cache memory. This means your maximum batch size will decrease, making this technique ideal for reducing latency on individual user requests rather than maximizing bulk offline throughput.

Advanced Tuning for Maximum Performance

To get the highest possible speedup, you must optimize your draft configurations. The first critical factor is the choice of draft model. If the draft model uses a different tokenizer than the target model, the system must translate tokens back and forth. This adds massive compute overhead. It is almost always best to use a draft model that shares the exact same tokenizer as your target model.

Mastering how to use speculative decoding for faster LLM inference also means choosing the right draft model. Look at this table to see how different draft pairings affect real-world speeds:

The second tuning lever is the speculation lookahead length, often called $K$. If you set $K$ too low, you limit the maximum speedup. If you set $K$ too high, the draft model will make guesses that are likely to be rejected, wasting compute power on verification passes that yield no accepted tokens. For most conversational tasks, a lookahead of 4 or 5 tokens provides the sweet spot for latency reduction.

You can also explore draft-less alternatives like Medusa. Instead of using a second, smaller model, Medusa adds multiple extra heads to the target model itself. These heads are trained to predict several steps ahead simultaneously. This removes the need to manage a separate draft model in memory, though it requires specific model architectures that support these custom heads.

Frequently Asked Questions

Does learning how to use speculative decoding for faster LLM inference require special hardware?

No, you do not need specialized hardware. Any GPU capable of running your target model can run speculative decoding, provided you have enough remaining VRAM to load the smaller draft model alongside it.

Can I learn how to use speculative decoding for faster LLM inference with closed-source APIs like OpenAI?

You cannot configure this yourself on closed APIs because the engine must run directly on the hosting infrastructure. However, providers like OpenAI and Anthropic already use speculative decoding behind the scenes to make their commercial endpoints faster and cheaper for users.

What happens if the draft model is completely wrong?

If the draft model makes an incorrect guess, the target model simply rejects those tokens during the verification pass. The generation remains perfectly accurate, but the system temporarily drops back to standard generation speeds for that step.