Every time a large language model generates a word, it spends the exact same amount of computing power. It does not matter if the model is writing a complex line of code or just outputting a simple comma. The neural network forces every single token through every single layer in its stack. Think of it as using a supercomputer to calculate two plus two. It is incredibly wasteful.
For more on this topic, you might also want to check out Ultimate Guide: Mastering AI's "Consciousness" with Self-Correction and Why Byte Latent Transformers Are the Future of AI Models and Why Group Relative Policy Optimization Is the Ultimate RL Upgrade and Ultimate Guide to Weight-Decomposed Low-Rank Adaptation for LLMs and How to Master Kahneman-Tversky Optimization for Better LLMs.
Engineers at Google DeepMind realized this flaw. They developed a method called Mixture of Depths to fix it. This technique allows models to choose which tokens deserve deep processing and which ones can skip ahead. By skipping unnecessary computations, models can run much faster while using far fewer resources.
How Mixture of Depths Works Under the Hood
To understand Mixture of Depths, think about how standard transformers work. In a normal model, every token visits every attention block and feed-forward network. This is a rigid pipeline. With a Mixture of Depths setup, the model introduces a router at each layer. This router calculates a scalar weight for every token. The weight represents how much the layer needs to process that specific token.
If the weight is high, the token goes through the attention and feed-forward blocks. If the weight is low, the token bypasses the layer entirely. It travels through a simple residual connection instead.
But how does the model decide who gets priority? It uses a top-k routing strategy. The system sets a strict compute budget for each layer. For example, a layer might have a capacity limit of fifty percent. This means only half of the tokens in a sequence can enter the layer. The router ranks all tokens by their predicted weights and selects the top fifty percent. The rest simply skip the layer.
The router operates as a simple linear layer. It takes the hidden state of a token and outputs a single number. This number represents the token’s priority score. Once the model has scores for all tokens in the sequence, it sorts them. It then keeps only the top-k tokens, where k is determined by the capacity limit. For instance, if your prompt is 1000 tokens long and your capacity is 30%, only the 300 tokens with the highest scores will go through the self-attention and MLP blocks. The other 700 tokens are passed directly to the next layer without any modification. This creates a massive shortcut in the network.
How Mixture of Depths Differs From Mixture of Experts
Many people confuse Mixture of Depths with Mixture of Experts. While both use routing mechanisms, they solve entirely different problems. Mixture of Experts divides a layer into multiple specialized sub-networks. Each token is routed to the best expert for its context. However, every token still gets processed by some expert. The total computational load remains high even if the active parameters are limited.
In contrast, Mixture of Depths focuses on skipping computation altogether. Instead of choosing which expert to use, the model decides whether to use any compute at all. This results in direct savings in floating-point operations. Let’s compare them directly to see the differences clearly.
| Feature | Mixture of Experts (MoE) | Mixture of Depths (MoD) |
|---|---|---|
| Primary Goal | Scale model capacity without increasing active parameters | Reduce total computation per token during inference |
| Routing Decision | Which specialized network should process the token | Whether a token needs to be processed by a layer at all |
| FLOPs Saved | Minimal compared to the total parameter size | Up to fifty percent reduction per forward pass |
| Memory Footprint | Requires holding multiple expert weights in memory | Keeps the standard model size but processes fewer steps |
To put this in perspective, think of Mixture of Experts as a large office with specialized departments. When a task comes in, a router sends it to the finance department, the legal department, or the marketing department. Every task still gets worked on by someone. Mixture of Depths is different. It is like having a manager who looks at the incoming tasks and decides that half of them do not need any work at all and can be filed away immediately. This is a much more aggressive way to save time and energy.
Training Challenges and Gradient Flow
In training a Mixture of Depths model, the main challenge is that the top-k selection process is not differentiable. You cannot easily pass gradients back through a sorting operation that picks the top fifty percent of tokens. If the router cannot receive gradients, it cannot learn which tokens are actually important.
DeepMind solved this by using the router weights themselves during training. The system scales the output of the layer by the router’s gating values. This allows gradients to flow backward, helping the router learn which tokens are critical for understanding the context. Once training is complete, we can turn off this scaling and use hard routing for maximum inference speed.
When planning a Mixture of Depths implementation, keep these architectural factors in mind:
- The router capacity ratio determines how many tokens skip each block.
- The placement of routers can be selective, meaning you do not have to put them at every single layer.
- Residual connections must be perfectly aligned to prevent token representation drift.
- Hardware compilation requires padding or specialized masking to handle the variable token paths.
If you are evaluating these models, you might want to inspect how the text output changes when you adjust the compute budget. We can use a Text Diff Checker to compare the outputs of an unoptimized model with its Mixture of Depths counterpart to ensure formatting remains identical. This tool helps you quickly spot if the model is dropping important punctuation or structural elements when skipping layers. Adjusting the compute budget lets you balance speed and accuracy. If you set the budget too low, the model might skip too many layers, causing the output quality to degrade. Finding the sweet spot is key.
One major hurdle when implementing this in practice is batching. GPUs love uniform computations. They perform best when every element in a batch undergoes the exact same operations. When different tokens in a sequence take different paths, it breaks this uniformity. This is known as the dynamic shape problem. To keep the GPU running at maximum efficiency, we must pack the selected tokens into a contiguous tensor. This requires specialized kernel writing. If you do not pack the tensors, the GPU will still allocate memory and compute cycles for the skipped tokens, defeating the entire purpose of the optimization.
Real-World Performance and Efficiency Benefits
Why should AI labs care about this approach? The answer lies in hardware efficiency. DeepMind’s research showed that models using Mixture of Depths can achieve the same performance as standard models while using a fraction of the computational resources. Specifically, they found that a model could maintain high accuracy while skipping up to half of the computations per forward pass.
This translates directly to faster token generation. Because the GPU does not have to compute self-attention for every single token at every layer, the memory bandwidth pressure drops significantly.
Let’s look at the memory implications. In standard autoregressive generation, we must store the Key-Value cache for every single token. With Mixture of Depths, since some tokens skip certain attention layers, we do not need to calculate or store their KV cache values for those skipped blocks. This drastically reduces the memory footprint of the KV cache, which is often the main bottleneck when serving large models to millions of users.
The speedup is especially noticeable during long-form generation. When generating a long report or processing a massive codebase, the accumulation of skipped computations leads to massive time savings. This is why mastering Mixture of Depths is becoming essential for developers who want to deploy large models cost-effectively.
Frequently Asked Questions
Does Mixture of Depths require training a model from scratch?
Yes, you typically need to train the model from scratch or perform extensive fine-tuning. Because the routing decisions affect how gradients flow through the network, you cannot easily apply this technique to an existing pre-trained model without retraining the weights.
How does Mixture of Depths affect the model’s accuracy?
It maintains virtually the same accuracy as standard models. By allowing the network to allocate compute where it is needed most, it prevents performance drops while using up to fifty percent fewer FLOPs per forward pass.
Can you combine Mixture of Depths with Mixture of Experts?
You can combine both methods in a single architecture. This hybrid approach allows a model to first decide whether to process a token at all, and then decide which specialized expert network should handle it if it does.
The Road to Custom Kernels
Engineers looking to deploy these models must focus on hardware-friendly routing implementations. Standard deep learning libraries are built for static tensor shapes, which makes variable sequence lengths per layer difficult to compile efficiently. Developing custom Triton kernels or using specialized compilation frameworks like TensorRT-LLM will be the next step in bringing these theoretical savings into actual production environments.
