How to Use Model Merging with MergeKit to Build Custom LLMs

The open-source AI community has a fascinating dirty secret.

For more on this topic, you might also want to check out Why the Byte Latent Transformer Will Kill the LLM Tokenizer and How to Use Direct Preference Optimization to Fine-Tune LLMs and Why Group Relative Policy Optimization Is the Future of AI and How LLM Speculative Decoding Doubles Your Generation Speed and How GraphRAG Solves the Biggest Limitations of Traditional RAG.

While tech giants spend millions of dollars training large language models from scratch, independent developers are building top-tier models for the price of a cup of coffee. They are not renting massive clusters of H100 GPUs for weeks. They are not scraping petabytes of new web text. Instead, they are taking existing, high-performing models and blending them together like master blenders combining fine whiskeys.

This technique, known as Model Merging with MergeKit, is changing how developers approach customization. It bypasses the massive compute barrier entirely. If you have a model that excels at writing Python and another that shines at creative storytelling, you no longer have to choose between them. You can merge them. The result is a single hybrid model that retains the strengths of both parent models without needing a single step of backpropagation.

Let us look at how this works under the hood, how you can do it on your own machine, and why this approach is reshaping the open-source landscape.

The Mathematics Behind Model Merging

To understand why Model Merging with MergeKit works so well, we have to look at the geometry of neural network weights.

When an LLM is trained, it navigates a vast landscape of high-dimensional parameter space to find a set of weights that minimizes loss. You might assume that two different fine-tuned versions of the same base model would end up in completely different, incompatible regions of this space. But they do not. Because they share the same starting point, the base model, they remain in the same localized basin of loss. They speak the same structural language.

When you use Model Merging with MergeKit, you aren’t just averaging numbers. Simple weight averaging usually results in a broken model because it ignores the path-dependent nature of training. Instead, the library implements advanced mathematical techniques to resolve conflicts between parameters. The table below outlines the primary methods used to accomplish this.

One of the primary methods is SLERP, or Spherical Linear Interpolation. Instead of drawing a straight line between two weight vectors, which often drags the values through low-performance valleys, SLERP interpolates along a spherical path. This preserves the geometric properties of the vector space, keeping the merged weights within highly functional regions. It is highly effective for combining two models.

But what if you want to combine three, four, or even eight models? That is where TIES comes in. When multiple models are merged, their parameter updates often conflict. One model might want to increase a specific weight, while another wants to decrease it. TIES resolves this by first trimming away the smallest, least significant weight changes. Then, it checks the sign of the remaining changes across all models and elects the dominant direction. Finally, it merges only the non-conflicting parameters.

Another popular algorithm is DARE. It randomly resets a massive portion of the fine-tuned weights, sometimes up to 90 percent, back to their base model values. It then rescales the remaining weights to compensate. This sounds destructive. Yet, it prevents the merged model from becoming a chaotic mess of overlapping signatures. It keeps the core architecture clean.

Step-by-Step Guide to Your First Merge

Getting started with Model Merging with MergeKit requires surprisingly little hardware.

You do not need an expensive cloud GPU instance to run a merge. Because the process only involves basic tensor mathematics rather than active training, you can merge 7B or 8B parameter models on a standard laptop with 16 gigabytes of RAM. The entire process takes minutes.

This is the beauty of Model Merging with MergeKit; you don’t need an entire cluster of servers to build something incredibly capable. You just need a configuration file.

MergeKit uses simple YAML files to define how models should be combined. Below is a complete, working example of a SLERP merge configuration. This setup blends a coding model with a general instruction model, using Mistral-7B as the foundation.

slices:
  - sources:
      - model: teknium/OpenHermes-2.5-Mistral-7B
        layer_range: [0, 32]
      - model: WizardLM/WizardCoder-Python-7B-V1.0
        layer_range: [0, 32]
merge_method: slerp
base_model: mistralai/Mistral-7B-v0.1
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: float16

Let us dissect this file. The slices block defines which layers we are pulling from each model. Here, we are taking all 32 layers from both parent models. The merge method is set to SLERP. The parameters block is where the weight distribution is decided. The t parameter represents the interpolation factor. A value of 0.5 means an equal blend. By applying filters, we can tell MergeKit to use different blending ratios for different parts of the network. For instance, we might want to favor the coding model’s attention layers while keeping the general model’s feed-forward network layers.

To run this merge, install the library using pip. Run this command in your terminal:

pip install mergekit

Once installed, you can execute the merge using the command-line utility. Ensure your YAML configuration is saved as config.yml, then run:

mergekit-yaml config.yml ./my-merged-model --copy-tokenizer

The tool downloads the necessary tensors directly from Hugging Face, performs the calculations in system memory, and writes the new model weights to your local disk. If you are short on system memory, you can add the --low-cpu-mem flag to write tensors directly to disk during computation.

Advanced Merge Architectures and Frankenmerging

Another advanced avenue of Model Merging with MergeKit is the creation of what the community calls Frankenmodels, or passthrough merges.

This technique does not blend weights. Instead, it stacks layers from the same model or different models to artificially increase the total parameter count. For example, you can take a highly optimized 7B model and copy some of its middle layers to turn it into a 10.7B model. This works because the middle layers of a transformer model are generally responsible for high-level reasoning and abstract concepts. By repeating these layers, you give the model more processing steps to think through complex prompts. It behaves like a deeper network without the massive cost of training a 10B model from scratch.

Here is what a passthrough configuration looks like:

slices:
  - sources:
      - model: mistralai/Mistral-7B-v0.1
        layer_range: [0, 24]
  - sources:
      - model: mistralai/Mistral-7B-v0.1
        layer_range: [8, 32]
merge_method: passthrough
dtype: float16

This configuration creates a 48-layer model from a 32-layer base. The first slice takes layers 0 through 24. The second slice takes layers 8 through 32. The overlapping layers are repeated, creating a deep network that can hold more complex representations. The famous Solar-10.7B model was built using this exact structural philosophy, proving that structural manipulation can rival traditional scaling.

The Hard Truths of Merging

While Model Merging with MergeKit offers an incredibly cheap way to build custom models, it is not a perfect solution.

Sometimes, merging two models results in a damaged output. The model might start repeating words or outputting complete gibberish. This is known as merge degradation. When you blend weights, you risk disrupting the fragile internal representations the models learned during training. A model that was perfectly aligned using reinforcement learning might lose its safety guardrails or its formatting consistency when merged with an unaligned model.

To combat this, you must run rigorous evaluations. Do not rely on simple testing. Use tools like the LM Evaluation Harness to test your merged model against standard benchmarks like MMLU or GSM8K. If you notice a drop in reasoning scores, you need to adjust your YAML configuration. Try changing the interpolation weights or switching from SLERP to TIES to resolve parameter conflicts.

This makes Model Merging with MergeKit an iterative science. You will rarely get a perfect model on your first try. Often, the best merged models undergo a very brief phase of post-merge training. Running a few hundred steps of low-learning-rate supervised fine-tuning on a clean dataset can smooth out the mathematical rough edges. This process acts like a light sanding on a freshly joined piece of wood, settling the merged parameters into a truly stable state.

Frequently Asked Questions

How does Model Merging with MergeKit differ from traditional fine-tuning?

Traditional fine-tuning updates a model’s weights by calculating gradients over a new dataset, requiring significant GPU compute power. Model Merging with MergeKit combines existing weights from pre-trained models using mathematical interpolation, requiring no training data and minimal compute.

Can I merge models with different base architectures, like Llama and Mistral?

No, you cannot merge models with completely different architectures because their layer dimensions and attention mechanisms do not align. Merging requires the models to share a common ancestral base or have identical structural configurations.

Can I run Model Merging with MergeKit on a consumer GPU?

Yes, you can run it on a consumer GPU, but you do not even need one. Because merging is a mathematical operation rather than a training process, it can run entirely on a CPU with sufficient system RAM.