← Back to blog

Why simple weight arithmetic can merge fine-tuned language models

August 9, 2026 · 22 min read

Abstract

A narrow but increasingly important question in LLM engineering is why two independently fine-tuned models can sometimes be combined by something as crude as averaging weights or adding task vectors. On its face, that should not work. Neural-network parameters are highly non-identifiable, optimization is non-convex, and separately trained models need not use the same internal coordinates. Yet the model merging literature keeps finding islands where simple weight arithmetic works surprisingly well. My reading is that these successes happen when fine-tuning mostly moves a large pretrained model inside a shared low-loss basin, and when the task-specific updates are both small relative to the base model and sufficiently aligned in sign and support. Under those conditions, arithmetic in weight space is not random interpolation between incompatible solutions; it is a cheap way of combining nearby behavioral edits. Merging breaks when updates conflict, when permutation symmetries leave units misaligned, or when averaging falls between basins rather than inside one.

Related Work

The primary source is Ilharco et al.'s Editing Models with Task Arithmetic (2023). Its central claim is that the difference between a pretrained checkpoint and its fine-tuned descendant can be treated as a task vector in weight space. Those vectors can then be negated, added, or combined analogically, and the resulting checkpoint often inherits the expected behavior. That is the clearest empirical statement of the phenomenon this memo is about: model edits can sometimes compose linearly even though the underlying network is nonlinear.

A useful geometric precursor is Wortsman et al.'s Model soups (2022). Their setting is slightly different: average multiple fine-tuned checkpoints and obtain a single model that often beats the best individual one without inference-time ensemble cost. The key observation is that large pretrained models fine-tuned under different hyperparameters often remain in a single low-error basin, so interpolation can stay on a low-loss path.

Two important qualifications come from Ainsworth et al.'s Git Re-Basin (2023) and Yadav et al.'s TIES-Merging (2023). Git Re-Basin shows that some apparent incompatibility is just a symmetry problem: two networks can implement nearly the same function while using permuted internal units, so naive averaging mismatches corresponding features. TIES-Merging shows a second obstacle: merged updates interfere when many parameters are tiny or move in opposite directions. Trimming small updates and resolving sign conflicts improves the arithmetic.

Method/Mechanism

The mechanism is easiest to understand by writing each fine-tuned model as base weights + task-specific delta. Suppose model A is the base model plus an update that specializes it for summarization, while model B is the same base plus an update for instruction following. If those deltas mostly modify disjoint or compatible parts of parameter space, adding or averaging them can preserve both improvements. The pretrained checkpoint acts as a stable reference frame, and the fine-tuning deltas behave like local edits around that anchor.

Why is this plausible in large language models? Because modern fine-tuning is often small compared with pretraining. The base model already contains broad lexical, syntactic, and world-model structure. Fine-tuning often nudges readout preferences, calibrates style, or reweights existing pathways rather than inventing new circuitry. When downstream tasks reuse that shared substrate, their updates can stay near one another in function space.

Three conditions seem to matter most. First, the checkpoints need a shared anchor, usually the same pretrained initialization. Second, their solutions must remain in a compatible basin so that linear interpolation does not cross a high-loss barrier. Third, the deltas must avoid severe interference. Git Re-Basin says that even functionally similar models can be misaligned by hidden-unit permutations, so "same basin" may only become visible after symmetry alignment. TIES says many coordinates should not be merged uniformly because tiny or sign-conflicted updates behave more like noise than reusable task edits.

What works is structured arithmetic around a common pretrained point. In that regime, merging approximates composing several small edits on top of one stable world model.

Key Findings

Two case studies make the mechanism concrete:

Four crisp insights follow:

For alignment, this matters because merging can combine safety tuning, task specialization, and stylistic control without retraining from scratch. But overlapping safety and capability edits can silently degrade either one.

Limitations

The cleanest limitation is that the strongest geometric explanations are still partly indirect. Model soups and task arithmetic show that certain interpolations and delta compositions work, but they do not by themselves prove a universal theory of LLM loss landscapes. "Single basin" is a good local description in some pretrained-and-fine-tuned settings, not a blanket statement about all independent language-model training runs. Much of the strongest evidence also came from classification, vision-language, or moderate-scale NLP settings rather than today's largest instruction-tuned LLMs. The safer claim is local: some fine-tuning deltas are composable around a shared base checkpoint under the right conditions.

Future Directions

The most useful next step is predictive diagnostics: before merging, can we estimate whether two fine-tuned LLMs are geometrically compatible, which layers will interfere, and whether safety-relevant behavior will survive composition? The field needs lightweight measures of basin compatibility, update sparsity, sign conflict, and representation alignment that predict merge quality before expensive evaluation.

Open question: can we build a reliable pre-merge criterion that tells us when two language-model updates correspond to separable capabilities that will compose cleanly, versus competing rewrites of the same internal mechanism?

Summary

The best current answer to why simple weight arithmetic can merge fine-tuned language models is that many downstream updates are small, structured edits around a common pretrained anchor. When those edits stay inside a shared low-loss basin and avoid severe sign or symmetry conflicts, averaging or adding them can preserve capabilities at once. Model soups makes the basin story vivid, task arithmetic shows that edits can compose linearly, Git Re-Basin explains alignment, and TIES shows how interference can be corrected. Merging works when fine-tuning preserves modularity, and fails when capabilities are fighting over the same parameters.

References