← Back to blog

Why muP makes hyperparameters transfer across scale

June 24, 2026 · 21 min read

Abstract

One of the least glamorous bottlenecks in large language model research is hyperparameter search. Learning rates, initialization scales, and scheduler choices that look harmless in a 50M parameter transformer can fail badly at 5B. The narrow question in this memo is why maximal update parameterization, or muP, often makes those choices transfer across width. The core claim from Yang et al. is that hyperparameter instability is not only a nuisance of scale; it is a consequence of using a parameterization where different parts of the network learn at incompatible rates as width grows. muP changes the scaling rules so that feature updates, logits, and residual pathways remain balanced in the wide-model limit. That is why tuning on a small proxy can predict what will work on a larger target. The deeper lesson is that optimizer settings are only meaningful relative to a scaling regime. muP matters because it reframes hyperparameter transfer as a property of training dynamics, not as folklore about picking a conservative learning rate.

Related Work

The primary source is Yang et al. (2022), "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer." That paper introduces the practical muTransfer recipe: parameterize the target network in muP, tune on a smaller model, and reuse those hyperparameters on the larger one without another expensive sweep. The empirical headline is strong because it was tested on exactly the settings people care about: transformers and large-scale language-model-style pretraining.

The conceptual precursor is Yang and Hu (2021), "Feature Learning in Infinite-Width Neural Networks." That work argues that common parameterizations either collapse toward lazy kernel behavior or produce badly imbalanced learning as width grows. The point is not merely asymptotic elegance. If the infinite-width limit suppresses useful feature updates, then width scaling will scramble the meaning of a fixed learning rate. muP emerges from that broader attempt to preserve feature learning rather than freezing the network into a near-linear regime.

A useful later extension is Blake et al. (2024) on u-muP, which combines muP with unit scaling to make implementation more robust, especially in low precision. The broader message is that the original idea was right but operationally fiddly: real training stacks still need careful engineering around initialization, width definitions, and numerical ranges.

Method/Mechanism

Why do hyperparameters stop transferring when width changes? In standard parameterization, different tensors effectively see different update magnitudes as the network gets wider. Some layers become too lazy, barely changing their learned features; others become too reactive, so the same nominal learning rate causes outsized functional movement. The optimizer step is the same in parameter space, but not in function space.

muP addresses this by choosing width-dependent scaling rules for initialization and learning rates so that updates remain order-one in the parts of the network that matter. In loose terms, it tries to keep the network in a regime where increasing width adds representational capacity without changing the basic "meaning" of an optimizer step. That is why a sweep on a small proxy model can stay predictive: the proxy and the target are no longer different optimization problems in disguise.

There is also a useful theoretical contrast with NTK-style scaling. NTK parameterizations are designed to stabilize training in the infinite-width limit by making the model behave almost linearly around initialization. That can make the dynamics analytically clean, but it downplays feature learning. muP instead aims for maximal feature learning while preserving stable large-width behavior. In other words, it tries to keep the network plastic rather than merely well behaved.

For transformers, that matters because widening attention and MLP channels should ideally give the model more room to learn richer features, not force a retuning of every optimizer knob. muP says that if the scaling is chosen correctly, width should mostly change capacity, while good hyperparameters should remain close to invariant. That is the mechanism behind zero-shot transfer across width.

Key Findings

Two case studies make the argument concrete:

Four crisp insights follow:

One alignment-adjacent implication is easy to miss. Safety interventions often depend on comparing many training runs under small recipe changes. If width scaling changes the interpretation of the optimizer, then observed safety effects may partly reflect accidental optimization-regime shifts rather than the intervention itself. muP offers a cleaner way to compare scaling experiments because it reduces one source of confounding.

Limitations

muP is not magic. First, its cleanest promise is widthwise transfer, not universal transfer across depth, data mixtures, architectures, optimizers, or regularization choices. If the larger model differs in more ways than width alone, hyperparameter portability can still fail. Second, the theory is more elegant than the implementation. Real training stacks need explicit base shapes, consistent width definitions, and careful handling of tied embeddings and residual branches. That friction partly explains why muP is respected but not yet universal.

There is also a scope issue. The central evidence shows that many optimal hyperparameters become more stable, not that all of them do. Scheduler length, batch size, regularization, and data curriculum can still interact with scale in ways muP does not erase. Finally, some of the broader tensor-programs intuition remains hard for practitioners to audit. When a method depends on subtle scaling identities, debugging mis-specification can become difficult enough that teams retreat to brute-force sweeps anyway.

Future Directions

The most important next step is broadening the invariance story beyond width. Large language model practice now changes depth, sequence length, optimizer families, precision, and MoE structure all at once. A parameterization that handles width but not those other axes still leaves a lot of tuning on the table. Recent work on Depth-muP and optimizer-specific extensions suggests progress, but it is not yet a single unified recipe for modern transformer stacks.

Another direction is interpretability of optimization itself. muP says that good scaling preserves comparable feature updates across model sizes. That invites a more direct empirical question: can we measure that invariance in activation space or residual-stream geometry, instead of inferring it from loss curves and validation sweeps? If yes, hyperparameter transfer could become easier to diagnose and perhaps partially automatable.

Open question: can we derive a scaling recipe that preserves hyperparameter transfer simultaneously across width, depth, optimizer choice, and sparse architectures without reintroducing a lazy-learning regime?

Summary

muP works because it treats width scaling as an optimization-dynamics problem rather than a purely architectural one. Yang et al. showed that, with the right scaling rules, a small proxy transformer can predict learning-rate choices for a much larger target. The mechanism is that feature updates remain balanced instead of drifting toward either instability or laziness as width grows. The broader lesson is that hyperparameters are not portable by default; they become portable when the parameterization makes model size preserve the meaning of a training step.

References