← Back to blog

Pre-LN vs Post-LN: why layer norm placement stabilizes Transformers

February 17, 2026 · 13 min read

Abstract

This memo asks a narrow but practical question: why does the placement of layer normalization (pre-LN vs post-LN) so strongly affect optimization stability in deep Transformers? The short answer is that layer norm placement controls how residual branches amplify parameter updates. In post-LN architectures, the residual stream is normalized after each sublayer, which can make early training dynamics sensitive to learning rates and cause gradient explosions near the output. Pre-LN moves normalization into each sublayer, letting the residual pathway act as a stable identity map at initialization. I review the mean-field and amplification analyses that justify this behavior, then connect them to two concrete engineering fixes: pre-LN training without warmup, and DeepNorm-style residual scaling that recovers post-LN performance at extreme depth.

Related Work

The canonical analysis is Xiong et al. (ICML 2020), which explains the learning rate warmup requirement in post-LN Transformers and shows that pre-LN yields well-behaved gradients at initialization. Liu et al. (EMNLP 2020) study training instability through an amplification lens, arguing that residual branches can either over-amplify or under-utilize updates depending on initialization and normalization choices. ReZero (Bachlechner et al., 2020) proposes a complementary idea: explicitly gate residual branches with a zero-initialized scalar to preserve dynamical isometry early in training. Finally, DeepNet (Wang et al., 2022) introduces DeepNorm, a residual scaling scheme that recovers post-LN performance while keeping signal propagation stable even at hundreds or thousands of layers.

Method/Mechanism

The core difference between pre-LN and post-LN is where normalization sits relative to the residual branch. In post-LN, each block computes x + Sublayer(x) and then normalizes. In pre-LN, the sublayer consumes LayerNorm(x) and the residual is added afterward. This tiny change alters the Jacobian of the block. Post-LN effectively couples the residual and the sublayer output inside a normalization operation, which means the gradient seen by upstream layers can be highly sensitive to the scale of the sublayer output. Pre-LN keeps the residual stream closer to the identity function at initialization, providing a stable gradient highway.

Xiong et al. formalize this intuition using a mean-field analysis of gradients at initialization. They show that in post-LN Transformers, the expected gradient magnitude grows with depth near the output layers, forcing practitioners to use warmup to avoid early divergence. In pre-LN, the gradient scale remains more uniform across depth, so the same learning rate can be used from step one. Liu et al. complement this with a perturbation perspective: the residual branch can amplify small parameter updates, and the balance of that amplification depends on how normalization interacts with the residual pathway.

Key Findings

Two concrete case studies help ground the theory:

From these analyses, several crisp insights emerge:

The overarching theme is that training stability is less about a single “correct” architecture and more about how normalization and residual pathways shape signal propagation over depth.

Limitations

Most analyses focus on initialization or early training. They explain why pre-LN is more stable but say less about late-training dynamics, where post-LN can sometimes yield better validation metrics. The mean-field results assume idealized conditions (e.g., random weights and simplified distributions), while real LLM training includes adaptive optimizers, dropout, and weight decay. DeepNet and ReZero demonstrate that residual scaling matters, but these techniques are mainly evaluated on translation or medium-scale language modeling rather than massive instruction-tuned LLMs.

Future Directions

A promising direction is to combine pre-LN stability with post-LN performance through adaptive normalization schedules. Another is to integrate residual scaling with optimizer dynamics to make the amplification analysis predictive of actual training curves. Finally, it would be valuable to map how pre-LN versus post-LN changes the interpretability of residual-stream features, since the normalization choice could alter linear probe behavior.

Open question: Can we design a layer norm or residual scaling scheme that provably keeps gradients well-conditioned throughout training while matching post-LN’s final accuracy at very large scale?

Summary

Layer norm placement is a deceptively small design choice that reshapes gradient flow in deep Transformers. Post-LN makes residual pathways sensitive to scale, requiring warmup or careful initialization, while pre-LN preserves an identity path that stabilizes optimization. DeepNorm and ReZero show that explicit residual scaling can keep post-LN viable at extreme depth. The broader lesson is that training stability is largely about controlling how residual branches amplify updates. That framing connects formal analyses to practical engineering choices and still leaves room for a hybrid approach that offers stability without sacrificing performance.

References