← Back to blog

Why label smoothing helps beam search but hurts distillation

August 16, 2026 · 22 min read

Abstract

Label smoothing looks like a tiny training tweak: replace a one-hot target with a mostly-correct distribution that leaves a little mass on the wrong classes. But the consequences are not tiny. The narrow question in this memo is why that trick often improves Transformer behavior at inference time, especially in sequence generation with beam search, while simultaneously making the trained model a worse teacher for knowledge distillation. My reading of the literature is that label smoothing helps because it prevents logits from becoming unnecessarily extreme, which improves calibration and makes sequence-level search less likely to chase brittle token-level overconfidence. The same mechanism also throws away some of the fine-grained similarity structure among wrong answers, the "dark knowledge" that a student needs when learning from a teacher's full probability distribution. So the method is not just generic regularization. It changes what information the model keeps in its logits.

Related Work

The primary source is Müller, Kornblith, and Hinton's When Does Label Smoothing Help? (NeurIPS 2019). That paper gives the clearest empirical answer to the puzzle. It argues that label smoothing improves calibration and can improve beam-search performance, but harms distillation because it compresses within-class variation and weakens the relative probabilities over incorrect classes. The paper matters because it moves the discussion beyond "regularization helps" and asks what specific information is being removed from the output distribution.

Two useful precursors frame the mechanism. Szegedy et al.'s Rethinking the Inception Architecture (2016) popularized label smoothing as a defense against pathological overconfidence. Vaswani et al.'s Attention Is All You Need (2017) then carried the trick into Transformer sequence modeling, where calibration affects decoding quality rather than only top-1 classification. A later extension from Lukasik et al. (2020) shows that the same machinery interacts nontrivially with label noise, reinforcing the view that label smoothing is best understood as a structured change to the training target, not a vague stabilizer.

Method/Mechanism

With ordinary cross-entropy, the optimization target tells the model to drive the correct token's probability toward one and every other token toward zero. In finite models that ideal is never reached, but the pressure is clear: keep widening the logit gap. Label smoothing relaxes that target. If the smoothing rate is epsilon, the correct class no longer needs probability one; some small mass is deliberately spread over the alternatives. That changes the optimum from "be maximally certain" to "be correct without claiming absurd certainty."

This has two downstream effects. First, the model becomes better calibrated. Probabilities line up more closely with actual correctness frequencies because the training loss no longer rewards extreme confidence for every easy example. Second, the output distribution carries less information about how the model ranks the wrong answers relative to one another. The logits become flatter in precisely the places where a distilled student would have learned useful structure, such as that "dog" is more confusable with "wolf" than with "tractor," or that one translation continuation is almost correct while another is nonsense.

For sequence models, calibration matters because decoding compounds local mistakes. Beam search keeps hypotheses that look best under the model's own token probabilities. If those probabilities are overconfident, search can overcommit early to a brittle path. Label smoothing often helps not because it makes the model uniformly better, but because it makes the model's token scores a better proxy for sequence quality. For distillation, the opposite pressure dominates: the student needs the teacher's full soft distribution, and label smoothing makes that signal less informative.

Key Findings

Two concrete cases make the mechanism visible:

Four crisp insights follow:

For modern LLMs, the immediate lesson is practical. Whenever post-training or fine-tuning pipelines use smoothed token targets, they are making an implicit trade: better-behaved confidence and possibly better decoding against weaker probability structure for downstream imitation or compression.

Limitations

The cleanest limitation is domain shift between the classic evidence base and today's largest decoder-only LLMs. Müller et al. analyze classification and translation settings, not frontier chat models trained with massive pretraining corpora, preference tuning, and tool-augmented decoding. So the mechanism should be treated as a strong explanatory template, not a fully established law of modern LLM training. Another limitation is that label smoothing interacts with vocabulary size, class imbalance, and target ambiguity. Uniformly moving probability mass away from the gold token may be too crude when some "wrong" tokens are near-synonyms and others are genuinely implausible. The method helps by discarding information, and that means the exact information being discarded matters.

Future Directions

The most interesting next step is to replace uniform smoothing with structure-aware smoothing. Instead of spreading probability mass equally over every alternative token, one could imagine targets that keep calibration benefits while preserving semantically meaningful similarity information for distillation. In language models, that could mean lexical, syntactic, or teacher-informed smoothing schemes rather than a flat uniform prior.

Open question: can we design a token-level smoothing rule for modern LLM training that preserves label smoothing's gains in calibration and decoding while retaining enough dark knowledge in the logits to remain an effective teacher for distillation?

Summary

The best current answer is that label smoothing helps beam search and hurts distillation for the same underlying reason: it reshapes the logit distribution away from brittle overconfidence and toward calibrated uncertainty, but in doing so it removes some of the relative-probability structure among non-gold options. That is good for direct decoding, where overconfident local scores can mislead search, and bad for teacher-student transfer, where those relative scores carry the teacher's richest signal. The method works not by making the model smarter in general, but by choosing which output information to preserve and which to compress.

References