Abstract
Can exposing a language model to its own mistakes teach it the wrong dependencies? Scheduled sampling makes this question precise: during training, it sometimes replaces a reference token with a generated token while retaining the original next-token target. The intended benefit is resilience during generation. The danger is that the target no longer belongs to the prefix. This memo examines that tension through foundational sequence-modeling research and a worked probability example. The lesson for LLM training is a testable design constraint: changing the distribution of contexts also requires checking whether the supervision remains valid in those contexts.
Related Work
Bengio et al. introduced scheduled sampling in 2015, gradually reducing reliance on reference histories. Their recurrent models improved on several sequence tasks. Huszár subsequently showed that the idealized objective is statistically inconsistent: even unlimited data and sufficient capacity need not recover the true joint distribution. These are compatible findings because task performance and distribution recovery are different criteria. [1] [2]
An earlier foundation is DAgger, from Ross, Gordon, and Bagnell. It collects states visited by the learner, obtains expert actions there, and aggregates those examples. Its guarantees require assumptions about the learning reduction and expert feedback. The useful comparison is the labeling operation: encountering a new state is paired with asking what to do in that state. [3]
Method/Mechanism
Teacher forcing minimizes the sum of negative log probabilities of reference tokens given reference prefixes. By the probability chain rule, that sum is the negative log probability of the complete reference sequence. Scheduled sampling instead flips a coin to choose a reference or model-generated token for the evolving history. The original experiments did not backpropagate through the discrete sampling decisions. [1]
Huszár's two-token analysis isolates the problem. When the first token is independently sampled from the model, the retained second token comes from an unrelated data example. As reference-token probability approaches zero, the idealized optimum becomes the product of the data marginals, discarding their dependency. This is a statement about the learning target, not merely noisy gradients or insufficient optimization. [2]
Example 1: a binary copying task. The following calculation is an illustrative derivation, not a reported experiment. Suppose a dataset contains only two-token sequences 00 and 11, equally often. A correct model samples a fair first bit and copies it. Assume its first-token distribution is already correct and held fixed. Let ε denote the probability of retaining the reference first bit during training.
Among examples whose presented prefix is 0, the retained-reference component always has target 0. In the replacement component, the independent sampled prefix says nothing about the reference target, which is equally likely to be either bit. The cross-entropy optimum therefore predicts a matching second bit with probability (1 + ε) / 2. At ε = 1 it copies perfectly; at ε = 0 it predicts a fair coin after either prefix.
This calculation gives a concrete diagnostic. At ε = 0.5, the learned conditional assigns 75% probability to copying. Free generation then produces invalid unequal pairs 25% of the time, despite a training dataset containing none. At ε = 0, all four pairs have probability 25%. The model's first-token marginal remains exactly right throughout. Evaluating only token frequencies would therefore miss the failure; evaluating the relation between tokens exposes it immediately.
Key Findings
Case study 2: image captioning. In Bengio et al.'s MSCOCO development evaluation, single-model BLEU-4 rose from 28.8 to 30.6 with scheduled sampling. Always sampling from the model instead scored 11.2. Those results support a benefit from the curriculum in that experimental setting; they do not establish consistent estimation. [1]
Four insights follow from separating those results from the worked example:
- Correct marginals can hide broken relations. In the binary task, individual token frequencies provide no warning that copying has deteriorated. Dependency-sensitive evaluation is essential.
- A better task score does not certify a probability model. Caption overlap and recovery of the training distribution answer different questions. Success on one leaves the other unresolved.
- Context changes require target checks. A useful audit records the presented prefix and asks whether the retained target is still appropriate, rather than checking only how often corruption occurred.
- The schedule is part of the intervention. Reporting a method name without its replacement rate and stopping point hides how much contradictory supervision the learner actually received.
Limitations
The copying calculation deliberately fixes the first-token marginal and assumes sufficient capacity to fit each conditional separately. Parameter sharing, finite data, external conditioning, and optimization can alter practical behavior. Huszár's argument concerns an idealized sampling objective; it does not predict that every finite training run will reach complete independence. [2]
Nor does this review establish the size of exposure bias in contemporary LLMs. The empirical anchor uses recurrent models, not a controlled comparison of large decoder-only transformers. Transferring the target-validity concern is a methodological inference; transferring the reported gains would require new evidence. Synthetic training on complete, coherent sequences also differs from attaching an old continuation to a newly sampled prefix. Those procedures should not be grouped together simply because both use model outputs.
Future Directions
A useful experiment would train matched small transformers on controlled sequence tasks with known dependencies. Hold compute, initialization, and dataset size fixed; vary only prefix replacement and labeling. Compare unchanged reference targets with targets supplied by an oracle for the actual presented prefix. Measure held-out likelihood, invalid-sequence probability, and recovery after deliberate perturbations separately. The binary task supplies an exact baseline before natural-language ambiguity enters.
DAgger motivates the oracle comparison, but language introduces a difficulty: a changed prefix may admit many valid continuations, and an expert action need not identify their full probability distribution. A repair policy and a generative density model therefore need separate evaluation. [3]
Open question: Can a practical LLM training procedure learn recovery from generated prefixes while preserving the correct distribution over valid continuations, without an oracle that labels every altered context?
Summary
Scheduled sampling makes the validity of supervision after a context change impossible to ignore. Its historical task gains deserve recognition, while the copying example shows exactly how a training modification can erase a dependency. For new proposals, inspect the prefix-target pair and evaluate sequence relations alongside aggregate task scores.
References
- [1] Bengio, Vinyals, Jaitly, and Shazeer (2015). Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. NeurIPS. Primary empirical anchor.
- [2] Huszár (2015). How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? Primary theoretical critique.
- [3] Ross, Gordon, and Bagnell (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS.