← Back to research notes

Why sampling token boundaries can improve robustness

October 12, 2026 · 7 min read

Abstract

Can a language model benefit from seeing the same sentence with different token boundaries? This review examines a precise intervention: sample alternative subword segmentations during training, then use a deterministic tokenizer at inference. The underlying text stays fixed, but the model encounters different sequences of token identifiers. The research suggests that this can reduce dependence on a single segmentation. The strongest evidence reviewed here comes from machine translation; extending it to large autoregressive models remains an empirical question. The useful distinction is between covering a string with a vocabulary and learning to interpret its possible encodings.

Related Work

The highly cited foundation is Sennrich, Haddow, and Birch’s subword translation work. It adapted byte-pair encoding (BPE) to represent rare words using smaller units, with gains on English–German and English–Russian translation. Its central contribution is vocabulary coverage without requiring a separate symbol for every word. [1]

Kudo’s Subword Regularization supplies the central training idea: resample segmentations on the fly using a unigram segmentation model. It reports improvements particularly in low-resource and out-of-domain translation. Provilkov, Emelianenko, and Voita subsequently introduced BPE-dropout, making stochastic segmentation compatible with an existing BPE vocabulary. [2] [3]

A separate comparison by Bostrom and Durrett found that unigram tokenization matched or exceeded BPE on downstream tasks for masked language models in English and Japanese. That establishes that tokenizer choice can matter beyond translation. It does not isolate the benefit of sampling boundaries, because changing the vocabulary construction method is a different intervention. [4]

Method/Mechanism

Ordinary BPE repeatedly applies learned merges to adjacent units. BPE-dropout randomly suppresses candidate merges during segmentation, producing alternative token sequences while retaining the vocabulary; inference uses ordinary BPE. Kudo’s approach instead assigns probabilities to segmentations using products of unigram token probabilities and samples from that distribution. These are different sampling procedures with a shared purpose. [3] [2]

Example 1: one string, two learning problems. Consider an illustrative vocabulary containing “play”, “ing”, “playing”, and the required characters. The text “playing” could be represented as [playing] or [play, ing]. Assume both segmentations are available; this example is not a claim about a particular production tokenizer.

For next-token training, the first representation asks the model to predict one identifier from the preceding context. The second asks it to predict “play”, then “ing” after a longer prefix. Detokenization yields the same text, but the loss terms and computation differ. Successful prediction of [playing] alone does not logically require successful prediction of [play, ing]. Sharing character content does not force independently learned token embeddings to share useful behavior.

To sharpen that distinction, consider an illustrative source-only objective with a fixed target y. Let q(s|x) assign probabilities to segmentations s of input text x. Training minimizes the average loss Es∼q[−log p(y|s)]. This is a simplified instance of sampling over segmentations, rather than a description of a new experiment. Kudo also samples target segmentations. [2]

Here is an original calculation. Suppose two source segmentations are equally likely and a model gives the correct target probabilities 0.9 and 0.1. Its average negative log probability is about 1.204 nats. A model assigning 0.5 under both receives about 0.693 nats. Both have the same arithmetic-average probability, 0.5, yet the average log loss prefers the second. Poor performance on one encoding cannot be fully hidden by good performance on another.

This also separates the average loss from −log Es[p(y|s)], which is 0.693 for both models. Jensen’s inequality makes the former at least as large as the latter. Sampling training encodings therefore should not casually be described as exact inference that averages over every encoding. Nor does the objective guarantee identical hidden states: it constrains predictions through training examples.

Key Findings

Case study 2: translation under synthetic misspellings. In Table 5 of the BPE-dropout paper, WMT14 English–German BLEU rises from 27.41 to 28.01 on original input and from 24.45 to 26.03 on corrupted input. The corruption modifies each source word with probability 10%, inserting, deleting, or substituting a character. Training did not introduce these misspellings. This supports transfer from segmentation variation to that particular noise test. [3]

My interpretation is bounded: the intervention helps with a perturbation that can alter token boundaries, but the result does not prove which internal representation caused the gain. Four practical insights follow from the mechanism and evidence:

Limitations

The translation evidence concerns conditional generation with an encoder and decoder. A decoder-only model trained on web text has a different objective and data regime. Treating these results as a recipe for frontier-scale pretraining would skip the experiment that matters.

There is also a resource confound. Splitting a token into several pieces increases sequence length. At fixed token count, a training run can then see fewer characters; at fixed text exposure, it can spend more computation. Either choice is defensible, but they answer different questions. Per-token perplexity also becomes difficult to compare when prediction units change. Evaluation should retain a common text unit and specify whether likelihood uses canonical tokenization or sums over alternative encodings.

Finally, sampling cannot supply missing facts or guarantee compositional understanding. It changes how existing text is presented. A model could improve through more useful embedding updates, broader prefix exposure, or another optimization effect without acquiring the human-interpretable mechanism suggested by the word “composition”.

Future Directions

A decisive LLM study would hold the raw corpus, architecture, and vocabulary fixed, and vary only segmentation sampling. Report both matched-compute and matched-text comparisons. Test clean text, controlled spelling errors, and held-out word formations separately, while recording sequence lengths and throughput.

To test the proposed mechanism directly, encode identical held-out strings in several valid ways and measure disagreement in predictions over a common set of continuation strings. That would distinguish sensitivity to token boundaries from ordinary task error. Repeated seeds and uncertainty intervals would help determine whether a small robustness gain survives training variability.

Open question: At fixed training compute, can stochastic segmentation reduce decoder-only LLM sensitivity to alternative encodings without sacrificing performance on clean text?

Summary

Subword sampling changes the prediction problem while preserving the text. Translation studies provide evidence that this can improve robustness; the loss calculation explains why performance across encodings matters. The next step for LLMs is a controlled test that accounts for both text exposure and computational cost.

References