← Back to blog

Why LLMs memorize rare repeated strings

July 8, 2026 · 18 min read

Abstract

A persistent confusion in discussions of language-model memorization is treating it as if models indiscriminately cache arbitrary chunks of their training data. The narrower and more useful question is: what kinds of sequences get memorized exactly enough to be reproduced verbatim? The strongest answer from the literature is that exact memorization is concentrated on strings with three properties: they are rare in semantic meaning but statistically distinctive in form, they often appear more than once in the corpus, and they are easy to identify once a prompt supplies enough matching context. Carlini et al. make this concrete by showing that memorization grows predictably with model size, duplication count, and prompt length. Subsequent work on training dynamics strengthens the point: memorization can appear well before classical overfitting, so it is not just a late-stage pathology. My view is that this matters because it turns "LLMs memorize data" from a vague fear into a specific mechanism. Models are especially good at preserving exact high-entropy surface forms when the training objective repeatedly rewards reconstructing them. That is a statement about gradient signal and data distribution, not about mysterious database-like storage.

Related Work

The primary source is Carlini et al., Quantifying Memorization Across Neural Language Models (2022/2023). Rather than only demonstrating isolated extraction anecdotes, the paper asks what governs extractable memorization and answers with three log-linear relationships: memorization rises with model capacity, with the number of times an example is duplicated in training, and with the amount of context given to the attacker. That framing is valuable because it shifts the conversation from existence proofs to scaling laws.

An earlier anchor is Carlini et al., Extracting Training Data from Large Language Models (2020/2021). That paper established the practical risk by recovering verbatim sequences from GPT-2, including names, contact details, code fragments, and random-looking identifiers. The important lesson was not only that extraction is possible, but that some single-occurrence sequences are so distinctive that a model can still regurgitate them when sampled aggressively.

A useful auxiliary perspective comes from Tirumala et al., Memorization Without Overfitting (2022/2023), which tracks memorization during training. Their result is awkward for a simple train-vs-test story: larger language models can begin exactly memorizing strings long before held-out loss shows obvious degradation. That means classical generalization metrics are not a reliable proxy for privacy or copyright risk.

Method/Mechanism

The mechanism is easiest to describe in terms of token prediction pressure. During pretraining, the model is rewarded for assigning high probability to the next token given a prefix. For ordinary text, many semantically equivalent continuations compete, so the model can generalize by learning broad patterns. High-entropy strings are different. A UUID, email address, boilerplate footer, or exact code line often has one sharply correct continuation and very few plausible alternatives. The training loss therefore pays a premium for reproducing the exact surface form.

Duplication amplifies that premium. If an unusual string appears repeatedly, its gradients align across many updates instead of being washed out by variation. The model does not need to infer abstract meaning; it only needs to carve out a reliable continuation path for a distinctive token sequence. This is why memorization is often more syntactic than semantic. The model is locking onto a narrow lexical trajectory, not necessarily forming a reusable concept.

Prompt context then acts as the retrieval key. Extraction attacks do not usually ask the model to dump its training set from scratch. They provide a prefix that partially matches a memorized example and let autoregressive continuation do the rest. More context disambiguates the target sequence, so the model can commit to the one continuation that training most strongly reinforced. In this sense, memorization is a three-part interaction between data duplication, model capacity, and prefix specificity.

Key Findings

Two case studies make the mechanism concrete:

Four crisp insights follow:

One alignment-adjacent implication is that post-training behavior does not erase pretraining memory by default. Later alignment or chatbot tuning may suppress easy extraction in ordinary use, but the base memorized trajectory can remain latent and sometimes be recovered with sufficiently adversarial prompting. The right mental model is suppression of expression, not guaranteed deletion of storage.

Limitations

The literature is strongest on extractable memorization, not on every possible notion of knowledge storage. A model might encode data in a distributed way that is hard to reproduce verbatim yet still influences outputs. So these papers likely undercount some forms of dependence on the training set even while capturing the most operationally important failures.

There is also a benchmark bias toward exact string reproduction. That makes sense for privacy and copyright risk, but it means the field knows less about fuzzy or mosaic memorization, where a model recombines overlapping training fragments. Finally, corpus duplication is easy to vary in controlled experiments but hard to eliminate in real web-scale data. Deduplication helps, yet near-duplicates, templated boilerplate, and syndicated text still create repeated gradient signals.

Future Directions

One direction is mechanistic: identify where exact string trajectories are stabilized inside modern decoder-only transformers. Are they mostly mediated by specific MLP memories, by attention-based prefix matching, or by a distributed interaction that only looks local at the output layer? Another is data-centric: build better measures of effective duplication that count fuzzy template reuse rather than only exact copies.

A second direction matters for safety practice. If memorization risk is driven by high-entropy strings plus repeated exposure, then mitigation should focus on those objects directly: aggressive deduplication, filtering of secrets, and training objectives that weaken multi-token exact recall without destroying normal language modeling. The research question is no longer whether memorization exists, but which interventions change the scaling curve most cheaply.

Open question: can we predict from dataset statistics alone which training examples are likely to become extractable before training begins, or does the answer depend irreducibly on the interaction between duplication, token entropy, model architecture, and later alignment tuning?

Summary

The most useful way to think about LLM memorization is not "models store random passages," but "exact surface forms with strong repeated gradient signal become retrievable continuations." Carlini et al. quantify that with scaling laws over model size, duplication, and prompt context; the earlier GPT-2 extraction paper shows the practical consequences; and training-dynamics work shows the effect arrives before classical overfitting becomes visible. The result is a narrower but sharper picture: LLM memorization is driven especially by rare, repeated, high-entropy strings, which makes data hygiene and duplication control central levers rather than afterthoughts.

References