← Back to research notes

Why long-context LLMs get lost in the middle

August 23, 2026 · 23 min read

Abstract

The narrow question in this memo is why language models that can technically ingest very long prompts still retrieve relevant information unreliably when it sits in the middle of the context. The best current answer is not simply that long contexts are hard. It is that many models allocate attention and trust unevenly across position: tokens near the beginning benefit from primacy, tokens near the end benefit from recency, and middle tokens are comparatively underused unless the model or prompt is explicitly adjusted. The result is a characteristic U-shaped retrieval pattern. That pattern matters because it means context-window size and context-window usefulness are different things. A model can advertise 100k or 1M tokens yet still fail at the practical task of finding and using the one passage that matters when it is buried halfway through.

Related Work

The primary source is Liu et al.'s Lost in the Middle: How Language Models Use Long Contexts (2023). It provides the cleanest baseline observation: performance on multi-document question answering and synthetic key-value retrieval often peaks when the relevant evidence appears at the start or end of the prompt, and drops sharply when the same evidence is moved to the middle. The paper matters because it reframes the long-context problem from "can the model fit more tokens?" to "how does the model distribute usable attention over position?"

Two auxiliary papers sharpen that interpretation. Hsieh et al.'s Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization (2024) argues that the failure is tied to an intrinsic U-shaped attention bias and shows that calibration can improve middle retrieval as well as downstream RAG quality. Zhang et al.'s Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding (2024) instead emphasizes the role of positional encoding, especially RoPE-style distance effects that decay badly over long spans. Taken together, these papers suggest that "lost in the middle" is not one bug but the combined consequence of architectural priors, training distribution, and inference heuristics.

Method/Mechanism

A useful mental model is that long-context usage requires two separate capabilities. First, the model must preserve enough information about distant tokens for them to remain representationally available. Second, it must allocate attention toward those tokens when they are actually relevant to the query. Many current LLMs are weaker on the second step than marketing about context length suggests.

Why do the edges win? Primacy can come from training and formatting conventions: instructions, task framing, and demonstrations often appear early, so models learn that early tokens are globally important. Recency comes from the ordinary mechanics of autoregressive decoding, where the most recent tokens are often easiest to condition on and most predictive of the next one. Middle tokens enjoy neither advantage. If positional encoding also weakens effective discrimination over long ranges, the model can retain a blurry memory that "something relevant was somewhere back there" without reliably routing attention to the right span.

This explains why simply scaling context windows is insufficient. If a model's positional bias remains U-shaped, then adding more room may just create a larger low-salience middle. It also explains why retrieval quality can improve when documents are reordered, chunked differently, or duplicated near the end. Those tricks do not necessarily increase knowledge; they reposition evidence into places the model already treats as trustworthy.

Key Findings

Two concrete cases make the mechanism visible:

Four crisp insights follow:

Hsieh et al. add an important extension: if positional attention bias is calibrated, performance gains appear not only on synthetic retrieval but also in retrieval-augmented generation. That suggests the issue is operationally important for real systems, not just a neat benchmark artifact. If a RAG system concatenates many passages and trusts the model to find the right one, it may underperform simply because the winning passage was inserted into the least favored region of the prompt.

Limitations

The evidence base still has boundaries. Much of it comes from controlled reorderings, synthetic lookup tasks, or specific long-context benchmarks rather than the full diversity of production agent workloads. Some newer models may also reduce the problem through better positional schemes, continued long-context training, or retrieval-aware finetuning. So "lost in the middle" should not be treated as a universal constant. It is better understood as a recurring failure mode that remains easy to trigger. A second limitation is explanatory granularity: current papers can show the bias and improve it, but they do not fully decompose how much comes from positional encoding, attention competition, instruction formatting priors, or supervised fine-tuning habits.

Future Directions

The strongest next step is to treat long-context use as a routing problem rather than a storage problem. That means combining architectural fixes, such as better positional calibration, with evaluation setups that explicitly vary where evidence appears. It also means designing retrieval systems that are position-aware instead of assuming that concatenation is neutral.

Open question: can we remove middle-position bias in long-context LLMs without weakening the short-context priors that make instruction following and local coherence work well near the beginning and end of prompts?

Summary

The best explanation for why long-context LLMs get lost in the middle is that their usable attention is distributed unevenly over position. Beginning and end tokens benefit from strong learned priors and easier access, while middle tokens are more likely to be ignored unless prompt structure or model design actively compensates. The consequence is practical: context length alone is a misleading capability measure. What matters is whether relevant evidence remains reachable when it appears anywhere in the prompt, especially in the least privileged region. Until that changes, many long-context systems will still be better described as edge-biased than truly position-robust.

References