What Is the Lost in the Middle Problem in LLMs?
Lost in the middle is the tendency of large language models to use information placed at the beginning or end of a long input context more reliably than the same information placed in the middle. Accuracy plotted against the position of the relevant passage forms a U-shaped curve. The effect is about where content sits inside a single prompt, not about whether that content is correct or current.
Updated
What is lost in the middle?
The name comes from the paper Lost in the Middle: How Language Models Use Long Contexts by Nelson F. Liu and six co-authors, first posted in July 2023 (arXiv:2307.03172) and published in Transactions of the ACL, vol. 12 (2024). The authors tested open and closed models, including GPT-3.5-Turbo, Claude-1.3, MPT-30B-Instruct and LongChat-13B, on two tasks: question answering over a set of retrieved documents, and retrieving a value for a key from a list of JSON key-value pairs.
They found what they called primacy and recency bias. Models did best when the relevant document was first or last, and worst when it was in the middle. In the multi-document setting, GPT-3.5-Turbo's accuracy could drop by more than 20%, and in the worst case, with 20 or 30 documents, it fell below its closed-book score of 56.1%, where the model answers with no documents at all (Liu et al., §2.3). When an input fit in both windows, extended-context versions of the same models scored nearly the same as the standard versions.
The finding matters for retrieval-augmented generation because the setup mirrors it: a retriever returns k passages, and the model must find the one that answers the question.
How lost in the middle works
A worked illustration of the experiment design (the layout is from the paper; the question is illustrative):
- 1.Take a question, such as "Who wrote the opinion in case X?"
- 2.Retrieve 20 passages. One contains the answer; 19 are related distractors.
- 3.Run the prompt with the answer passage at different positions, from first to last (the paper reports positions 1, 5, 10, 15 and 20).
- 4.Plot accuracy by position. The curve is high at both ends and sags in the middle.
Several things shape the effect, according to the same paper. Encoder-decoder models were relatively robust when inputs stayed within the length they were trained on. Placing the question both before and after the documents nearly solved the synthetic key-value task but barely changed multi-document question answering. And retrieving more passages saturated early: going from 20 to 50 retrieved documents improved GPT-3.5-Turbo by only about 1.5% (Liu et al., §1 and §5).
Later work tied the effect to attention. Hsieh et al. found a U-shaped attention bias in which tokens at the start and end of the input receive more attention regardless of relevance, and proposed a calibration method, "found-in-the-middle," that outperformed existing methods on retrieval-augmented generation by up to 15 percentage points across the tasks they tested (ACL Findings 2024). Whether a given newer model still shows the effect is an empirical question, so measure it on your own model and prompt layout rather than assuming either way.
Why lost in the middle matters in real-time systems
Lost in the middle is an in-prompt failure: the right fact was in the prompt and the model did not use it. That differs from the data underneath the prompt being wrong: a balance that trails the latest transaction, a policy that changed after the index was built, or two retrieved records that describe different moments. That second class is a data-layer failure, often called context lag, and no prompt layout repairs it. Better ordering cannot rescue a stale fact, and a fresh, consistent fact can still be lost in the middle of a 30-document prompt.
Agents in production can have both problems at once, so diagnose them separately. If the correct, current fact was in the prompt, look at position, length and reranking. If the fact in the prompt was already wrong when the prompt was assembled, look at how the context was retrieved and how current it was. The two are compared in context rot vs. context lag.
FAQ
Related terms
Context infrastructure is the data layer beneath AI applications and agent frameworks that determines whether the context they consume is fresh and coherent.
Semantic context is the meaning-bearing layer of decision context — aggregations, vector similarity, LLM-derived signals — computed from the same events.
A context gap is the difference between the state of the world an automated decision acts on and the actual state at the moment the decision commits.
Derived context is state computed from events rather than stored directly — aggregates, velocity counts, features, embeddings — that decisions consume.
Decision context is everything an automated decision reads and everything earlier decisions committed, as it stands at the moment the decision executes.
Context distraction is when a context grows so long that a language model leans on it, such as its own past actions, over what it learned in training.
A token budget limits how many tokens an LLM call or agent task may use: how the context window is split, how much output and reasoning, and total spend.