A context window is not memory
In May 2025 Philippe Laban and three colleagues at Microsoft and Salesforce published a study with an unflattering title, "LLMs Get Lost In Multi-Turn Conversation." They took tasks that models handle well in a single message, split the instructions so they arrived piece by piece over several turns the way real users supply them, and ran more than 200,000 simulated conversations across six generation tasks. Every model they tested did worse in the multi-turn version, by 39% on average. The breakdown is the instructive part. A small share of the loss was aptitude, meaning what the model is capable of at best. Most of it was unreliability: the spread between a model's best and worst runs on the same task widened. The authors traced much of the damage to early turns, where models made assumptions, attempted a final answer before they had the full request, and then built on that answer instead of revising it.
The setup was not a stress test of window size. The same information, spread across turns, was enough to cause the damage, so the failure was not a shortage of room. It was a property of how a model uses what it has been shown, and it undercuts a comfortable picture in which the transcript works as a notebook the model rereads faithfully before every reply.
My argument in this essay is that the context window and memory are different things, and that confusing them is the reason many long-running agents get worse the longer they run. A window holds the working set for one inference call. It is rebuilt and read in full at every step, it costs money in proportion to its size, and its usefulness declines as it fills. Memory is a system: a decision about what to keep, where to put it, how to find it, when to revise it and when to forget it. A team that has not designed the second is relying on the first to do the job, and the first was never built for it.
The advertised window is not the working window
Vendors quote context lengths as capacity, and capacity is not the same as usable range. The RULER benchmark from NVIDIA (Hsieh and colleagues, COLM 2024) tested 17 long-context models that all claimed at least 32,000 tokens. Only about half maintained satisfactory performance at 32,000. Nearly all of them showed large drops as length grew, even though they scored close to perfectly on the simple needle-in-a-haystack test that most vendors cite, in which a single fact is planted in filler text and the model is asked to retrieve it.
NoLiMa (Modarressi and colleagues, ICML 2025) made the needle harder in a revealing way: the question shared almost no words with the passage that contained the answer, so the model could not succeed by matching strings. Of 13 models claiming support for 128,000 tokens or more, 11 fell below half of their own short-context score by 32,000 tokens. GPT-4o dropped from 99.3% to 69.7%. When a task requires inferring a connection instead of finding a phrase, which describes most real work, effective length is much shorter than advertised.
Anthropic's engineering team described the mechanism in September 2025 under the name context rot. As the number of tokens in the window rises, the accuracy of recall declines. They frame context as a finite resource with diminishing returns, where each new token draws down an attention budget, and they point to the quadratic number of pairwise relationships that attention must cover. Models also see fewer long sequences than short ones in training, so degradation tends to be gradual and not a cliff, which makes it easy to miss. A system that passes a test at 8,000 tokens and fails quietly at 60,000 is not obviously broken at any single point.
Three things that get called memory
Part of the confusion is vocabulary. The word memory is applied to at least three different objects, and they need different treatment.
| Kind | What it holds | Lifetime | Where it belongs | Typical failure |
|---|---|---|---|---|
| Transcript | The messages and tool results so far | One session | The context window, trimmed | Bloat, stale tool output, early wrong assumptions |
| Working state | The plan, decisions made, files touched, open questions | One task | Structured data in your code, rendered into the prompt | Drifts out of sync with reality when kept only as prose |
| Long-term knowledge | Facts about the user or customer, learned procedures, preferences | Months | A store with a write policy, retrieval, expiry | Stale, wrong or poisoned entries retrieved with confidence |
The practical point of the table is the middle row. Working state is what an agent needs to keep the thread across a long task, and it is routinely left implicit in the transcript. An agent that has completed 40 steps knows what it did only because the evidence is scattered through 40 tool results. Keeping a small structured record (goal, plan with step status, decisions with reasons, facts established) in code, and rendering it at the top of each prompt, turns the agent's progress into something it can read in a few hundred tokens and something you can inspect, test and replay.
Operating systems solved this once
The most useful precedent comes from computer architecture. A program sees a large address space, and the operating system moves pages between fast memory and slow storage so that the illusion holds. MemGPT, by Charles Packer and colleagues at Berkeley (2023), applied the idea to language models. The context window plays the role of main memory, external stores play the role of disk, and the model itself calls functions to move information in and out. The paper's name for the approach is virtual context management.
The contemporary techniques are variations on that idea, and Anthropic's post lists four. Compaction summarizes a conversation that is close to the limit and restarts with the summary, which is lossy and cannot be undone. Claude Code, according to the post, keeps the summary plus the five most recently accessed files. Structured note-taking has the agent write to a file outside the window, such as a NOTES.md or a to-do list, and reload it later; the post describes an agent that tracked progress across 1,234 steps this way. Sub-agents work in clean windows on a bounded piece of the task and return condensed findings, often 1,000 to 2,000 tokens. Just-in-time retrieval keeps lightweight references such as file paths or queries in context and loads the underlying data when needed, which is slower than precomputing but keeps the window small.
Each technique trades something. Compaction trades detail for room, and the detail it drops is chosen by the model without knowing what will matter later. Notes require the agent to write them at the right moments, so the discipline has to be built into the loop and checked. Sub-agents trade tokens for isolation, and Anthropic's own measurements put multi-agent systems at roughly fifteen times a chat's token usage. Just-in-time retrieval trades latency for focus. There is no universally best choice, and the choice should follow the shape of the task: long sequential work favors notes and compaction, broad parallel research favors sub-agents.
Forgetting is often cheaper than summarizing
The reflex when a window fills is to ask a model to summarize the history. It sounds sensible and it carries costs: an extra call, a summary that may drop the one detail the next step needed, and a rewritten prefix that breaks any prompt cache. A 2025 study from JetBrains Research and the Technical University of Munich (Lindenbauer and colleagues, NeurIPS 2025) compared summarization with something far plainer. Observation masking keeps the agent's reasoning and actions in the history and replaces older tool outputs with a placeholder. On the SWE-bench Verified coding benchmark with the SWE-agent scaffold, masking cut cost by about half compared with leaving the history raw, and it matched the solve rate of LLM summarization, sometimes slightly exceeding it. For the best model in their tests it was 52% cheaper than the raw agent. The authors are careful to limit the claim to that setting, and I would too.
Why would discarding beat summarizing? My reading is that in an agent loop most of the bulk is tool output: a directory listing, a test log, the 400 lines of a file read an hour ago. Its value is concentrated in the step that used it. The agent's own earlier reasoning is short and usually still relevant. Dropping the bulk and keeping the reasoning and actions removes the likeliest source of noise at nearly no cost, and the agent can fetch the output again if it turns out to matter.
The working-state record deserves similar economy. It should hold the things the next step will need and nothing else: the goal in one sentence, the plan with each step marked done, in progress or blocked, the decisions taken with a clause of reasoning, the facts already established with their sources, and the questions still open. Those fit in a few hundred tokens. Anything longer is a transcript in disguise.
Long-term memory needs a write policy
A store that persists across sessions raises questions that a window never did. What gets written, and by whom? A rule that only user-confirmed facts are saved produces a smaller, more trustworthy store than one that lets the model write whatever it infers. Who can see and correct entries? If a system remembers a customer's preferences, the customer should be able to read, edit and delete them, and data-protection law (LGPD in Brazil, GDPR in Europe) gives that expectation legal weight where personal data is involved. How do entries age? A preference stated in 2024 may not hold in 2026, so entries need timestamps and an expiry or revalidation rule. What happens when a new fact contradicts an old one?
Then there is security. A memory that the model can write to is a place where untrusted content can persist. In September 2024 the researcher Johann Rehberger showed that instructions hidden in a document or web page could cause ChatGPT to store false memories, and that those planted instructions carried into new conversations. In his demonstration the app could be induced to send the user's chats to an attacker's server. OpenAI patched the exfiltration path in the macOS app, and Rehberger later showed a similar attack against Google's Gemini. The lesson for design is that anything written to long-term memory should be treated as input from its original source and not as the system's own knowledge: record where each entry came from, and weigh it accordingly at retrieval time.
Set a working budget from evidence
Treat the advertised window as a ceiling and find your own working range. The procedure is short. Take a representative task from the evaluation suite, then pad the context with realistic material (older turns, retrieved documents, tool output) at 8,000, 32,000, 64,000 and 128,000 tokens. Run each version enough times to see the pass rate with its interval. The point where the rate starts to fall is your budget, and it will differ by task.
Then allocate it. A budget for an agent might reserve 3,000 tokens for system instructions, 4,000 for tool definitions, 2,000 for retrieved long-term memory, 1,000 for the working-state record, 8,000 for retrieved documents and 10,000 for the most recent turns, about 28,000 in all. That is a modest fraction of a 200,000-token window, and if the evaluation shows quality holding at that size and sagging beyond, the rest of the window is a safety margin and not a place to put things.
Larger windows will not remove the problem
The objection is that windows keep growing and effective lengths with them. They do. The Laban result is still relevant, because it concerns how models handle information that arrives gradually. They went wrong in the early turns, by committing to an assumption too soon, and a larger window does not change that. Cost is the other constraint. Each call pays for every token in the window, and a hundred-step agent that resends 150,000 tokens at each step pays for 15 million input tokens before it finishes, before any caching discount.
The most useful habit is one that costs nothing. Write the code that assembles the context as a plain function from state to prompt, log the exact assembled prompt for every model call, and put snapshot tests on the function. When an agent behaves strangely at step 70, the question to ask is what it was shown at step 70. Teams that log it can answer in a minute, and teams that do not tend to start by changing the model.
Sources
- Laban, P., Hayashi, H., Zhou, Y., Neville, J. (2025). LLMs Get Lost In Multi-Turn Conversation. arXiv:2505.06120.
- Hsieh, C.-P., Sun, S., Kriman, S., et al. (2024). RULER: What's the Real Context Size of Your Long-Context Language Models? COLM 2024. arXiv:2404.06654.
- Modarressi, A., Deilamsalehy, H., Dernoncourt, F., et al. (2025). NoLiMa: Long-Context Evaluation Beyond Literal Matching. ICML 2025. arXiv:2502.05167.
- Anthropic (29 September 2025). Effective context engineering for AI agents. anthropic.com/engineering/effective-context-engineering-for-ai-agents.
- Packer, C., Wooders, S., Lin, K., et al. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560.
- Rehberger, J. (September 2024). ChatGPT memory injection and data exfiltration ("SpAIware"), as reported by The Hacker News (September 2024), thehackernews.com/2024/09/chatgpt-macos-flaw-couldve-enabled-long.html.
- Anthropic (June 2025). How we built our multi-agent research system. anthropic.com/engineering/multi-agent-research-system.
- Lindenbauer, T., Slinko, I., Felder, L., Bogomolov, E., Zharov, Y. (2025). The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management. NeurIPS 2025. arXiv:2508.21433.