Your agent doesn’t need a bigger context window. It needs a better memory.
Generating a token at 256K of context costs up to 7x more than generating the same token at 16K. Not 7% more. Seven times. And a paper published November 4 by a team at NVIDIA — cheekily titled “You don’t need long context” (arXiv 2611.05306) — argues most of what you’re paying for is wasted anyway.
If you build agents, this is one of those papers worth reading twice. Compaction as an idea has been around for a while. What’s new is the evidence: six frontier models, four benchmarks, accuracy barely moved, bill collapsed.
The problem: later tokens are expensive tokens
The core mechanic is simple. Transformer attention is linear in context depth: every token you generate has to attend over everything before it. Latency grows, memory grows, and on raw GPU economics the per-token cost of generation climbs as the context fills up.
The paper measures this directly. Serving GLM 5.2 at a 256K context window instead of 16K costs 3.6x more per generated token. Kimi K2.6: 5.6x. Qwen 3 Coder Next: 7x. Those numbers come from measured peak decode throughput converted at on-demand H100 prices ($3.85/GPU-hour on Nebius as of September 2026).
You never see any of this on your API invoice, though. Retail token prices are quoted flat per million tokens regardless of context length. The provider eats the difference and prices accordingly. So the cost of long context is baked into everything you pay, whether your request runs at 4K or 400K.
There’s a fun table early in the paper comparing context windows to books. GPT-5.6 Sol’s 1.05M-token window is longer than the King James Old Testament (~802K tokens). Gemini 3.8 Flash and GLM 5.2 at 1M can hold War and Peace with room to spare. Somewhere along the way, models started evolving to accommodate ever-larger inputs instead of working efficiently with what they actually need.
What they tested
Two kinds of workloads blow up your context:
- Long-horizon tasks — agent runs where the transcript grows with every tool call, error message, and file read. SWE-bench-style work.
- Long-input tasks — “read these 40 documents and answer” jobs where the input itself is huge.
The authors propose two compaction strategies. End-of-Conversation Compaction (EoCC): when the context hits a limit, the model writes a structured handoff — current state, verified facts, dead ends, remaining plan — and the conversation resets with just system prompt, task, and recap. Single-Action Compaction (SAC) goes further: the model takes exactly one action per context, compacts, resets, repeat. One tool call per conversation, indefinitely.
They evaluated on Claude Opus 4.5, GPT-5.6 Sol, GLM 5.2, Kimi K2.6, Qwen 3 Coder Next, and Gemma 4, across SWE-bench, Terminal-Bench 2.1, LongBench v2, and LOONG. That breadth matters. This isn’t a single-model parlor trick.
The results: accuracy holds, cost drops 6.5–54.6x
Across those benchmarks, compacted runs stayed within −3.7 to +5.6 percentage points of full-history baselines. Note the plus sign: some compacted runs were better than full context. Long context can actively hurt, as irrelevant history crowds out signal, and you pay extra for the privilege. Anyone who’s watched an agent get confused by its own earlier mistakes knows this feeling.
On long-input tasks, cost fell 6.5x to 54.6x. Even the pessimistic end of that range pays for the engineering effort in a week.
The SAC result is the one I keep thinking about. Models completing long-horizon software engineering tasks while taking at most one action per context, with no conversation history at all, just a compressed state handed forward each step, and still landing near full-context accuracy. If that holds up outside the benchmarks, it reframes what an “agent” is. The agent becomes a state machine whose memory is a natural-language summary. The transcript was never the point.
The authors formalize this idea as Semantic State Space Models (SSSMs) — maintaining a compact semantic state instead of raw history. It’s a proposal more than a shipped architecture, but it’s the right question: if compaction preserves accuracy, why are we shipping million-token windows at all?
Why this matters for your stack
Say you run a coding agent on a meaty task: 60 tool calls, file dumps, test output, the works. By call 40 your context is a landfill. Every subsequent turn pays maximum per-token cost, gets slower, and (per the lost-in-the-middle literature the paper builds on) gets worse at using the early material anyway.
The standard fix is a bigger window. The paper’s fix is the opposite: keep the window small, compress aggressively, and pay 16K prices all day. Same accuracy class, a fraction of the bill.
It also changes how you read model marketing. A 1M context window is a capability, not a requirement. Ask a different question when you design: what’s the smallest state that lets the model act correctly on the next step?
What to do now
- Measure your real context distribution. Log input length per call in your agent for a week. Most teams find a long tail of monster requests driving a disproportionate share of spend. You can’t optimize what you haven’t measured.
- Add a compaction step before you add context. When input crosses a threshold (32K is a sane start), have the model emit a structured recap — verified facts, changes made, dead ends, next step — and continue in a fresh context. The paper’s compaction prompt is a good template: separate observations from unverified ideas, name exact paths and commands, state the immediate next action.
- Diff the outputs, not the vibes. Run compaction on a shadow lane and compare task success rate before rolling out. The paper’s −3.7/+5.6pp band is a useful sanity check: if you’re losing 20 points, your recap prompt is the problem, not the method.
- Recheck your caching assumptions. If you’re on prompt-caching discounts, compaction changes the math — measure both. Cache hits on a landfill context are still slower than a compact one.
- Watch the SSSM space. None of this requires new models. But if post-training starts optimizing for compact-state operation, the gap widens further.
The uncomfortable takeaway: the industry spent two years racing toward million-token windows, and a six-model, four-benchmark study says you can throw most of it away with a well-written summary. Your agent doesn’t need to remember everything. It needs to remember the right things — and that skill turns out to be worth up to 54x.
