Stop Deleting Frames: The Memory Pattern That Made a Vision Agent Beat Humans
The most capable vision agent MIT could build doesn’t have a smarter model inside it. It has a better memory.
Last week an MIT team posted VISTA, an agent harness that finished ARC-AGI-3, a benchmark of interactive visual puzzles built to frustrate machines, using 57.4 percent fewer actions than first-time human players. With Claude Opus 5.0 doing the reasoning, it hit a perfect 100 Relative Human Action Efficiency: every single puzzle solved in fewer steps than a person seeing it for the first time.
The recipe is almost embarrassingly simple. VISTA stores every frame it sees and never throws one away.
Your agent forgets on purpose
If you’ve built any agent that works with images, whether that’s computer-use agents, browser automation, or screen-scraping helpers, you’ve probably made the same tradeoff everyone makes. Context windows are finite, frames are big, so you embed each frame, keep the embedding, and drop the pixels.
That approach loses information twice. Embeddings are lossy: compress a screenshot into a vector and you keep the gist while throwing away the layout, the fine detail, and the thing in the corner you didn’t know mattered yet. And eviction is permanent. Once a frame leaves the context window, it’s gone. If the agent needs to check what a toggle looked like three steps ago, it can’t. It guesses.
VISTA’s bet is that both losses are unnecessary. Storage is cheap. Attention is expensive. So store the raw frames in persistent memory outside the context window, and give the model tools to pull any frame back in, including a zoom tool for inspecting a region at full resolution.
The agent decides when to look back. Not the framework.
The numbers
On ARC-AGI-3, VISTA running Claude Opus 5.0 scored the full 100 RHAE. With GPT-5.6 Sol it hit 99. Then came the ablation, and this is the number I’d tattoo on the wall of any agent-engineering team: remove frame storage, keep everything else identical, and GPT-5.6 Sol drops from 99 to 70.
Same model. Same prompts. Same tools. Thirty percent of the performance lived in the memory design rather than the model.
On GameWorld, a second benchmark, VISTA with GPT-5.6 Sol completed 63.3 percent of tasks. First-time humans managed 55.3 percent. The harness shipped with the environment, the “native” setup you’d actually get out of the box, completed 40 percent.
The code is MIT-licensed on GitHub, and the harness is model-agnostic. It works with frontier models you can rent by the token today.
The pattern to steal
Forget the leaderboard. Here’s the part you can put into production this week.
Store raw, retrieve on demand. Keep every screenshot your computer-use agent takes in object storage or a local directory. Give the model a tool or two, something like get_frame(n) and zoom(frame, region), and let it decide when memory is worth spending attention on. This is the exact playbook text agents converged on in 2025: store everything, retrieve only when it matters. Eviction is the bug.
Keep two kinds of notes. VISTA maintains facts about the environment that carry across tasks (how this app’s menu behaves) separately from facts about the current task (which tab I already checked). Mixing the two is how agents re-learn the same interface on every run, or pollute a fresh task with stale assumptions.
Log every action-result pair. VISTA stores what it did and what happened, so the agent can audit its own history when something goes sideways. When your browser agent claims it submitted the form, the log is right there. This costs almost nothing and turns “why did it do that” from archaeology into a query.
One honest caveat: memory grows with run length, and a long computer-use session produces a lot of frames. You’ll want an index alongside the pile, not just the pile.
None of this requires a new model. That’s the point.
Why memory beats model size here
The text-agent world already learned this lesson the hard way. Retrieval beat bigger context windows. External stores beat stuffing everything into the prompt. The entire field of context engineering exists because memory architecture is usually the binding constraint, long before model intelligence is.
Vision agents have been slow to catch up because frames felt too big to keep. But storage keeps getting cheaper and models keep getting better at deciding what to look at. VISTA is the cleanest demonstration I’ve seen that the text playbook transfers wholesale: persistent raw memory plus retrieval tools beats compressing and forgetting, in pixels just like in prose.
A year from now, I suspect every serious computer-use agent ships with a frame store the way every text agent today ships with a retrieval layer. An ablation this clean, 99 down to 70 with nothing else changed, is the kind of evidence that gets an architecture adopted.
What to do now
- Audit your vision agent’s memory. If it embeds and evicts, count how often a run fails because the agent needed information it had already thrown away.
- Add a frame store. A directory of PNGs with an index beats what you have now. Object storage if you’re on cloud; the cost works out to cents per thousand frames.
- Expose two tools to the model: fetch a frame by index, and zoom into a region. Watch for the model citing old frames in its reasoning. That’s when you know it’s actually using them.
- Split your agent’s notes into environment facts and task facts. Persist the first, reset the second between runs.
- Log action-result pairs from day one. Future-you, debugging an agent failure at 3 a.m., will be grateful.
The MIT paper (arXiv 2610.02200) has the full harness details, and the code is worth an afternoon of reading even if you never run ARC-AGI. The headline is the memory, and the memory is borrowable.
