Every few months a new agent-memory system ships, posts a benchmark number, and a competitor disputes it. The LOCOMO results have been openly contested between vendors — a public argument over methodology and figures that has its own nickname, the "benchmark war." I don't think that argument is a sign the field is immature. I think it's a sign the field is measuring the wrong thing, in a way that can't settle.
Here's the deeper issue. There is no shared agreement on what an agent-memory system is even supposed to do. Each system makes its own private choices — whether facts can be superseded, whether history is queryable, what gets evicted under pressure, whether a deletion is honored on read, whether one tenant's data can surface for another — and those choices are rarely stated, almost never tested against ground truth, and impossible to compare across systems. We've been building the substrate that agent reliability and user trust both rest on, entirely ad hoc.
The usual fix for "no way to compare" is "build a leaderboard." We don't think a leaderboard is the answer, because the problem isn't a missing score. The problem is the absence of agreed axes along which memory systems differ at all. A single recall number can't tell you whether good recall came from the retriever or a generous budget; can't distinguish a store that supersedes stale facts from one that hoards them; and — most consequentially — cannot detect that a system reporting a fact as deleted still returns it. The interesting failures of agent memory aren't "low recall." They're category errors about what the system is even claiming to provide.
So we built a benchmark designed to expose the axes instead of averaging over them, and it pointed at something larger: agent memory needs a protocol. Not a model, not a leaderboard — a contract. Here's what the evidence says that contract has to specify.
Versioning and validity, as first-class operations. A protocol that treats memory as an append-and-retrieve log can't express the most common real-world event — a fact changing — and hands you stale data alongside current data with no signal to separate them. It must define what it means to supersede a fact and to query the past.
A retrieval contract that is embedding-agnostic. Embedding quality is a real lever, but it's the one axis that improves on the timescale of model releases, not protocol revisions. A standard should specify the retrieval interface and a budget contract — and deliberately not mandate an embedder. Fixing a model into the standard would couple an axis the evidence shows is independent, and date the whole thing to one model generation.
A declared retention policy. Every store makes a retention choice, and that choice imposes a hard structural limit on how far back it can answer. A protocol must let a store declare its policy and the footprint/coverage contract it implies, so an agent depending on a months-old fact knows whether the store can still be expected to hold it.
Two-sided privacy, enforced on the read path. Deletion and tenant isolation are the same object seen twice, and each fails in two directions: leakage (returning what it must not) and over-restriction (withholding what it must). The guarantee has to live on the read path, because the failures we see are precisely stores that accept the write-side call — a tombstone, a tenant tag — and ignore it when they rank. "Supports deletion" is not a guarantee; what retrieval returns is.
A declared, honored isolation policy. A store that serves more than one writer makes a concurrency choice, and declaring a level is not delivering it. A protocol must treat the declaration as falsifiable: a store that admits an anomaly its level forbids has over-claimed, full stop — and a committed deletion must not be resurrected by a racing write.
And then the requirement that makes the rest enforceable rather than aspirational:
A conformance suite is part of the protocol. This is the one I'd defend hardest, because the benchmark produced its own existence proof. We built backends that advertise the correct capability flag, accept every relevant call, return success — and leak anyway. One of them certifies its own compliance through an audit log while its retrieval path quietly hands back deleted data. The type contract is impeccable; the semantic contract is broken; and nothing in the interface reveals it. A protocol therefore cannot define conformance by inspecting declared capabilities. It has to ship an executable, ground-truth test, and define "supports capability X" to mean "passes the suite for X" — not "declares X."
That reframing is the whole thing. It's the difference between a spec sheet and a guarantee — the same difference that separates "this material is fire-resistant" from a UL listing, or "this query is valid SQL" from a database that actually runs it. Agent memory has had spec sheets. It has not had the listing.
We've started building it. There's an open spec, a runnable two-sided conformance suite, a content-addressed corpus whose numbers reproduce byte-for-byte across machines — so the next dispute can be settled by re-running, not by arguing — and three independent implementations that pass. It is early, and it is deliberately open: the spec, the suite, and the corpus are all MIT-licensed, because a standard that one company owns isn't a standard.
If you build agent memory, I'd genuinely rather you ran our suite and found it wanting than ignored it. Run it against your own store. Tell us where the spec is wrong. The goal was never to crown a winner — it was to make "trustworthy" a property you can prove instead of a checkbox you can claim.
A declared capability is not a guarantee. Let's build the thing that turns claims into proofs.
The protocol requirements, the full benchmark, and the public conformance registry are at grafomem.com. The spec, suite, and corpus are open source on GitHub.