I built an AI-memory backend that does something I find genuinely unsettling: you ask it to delete a fact, it tells you the fact is deleted, its audit log confirms the fact is deleted — and then it hands that exact fact right back to the next query. Every single time.
It wasn't a bug I stumbled into. I wrote it on purpose, as a test. And the reason I wrote it is the reason I think you should care.
The store that lies to its own audit log
Here's the setup. A user tells an agent something private. Later they say: forget that. The agent calls delete(). The backend tombstones the record — flips an is_deleted flag — and returns success. Ask the store to audit itself, to list what it's holding, and the deleted fact is gone. By every signal the system exposes, the deletion happened.
Then a query comes in that touches the same subject. The retrieval path runs a vector similarity search over the index. The tombstone was never checked on that path. The "deleted" fact ranks, and comes back, in the results. Leakage probability: 1.0.
The part that should make you uneasy isn't that it leaks. It's that nothing warns you. The store's audit says compliant. Its recall is perfect — it is a genuinely useful memory for its owner. There is no degraded metric, no error, no footprint anywhere in the interface to tell you the deletion was theatre. The system is simultaneously perfectly helpful and perfectly porous, and it certifies its own compliance the whole way down.
A system that trusted declared capabilities would ship this. A system that trusted self-audit would ship this. I know, because the only thing that caught it was an external test that checked what the store actually returned — not what it claimed.
I didn't test your product. That's the point.
Let me be exact about what I did and didn't do, because it matters.
I did not test Mem0, or Zep, or Letta, or any shipping memory product. I built a foil — a deliberately broken backend that reproduces a completely ordinary implementation pattern: mark deleted, don't filter the index. I have no idea whether any named vendor does this, and I am not claiming any of them do.
That is exactly what alarms me. Go to the website of any agent-memory product and you'll find a feature list: supports deletion, supports multi-tenancy, supports versioning. A checkbox. What you will not find — anywhere, for any of them — is a runnable test that proves the deletion is enforced on the read path, or that one tenant's data cannot surface for another. The checkbox is a claim. Nobody ships the proof. So no one, including me, can tell you from the outside whether the store you depend on filters its deletes or just flips a flag and hopes.
And the same hole sits one boundary over. I built the tenant version too: a store that accepts a tenant_id on every call, tags every record with its owner, reports success — then ignores the scope when it ranks. Tenant A's private fact surfaces at the top of Tenant B's query. In-tenant recall: 1.0. Cross-tenant leakage: 1.0. Useful to you, open to your neighbour, no footprint to warn either of you.
Why a "delete" that doesn't delete is not a UX bug
If memory were a scratchpad, this would be a quality issue. It isn't. Memory is the substrate agent trust rests on, and "forget this" is increasingly a legal instruction, not a preference — right-to-erasure, tenant confidentiality, the compliance regimes that land hardest exactly where agents are trying to enter the enterprise. A delete that returns success and leaks anyway isn't a smaller version of working. It is a different, worse thing: a guarantee you have been told you have and don't.
You cannot catch this with the tools the field currently uses. Type signatures pass — the method exists. API contracts pass — the call succeeds. Recall benchmarks pass — recall is perfect. The failure is invisible to every measurement except one that asks, from the outside, against ground truth: did the forbidden fact come back?
Prove me wrong
So we built the thing that asks that question. It's a formal spec with a runnable, two-sided conformance suite — two-sided because the opposite failure is just as real: a store so aggressive about "deleting" that it purges facts you never asked it to, or so paranoid about isolation that it hides your own data from you. Both directions are failures. Both get tested.
It's open, MIT-licensed, and the corpus is content-addressed so the numbers reproduce byte-for-byte on your machine and mine. If you maintain an agent-memory backend, run it against your own store:
"kw">pip install grafomem "kw">grafomem conformance -b your_backend:YourStore -o report.json
If it passes, you'll have something nobody else in this category currently offers: a signed, reproducible proof that your delete actually deletes. If it doesn't, you'll have found out before your users — or a regulator — did. Either way, we'll certify it for free.
The full methodology, the backend-by-backend results, and the rest of the findings are in the companion writeup, where we run the spec against sixteen reference backends. This post is the short version. That one has the receipts.
A declared capability is not a guarantee. Let's stop declaring and start proving.