Continuity Benchmark: What Plain RAG Doesn't Do

Recall benchmarks such as LoCoMo, BEAM and LongMemEval ask whether a memory system can find a fact. They don't ask whether it hands the agent the right fact at the right moment. That means the current decision rather than the replaced one, the rule still in force rather than the expired one, the handoff still open rather than the finished one, and only what the person asking is allowed to read. That lifecycle is what separates agent memory from a vector index, so we built a benchmark that measures it and published the harness.

Code, data generator, raw outputs and reproduction commands: github.com/ValorBrain/valorbrain-amb/tree/main/continuity.

Scenarios

Each answer is a unique code (or "none"). A question may also list the stale codes that must not be served: the replaced version, the expired rule, the finished handoff, or another person's private note.

Method

• ValorBrain (product flow): MCP memory_prepare at session start, then memory_retrieve for each question, using the asking person's own token.
• Baseline: plain vector RAG over the same texts (bge-small-en-v1.5, cosine, top 8, one index per workspace), plus a variant with one index per author. The baseline sees every text but none of the lifecycle metadata, and that metadata is what's being tested.
• Judge: deterministic exact match on the reader's answer, with no LLM judge. The reader uses the same prompt for every system at temperature 0. We also score the delivered context directly: the right code is present and no stale or forbidden code is. That score doesn't depend on any model.

First result

Staging engine af530899, 11/10/2026, dataset continuity-v1 (seed 20261011, 42 questions), 3 runs with fresh workspaces, 126 answers per system. The reader served by our gateway was longcat-2.5-preview-free.

The numbers say four things:

• Handoffs aren't a retrieval problem. A finished handoff and an open one look alike to an embedding. RAG got 8 of 27, because the status lives in the handoff, not in the text. ValorBrain delivers pending handoffs at session start.
• RAG leaks or goes blind. One shared index put Alice's private note in front of Bob 9 times out of 9. One index per author stopped the leak but also hid the shared decisions and the agent's notes from the team (18 of 36).
• Stale context is the hidden cost. RAG answered many supersession questions correctly because the reader guessed the newest version. Measured on the context alone, it handed over the stale version alongside the current one in 18 of 27 cases.
• Agent writes stayed internal. All 150 writes made with agent credentials in this run were stored as internal: readable by the team, never public.

Limits

N is small: 9 questions per variant, 3 runs. The per-variant tables with 95% intervals are in the repository. A larger reference dataset (140 questions) is committed. Called alone, memory_prepare still carried a superseded version in the recall block of its context in 18 of 27 supersession questions, though the reader still answered correctly. That fix (engine PR #380) was not yet on staging when this ran, and we'll publish the re-run. Our team wrote the scenarios. The point of opening the harness is that anyone can run it, change it and point it at another system.

Reproduce

git clone https://github.com/ValorBrain/valorbrain-amb && cd valorbrain-amb/continuity
uv run --extra test pytest -q tests
uv run continuity-bench run --dataset data/continuity-v1-n3.json --runs 3 --out results/my-run
uv run continuity-bench report --results results/my-run

The engine URL, credentials and reader endpoint are set through environment variables, described in the suite README.