Proof
How often the memory gets it right
LoCoMo checks whether the session holding the answer lands in the top 10 results; BEAM checks whether the final answer is correct. Every number carries one metric, its run, the reader and whether the configuration is the product default, and what the scores don't measure is on the same page.
LoCoMo · per-session recall@10
96.58%
1918/1986 questions · run 2026-07-29
10 sessions picked at random
42.25%
Each conversation has 19 to 32 sessions, and a hit means the session holding the evidence is in the top 10. Returning 10 sessions at random already hits 42.25% of the time. Read the 96.58% against that number.
Configuration: retrieval only, no reader; one document per session (19 to 32 per conversation), the benchmark's own segmentation (the product ingests conversations as 6-turn windows); ranking uses the production configuration, 0 LLM calls; picking 10 sessions at random scores 42.25%; run 2026-07-29
Full per-conversation table
Run of 2026-07-29 with 1986 questions across 10 conversations. The table total is the number above, same math; the last column is 10 random sessions in the same conversation.
| Conversation | Sessions | Hits | Questions | recall@10 | Random |
|---|---|---|---|---|---|
| 26 | 19 | 191 | 199 | 96.0% | 56.4% |
| 30 | 19 | 102 | 105 | 97.1% | 55.8% |
| 41 | 32 | 188 | 193 | 97.4% | 36.5% |
| 42 | 29 | 252 | 260 | 96.9% | 40.2% |
| 43 | 29 | 236 | 242 | 97.5% | 40.1% |
| 44 | 28 | 154 | 158 | 97.5% | 44.4% |
| 47 | 31 | 180 | 190 | 94.7% | 35.4% |
| 48 | 30 | 230 | 239 | 96.2% | 37.2% |
| 49 | 25 | 187 | 196 | 95.4% | 47.2% |
| 50 | 30 | 198 | 204 | 97.1% | 38.0% |
| Total | 272 | 1918 | 1986 | 96.58% | 42.25% |
LoCoMo · answer accuracy
No published number yet. The full staging run finished, but it does not count as a product number: staging's database is remote, search runs out of its time budget and the reranking step is skipped on almost every query, while in production it runs. The number would understate what the product delivers. It goes here, with reader, judge and date, once staging runs with production's ranking.
BEAM-100K · judge average
75.5%
Here the score goes to the final answer, after search and reading. The average is the AMB scoreboard metric: the judge scores each question from 0 to 1.
Binary (correct over total): 80.8% (323/400)
Configuration: reader GLM-5.3 Flash (Ox Alpha) at effort=max — a benchmark configuration, not the product default (the reader is the customer's LLM; ~44% slower than default effort); judge GLM-5.2; memory through the production path (memory_prepare); 20 conversations, 400 questions; N = 1 run; run 2026-08-23
The ablation behind itBEAM-10M · judge average
51.2%
The 10-million-tokens-per-conversation scale, with the same reader and judge as BEAM-100K.
Binary (correct over total): 56.0% (112/200)
Configuration: same reader (GLM-5.3 Flash (Ox Alpha), effort=max, not the product default) and same judge (GLM-5.2); 10 conversations of ~10M tokens, 200 questions; N = 2 runs (the second averaged 50.7%); run 2026-08-24. Context: on the same AMB scoreboard, Hindsight averages 64.1% with a different reader (Gemini 3.1 Pro) and judge (Gemini 2.5 Flash Lite), so the two numbers are not directly comparable
Rerun the math on GitHubWhat we don't measure
Both benchmarks use synthetic or public conversations. Permission, origin and continuity across people and agents are product capabilities outside the scores.
What the scores cover
- Evidence retrieval in long conversations (LoCoMo, per-session recall@10)
- Final answers in long conversations (BEAM, judge average and binary)
- Ranking without calling an LLM in the retrieval path
Outside the scores
- Multi-tenant isolation or role-based permission filtering
- Team handoffs, knowledge gaps, or veracity/origin badges
- Production Ask citation fidelity or latency under live tenant workloads
Read before citing
- Internal benchmarks, not peer-reviewed; every number states its metric, reader and run.
- Per-session recall@10: the session that holds the evidence is among the top 10 results. The unit is the whole session; passage-level recall is left out.
- Answer accuracy: share of answers the judge marked correct.
- Judge average (BEAM): mean of the 0-to-1 score per question. Binary: share of questions marked correct.
- Each run states the reader it used. In the product, the reader is the customer's LLM.