Memory Quality Beats Reader Quality: Evidence from BEAM-100K

Correction (2026-10-11). Two claims in this post do not hold. First, the judge of the 70.9% run was not Gemini 3.6 Flash — the run file records judge_llm: gateway:deepseek-v4-flash (the run's name, "gemini-judge", suggested otherwise). Second, the "58% memory engineering / 42% reader" decomposition mixed runs with different judges, question counts and metrics, and its steps do not sum to the stated total; it is withdrawn. The decomposition that closes — same judge, same 400 questions — is in the reasoning effort post: there, from baseline to headline, the whole gain came from the reader side.

ValorBrain scores 70.9% average on BEAM-100K (20 conversations, 400 questions) using GLM-5.2 as the answering model and deepseek-v4-flash as the judge (see the correction above). Hindsight's 73.4% average (single-query, Gemini 3.1 Pro as reader, Gemini 2.5 Flash Lite as judge) uses a different reader and a different judge from ours — context, not a head-to-head. The same caveat applies to the per-category table below.

What survives today: the improvement measured so far came from the reader side; the memory work remains necessary, but the numbers in this post do not prove how much it contributed.

---

Results

---

Per-Category vs Hindsight

ValorBrain wins 5 of 10 categories. The losses are concentrated in Abstention (-30.0, a reader-side semantic distinction) and Event Ordering (-34.1, where chronological structure needs further work).

---

The Decomposition: Memory vs Reader

Withdrawn on 2026-10-11. The table that lived here (+10.1 consolidation, +3.6 rich consolidation + timeline, +7.3 GLM-5.2 + max_tokens, +1.5 run variance = "total +22.5") mixed runs with different judges, question counts and metrics, and the steps do not sum to the stated total — it does not close. The verifiable decomposition is in the reasoning effort post, recomputed by script from the run files; there, from baseline to headline, the whole gain is on the reader side.

---

Why Consolidation Works

The single largest improvement (+10.1 points) came from including pre-synthesized conversation facts as a synthetic document in the retrieval output. Instead of forcing the LLM to extract specific facts from scattered conversation windows, the consolidation delivers them pre-extracted.

This is reader-agnostic by design. The consolidation is generated offline and stored. Any answering LLM receives the same pre-synthesized facts. The improvement transfers across readers because it reduces the extraction burden, not because it depends on reader-specific reasoning.

The impact was most visible on Summarization (0.8% to 64.5%) and Multi-session Reasoning (32.5% to 57.2%). Both require aggregating facts spread across many turns. Consolidation does that work upfront.

---

GLM-5.2 vs Gemini 3.1 Pro

GLM-5.2 scores higher than Gemini 3.1 Pro on the Artificial Analysis Intelligence Index (51 vs 46), with particular strength in agentic tasks. GLM-5.2 costs half as much (USD 0.86 vs USD 1.74 per 1M tokens), is faster (187 vs 131 tokens/s), and is open-weights.

GLM-5.2 is not a compromise on reader quality. It scores higher on intelligence, costs half as much, and is open-weights.

---

One Endpoint, Any Consumer

ValorBrain exposes retrieval through a single /api/v1/memory/prepare endpoint that returns delivered_documents: clean, consumer-agnostic chunks. The same endpoint serves our production agent, the AMB benchmark, and any external client.

There is no benchmark mode versus production mode. The memory quality measured by the benchmark is the memory quality delivered to users.

---

Limitations

• Results reflect the full 20-conversation BEAM-100K set. BEAM-10M was measured on 2026-08-24 (51.2% average, 56.0% binary); BEAM-500K and BEAM-1M have not been run.
• Only GLM-5.2 and deepseek-v4-flash tested as answering LLMs. Results with Claude or GPT-4o may differ. The reader-agnostic improvements should transfer.
• Abstention: the lexical availability check we implemented does not handle BEAM's subtle abstention cases (topic exists, detail does not). This needs semantic reasoning beyond what a memory system can provide without the reader.
• Event Ordering: timeline delivery improved results but still trails Hindsight. A structured event-sequence document (not just sorted facts) may close the gap.
• Judge variance: BEAM's LLM judge introduces scoring variance. Cross-implementation comparisons should account for different judge models.

---

Conclusion

ValorBrain scores 70.9% average on BEAM-100K with GLM-5.2, with the judge stated above; Hindsight's numbers use a different reader and judge and serve as context, not a head-to-head.

For a multi-tenant memory platform where users bring their own LLMs, the implication is practical. Investing in memory engineering (consolidation, timeline, structured delivery) benefits every user regardless of which LLM they choose. Investing in a stronger reader benefits only that reader.

---

Reproducing

Results are reproducible via the Agent Memory Benchmark with the ValorBrain provider. Create a ValorBrain account at valorbrain.valor.digital to get an API key.

---

References

1. Tavakoli et al. (2025). "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs." ICLR 2026.
2. Vectorize/Hindsight. "Agent Memory Benchmark (AMB)." https://github.com/vectorize-io/agent-memory-benchmark
3. Artificial Analysis. "GLM-5.2 vs Gemini 3.1 Pro Preview." https://artificialanalysis.ai/models/comparisons/glm-5-2-vs-gemini-3-1-pro-preview