Skip to content

EngineeringJuly 5, 20266 min read

96.58% recall on LoCoMo: how we built a CPU retrieval engine

The architecture behind our retrieval pipeline: BM25 + dense vectors + Personalized PageRank. No LLM in the ranking path.

Gustavo Franson

Our canonical full-table run measured 96.58% recall@10 on LoCoMo. Here is the retrieval architecture behind it and why ranking needs no LLM.

The benchmark

LoCoMo (Long Context Memory) is a public, peer-reviewed benchmark for conversational memory. Our full table has 1986 questions across 10 conversations. The published open-source leader (pmb) scored 94.5%. ValorBrain measured 96.58%, a margin of +2.1 percentage points.

The pipeline

Our retrieval has four stages, fused into a single index scan:

  1. BM25 (keyword search) catches exact-match queries.
  2. Dense vectors (LFM2.5-Embedding-350M) handle semantic similarity. Swapping from Jina v5-small to LFM2.5 moved dense-only recall from 32.7% to 63.5% in the original ablation.
  3. RRF (Reciprocal Rank Fusion) merges BM25 and dense results.
  4. Personalized PageRank re-ranks the fused results by graph centrality. In that ablation, this stage added 2.4 points.

No LLM in the ranking

The ranking pipeline runs on CPU, sub-second, with zero token cost. Personalized PageRank is linear algebra; it needs no model inference.

That keeps search cost independent of OpenAI or Anthropic tokens. Answer generation remains a separate step and may use a language model.

What we learned

  • The embedding model drove most of the gain. The Jina-to-LFM2.5 swap added 30.8 points in the original ablation.
  • Hybrid search covers different failure modes. Dense-only misses exact matches; BM25-only misses semantic matches.
  • Personalized PageRank is cheap. One matrix operation per query delivered the final graph-based lift.

The end-to-end layer

LoCoMo measures retrieval. On BEAM-100K, the end-to-end pipeline answered 323/400 questions correctly (80.8% binary accuracy) across 20 conversations. The 2026-08-23 run used GLM-5.3 Flash (Ox Alpha), effort=max, and judge GLM-5.2. Max effort added about 44% latency over the default, so it is a benchmark configuration rather than the chat default. See the methodology and the full run breakdown.

All posts

Next step

Less starting over. More continuity

Bring a real project. In one conversation we map where context gets lost between the people and agents on your team, and where to start.