96.58% recall on LoCoMo: how we built a CPU retrieval engine

Our canonical full-table run measured 96.58% per-session recall@10 on LoCoMo — the session holding the evidence appears among the top 10 results. Picking 10 sessions at random gets 42.25% of the way there; read the 96.58% against that number. Here is the retrieval architecture behind it and why ranking needs no LLM.

The benchmark

LoCoMo (Long Context Memory) is a public, peer-reviewed benchmark for conversational memory. Our full table has 1986 questions across 10 conversations of 19 to 32 sessions each, with one document per session.

The pipeline

There is one production pipeline, the same one described in the docs and in the benchmark provider:

1. BM25 (keyword search, Portuguese and English) catches exact-match queries.
2. Dense vectors (LFM2.5-Embedding-350M, 1024 dimensions) handle semantic similarity.
3. RRF (Reciprocal Rank Fusion) merges BM25 and dense results.
4. Cross-encoder (BGE-Reranker-v2-m3) reranks the top; MMR keeps it diverse.

An LLM only expands the query; it never ranks. Personalized PageRank, which appears in this post's original ablation, exists behind a flag and has been off in production since 2026-07 — in that June ablation it added 2.4 points, and swapping from Jina v5-small to LFM2.5 moved dense-only recall from 32.7% to 63.5%.

No LLM in the ranking

The ranking pipeline runs on CPU, sub-second, with zero token cost. The cross-encoder rerank and MMR are algebra and similarity, not model inference.

That keeps search cost independent of OpenAI or Anthropic tokens. Answer generation remains a separate step and may use a language model.

What we learned

• The embedding model drove most of the gain. The Jina-to-LFM2.5 swap added 30.8 points in the original ablation.
• Hybrid search covers different failure modes. Dense-only misses exact matches; BM25-only misses semantic matches.
• Personalized PageRank is cheap — and off. In the June ablation, one matrix operation per query gained 2.4 points; in production it sits behind a flag, disabled.

The end-to-end layer

LoCoMo measures retrieval; answer accuracy in the same harness has no published full run yet (the benchmarks page states where it stands). On BEAM-100K, the end-to-end pipeline answered 323/400 questions correctly (average 75.5%; binary 80.8%) across 20 conversations. The 2026-08-23 run used GLM-5.3 Flash (Ox Alpha), effort=max, and judge GLM-5.2 — a benchmark configuration, not the product default: the reader is the customer's LLM. See the methodology and the full run breakdown.