We asked every document what it is about. One in four was not found

"Improved" is easy to say and hard to prove. A customer corpus does not come with an answer key: nobody knows which document should answer each question. Public benchmarks measure another corpus, written by someone else, with a different question in mind.

So we built the truth the corpus allows. For each document, the question is its own subject, title plus the beginning of the body. The right answer is that document. If search cannot find a document from its own subject, it will not find it from a real question either.

This is a floor, not a ceiling. It does not measure whether the context answers, nor whether the model reading it gets it right. It measures one thing: does the document reach the person asking.

Three bugs in the instrument before the first number

The first run returned zero percent across 100 documents. Zero is absurd enough that I distrusted the instrument before distrusting the product.

The bug was in the comparison: I matched the path stored in the database against the path search returns, and search returns the path with the collection in front. Nothing matched. With that fixed, the number jumped to almost 50%.

The second run gave different numbers on every execution, because the document sample had no seed. Two measurements of the same system were not comparable.

The third was subtler. The question I built started with the raw text of the document, and many meeting documents open with a repeated header, something like "Corpus index, not a source". I was asking search to find the header.

Only after those three fixes did the number mean anything.

What was left once the instrument stood up

One in four documents did not show up among the first ten results, with its own subject as the question. Both causes were in the product, not in the measurement.

The first: the filter that avoids returning two similar documents. It compared text, and when two documents mentioned the same client or the same meeting, it treated the second as a version of the first and pushed it to the end of the queue. In practice, the document that answered fell behind for looking like another. The filter now only demotes an earlier version of the same document, when the date proves which one is new, or a near-identical copy.

The second: the cap on documents per answer. Dropping from 12 to 6 looked good in the test set, where the answer is almost always near the top. On a real corpus with 6,000 documents, it cost seven points of recall. The cap is now 8, where the curve stops rising on a large corpus and the delivered context is still smaller than with 12.

The number, and what it does not say

After the fixes, the chance that the right document appears among the first ten went from 79% to 84% in the account we use as reference, with context about 30% smaller than it was with 12 documents.

That number does not say the answer is right. Retrieving the document is a condition for answering, not the answer itself. It also does not say whether the reader uses what it received, and the question we use is synthetic, built from the document subject, not from a question a person actually asked.

What it does say is where the problem was and what it cost. Before, the argument about scope and ranking was opinion against opinion. Now there is a ruler anyone can run on their own corpus and compare with ours.