Introduction

A paired experiment across 230 questions compared two RAG interfaces using the same BM25 retriever, revealing that higher search recall does not automatically translate to better answer accuracy. The study interrogates the boundary between retrieving relevant documents and delivering useful passages to the model.

What Happened

The project interface permitted search returning one to twenty documents, each with 2000-character excerpts, source IDs, offsets, and pagination support, plus a default read tool for 4000-character spans continuing from an offset. The standard interface returned five results per query, each capped at 512 tokens from the document beginning, with no separate read capability. On the same 230 BrowseComp-Plus questions, the project interface answered 217 correctly versus 194 for the standard interface, despite achieving lower search-only reference-document recall (86.12% versus 89.71%). Mean search calls averaged 12.16 for the project interface versus 19.84 for the standard one, while mean read calls were 3.33 versus zero.

Why This Matters

The results suggest that simply retrieving more documents does not guarantee the model accesses the needed passage. The recall metric counts reference-document IDs returned by search but does not measure whether the decisive text reached the model. Interface design including returned text range truncation indicators and continuation actions shapes what the model actually sees and can use. The experiment also notes that several interface properties changed together, so the aggregate result cannot isolate any single factor's causal contribution.

Key Takeaways

  • Search recall alone doesn't measure evidence quality delivered to the model
  • Returned text with source IDs and offsets makes observations interpretable
  • Document length and truncation indicators help the model recognize partial views
  • Continuation actions (reading more, searching within a source) give the model concrete next steps
  • Interface properties changed together, so the aggregate result can't allocate gain among read access, passage placement, parameters, and prompts

Conclusion

The experiment provides a reason to carefully evaluate evidence interfaces alongside retrievers, checking whether models can locate useful passages, recognize partial views, and continue gathering evidence. Higher recall is not a standalone solution; interface design and evidence delivery remain critical levers for RAG effectiveness. The authors maintain the project linked in the article, and the paper references explain the research context while project records support the reported comparison.