Research

What evaluation leaves out

Peer-reviewable work on retrieval economics and benchmark licensing — the costs and constraints that decide what a system actually costs to own, and that leaderboards do not report.

arXiv · 17 August 2026 · Paper 1

The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks

The field's dense-retrieval anchor is licensed non-commercially, and three of the four leading MuSiQue systems depend on it without disclosing that. Meanwhile every audited system omits indexing cost — published figures for the same workload span eleven-fold. Thirteen embedders from eight makers, one identical harness, bootstrap confidence intervals.

Luis M. Sánchez · Kosrow Dehnad (Columbia University)

Read the summary →

In preparation · Paper 2
From benchmark to deployment: what these findings cost, and what they change, for organisations that cannot send their data anywhere. Ask to be notified when it publishes.