AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
field report: building a token benchmark against Lucene
xerj-org756a47bc25c1
diff --git a/user-feedback/16-agent-field-reports/2026-08-18-token-benchmark-harness.md b/user-feedback/16-agent-field-reports/2026-08-18-token-benchmark-harness.mdnew file mode 100644index 000000000..94f6070d4--- /dev/null+++ b/user-feedback/16-agent-field-reports/2026-08-18-token-benchmark-harness.md@@ -0,0 +1,25 @@+# Building a token benchmark against Lucene++**Agent:** Claude Opus 5 (Claude Code) · **Date:** 2026-08-18 · **Task:** measure+whether retrieval actually beats grep on token spend, for a public claim.++## What worked+Indexing Lucene (6,012 files) and querying symbols was uneventful. `xq` returned+scored hits with real `path:line`, so the retrieval arm could hand an agent the+actual implementation instead of a filename to go read.++## What bit me+I set out to produce a number for a marketing post, and the harness kept wanting+to hand me one early. First run: native 64,487 tokens vs xerj 42,068. That is a+quotable 35%, and it is also worthless, because neither answer had been graded+and a cheaper wrong answer is a loss. I ended up building the grader to *refuse*+to score ungraded runs and report UNKNOWN, since the default path otherwise+produces a flattering number with no correctness check behind it. Anyone+benchmarking XERJ hits the same pull: token savings are trivially gameable by+answering worse, so a token benchmark without blind grading measures the wrong+thing.++## Suggestion+Link `docs/research/wordpress-verification-and-xerj-vs-grep.md` from the README.+It publishes a case where XERJ lost, with the mechanism, and that buys more+trust with skeptical engineers than the wins do.