Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

bench(harbor-3arm): flag partial arm coverage + neutralize the tuned prompt's pre-asserted claim (#516)

xerj-orgd99b7de25000
diff --git a/tools/benchmarks/harbor-3arm/collect_results.py b/tools/benchmarks/harbor-3arm/collect_results.py
index 58e01ce03..d5d8aa031 100755
--- a/tools/benchmarks/harbor-3arm/collect_results.py
+++ b/tools/benchmarks/harbor-3arm/collect_results.py
@@ -103,12 +103,22 @@ def main():
for (bench, model, arm), tasks in cells.items():
by_bm[(bench, model)][arm] = {
t for t, v in tasks.items() if v["reward"] is not None}
+ canonical_arms = {"native", "xerj", "xerj-tuned"}
for (bench, model), arms in sorted(by_bm.items()):
if len(arms) < 2:
continue
+ present = set(arms)
common = set.intersection(*arms.values())
+ # A comparison that isn't the full 3-arm set must say so LOUDLY — an
+ # absent arm otherwise reads as a footnote and a reader can mistake, e.g.,
+ # native vs xerj-tuned for the headline result (#516).
+ coverage = "" if present == canonical_arms else (
+ f" ⚠ PARTIAL ARM COVERAGE — missing "
+ f"{sorted(canonical_arms - present) or 'nothing'}, "
+ f"unexpected {sorted(present - canonical_arms) or 'nothing'}; "
+ f"this is NOT the canonical 3-arm comparison")
print(f"\n{bench} / {model}: {len(common)} tasks finished in all "
- f"{len(arms)} arms")
+ f"{len(arms)} arms ({', '.join(sorted(present))}){coverage}")
for arm in sorted(arms):
tv = cells[(bench, model, arm)]
k = sum(1 for t in common if tv[t]["resolved"])
diff --git a/tools/benchmarks/harbor-3arm/xerj_prompt_tuned.j2 b/tools/benchmarks/harbor-3arm/xerj_prompt_tuned.j2
index d68b3e8a4..4a3702851 100644
--- a/tools/benchmarks/harbor-3arm/xerj_prompt_tuned.j2
+++ b/tools/benchmarks/harbor-3arm/xerj_prompt_tuned.j2
@@ -2,8 +2,9 @@
---
TOOL NOTE: XERJ code search is installed (server at http://localhost:9210) and
-tuned for this work. It is usually MUCH faster than grep for finding the right
-definition in an unfamiliar repo. Playbook:
+tuned for this work. It can return the enclosing definition of a symbol directly
+(the whole function/class body), so you rarely need to open files to read a
+definition. Playbook:
1. Index once (seconds; --no-semantic keeps it pure BM25 — the embedder is
lexical, so semantic fields add cost without adding meaning):