Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(reranker): ask the choice adapter for the specific answering page

evanmaranzano3cc78985d298
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d3cb5f1d..0dd7e2a2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,13 +12,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Jev reranker adapter
(`docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py`): one
`choice` question over the whole candidate list instead of one rubric
- `score` per candidate, choice probability mapped to `relevance`. On the
- same live golden set and 35B backend it measured hit@1 0.778 / NDCG@10
- 0.893 at 0.75 s mean rerank latency versus 0.636 / 0.833 at 2.2 s for
- the rubric adapter, and it is the shape in which replay-trained small
- judges hold their quality (4B: 0.808 vs 0.596). Safe because the
- reranker leg is sort-only; the rubric-adapter caveat on absolute
- relevance semantics is scoped to consumers that read absolute values.
+ `score` per candidate, choice probability mapped to `relevance`. The
+ example asks for the specific page that contains the answer, and not to
+ pick an index, catalog, or summary when a more specific page answers.
+ On the same live golden set and 35B backend the choice shape measured
+ hit@1 0.778 / NDCG@10 0.893 at 0.75 s mean rerank latency versus 0.636 /
+ 0.833 at 2.2 s for the rubric adapter, and it is the shape in which
+ replay-trained small judges hold their quality (4B: 0.808 vs 0.596).
+ Safe because the reranker leg is sort-only; the rubric-adapter caveat
+ on absolute relevance semantics is scoped to consumers that read
+ absolute values.
- `ai-memory reclaim-ledger-versions` drops the superseded versions of the raw
hook event ledger that the pre-2.1.1 indexer left behind (#660), and nothing
else. It reports what it would remove and changes nothing unless
diff --git a/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py b/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py
index bf052e10..1079f684 100644
--- a/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py
+++ b/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py
@@ -86,8 +86,14 @@ def jev_rerank(payload):
desc = (c.get("title") or f"candidate {n}")[:120]
lines.append(f"{n}. {desc} — {c['text'][:160]}")
criteria[f"c{n}"] = desc
+ # Pool v3 on the 35B backend: this wording beat "most relevant" 0.828 -> 0.879
+ # hit@1 (5 fixed, 0 broken). Relevance stays the raw choice probability.
questions = {"best": {"type": "choice",
- "instructions": "Which candidate document is most relevant to answering the user query?",
+ "instructions": (
+ "Which candidate directly answers the user query? "
+ "Choose the specific page that contains the answer. "
+ "Do not choose an index, catalog, or summary page if a more specific page answers the query."
+ ),
"criteria": criteria}}
body = {"model": JEV_MODEL, "state": "\n".join(lines), "questions": questions}
t0 = time.perf_counter()
diff --git a/docs/jev-reranker-adapter.md b/docs/jev-reranker-adapter.md
index 4acf8ae2..b173f241 100644
--- a/docs/jev-reranker-adapter.md
+++ b/docs/jev-reranker-adapter.md
@@ -138,9 +138,13 @@ of magnitude under any hosted reasoning model.
[`jev_rerank_shim_choice.py`](examples/jev-reranker-adapter/jev_rerank_shim_choice.py)
flips the question shape: instead of one rubric `score` per candidate, it
-asks a single `choice` question over the whole candidate list ("which
-document is most relevant?") and maps each candidate's choice probability
-directly to `relevance`.
+asks a single `choice` question over the whole candidate list and maps each
+candidate's choice probability directly to `relevance`. The production
+wording asks which specific page contains the answer, and tells the model
+not to pick an index, catalog, or summary when a more specific page answers.
+On a 30-candidate pool that wording moved the 35B from 0.828 to 0.879 hit@1
+(5 fixed, 0 broken) versus "which document is most relevant?". The table
+below is the earlier choice-versus-rubric comparison, not that wording test.
That is deliberately the substitution the caveat below used to warn about.
It is sound here because the reranker's consumer is **sort-only**: the