Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(reranker): choice-contrastive variant of the Jev reranker adapter

evanmaranzanof64ef68dbfd9
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3a2a13e4..d3cb5f1d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- `docs/jev-reranker-adapter.md` gains a choice-contrastive variant of the
+ Jev reranker adapter
+ (`docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py`): one
+ `choice` question over the whole candidate list instead of one rubric
+ `score` per candidate, choice probability mapped to `relevance`. On the
+ same live golden set and 35B backend it measured hit@1 0.778 / NDCG@10
+ 0.893 at 0.75 s mean rerank latency versus 0.636 / 0.833 at 2.2 s for
+ the rubric adapter, and it is the shape in which replay-trained small
+ judges hold their quality (4B: 0.808 vs 0.596). Safe because the
+ reranker leg is sort-only; the rubric-adapter caveat on absolute
+ relevance semantics is scoped to consumers that read absolute values.
- `ai-memory reclaim-ledger-versions` drops the superseded versions of the raw
hook event ledger that the pre-2.1.1 indexer left behind (#660), and nothing
else. It reports what it would remove and changes nothing unless
diff --git a/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py b/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py
new file mode 100644
index 00000000..bf052e10
--- /dev/null
+++ b/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py
@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+"""OpenAI-compat -> Jev `/v1/systemone` reranker adapter for ai-memory.
+
+ai-memory's LLM reranker (`AI_MEMORY_RERANKER=llm`) sends one structured chat
+request per `memory_query`: a fixed system prompt plus a user JSON payload
+`{"query", "candidates":[{"candidate","title","text"}]}`, and expects the
+first balanced JSON object in the reply content to be
+`{"scores":[{"candidate","relevance"}]}` (1-based indices, relevance in
+[0,1]; a timeout, error, or invalid score set preserves the server's own
+order).
+
+This adapter recognises exactly that request (by the system-prompt prefix),
+scores every candidate with one batched Jev `score` question whose rubric
+mirrors the reranker prompt's own 1.0 / 0.7 / 0.3 / 0.0 guidance, and returns
+the judgement as plain chat content. Everything else — consolidation, lint,
+bootstrap — is reverse-proxied unchanged to the configured upstream, so only
+reranking rides the Jev endpoint.
+
+Why: a hosted reranker model that answers in tens of seconds dwarfs the rest
+of a `memory_query`. A judge endpoint that scores a fixed rubric answers in
+well under a second, which keeps `AI_MEMORY_RERANKER=llm` usable in
+interactive sessions. On a 102-query golden set (see
+docs/jev-reranker-adapter.md) this adapter matched the hosted reranker's
+hit@1 / MRR / NDCG@10 while cutting mean rerank latency from ~20s to ~0.2s.
+
+Env:
+ JEV_URL Jev systemone endpoint (default http://127.0.0.1:18095/v1/systemone)
+ JEV_MODEL model name sent to Jev (default jev-latest)
+ UPSTREAM OpenAI-compat upstream base (default http://127.0.0.1:8000)
+ LISTEN host:port to bind (default 127.0.0.1:18097)
+
+Stdlib only. No secrets are stored: Authorization headers are forwarded
+verbatim to the upstream.
+"""
+import json
+import os
+import sys
+import time
+import urllib.error
+import urllib.request
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+JEV_URL = os.environ.get("JEV_URL", "http://127.0.0.1:18095/v1/systemone").rstrip("/")
+JEV_MODEL = os.environ.get("JEV_MODEL", "jev-latest")
+UPSTREAM = os.environ.get("UPSTREAM", "http://127.0.0.1:8000").rstrip("/")
+LISTEN = os.environ.get("LISTEN", "127.0.0.1:18097")
+
+# Fixed prefix of the reranker's system prompt; see
+# crates/ai-memory-llm/src/reranker.rs in the ai-memory tree.
+RERANK_SYSTEM_PREFIX = "You are a retrieval reranker for a software project's memory wiki."
+
+# Mirrors the four grades of the reranker prompt (1.0 / 0.7 / 0.3 / 0.0).
+RUBRIC = [
+ "0.0 unrelated — does not address the query at all",
+ "0.3 tangential — loosely related to the query",
+ "0.7 same topic — same subsystem or topic, useful supporting context",
+ "1.0 direct answer — directly answers the query",
+]
+
+
+def log(msg):
+ sys.stderr.write(f"[jev-rerank] {time.strftime('%H:%M:%S')} {msg}\n")
+ sys.stderr.flush()
+
+
+def jev_rerank(payload):
+ """Translate a reranker chat request into ONE batched Jev choice call.
+
+ Choice-contrastive variant of the rubric adapter: a single `choice`
+ question over all candidates, ranked by probability. On the same
+ golden set and backend the rubric adapter answers from, comparative
+ ranking beat per-candidate rubric scoring by double-digit hit@1 on
+ both backends measured (see the comparison table in the docs), and it
+ is the only shape in which replay-trained small models hold their
+ quality. Relevance is the raw choice probability: monotonic in the
+ probability order, which is all the reranker sorts on.
+ """
+ user = next((m.get("content", "") for m in payload.get("messages", [])
+ if m.get("role") == "user"), "")
+ data = json.loads(user)
+ query, cands = data["query"], data["candidates"]
+ lines = [f"User query: {query}", "", "Candidate documents:"]
+ criteria = {}
+ for c in cands:
+ n = c["candidate"]
+ desc = (c.get("title") or f"candidate {n}")[:120]
+ lines.append(f"{n}. {desc} — {c['text'][:160]}")
+ criteria[f"c{n}"] = desc
+ questions = {"best": {"type": "choice",
+ "instructions": "Which candidate document is most relevant to answering the user query?",
+ "criteria": criteria}}
+ body = {"model": JEV_MODEL, "state": "\n".join(lines), "questions": questions}
+ t0 = time.perf_counter()
+ req = urllib.request.Request(JEV_URL, data=json.dumps(body).encode(),
+ headers={"Content-Type": "application/json"})
+ with urllib.request.urlopen(req, timeout=25) as resp:
+ out = json.loads(resp.read())
+ dt = time.perf_counter() - t0
+ probs = ((out.get("answers") or {}).get("best") or {}).get("probabilities") or {}
+ scores = []
+ for c in cands:
+ n = c["candidate"]
+ p = probs.get(f"c{n}", 0.0)
+ p = float(p) if isinstance(p, (int, float)) else 0.0
+ scores.append({"candidate": n, "relevance": round(p, 4)})
+ log(f"jev-choice {len(cands)} candidates in {dt:.3f}s query={query[:60]!r}")
+ return {"scores": scores}
+
+
+class Handler(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def log_message(self, fmt, *args):
+ pass
+
+ def _reply(self, status, body=b"", content_type="application/json"):
+ self.send_response(status)
+ if content_type:
+ self.send_header("Content-Type", content_type)
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ if body:
+ self.wfile.write(body)
+
+ def _proxy(self, raw):
+ """Reverse-proxy this request unchanged to the upstream."""
+ req = urllib.request.Request(UPSTREAM + self.path, data=raw, method=self.command)
+ for k, v in self.headers.items():
+ if k.lower() in ("host", "content-length", "connection", "accept-encoding"):
+ continue
+ req.add_header(k, v)
+ try:
+ with urllib.request.urlopen(req, timeout=900) as resp:
+ body = resp.read()
+ self.send_response(resp.status)
+ for k, v in resp.headers.items():
+ if k.lower() in ("transfer-encoding", "connection", "content-length"):
+ continue
+ self.send_header(k, v)
+ self._reply_finish(body)
+ except urllib.error.HTTPError as e:
+ self._reply(e.code, e.read())
+ except Exception as e: # transport failure upstream
+ log(f"forward error {self.command} {self.path}: {e}")
+ self._reply(502, json.dumps({"error": str(e)[:200]}).encode())
+
+ def _reply_finish(self, body):
+ # headers besides Content-Length were already sent by the caller
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def do_POST(self):
+ length = int(self.headers.get("Content-Length") or 0)
+ raw = self.rfile.read(length) if length else b""
+ if not self.path.rstrip("/").endswith("/chat/completions"):
+ self._proxy(raw)
+ return
+ try:
+ payload = json.loads(raw or b"{}")
+ except json.JSONDecodeError:
+ self._reply(400)
+ return
+ system = next((m.get("content", "") for m in payload.get("messages", [])
+ if m.get("role") == "system"), "")
+ if not system.startswith(RERANK_SYSTEM_PREFIX):
+ self._proxy(raw) # consolidation / lint / bootstrap traffic
+ return
+ try:
+ result = jev_rerank(payload)
+ out = {
+ "id": "jev-rerank",
+ "object": "chat.completion",
+ "model": payload.get("model", JEV_MODEL),
+ "choices": [{
+ "index": 0,
+ "message": {"role": "assistant", "content": json.dumps(result)},
+ "finish_reason": "stop",
+ }],
+ "usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},
+ }
+ self._reply(200, json.dumps(out).encode())
+ except Exception as e:
+ # The server treats any failure as "keep the original order".
+ log(f"jev rerank failed: {e}")
+ self._reply(500)
+
+ def do_GET(self):
+ self._proxy(None)
+
+
+def main():
+ host, port = LISTEN.rsplit(":", 1)
+ srv = ThreadingHTTPServer((host, int(port)), Handler)
+ log(f"listening {LISTEN} jev={JEV_URL} upstream={UPSTREAM}")
+ try:
+ srv.serve_forever()
+ except KeyboardInterrupt:
+ pass
+
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/jev-reranker-adapter.md b/docs/jev-reranker-adapter.md
index e22bafbb..4acf8ae2 100644
--- a/docs/jev-reranker-adapter.md
+++ b/docs/jev-reranker-adapter.md
@@ -91,6 +91,11 @@ AI_MEMORY_LLM_BASE_URL=http://127.0.0.1:18097/v1
AI_MEMORY_RERANKER=llm
```
+To run the choice-contrastive variant instead, point `ExecStart` at
+`jev_rerank_shim_choice.py`; reranker requests then log
+`jev-choice N candidates in X.XXXs`. Everything else (env, unit,
+provider config) is identical.
+
Verify the split with `journalctl -u <unit>`: reranker requests log
`jev N candidates in X.XXXs`, everything else is silent (proxied).
@@ -129,15 +134,60 @@ question is asked once per candidate): ~0.2 s for 10 candidates, ~1.6 s
mean for the 15–30-candidate batches the live server sends, still an order
of magnitude under any hosted reasoning model.
+## Choice-contrastive variant
+
+[`jev_rerank_shim_choice.py`](examples/jev-reranker-adapter/jev_rerank_shim_choice.py)
+flips the question shape: instead of one rubric `score` per candidate, it
+asks a single `choice` question over the whole candidate list ("which
+document is most relevant?") and maps each candidate's choice probability
+directly to `relevance`.
+
+That is deliberately the substitution the caveat below used to warn about.
+It is sound here because the reranker's consumer is **sort-only**: the
+server reorders candidates by `relevance` and never thresholds or sums its
+absolute value, and a per-question monotonic map preserves order exactly.
+The old warning applies to consumers that *read* absolute relevance
+grades; the reranker leg does not, and the caveat below is now scoped to
+that case.
+
+Measured on one production deployment (same 102-query golden set, same
+live server and backend, end-to-end through the adapter):
+
+| Backend model | Adapter | hit@1 | NDCG@10 | mean rerank latency |
+|---|---|---|---|---|
+| 35B (Qwen3.6-35B-A3B) | rubric `score` | 0.636 | 0.833 | 2.2 s |
+| 35B (Qwen3.6-35B-A3B) | **choice** | **0.778** | **0.893** | **0.75 s** |
+| 4B replay-trained | rubric `score` | 0.596 | 0.776 | 8.2 s |
+| 4B replay-trained | **choice** | **0.808** | **0.895** | 1.16 s |
+
+Two takeaways. First, the choice shape beat the rubric shape by 14–21
+hit@1 points *on the same backend model*: relative comparison is an easier
+judgement than absolute grading, and it shows up in both quality and
+latency (one batched question for the whole list instead of one question
+per candidate). Second, the rubric shape is where replay-trained small
+models collapse (0.596 on 4B vs 0.636 on 35B); in the choice shape the
+same 4B overtakes the 35B. If your backend is a small replay-trained
+judge, the choice adapter is the difference between usable and not.
+
+Prefer the choice adapter when the reranker leg is the only consumer.
+Keep the rubric adapter if anything downstream reads absolute relevance
+grades (thresholding, logging heuristics, score fusion) — choice
+probabilities carry no absolute meaning and their scale shifts with
+candidate count.
+
## Caveats
- The adapter keys on the exact system-prompt prefix above. If the
reranker prompt wording changes in a future release, detection (not
scoring) breaks first: reranker requests would be proxied to the hosted
model, which is the pre-adapter behavior, not an outage.
-- Scoring uses Jev's calibrated rubric grades. Do not substitute
- per-candidate choice probabilities: those are normalized across options
- (winner ≈ 0.99, rest ≈ 0.001) and do not fit the reranker's 0–1
- relevance semantics.
+- Rubric scoring produces absolute relevance grades. If anything
+ downstream of `relevance` reads absolute values (thresholding, fusion,
+ logged heuristics), keep the rubric adapter: per-candidate choice
+ probabilities from the choice adapter are normalized across the whole
+ candidate list (winner ≈ 0.99, rest ≈ 0.001, scale shifts with
+ candidate count) and carry ordering information only. The sort-only
+ reranker leg itself reads no absolute values, which is why the choice
+ variant is safe to substitute for it.
- Run the adapter on loopback or a trusted private network. It forwards
`Authorization` headers verbatim and adds no authentication of its own.