Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(reranker): choice-contrastive variant of the Jev reranker adapter
evanmaranzanof64ef68dbfd9
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3a2a13e4..d3cb5f1d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Added+- `docs/jev-reranker-adapter.md` gains a choice-contrastive variant of the+ Jev reranker adapter+ (`docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py`): one+ `choice` question over the whole candidate list instead of one rubric+ `score` per candidate, choice probability mapped to `relevance`. On the+ same live golden set and 35B backend it measured hit@1 0.778 / NDCG@10+ 0.893 at 0.75 s mean rerank latency versus 0.636 / 0.833 at 2.2 s for+ the rubric adapter, and it is the shape in which replay-trained small+ judges hold their quality (4B: 0.808 vs 0.596). Safe because the+ reranker leg is sort-only; the rubric-adapter caveat on absolute+ relevance semantics is scoped to consumers that read absolute values.- `ai-memory reclaim-ledger-versions` drops the superseded versions of the rawhook event ledger that the pre-2.1.1 indexer left behind (#660), and nothingelse. It reports what it would remove and changes nothing unlessdiff --git a/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py b/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.pynew file mode 100644index 00000000..bf052e10--- /dev/null+++ b/docs/examples/jev-reranker-adapter/jev_rerank_shim_choice.py@@ -0,0 +1,203 @@+#!/usr/bin/env python3+"""OpenAI-compat -> Jev `/v1/systemone` reranker adapter for ai-memory.++ai-memory's LLM reranker (`AI_MEMORY_RERANKER=llm`) sends one structured chat+request per `memory_query`: a fixed system prompt plus a user JSON payload+`{"query", "candidates":[{"candidate","title","text"}]}`, and expects the+first balanced JSON object in the reply content to be+`{"scores":[{"candidate","relevance"}]}` (1-based indices, relevance in+[0,1]; a timeout, error, or invalid score set preserves the server's own+order).++This adapter recognises exactly that request (by the system-prompt prefix),+scores every candidate with one batched Jev `score` question whose rubric+mirrors the reranker prompt's own 1.0 / 0.7 / 0.3 / 0.0 guidance, and returns+the judgement as plain chat content. Everything else — consolidation, lint,+bootstrap — is reverse-proxied unchanged to the configured upstream, so only+reranking rides the Jev endpoint.++Why: a hosted reranker model that answers in tens of seconds dwarfs the rest+of a `memory_query`. A judge endpoint that scores a fixed rubric answers in+well under a second, which keeps `AI_MEMORY_RERANKER=llm` usable in+interactive sessions. On a 102-query golden set (see+docs/jev-reranker-adapter.md) this adapter matched the hosted reranker's+hit@1 / MRR / NDCG@10 while cutting mean rerank latency from ~20s to ~0.2s.++Env:+ JEV_URL Jev systemone endpoint (default http://127.0.0.1:18095/v1/systemone)+ JEV_MODEL model name sent to Jev (default jev-latest)+ UPSTREAM OpenAI-compat upstream base (default http://127.0.0.1:8000)+ LISTEN host:port to bind (default 127.0.0.1:18097)++Stdlib only. No secrets are stored: Authorization headers are forwarded+verbatim to the upstream.+"""+import json+import os+import sys+import time+import urllib.error+import urllib.request+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++JEV_URL = os.environ.get("JEV_URL", "http://127.0.0.1:18095/v1/systemone").rstrip("/")+JEV_MODEL = os.environ.get("JEV_MODEL", "jev-latest")+UPSTREAM = os.environ.get("UPSTREAM", "http://127.0.0.1:8000").rstrip("/")+LISTEN = os.environ.get("LISTEN", "127.0.0.1:18097")++# Fixed prefix of the reranker's system prompt; see+# crates/ai-memory-llm/src/reranker.rs in the ai-memory tree.+RERANK_SYSTEM_PREFIX = "You are a retrieval reranker for a software project's memory wiki."++# Mirrors the four grades of the reranker prompt (1.0 / 0.7 / 0.3 / 0.0).+RUBRIC = [+ "0.0 unrelated — does not address the query at all",+ "0.3 tangential — loosely related to the query",+ "0.7 same topic — same subsystem or topic, useful supporting context",+ "1.0 direct answer — directly answers the query",+]+++def log(msg):+ sys.stderr.write(f"[jev-rerank] {time.strftime('%H:%M:%S')} {msg}\n")+ sys.stderr.flush()+++def jev_rerank(payload):+ """Translate a reranker chat request into ONE batched Jev choice call.++ Choice-contrastive variant of the rubric adapter: a single `choice`+ question over all candidates, ranked by probability. On the same+ golden set and backend the rubric adapter answers from, comparative+ ranking beat per-candidate rubric scoring by double-digit hit@1 on+ both backends measured (see the comparison table in the docs), and it+ is the only shape in which replay-trained small models hold their+ quality. Relevance is the raw choice probability: monotonic in the+ probability order, which is all the reranker sorts on.+ """+ user = next((m.get("content", "") for m in payload.get("messages", [])+ if m.get("role") == "user"), "")+ data = json.loads(user)+ query, cands = data["query"], data["candidates"]+ lines = [f"User query: {query}", "", "Candidate documents:"]+ criteria = {}+ for c in cands:+ n = c["candidate"]+ desc = (c.get("title") or f"candidate {n}")[:120]+ lines.append(f"{n}. {desc} — {c['text'][:160]}")+ criteria[f"c{n}"] = desc+ questions = {"best": {"type": "choice",+ "instructions": "Which candidate document is most relevant to answering the user query?",+ "criteria": criteria}}+ body = {"model": JEV_MODEL, "state": "\n".join(lines), "questions": questions}+ t0 = time.perf_counter()+ req = urllib.request.Request(JEV_URL, data=json.dumps(body).encode(),+ headers={"Content-Type": "application/json"})+ with urllib.request.urlopen(req, timeout=25) as resp:+ out = json.loads(resp.read())+ dt = time.perf_counter() - t0+ probs = ((out.get("answers") or {}).get("best") or {}).get("probabilities") or {}+ scores = []+ for c in cands:+ n = c["candidate"]+ p = probs.get(f"c{n}", 0.0)+ p = float(p) if isinstance(p, (int, float)) else 0.0+ scores.append({"candidate": n, "relevance": round(p, 4)})+ log(f"jev-choice {len(cands)} candidates in {dt:.3f}s query={query[:60]!r}")+ return {"scores": scores}+++class Handler(BaseHTTPRequestHandler):+ protocol_version = "HTTP/1.1"++ def log_message(self, fmt, *args):+ pass++ def _reply(self, status, body=b"", content_type="application/json"):+ self.send_response(status)+ if content_type:+ self.send_header("Content-Type", content_type)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ if body:+ self.wfile.write(body)++ def _proxy(self, raw):+ """Reverse-proxy this request unchanged to the upstream."""+ req = urllib.request.Request(UPSTREAM + self.path, data=raw, method=self.command)+ for k, v in self.headers.items():+ if k.lower() in ("host", "content-length", "connection", "accept-encoding"):+ continue+ req.add_header(k, v)+ try:+ with urllib.request.urlopen(req, timeout=900) as resp:+ body = resp.read()+ self.send_response(resp.status)+ for k, v in resp.headers.items():+ if k.lower() in ("transfer-encoding", "connection", "content-length"):+ continue+ self.send_header(k, v)+ self._reply_finish(body)+ except urllib.error.HTTPError as e:+ self._reply(e.code, e.read())+ except Exception as e: # transport failure upstream+ log(f"forward error {self.command} {self.path}: {e}")+ self._reply(502, json.dumps({"error": str(e)[:200]}).encode())++ def _reply_finish(self, body):+ # headers besides Content-Length were already sent by the caller+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_POST(self):+ length = int(self.headers.get("Content-Length") or 0)+ raw = self.rfile.read(length) if length else b""+ if not self.path.rstrip("/").endswith("/chat/completions"):+ self._proxy(raw)+ return+ try:+ payload = json.loads(raw or b"{}")+ except json.JSONDecodeError:+ self._reply(400)+ return+ system = next((m.get("content", "") for m in payload.get("messages", [])+ if m.get("role") == "system"), "")+ if not system.startswith(RERANK_SYSTEM_PREFIX):+ self._proxy(raw) # consolidation / lint / bootstrap traffic+ return+ try:+ result = jev_rerank(payload)+ out = {+ "id": "jev-rerank",+ "object": "chat.completion",+ "model": payload.get("model", JEV_MODEL),+ "choices": [{+ "index": 0,+ "message": {"role": "assistant", "content": json.dumps(result)},+ "finish_reason": "stop",+ }],+ "usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},+ }+ self._reply(200, json.dumps(out).encode())+ except Exception as e:+ # The server treats any failure as "keep the original order".+ log(f"jev rerank failed: {e}")+ self._reply(500)++ def do_GET(self):+ self._proxy(None)+++def main():+ host, port = LISTEN.rsplit(":", 1)+ srv = ThreadingHTTPServer((host, int(port)), Handler)+ log(f"listening {LISTEN} jev={JEV_URL} upstream={UPSTREAM}")+ try:+ srv.serve_forever()+ except KeyboardInterrupt:+ pass+++if __name__ == "__main__":+ main()diff --git a/docs/jev-reranker-adapter.md b/docs/jev-reranker-adapter.mdindex e22bafbb..4acf8ae2 100644--- a/docs/jev-reranker-adapter.md+++ b/docs/jev-reranker-adapter.md@@ -91,6 +91,11 @@ AI_MEMORY_LLM_BASE_URL=http://127.0.0.1:18097/v1AI_MEMORY_RERANKER=llm```+To run the choice-contrastive variant instead, point `ExecStart` at+`jev_rerank_shim_choice.py`; reranker requests then log+`jev-choice N candidates in X.XXXs`. Everything else (env, unit,+provider config) is identical.+Verify the split with `journalctl -u <unit>`: reranker requests log`jev N candidates in X.XXXs`, everything else is silent (proxied).@@ -129,15 +134,60 @@ question is asked once per candidate): ~0.2 s for 10 candidates, ~1.6 smean for the 15–30-candidate batches the live server sends, still an orderof magnitude under any hosted reasoning model.+## Choice-contrastive variant++[`jev_rerank_shim_choice.py`](examples/jev-reranker-adapter/jev_rerank_shim_choice.py)+flips the question shape: instead of one rubric `score` per candidate, it+asks a single `choice` question over the whole candidate list ("which+document is most relevant?") and maps each candidate's choice probability+directly to `relevance`.++That is deliberately the substitution the caveat below used to warn about.+It is sound here because the reranker's consumer is **sort-only**: the+server reorders candidates by `relevance` and never thresholds or sums its+absolute value, and a per-question monotonic map preserves order exactly.+The old warning applies to consumers that *read* absolute relevance+grades; the reranker leg does not, and the caveat below is now scoped to+that case.++Measured on one production deployment (same 102-query golden set, same+live server and backend, end-to-end through the adapter):++| Backend model | Adapter | hit@1 | NDCG@10 | mean rerank latency |+|---|---|---|---|---|+| 35B (Qwen3.6-35B-A3B) | rubric `score` | 0.636 | 0.833 | 2.2 s |+| 35B (Qwen3.6-35B-A3B) | **choice** | **0.778** | **0.893** | **0.75 s** |+| 4B replay-trained | rubric `score` | 0.596 | 0.776 | 8.2 s |+| 4B replay-trained | **choice** | **0.808** | **0.895** | 1.16 s |++Two takeaways. First, the choice shape beat the rubric shape by 14–21+hit@1 points *on the same backend model*: relative comparison is an easier+judgement than absolute grading, and it shows up in both quality and+latency (one batched question for the whole list instead of one question+per candidate). Second, the rubric shape is where replay-trained small+models collapse (0.596 on 4B vs 0.636 on 35B); in the choice shape the+same 4B overtakes the 35B. If your backend is a small replay-trained+judge, the choice adapter is the difference between usable and not.++Prefer the choice adapter when the reranker leg is the only consumer.+Keep the rubric adapter if anything downstream reads absolute relevance+grades (thresholding, logging heuristics, score fusion) — choice+probabilities carry no absolute meaning and their scale shifts with+candidate count.+## Caveats- The adapter keys on the exact system-prompt prefix above. If thereranker prompt wording changes in a future release, detection (notscoring) breaks first: reranker requests would be proxied to the hostedmodel, which is the pre-adapter behavior, not an outage.-- Scoring uses Jev's calibrated rubric grades. Do not substitute- per-candidate choice probabilities: those are normalized across options- (winner ≈ 0.99, rest ≈ 0.001) and do not fit the reranker's 0–1- relevance semantics.+- Rubric scoring produces absolute relevance grades. If anything+ downstream of `relevance` reads absolute values (thresholding, fusion,+ logged heuristics), keep the rubric adapter: per-candidate choice+ probabilities from the choice adapter are normalized across the whole+ candidate list (winner ≈ 0.99, rest ≈ 0.001, scale shifts with+ candidate count) and carry ordering information only. The sort-only+ reranker leg itself reads no absolute values, which is why the choice+ variant is safe to substitute for it.- Run the adapter on loopback or a trusted private network. It forwards`Authorization` headers verbatim and adds no authentication of its own.