Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(reranker): add Jev judge-endpoint adapter for AI_MEMORY_RERANKER=llm
evanmaranzano45d970d2206b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e928436e..9e852b42 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Added+- `docs/jev-reranker-adapter.md` documents a stdlib-only adapter+ (`docs/examples/jev-reranker-adapter/jev_rerank_shim.py`) that serves the+ `AI_MEMORY_RERANKER=llm` request leg from a Jev `/v1/systemone` judge+ endpoint while reverse-proxying consolidation/lint/bootstrap traffic to+ the configured provider unchanged. On a 102-query golden set the judge+ matched the hosted reranker's hit@1/MRR/NDCG@10 (0.778/0.838/0.873 vs+ 0.778/0.840/0.875) at 0.205 s mean latency instead of 20.2 s — the+ hosted mean sat on the server's 20 s completion timeout, which made the+ reranker stall every query before falling back in production.+### Fixed- `bootstrap` now retries a chunk's LLM call on a transient error beforegiving up, instead of letting one blip discard the whole multi-chunk rundiff --git a/docs/examples/jev-reranker-adapter/jev_rerank_shim.py b/docs/examples/jev-reranker-adapter/jev_rerank_shim.pynew file mode 100755index 00000000..98acb5a2--- /dev/null+++ b/docs/examples/jev-reranker-adapter/jev_rerank_shim.py@@ -0,0 +1,197 @@+#!/usr/bin/env python3+"""OpenAI-compat -> Jev `/v1/systemone` reranker adapter for ai-memory.++ai-memory's LLM reranker (`AI_MEMORY_RERANKER=llm`) sends one structured chat+request per `memory_query`: a fixed system prompt plus a user JSON payload+`{"query", "candidates":[{"candidate","title","text"}]}`, and expects the+first balanced JSON object in the reply content to be+`{"scores":[{"candidate","relevance"}]}` (1-based indices, relevance in+[0,1]; a timeout, error, or invalid score set preserves the server's own+order).++This adapter recognises exactly that request (by the system-prompt prefix),+scores every candidate with one batched Jev `score` question whose rubric+mirrors the reranker prompt's own 1.0 / 0.7 / 0.3 / 0.0 guidance, and returns+the judgement as plain chat content. Everything else — consolidation, lint,+bootstrap — is reverse-proxied unchanged to the configured upstream, so only+reranking rides the Jev endpoint.++Why: a hosted reranker model that answers in tens of seconds dwarfs the rest+of a `memory_query`. A judge endpoint that scores a fixed rubric answers in+well under a second, which keeps `AI_MEMORY_RERANKER=llm` usable in+interactive sessions. On a 102-query golden set (see+docs/jev-reranker-adapter.md) this adapter matched the hosted reranker's+hit@1 / MRR / NDCG@10 while cutting mean rerank latency from ~20s to ~0.2s.++Env:+ JEV_URL Jev systemone endpoint (default http://127.0.0.1:18095/v1/systemone)+ JEV_MODEL model name sent to Jev (default jev-latest)+ UPSTREAM OpenAI-compat upstream base (default http://127.0.0.1:8000)+ LISTEN host:port to bind (default 127.0.0.1:18097)++Stdlib only. No secrets are stored: Authorization headers are forwarded+verbatim to the upstream.+"""+import json+import os+import sys+import time+import urllib.error+import urllib.request+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++JEV_URL = os.environ.get("JEV_URL", "http://127.0.0.1:18095/v1/systemone").rstrip("/")+JEV_MODEL = os.environ.get("JEV_MODEL", "jev-latest")+UPSTREAM = os.environ.get("UPSTREAM", "http://127.0.0.1:8000").rstrip("/")+LISTEN = os.environ.get("LISTEN", "127.0.0.1:18097")++# Fixed prefix of the reranker's system prompt; see+# crates/ai-memory-llm/src/reranker.rs in the ai-memory tree.+RERANK_SYSTEM_PREFIX = "You are a retrieval reranker for a software project's memory wiki."++# Mirrors the four grades of the reranker prompt (1.0 / 0.7 / 0.3 / 0.0).+RUBRIC = [+ "0.0 unrelated — does not address the query at all",+ "0.3 tangential — loosely related to the query",+ "0.7 same topic — same subsystem or topic, useful supporting context",+ "1.0 direct answer — directly answers the query",+]+++def log(msg):+ sys.stderr.write(f"[jev-rerank] {time.strftime('%H:%M:%S')} {msg}\n")+ sys.stderr.flush()+++def jev_rerank(payload):+ """Translate a reranker chat request into one batched Jev score call."""+ user = next((m.get("content", "") for m in payload.get("messages", [])+ if m.get("role") == "user"), "")+ data = json.loads(user)+ query, cands = data["query"], data["candidates"]+ lines = [f"User query: {query}", "", "Candidate documents:"]+ questions = {}+ for c in cands:+ n = c["candidate"]+ lines.append(f"{n}. {c['title']} — {c['text']}")+ questions[f"c{n}"] = {+ "type": "score",+ "instructions": (+ f"Apply the relevance rubric to candidate {n}: how well "+ "does it answer the user query?"+ ),+ "criteria": RUBRIC,+ }+ body = {"model": JEV_MODEL, "state": "\n".join(lines), "questions": questions}+ t0 = time.perf_counter()+ req = urllib.request.Request(JEV_URL, data=json.dumps(body).encode(),+ headers={"Content-Type": "application/json"})+ with urllib.request.urlopen(req, timeout=25) as resp:+ out = json.loads(resp.read())+ dt = time.perf_counter() - t0+ scores = []+ for c in cands:+ ans = (out.get("answers") or {}).get(f"c{c['candidate']}") or {}+ raw = ans.get("score")+ # criteria index 0..3 -> relevance 0.0 / 0.3 / 0.7 / 1.0+ relevance = max(0.0, min(1.0, raw / (len(RUBRIC) - 1))) \+ if isinstance(raw, (int, float)) else 0.0+ scores.append({"candidate": c["candidate"], "relevance": round(relevance, 4)})+ log(f"jev {len(cands)} candidates in {dt:.3f}s query={query[:60]!r}")+ return {"scores": scores}+++class Handler(BaseHTTPRequestHandler):+ protocol_version = "HTTP/1.1"++ def log_message(self, fmt, *args):+ pass++ def _reply(self, status, body=b"", content_type="application/json"):+ self.send_response(status)+ if content_type:+ self.send_header("Content-Type", content_type)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ if body:+ self.wfile.write(body)++ def _proxy(self, raw):+ """Reverse-proxy this request unchanged to the upstream."""+ req = urllib.request.Request(UPSTREAM + self.path, data=raw, method=self.command)+ for k, v in self.headers.items():+ if k.lower() in ("host", "content-length", "connection", "accept-encoding"):+ continue+ req.add_header(k, v)+ try:+ with urllib.request.urlopen(req, timeout=900) as resp:+ body = resp.read()+ self.send_response(resp.status)+ for k, v in resp.headers.items():+ if k.lower() in ("transfer-encoding", "connection", "content-length"):+ continue+ self.send_header(k, v)+ self._reply_finish(body)+ except urllib.error.HTTPError as e:+ self._reply(e.code, e.read())+ except Exception as e: # transport failure upstream+ log(f"forward error {self.command} {self.path}: {e}")+ self._reply(502, json.dumps({"error": str(e)[:200]}).encode())++ def _reply_finish(self, body):+ # headers besides Content-Length were already sent by the caller+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_POST(self):+ length = int(self.headers.get("Content-Length") or 0)+ raw = self.rfile.read(length) if length else b""+ if not self.path.rstrip("/").endswith("/chat/completions"):+ self._proxy(raw)+ return+ try:+ payload = json.loads(raw or b"{}")+ except json.JSONDecodeError:+ self._reply(400)+ return+ system = next((m.get("content", "") for m in payload.get("messages", [])+ if m.get("role") == "system"), "")+ if not system.startswith(RERANK_SYSTEM_PREFIX):+ self._proxy(raw) # consolidation / lint / bootstrap traffic+ return+ try:+ result = jev_rerank(payload)+ out = {+ "id": "jev-rerank",+ "object": "chat.completion",+ "model": payload.get("model", JEV_MODEL),+ "choices": [{+ "index": 0,+ "message": {"role": "assistant", "content": json.dumps(result)},+ "finish_reason": "stop",+ }],+ "usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0},+ }+ self._reply(200, json.dumps(out).encode())+ except Exception as e:+ # The server treats any failure as "keep the original order".+ log(f"jev rerank failed: {e}")+ self._reply(500)++ def do_GET(self):+ self._proxy(None)+++def main():+ host, port = LISTEN.rsplit(":", 1)+ srv = ThreadingHTTPServer((host, int(port)), Handler)+ log(f"listening {LISTEN} jev={JEV_URL} upstream={UPSTREAM}")+ try:+ srv.serve_forever()+ except KeyboardInterrupt:+ pass+++if __name__ == "__main__":+ main()diff --git a/docs/jev-reranker-adapter.md b/docs/jev-reranker-adapter.mdnew file mode 100644index 00000000..e22bafbb--- /dev/null+++ b/docs/jev-reranker-adapter.md@@ -0,0 +1,143 @@+# Jev Reranker Adapter++> Route ai-memory's LLM reranker to a Jev judge endpoint while keeping the+> hosted LLM for everything else. Stdlib-only Python, no Rust changes.++[`AI_MEMORY_RERANKER=llm`](llm-providers.md) reorders `memory_query`+candidates with the configured chat provider. That works until the provider+is a hosted reasoning model: the reranker prompt is long, a graded judgement+over up to 30 candidates, and a slow model answers in tens of seconds —+longer than one `memory_query` should ever take, and long enough to trip the+server's completion timeout, which turns the reranker into a dead feature+that stalls every query before falling back to the original order.++The reranker prompt, though, is already a scoring rubric: grade each+candidate 1.0 (direct answer) / 0.7 (same topic) / 0.3 (tangential) /+0.0 (unrelated). A judge endpoint that scores a fixed rubric answers the+same question without autoregressive decoding. The adapter in+[`docs/examples/jev-reranker-adapter/jev_rerank_shim.py`](examples/jev-reranker-adapter/jev_rerank_shim.py)+sits between ai-memory and the provider, translates exactly that request+into one batched Jev `score` call, and proxies everything else unchanged.++## How it works++ai-memory 2.4.0 has a single chat-provider configuration, so the reranker+and consolidation share one base URL. The adapter does not try to split+configuration; it splits traffic by request shape:++1. The reranker's system prompt starts with the fixed sentence+ `You are a retrieval reranker for a software project's memory wiki.`+ (see `crates/ai-memory-llm/src/reranker.rs`). Requests whose system+ message has that prefix are reranker requests.+2. The user message is `{"query", "candidates": [{"candidate", "title",+ "text"}]}` with 1-based indices. The adapter renders one `state` block+ (query + numbered candidates) and one `score` question per candidate+ whose `criteria` mirror the reranker prompt's four grades, then makes a+ single POST to the Jev `/v1/systemone` endpoint.+3. Each answer's rubric index maps back to `relevance` 0.0 / 0.3 / 0.7 /+ 1.0 (index / 3), and the adapter returns+ `{"scores": [{"candidate": n, "relevance": f}]}` as plain chat-completion+ content — the exact shape the reranker's tolerant parser expects.+4. Every other request — consolidation, lint, bootstrap, plain chats — is+ reverse-proxied to the real upstream byte-for-byte, `Authorization`+ forwarded verbatim. The adapter stores no secrets.++Failure semantics match the server contract: if the Jev call fails, the+adapter answers HTTP 500 and ai-memory keeps its own candidate order, the+same as any provider outage. A judge endpoint that is down degrades to+"no reranking", never to "no search".++One cosmetic note: the server logs still show the provider's configured+model name for reranking (it comes from provider config, not from the+adapter's reply), so the reranker leg will be attributed to your hosted+model in `docker logs` even while the adapter serves it.++## Deploy++The adapter is stdlib-only Python 3. Configure with environment variables+(all defaults are loopback examples):++| Env | Meaning | Default |+|---|---|---|+| `JEV_URL` | Jev `/v1/systemone` endpoint | `http://127.0.0.1:18095/v1/systemone` |+| `JEV_MODEL` | model name sent to Jev | `jev-latest` |+| `UPSTREAM` | real OpenAI-compat base URL | `http://127.0.0.1:8000` |+| `LISTEN` | adapter bind address | `127.0.0.1:18097` |++Run it next to the server (systemd unit adapted to your paths):++```ini+[Unit]+Description=ai-memory Jev reranker adapter+After=network-online.target++[Service]+ExecStart=/usr/bin/python3 /opt/ai-memory/ops/jev_rerank_shim.py+Environment=JEV_URL=http://127.0.0.1:18095/v1/systemone+Environment=UPSTREAM=http://127.0.0.1:8000+Environment=LISTEN=127.0.0.1:18097+Restart=always+RestartSec=3++[Install]+WantedBy=multi-user.target+```++Then point the provider at the adapter instead of the upstream:++```console+AI_MEMORY_LLM_PROVIDER=openai-compat+AI_MEMORY_LLM_BASE_URL=http://127.0.0.1:18097/v1+AI_MEMORY_RERANKER=llm+```++Verify the split with `journalctl -u <unit>`: reranker requests log+`jev N candidates in X.XXXs`, everything else is silent (proxied).++## Benchmarks++Offline A/B, same candidate pool per query, 102 queries / 99 scored against+a hand-curated golden set for a real production wiki (FTS5 + vector + graph+hybrid, `explain=true` score details, identical server build):++| Reranker | hit@1 | MRR | NDCG@10 | mean rerank latency |+|---|---|---|---|---|+| none (server order) | 0.495 | 0.651 | 0.739 | — |+| hosted model A (reasoning, max) | 0.778 | 0.840 | 0.875 | **20.2 s** |+| hosted model B (reasoning) | 0.768 | 0.834 | 0.870 | 13.9 s |+| **Jev via this adapter** | **0.778** | **0.838** | **0.873** | **0.205 s** |++The judge endpoint matches the hosted reasoning models' quality at+~100× lower latency. Hosted model A's 20.2 s mean sat exactly on the+server's 20 s completion timeout, so in production every rerank call timed+out and the feature was effectively off (queries stalled 20 s, then fell+back to the original order).++Live end-to-end (production server, `AI_MEMORY_RERANKER=llm` through the+adapter, full golden set): hit@1 0.657 / MRR 0.788 / NDCG@10 0.843,+0 errors in 102 queries, mean `memory_query` latency 2.2 s (p50 2.21 s,+max 3.07 s), 110/110 rerank translations served, 0 adapter failures. The+gap to the offline arm is candidate-pool shape, not scoring: the live+server over-fetches 15–30 candidates with its own bounded snippets (the+offline arm rescored a fixed top-10 pool), and in 32 of 34 non-top-1+queries the expected page was still returned — mostly at rank 2 — with+hit@5 at 0.949. Versus the no-reranker baseline that is +16.2 points+hit@1 and +10.4 points NDCG@10 end to end.++Judge latency scales with the candidate count in the batch (the rubric+question is asked once per candidate): ~0.2 s for 10 candidates, ~1.6 s+mean for the 15–30-candidate batches the live server sends, still an order+of magnitude under any hosted reasoning model.++## Caveats++- The adapter keys on the exact system-prompt prefix above. If the+ reranker prompt wording changes in a future release, detection (not+ scoring) breaks first: reranker requests would be proxied to the hosted+ model, which is the pre-adapter behavior, not an outage.+- Scoring uses Jev's calibrated rubric grades. Do not substitute+ per-candidate choice probabilities: those are normalized across options+ (winner ≈ 0.99, rest ≈ 0.001) and do not fit the reranker's 0–1+ relevance semantics.+- Run the adapter on loopback or a trusted private network. It forwards+ `Authorization` headers verbatim and adds no authentication of its own.diff --git a/docs/llm-providers.md b/docs/llm-providers.mdindex 3f296996..2719665a 100644--- a/docs/llm-providers.md+++ b/docs/llm-providers.md@@ -131,8 +131,11 @@ request sends the query plus at most 30 bounded page titles and search snippetsto the configured provider; all values are JSON-encoded and treated asuntrusted data. A timeout, provider error, or incomplete/invalid score setpreserves the normal order. `global=true` and supplemental global-preference-hits keep their existing non-RRF ranking. Concurrent provider calls are capped-at four; saturated queries keep their local ranking without waiting.+at four; saturated queries keep their local ranking without waiting. If the+configured provider is too slow for interactive reranking, a judge-endpoint+adapter can serve the reranker leg in sub-second time while consolidation+keeps the hosted model — see+[`docs/jev-reranker-adapter.md`](jev-reranker-adapter.md).Embeddings are optional and separate from the LLM provider. Set`AI_MEMORY_EMBEDDING_PROVIDER=openai`, `voyage`, `google`/`gemini`, or