Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(benchmarks): refresh LongMemEval-S + populate R2 A/B on the 2.4 tree
AkitaOnRails1c5303ef7da8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 348aa343..23645646 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -246,6 +246,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`OPENCODE_API_KEY` as the only credential, Go as the default endpoint, Zenvia `AI_MEMORY_LLM_BASE_URL`, the built-in default model, per-cataloguemodel ids, and which model goes through the Responses endpoint (#763).+- Refreshed the LongMemEval-S retrieval benchmarks on the 2.4 tree and+ populated the full-dataset R2 A/B (`docs/benchmarks/`): local embeddings add+ +0.149 hit@5 / +0.254 recall@10 over zero-LLM FTS, with a clean+ baseline-vs-baseline determinism check and **no default-ranking regression**+ vs 2.3.x (the 2.4 features are opt-in / off by default).### Fixed- The Windows Docker wrapper (`bin/ai-memory.ps1`) now forwards the samediff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.mdindex 95c17e81..a51a63a4 100644--- a/docs/benchmarks/README.md+++ b/docs/benchmarks/README.md@@ -13,10 +13,18 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch| date | dataset | mode | overall hit@5 | file ||---|---|---|---|---|-| 2026-09-01 | LongMemEval-S (v1) | **local embeddings (2.0 default)** | **0.823** | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) |-| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |+| **2026-09-21** | LongMemEval-S (v1) | **local embeddings (2.4 RC, current)** | **0.815** | [retrieval-ab-r2.md](retrieval-ab-r2.md) |+| 2026-09-21 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (2.4 RC) | 0.666 | [retrieval-ab-r2.md](retrieval-ab-r2.md) |+| 2026-09-01 | LongMemEval-S (v1) | local embeddings (2.0 default, prior) | 0.823 | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) |+| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) || 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |+The **2026-09-21 (2.4 RC)** re-run confirms no default-ranking regression: local+`hit@5` 0.823 → 0.815 and FTS `hit@5` 0.668 → 0.666 are identical within+run-to-run noise (2.4's aging/retrieval features are opt-in / off by default, so+default retrieval is unchanged). Full triple + provenance in+[retrieval-ab-r2.md](retrieval-ab-r2.md).+The 2.0 retrieval work moved overall hit@5 from **0.617 → 0.823**(+20.6 points; hit@1 0.449 → 0.536, recall@5 0.472 → 0.680) in twomeasured steps: dropping stopwords from bare-query FTS OR-joins@@ -32,8 +40,11 @@ embedding-based numbers on this dataset: agentmemory 0.967 R@5The harness can also A/B two configs over the same question set and reportan **accuracy + latency + context-tokens** triple with a baseline→candidate-delta — see [retrieval-ab-r2.md](retrieval-ab-r2.md). Full-dataset A/B-baselines are published there once run.+delta — see [retrieval-ab-r2.md](retrieval-ab-r2.md). The full-dataset A/B+(FTS baseline → local candidate, 470 scored) is published there as of the+**2026-09-21 (2.4 RC)** run: local embeddings add **+0.149 hit@5 / +0.254+recall@10** over zero-LLM FTS for ~90 ms p50 latency, with a clean+baseline-vs-baseline determinism check (all accuracy/token deltas `0.000`).## Reading the numbersdiff --git a/docs/benchmarks/retrieval-ab-r2.md b/docs/benchmarks/retrieval-ab-r2.mdindex b0d808f7..6a263838 100644--- a/docs/benchmarks/retrieval-ab-r2.md+++ b/docs/benchmarks/retrieval-ab-r2.md@@ -37,30 +37,54 @@ deltas are exactly zero (latency wobbles at wall-clock noise).## Published A/B numbers-> Full-dataset (500-question) A/B baselines go here, one section per-> comparison, pasted verbatim from `evals/runs/<stamp>-retrieval/report.md`-> with its provenance header. **Not yet populated** — running the full-> matrix is a deliberate, separately-scheduled pass (the dataset is 278 MB-> and a full local-embeddings leg is minutes, not seconds). Do not paste-> `--sample` numbers here as if they were baselines.+Full-dataset run, **2.4 release candidate** (470 scored; 30 abstention+questions excluded).-### Illustrative smoke (NOT a baseline)+- commit: `79fea4009e74cf83561c6e2170dccef80360c079` (release/2.4)+- dataset: `longmemeval_s` (sha256 `08d8dad4be43ee20…`)+- hardware: AMD Ryzen 9 7950X3D 16-Core (32 threads)+- baseline: `embeddings=none reranker=off` (zero-LLM, FTS + entity + graph)+- candidate: `embeddings=local reranker=off` (in-process all-MiniLM-L6-v2)-Recorded only to show the report shape. `--sample 4`, four-`single-session-user` questions, so the accuracy figures are noise;-latency and context-tokens show the expected direction (local embeddings-add query latency):+**Overall (470 questions):**-| metric | baseline (none) | candidate (local) | delta |+| metric | baseline (FTS, zero-LLM) | candidate (local) | delta ||---|---|---|---|-| hit@5 | 0.750 | 0.500 | -0.250 |-| latency p50 ms | 2 | 31 | +29 |-| latency p95 ms | 2 | 35 | +33 |-| ctx tok mean | 408.5 | 393.2 | -15.2 |-| ctx tok median | 494.5 | 389.0 | -105.5 |--Hardware: AMD Ryzen 9 7950X3D (32 threads). Replace this section with a-full-dataset run before citing any A/B number.+| hit@1 | 0.532 | 0.536 | +0.004 |+| hit@3 | 0.645 | 0.730 | +0.085 |+| hit@5 | 0.666 | 0.815 | +0.149 |+| hit@10 | 0.694 | 0.891 | +0.198 |+| recall@5 | 0.536 | 0.677 | +0.142 |+| recall@10 | 0.564 | 0.817 | +0.254 |+| latency p50 ms | 5 | 94 | +89 |+| latency p95 ms | 40 | 162 | +122 |+| ctx tok mean | 350.3 | 415.7 | +65.4 |+| ctx tok median | 316.5 | 367.5 | +51.0 |++Local embeddings buy a large recall gain (hit@10 +0.198, recall@10 +0.254)+for ~90 ms added p50 query latency and ~65 more context tokens — the expected+tradeoff, now measured on the full set rather than a smoke.++Per-slice `hit@5` (candidate, local): knowledge-update 0.875, multi-session+0.868, temporal-reasoning 0.780, single-session-assistant 0.821,+single-session-preference 0.767, single-session-user 0.734.++**Determinism check (baseline vs baseline, `--candidate`):** every accuracy+and context-token delta is exactly `0.000` (latency wobbles ±5 ms at wall-clock+noise) — the harness is sound on the 2.4 tree.++**No default-ranking regression on 2.4.** The 2.4 aging/retrieval features are+opt-in / off by default, so default retrieval is unchanged from 2.3.x: overall+FTS `hit@5` 0.668 → 0.666 and local `hit@5` 0.823 → 0.815 versus the+2026-09-01 snapshot (commit `0ac0dcf`) — identical within run-to-run noise.++Reproduce:++```bash+cargo run --release -p ai-memory-eval -- retrieval --candidate-embeddings local+# determinism (deltas must be 0):+cargo run --release -p ai-memory-eval -- retrieval --candidate+```### Illustrative sample-20 run (triple + QA-accuracy, NOT a baseline)