Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(benchmarks): refresh LongMemEval-S + populate R2 A/B on the 2.4 tree

AkitaOnRails1c5303ef7da8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 348aa343..23645646 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -246,6 +246,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`OPENCODE_API_KEY` as the only credential, Go as the default endpoint, Zen
via `AI_MEMORY_LLM_BASE_URL`, the built-in default model, per-catalogue
model ids, and which model goes through the Responses endpoint (#763).
+- Refreshed the LongMemEval-S retrieval benchmarks on the 2.4 tree and
+ populated the full-dataset R2 A/B (`docs/benchmarks/`): local embeddings add
+ +0.149 hit@5 / +0.254 recall@10 over zero-LLM FTS, with a clean
+ baseline-vs-baseline determinism check and **no default-ranking regression**
+ vs 2.3.x (the 2.4 features are opt-in / off by default).
### Fixed
- The Windows Docker wrapper (`bin/ai-memory.ps1`) now forwards the same
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md
index 95c17e81..a51a63a4 100644
--- a/docs/benchmarks/README.md
+++ b/docs/benchmarks/README.md
@@ -13,10 +13,18 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| date | dataset | mode | overall hit@5 | file |
|---|---|---|---|---|
-| 2026-09-01 | LongMemEval-S (v1) | **local embeddings (2.0 default)** | **0.823** | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) |
-| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |
+| **2026-09-21** | LongMemEval-S (v1) | **local embeddings (2.4 RC, current)** | **0.815** | [retrieval-ab-r2.md](retrieval-ab-r2.md) |
+| 2026-09-21 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (2.4 RC) | 0.666 | [retrieval-ab-r2.md](retrieval-ab-r2.md) |
+| 2026-09-01 | LongMemEval-S (v1) | local embeddings (2.0 default, prior) | 0.823 | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) |
+| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |
| 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |
+The **2026-09-21 (2.4 RC)** re-run confirms no default-ranking regression: local
+`hit@5` 0.823 → 0.815 and FTS `hit@5` 0.668 → 0.666 are identical within
+run-to-run noise (2.4's aging/retrieval features are opt-in / off by default, so
+default retrieval is unchanged). Full triple + provenance in
+[retrieval-ab-r2.md](retrieval-ab-r2.md).
+
The 2.0 retrieval work moved overall hit@5 from **0.617 → 0.823**
(+20.6 points; hit@1 0.449 → 0.536, recall@5 0.472 → 0.680) in two
measured steps: dropping stopwords from bare-query FTS OR-joins
@@ -32,8 +40,11 @@ embedding-based numbers on this dataset: agentmemory 0.967 R@5
The harness can also A/B two configs over the same question set and report
an **accuracy + latency + context-tokens** triple with a baseline→candidate
-delta — see [retrieval-ab-r2.md](retrieval-ab-r2.md). Full-dataset A/B
-baselines are published there once run.
+delta — see [retrieval-ab-r2.md](retrieval-ab-r2.md). The full-dataset A/B
+(FTS baseline → local candidate, 470 scored) is published there as of the
+**2026-09-21 (2.4 RC)** run: local embeddings add **+0.149 hit@5 / +0.254
+recall@10** over zero-LLM FTS for ~90 ms p50 latency, with a clean
+baseline-vs-baseline determinism check (all accuracy/token deltas `0.000`).
## Reading the numbers
diff --git a/docs/benchmarks/retrieval-ab-r2.md b/docs/benchmarks/retrieval-ab-r2.md
index b0d808f7..6a263838 100644
--- a/docs/benchmarks/retrieval-ab-r2.md
+++ b/docs/benchmarks/retrieval-ab-r2.md
@@ -37,30 +37,54 @@ deltas are exactly zero (latency wobbles at wall-clock noise).
## Published A/B numbers
-> Full-dataset (500-question) A/B baselines go here, one section per
-> comparison, pasted verbatim from `evals/runs/<stamp>-retrieval/report.md`
-> with its provenance header. **Not yet populated** — running the full
-> matrix is a deliberate, separately-scheduled pass (the dataset is 278 MB
-> and a full local-embeddings leg is minutes, not seconds). Do not paste
-> `--sample` numbers here as if they were baselines.
+Full-dataset run, **2.4 release candidate** (470 scored; 30 abstention
+questions excluded).
-### Illustrative smoke (NOT a baseline)
+- commit: `79fea4009e74cf83561c6e2170dccef80360c079` (release/2.4)
+- dataset: `longmemeval_s` (sha256 `08d8dad4be43ee20…`)
+- hardware: AMD Ryzen 9 7950X3D 16-Core (32 threads)
+- baseline: `embeddings=none reranker=off` (zero-LLM, FTS + entity + graph)
+- candidate: `embeddings=local reranker=off` (in-process all-MiniLM-L6-v2)
-Recorded only to show the report shape. `--sample 4`, four
-`single-session-user` questions, so the accuracy figures are noise;
-latency and context-tokens show the expected direction (local embeddings
-add query latency):
+**Overall (470 questions):**
-| metric | baseline (none) | candidate (local) | delta |
+| metric | baseline (FTS, zero-LLM) | candidate (local) | delta |
|---|---|---|---|
-| hit@5 | 0.750 | 0.500 | -0.250 |
-| latency p50 ms | 2 | 31 | +29 |
-| latency p95 ms | 2 | 35 | +33 |
-| ctx tok mean | 408.5 | 393.2 | -15.2 |
-| ctx tok median | 494.5 | 389.0 | -105.5 |
-
-Hardware: AMD Ryzen 9 7950X3D (32 threads). Replace this section with a
-full-dataset run before citing any A/B number.
+| hit@1 | 0.532 | 0.536 | +0.004 |
+| hit@3 | 0.645 | 0.730 | +0.085 |
+| hit@5 | 0.666 | 0.815 | +0.149 |
+| hit@10 | 0.694 | 0.891 | +0.198 |
+| recall@5 | 0.536 | 0.677 | +0.142 |
+| recall@10 | 0.564 | 0.817 | +0.254 |
+| latency p50 ms | 5 | 94 | +89 |
+| latency p95 ms | 40 | 162 | +122 |
+| ctx tok mean | 350.3 | 415.7 | +65.4 |
+| ctx tok median | 316.5 | 367.5 | +51.0 |
+
+Local embeddings buy a large recall gain (hit@10 +0.198, recall@10 +0.254)
+for ~90 ms added p50 query latency and ~65 more context tokens — the expected
+tradeoff, now measured on the full set rather than a smoke.
+
+Per-slice `hit@5` (candidate, local): knowledge-update 0.875, multi-session
+0.868, temporal-reasoning 0.780, single-session-assistant 0.821,
+single-session-preference 0.767, single-session-user 0.734.
+
+**Determinism check (baseline vs baseline, `--candidate`):** every accuracy
+and context-token delta is exactly `0.000` (latency wobbles ±5 ms at wall-clock
+noise) — the harness is sound on the 2.4 tree.
+
+**No default-ranking regression on 2.4.** The 2.4 aging/retrieval features are
+opt-in / off by default, so default retrieval is unchanged from 2.3.x: overall
+FTS `hit@5` 0.668 → 0.666 and local `hit@5` 0.823 → 0.815 versus the
+2026-09-01 snapshot (commit `0ac0dcf`) — identical within run-to-run noise.
+
+Reproduce:
+
+```bash
+cargo run --release -p ai-memory-eval -- retrieval --candidate-embeddings local
+# determinism (deltas must be 0):
+cargo run --release -p ai-memory-eval -- retrieval --candidate
+```
### Illustrative sample-20 run (triple + QA-accuracy, NOT a baseline)