Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(benchmarks): publish the 2.0 retrieval progression

AkitaOnRails6cc0390a7c42
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md
index 5473ea33..78e565b1 100644
--- a/docs/benchmarks/README.md
+++ b/docs/benchmarks/README.md
@@ -13,16 +13,28 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| date | dataset | mode | overall hit@5 | file |
|---|---|---|---|---|
-| 2026-09-01 | LongMemEval-S (v1) | zero-llm (FTS only) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |
+| 2026-09-01 | LongMemEval-S (v1) | **local embeddings (2.0 default)** | **0.823** | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) |
+| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |
+| 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |
+
+The 2.0 retrieval work moved overall hit@5 from **0.617 → 0.823**
+(+20.6 points; hit@1 0.449 → 0.536, recall@5 0.472 → 0.680) in two
+measured steps: dropping stopwords from bare-query FTS OR-joins
+(+5.1 hit@5, +8.5 hit@1), then the in-process local embedder with
+correct masked-mean pooling — the pooling fix alone was worth ~6.6
+points over a padded-attention implementation, caught by the
+calibration tests. Every intermediate number was measured on this
+harness before the next change landed. For context, published
+embedding-based numbers on this dataset: agentmemory 0.967 R@5
+(hybrid + reranking), doobidoo/mcp-memory-service 0.804 R@5.
## Reading the numbers
-- **mode: zero-llm** is the deterministic floor: no consolidation LLM, no
- embedder, no reranker — pure FTS5 + entity/graph fusion over
- hook-captured observations. It is the configuration every install has
- with no API key at all. Embedding-assisted modes are expected to score
- higher and will be published alongside when the local-embedding work
- lands (roadmap item 5).
+- **mode: local embeddings** is the 2.0 default: the in-process
+ all-MiniLM embedder (no API key, no egress) fused with FTS5 + entity +
+ graph. **mode: zero-llm** is the deterministic floor (`embedding_provider
+ = "none"`, or any host where the model cannot load): FTS5 +
+ entity/graph only.
- **Comparability.** Published numbers from other systems on this dataset
(agentmemory 0.967 R@5, doobidoo/mcp-memory-service 0.804 R@5) are
embedding-based retrieval over raw chat logs. Our `hit@5` is the
diff --git a/docs/benchmarks/longmemeval-s-2026-09-01-fts.md b/docs/benchmarks/longmemeval-s-2026-09-01-fts.md
new file mode 100644
index 00000000..ccf9b07b
--- /dev/null
+++ b/docs/benchmarks/longmemeval-s-2026-09-01-fts.md
@@ -0,0 +1,19 @@
+# LongMemEval-S retrieval — 2026-09-01T22:19:24.828453944Z
+
+- commit: `0ac0dcf8f89a1f12d38f68e776e8aa1ab08a0f96`
+- dataset: `longmemeval_s` (sha256 `08d8dad4be43ee20…`)
+- mode: zero-llm
+- hardware: AMD Ryzen 9 7950X3D 16-Core Processor (32 threads)
+- questions scored: 470 (30 abstention questions excluded)
+
+| slice | n | hit@1 | hit@3 | hit@5 | hit@10 | recall@1 | recall@3 | recall@5 | recall@10 |
+|---|---|---|---|---|---|---|---|---|---|
+| knowledge-update | 72 | 0.583 | 0.750 | 0.764 | 0.778 | 0.292 | 0.549 | 0.562 | 0.590 |
+| multi-session | 121 | 0.446 | 0.554 | 0.579 | 0.603 | 0.193 | 0.350 | 0.385 | 0.403 |
+| overall | 470 | 0.534 | 0.647 | 0.668 | 0.696 | 0.351 | 0.510 | 0.538 | 0.566 |
+| single-session-assistant | 56 | 0.679 | 0.786 | 0.804 | 0.839 | 0.679 | 0.786 | 0.804 | 0.839 |
+| single-session-preference | 30 | 0.300 | 0.467 | 0.533 | 0.667 | 0.300 | 0.467 | 0.533 | 0.667 |
+| single-session-user | 64 | 0.625 | 0.656 | 0.688 | 0.703 | 0.625 | 0.656 | 0.688 | 0.703 |
+| temporal-reasoning | 127 | 0.535 | 0.654 | 0.661 | 0.677 | 0.264 | 0.456 | 0.478 | 0.493 |
+
+Notes: hit@k = any evidence session in top k (the "Recall@k" most systems publish); recall@k = fraction of evidence sessions found. Session attribution: `sessions/<id>.md` pages and raw observation hits; unattributable pages never score. Capture is production-shaped: excerpts bounded at the 2 KB privacy boundary.
diff --git a/docs/benchmarks/longmemeval-s-2026-09-01-local.md b/docs/benchmarks/longmemeval-s-2026-09-01-local.md
new file mode 100644
index 00000000..3862d3ba
--- /dev/null
+++ b/docs/benchmarks/longmemeval-s-2026-09-01-local.md
@@ -0,0 +1,19 @@
+# LongMemEval-S retrieval — 2026-09-01T22:38:38.981115401Z
+
+- commit: `0ac0dcf8f89a1f12d38f68e776e8aa1ab08a0f96`
+- dataset: `longmemeval_s` (sha256 `08d8dad4be43ee20…`)
+- mode: local-embeddings
+- hardware: AMD Ryzen 9 7950X3D 16-Core Processor (32 threads)
+- questions scored: 470 (30 abstention questions excluded)
+
+| slice | n | hit@1 | hit@3 | hit@5 | hit@10 | recall@1 | recall@3 | recall@5 | recall@10 |
+|---|---|---|---|---|---|---|---|---|---|
+| knowledge-update | 72 | 0.625 | 0.806 | 0.903 | 0.944 | 0.312 | 0.590 | 0.736 | 0.875 |
+| multi-session | 121 | 0.471 | 0.760 | 0.876 | 0.950 | 0.203 | 0.462 | 0.624 | 0.801 |
+| overall | 470 | 0.536 | 0.726 | 0.823 | 0.889 | 0.343 | 0.566 | 0.680 | 0.812 |
+| single-session-assistant | 56 | 0.714 | 0.804 | 0.821 | 0.839 | 0.714 | 0.804 | 0.821 | 0.839 |
+| single-session-preference | 30 | 0.433 | 0.633 | 0.767 | 0.867 | 0.433 | 0.633 | 0.767 | 0.867 |
+| single-session-user | 64 | 0.453 | 0.656 | 0.750 | 0.812 | 0.453 | 0.656 | 0.750 | 0.812 |
+| temporal-reasoning | 127 | 0.535 | 0.669 | 0.780 | 0.866 | 0.255 | 0.485 | 0.584 | 0.763 |
+
+Notes: hit@k = any evidence session in top k (the "Recall@k" most systems publish); recall@k = fraction of evidence sessions found. Session attribution: `sessions/<id>.md` pages and raw observation hits; unattributable pages never score. Capture is production-shaped: excerpts bounded at the 2 KB privacy boundary.