Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(benchmarks): publish the 2.0 retrieval progression
AkitaOnRails6cc0390a7c42
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.mdindex 5473ea33..78e565b1 100644--- a/docs/benchmarks/README.md+++ b/docs/benchmarks/README.md@@ -13,16 +13,28 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch| date | dataset | mode | overall hit@5 | file ||---|---|---|---|---|-| 2026-09-01 | LongMemEval-S (v1) | zero-llm (FTS only) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |+| 2026-09-01 | LongMemEval-S (v1) | **local embeddings (2.0 default)** | **0.823** | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) |+| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |+| 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |++The 2.0 retrieval work moved overall hit@5 from **0.617 → 0.823**+(+20.6 points; hit@1 0.449 → 0.536, recall@5 0.472 → 0.680) in two+measured steps: dropping stopwords from bare-query FTS OR-joins+(+5.1 hit@5, +8.5 hit@1), then the in-process local embedder with+correct masked-mean pooling — the pooling fix alone was worth ~6.6+points over a padded-attention implementation, caught by the+calibration tests. Every intermediate number was measured on this+harness before the next change landed. For context, published+embedding-based numbers on this dataset: agentmemory 0.967 R@5+(hybrid + reranking), doobidoo/mcp-memory-service 0.804 R@5.## Reading the numbers-- **mode: zero-llm** is the deterministic floor: no consolidation LLM, no- embedder, no reranker — pure FTS5 + entity/graph fusion over- hook-captured observations. It is the configuration every install has- with no API key at all. Embedding-assisted modes are expected to score- higher and will be published alongside when the local-embedding work- lands (roadmap item 5).+- **mode: local embeddings** is the 2.0 default: the in-process+ all-MiniLM embedder (no API key, no egress) fused with FTS5 + entity ++ graph. **mode: zero-llm** is the deterministic floor (`embedding_provider+ = "none"`, or any host where the model cannot load): FTS5 ++ entity/graph only.- **Comparability.** Published numbers from other systems on this dataset(agentmemory 0.967 R@5, doobidoo/mcp-memory-service 0.804 R@5) areembedding-based retrieval over raw chat logs. Our `hit@5` is thediff --git a/docs/benchmarks/longmemeval-s-2026-09-01-fts.md b/docs/benchmarks/longmemeval-s-2026-09-01-fts.mdnew file mode 100644index 00000000..ccf9b07b--- /dev/null+++ b/docs/benchmarks/longmemeval-s-2026-09-01-fts.md@@ -0,0 +1,19 @@+# LongMemEval-S retrieval — 2026-09-01T22:19:24.828453944Z++- commit: `0ac0dcf8f89a1f12d38f68e776e8aa1ab08a0f96`+- dataset: `longmemeval_s` (sha256 `08d8dad4be43ee20…`)+- mode: zero-llm+- hardware: AMD Ryzen 9 7950X3D 16-Core Processor (32 threads)+- questions scored: 470 (30 abstention questions excluded)++| slice | n | hit@1 | hit@3 | hit@5 | hit@10 | recall@1 | recall@3 | recall@5 | recall@10 |+|---|---|---|---|---|---|---|---|---|---|+| knowledge-update | 72 | 0.583 | 0.750 | 0.764 | 0.778 | 0.292 | 0.549 | 0.562 | 0.590 |+| multi-session | 121 | 0.446 | 0.554 | 0.579 | 0.603 | 0.193 | 0.350 | 0.385 | 0.403 |+| overall | 470 | 0.534 | 0.647 | 0.668 | 0.696 | 0.351 | 0.510 | 0.538 | 0.566 |+| single-session-assistant | 56 | 0.679 | 0.786 | 0.804 | 0.839 | 0.679 | 0.786 | 0.804 | 0.839 |+| single-session-preference | 30 | 0.300 | 0.467 | 0.533 | 0.667 | 0.300 | 0.467 | 0.533 | 0.667 |+| single-session-user | 64 | 0.625 | 0.656 | 0.688 | 0.703 | 0.625 | 0.656 | 0.688 | 0.703 |+| temporal-reasoning | 127 | 0.535 | 0.654 | 0.661 | 0.677 | 0.264 | 0.456 | 0.478 | 0.493 |++Notes: hit@k = any evidence session in top k (the "Recall@k" most systems publish); recall@k = fraction of evidence sessions found. Session attribution: `sessions/<id>.md` pages and raw observation hits; unattributable pages never score. Capture is production-shaped: excerpts bounded at the 2 KB privacy boundary.diff --git a/docs/benchmarks/longmemeval-s-2026-09-01-local.md b/docs/benchmarks/longmemeval-s-2026-09-01-local.mdnew file mode 100644index 00000000..3862d3ba--- /dev/null+++ b/docs/benchmarks/longmemeval-s-2026-09-01-local.md@@ -0,0 +1,19 @@+# LongMemEval-S retrieval — 2026-09-01T22:38:38.981115401Z++- commit: `0ac0dcf8f89a1f12d38f68e776e8aa1ab08a0f96`+- dataset: `longmemeval_s` (sha256 `08d8dad4be43ee20…`)+- mode: local-embeddings+- hardware: AMD Ryzen 9 7950X3D 16-Core Processor (32 threads)+- questions scored: 470 (30 abstention questions excluded)++| slice | n | hit@1 | hit@3 | hit@5 | hit@10 | recall@1 | recall@3 | recall@5 | recall@10 |+|---|---|---|---|---|---|---|---|---|---|+| knowledge-update | 72 | 0.625 | 0.806 | 0.903 | 0.944 | 0.312 | 0.590 | 0.736 | 0.875 |+| multi-session | 121 | 0.471 | 0.760 | 0.876 | 0.950 | 0.203 | 0.462 | 0.624 | 0.801 |+| overall | 470 | 0.536 | 0.726 | 0.823 | 0.889 | 0.343 | 0.566 | 0.680 | 0.812 |+| single-session-assistant | 56 | 0.714 | 0.804 | 0.821 | 0.839 | 0.714 | 0.804 | 0.821 | 0.839 |+| single-session-preference | 30 | 0.433 | 0.633 | 0.767 | 0.867 | 0.433 | 0.633 | 0.767 | 0.867 |+| single-session-user | 64 | 0.453 | 0.656 | 0.750 | 0.812 | 0.453 | 0.656 | 0.750 | 0.812 |+| temporal-reasoning | 127 | 0.535 | 0.669 | 0.780 | 0.866 | 0.255 | 0.485 | 0.584 | 0.763 |++Notes: hit@k = any evidence session in top k (the "Recall@k" most systems publish); recall@k = fraction of evidence sessions found. Session attribution: `sessions/<id>.md` pages and raw observation hits; unattributable pages never score. Capture is production-shaped: excerpts bounded at the 2 KB privacy boundary.