Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(benchmarks): publish the 3.2.0 LongMemEval-S release run
AkitaOnRails394fd873b974
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.mdindex 99cb93bb..35d17045 100644--- a/docs/benchmarks/README.md+++ b/docs/benchmarks/README.md@@ -13,6 +13,8 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch| date | dataset | mode | overall hit@5 | file ||---|---|---|---|---|+| **2026-10-22** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.2.0 release)** | **0.951** | [longmemeval-s-3.2.0-release.md](longmemeval-s-3.2.0-release.md) |+| **2026-10-22** | LongMemEval-S (v1) | **local embeddings + observation stream (3.2.0 release)** | **0.938** | [longmemeval-s-3.2.0-release.md](longmemeval-s-3.2.0-release.md) || **2026-10-20** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.1.2 release)** | **0.951** | [longmemeval-s-3.1.2-release.md](longmemeval-s-3.1.2-release.md) || **2026-10-20** | LongMemEval-S (v1) | **local embeddings + observation stream (3.1.2 release)** | **0.947** | [longmemeval-s-3.1.2-release.md](longmemeval-s-3.1.2-release.md) || **2026-10-16** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.1 release)** | **0.951** | [longmemeval-s-3.1.0-release.md](longmemeval-s-3.1.0-release.md) |@@ -27,6 +29,13 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) || 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |+The 2026-10-22 minor-release run found FTS identical to 3.1.2 on every+question. Local embeddings fell from 0.947 to 0.938 `hit@5` (five questions+lost, one gained), with higher query latency and 75% longer ingest. That is+outside the +/-0.005 noise band and is tracked as a possible regression for+the next release. See the [3.2.0 release report](longmemeval-s-3.2.0-release.md)+for every slice, ingest performance, the suspected commit and provenance.+The 2026-10-20 patch-release run found no retrieval regression against 3.1.FTS is identical at every accuracy cutoff. Local embeddings improved from0.940 to 0.947 `hit@5`, with stable latency and context size. See thediff --git a/docs/benchmarks/longmemeval-s-3.2.0-release.md b/docs/benchmarks/longmemeval-s-3.2.0-release.mdnew file mode 100644index 00000000..c2a795ff--- /dev/null+++ b/docs/benchmarks/longmemeval-s-3.2.0-release.md@@ -0,0 +1,104 @@+# LongMemEval-S release benchmark: 3.2.0++The 3.2.0 release benchmark runs all 470 answerable LongMemEval-S questions+through the shipped hook capture pipeline. It measures the supported default+retrieval paths: zero-LLM FTS and the in-process local embedder, both with the+production-default observation stream enabled.++## Provenance++- code: both modes used `1c86aebc14addccc7ac875913d565c681bfa4a50`, the+ `v3.2.0` tag. The benchmark ran after the release was published, and this+ report is the only change since.+- dataset: `longmemeval_s`, sha256+ `08d8dad4be43ee2049a22ff5674eb86725d0ce5ff434cde2627e5e8e7e117894`.+- hardware: AMD Ryzen 9 7950X3D (32 threads). No build or test workload ran+ during either measurement. Static code review and a few single-threaded+ shell timings in throwaway containers overlapped the FTS run and the first+ minutes of the local run; the FTS result matches 3.1.2 exactly and the local+ slowdown below lasts the whole run, so that work does not explain either.+- commands, after `cargo clean` and building `ai-memory` and `ai-memory-eval`+ in release mode:++ ```bash+ ai-memory-eval retrieval \+ --server-bin target/release/ai-memory \+ --concurrency 6 --http-timeout-secs 900 \+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true++ ai-memory-eval retrieval \+ --server-bin target/release/ai-memory \+ --concurrency 6 --http-timeout-secs 900 \+ --embeddings local \+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true+ ```++- run dirs: `evals/runs/2026-10-22T02-00-49-545842361Z-retrieval` (FTS)+ and `evals/runs/2026-10-22T03-05-33-423353129Z-retrieval` (local).++## Results++FTS is identical to 3.1.2 on every question at every cutoff. Local embeddings+lost five questions at `hit@5` and gained one, moving from 0.947 to 0.938, and+query latency and ingest time rose. That change is outside the+pre-registered +/-0.005 overall noise band and is tracked as a possible+regression for the next release.++| mode | hit@1 | hit@5 | hit@10 | p50 | p95 | mean context tokens |+|---|---:|---:|---:|---:|---:|---:|+| FTS 3.1.2 | 0.857 | 0.951 | 0.972 | 12 ms | 103 ms | 528 |+| **FTS 3.2.0** | **0.857** | **0.951** | **0.972** | **12 ms** | **97 ms** | **528** |+| local 3.1.2 | 0.768 | 0.947 | 0.977 | 98 ms | 225 ms | 528 |+| **local 3.2.0** | **0.757** | **0.938** | **0.974** | **118 ms** | **295 ms** | **528** |++Recall moved with hit rate: local `recall@5` went from 0.879 to 0.873 and+`recall@10` from 0.947 to 0.945. FTS recall is unchanged at every cutoff.++Per-slice `hit@5`:++| slice | n | FTS 3.1.2 | FTS 3.2.0 | local 3.1.2 | local 3.2.0 |+|---|---:|---:|---:|---:|---:|+| knowledge-update | 72 | 1.000 | 1.000 | 1.000 | 1.000 |+| multi-session | 121 | 0.967 | 0.967 | 0.975 | 0.975 |+| temporal-reasoning | 127 | 0.921 | 0.921 | 0.921 | 0.890 |+| single-session-assistant | 56 | 1.000 | 1.000 | 0.893 | 0.875 |+| single-session-preference | 30 | 0.700 | 0.700 | 0.900 | 0.900 |+| single-session-user | 64 | 1.000 | 1.000 | 0.953 | 0.969 |++The local losses are four temporal-reasoning questions (`71017276`,+`gpt4_68e94288`, `gpt4_b5700ca0`, `gpt4_e061b84f`) and one+single-session-assistant question (`5809eb10`); `001be529`+(single-session-user) was gained. The temporal-reasoning drop of 0.031 sits at+the upper edge of the +/-0.02 to 0.03 small-slice band, and the other two+slice moves are one question each.++Local query latency rose on every slice, from 98 to 118 ms at p50 and from+225 to 295 ms at p95. Returned context is unchanged.++## Ingest++| mode | items | requests | throughput | wall clock | median ingest per question | fallback answers |+|---|---:|---:|---:|---:|---:|---:|+| FTS 3.1.2 | 278,946 | 1,403 | 12 items/s per question stream | 3,837.0 s | 48.7 s | 0/470 |+| FTS 3.2.0 | 278,946 | 1,403 | 12 items/s per question stream | 3,882.6 s | 48.5 s | 0/470 |+| local 3.1.2 | 278,946 | 1,403 | 10 items/s per question stream | 4,899.8 s | 60.1 s | 0/470 |+| local 3.2.0 | 278,946 | 1,403 | 5 items/s per question stream | 8,590.6 s | 101.3 s | 0/470 |++FTS ingest is unchanged, so the CJK search index added in 3.2.0 does not+account for the local slowdown. Local ingest took 75% longer end to end.++No document exceeded the 16 KiB capture bound. The raw-observation fallback+was not needed, so both results measure indexed retrieval.++## Suspected cause++3.2.0 reserved local-embedding capacity for interactive queries (`107ada8d`,+"Reserved local-embedding capacity for interactive search queries" in the+3.2.0 changelog). The local embedder is now wrapped in a two-slot limit: one+slot for background document embedding and one held back for queries. Page+writes therefore embed one document at a time per server, which matches the+halved ingest throughput. Whether the slower embedding also explains the+accuracy loss, for example by leaving some pages without a vector when their+question is asked, has not been measured. The follow-up issue for the next+release ([#1580](https://github.com/akitaonrails/ai-memory/issues/1580)) records the deltas, run dirs and this commit so the trade can be+measured and decided there.diff --git a/docs/comparison.md b/docs/comparison.mdindex 1eaa7986..eeee8dfd 100644--- a/docs/comparison.md+++ b/docs/comparison.md@@ -55,7 +55,8 @@ LongMemEval-S hit@5 **0.666** zero-LLM and **0.815** with the default localembeddings in the published 2.6 line. The observation stream in 3.0(on by default) raises zero-LLM to **0.951** and local embeddingsto **0.945** on all 470 questions, measured on one build with the stream off-and on. With capture skipped and each raw session indexed whole (component+and on. The latest release run (3.2.0) measures **0.951** zero-LLM and+**0.938** local ([report](benchmarks/longmemeval-s-3.2.0-release.md)). With capture skipped and each raw session indexed whole (componentmode, ranking only), zero-LLM FTS scores 0.972; the[decomposition report](benchmarks/longmemeval-s-decomposition-2026-10-12.md)explains the difference.@@ -441,7 +442,8 @@ ships something ai-memory does not, it is named.card showing the user which memories were injected.- **Retrieval through capture, and no local reranker.** Through the shippedpipeline, LongMemEval-S hit@5 is 0.815 with local embeddings and 0.666- zero-LLM (2.6); the 3.0 observation stream raises them to 0.945 and 0.951. Ranking+ zero-LLM (2.6); the 3.0 observation stream raises them to 0.945 and 0.951+ (3.2.0: 0.938 and 0.951). Rankingthe same raw sessions whole scores 0.972 zero-LLM. The[decomposition](benchmarks/longmemeval-s-decomposition-2026-10-12.md) tracedthe gap to capture, and within capture not to the 2 KB privacy cap (pages holding exactly the captured text@@ -462,7 +464,7 @@ ships something ai-memory does not, it is named.| Tool | Number | Kind | Setup ||---|---|---|---|- | ai-memory 3.0 | LongMemEval-S hit@5 0.951 FTS / 0.945 local (2.6: 0.666 / 0.815) | Retrieval through capture | In-repo harness, all 470 questions; component mode FTS 0.972 |+ | ai-memory 3.2 | LongMemEval-S hit@5 0.951 FTS / 0.938 local (3.0: 0.951 / 0.945; 2.6: 0.666 / 0.815) | Retrieval through capture | In-repo harness, all 470 questions; component mode FTS 0.972 || MemPalace | 96.6% R@5 raw, 98.4% held-out (450 q) | Retrieval only | Different harness || agentmemory | 95.2% `recall_any@5`; 2026-10-09 baseline 96.2% any-hit, 84.68% strict recall-all@5 | Retrieval, component | Whole raw sessions, reranker disabled; the authors say the baseline supports no competitive claim || Mem0 | LongMemEval 94.4, LoCoMo 92.5, BEAM 64.1 (1M) / 48.6 (10M) | End-to-end QA, LLM-judged | Managed platform, top-200 budget; harness defaults to gpt-4o reader and judge; headline models not stated |diff --git a/docs/competitive-parity.md b/docs/competitive-parity.mdindex 04af326a..93d30cd8 100644--- a/docs/competitive-parity.md+++ b/docs/competitive-parity.md@@ -47,7 +47,7 @@ None of the competitors below has more than one or two of these; none has all.#16), self-hosted, with no paid tier.- **Reproducible retrieval benchmarks**: LongMemEval-S hit@5 0.815 local and0.666 zero-LLM through the shipped 2.6 pipeline, 0.951 zero-LLM and 0.945 local with the 3.0- observation stream, and a published pipeline-vs-component decomposition+ observation stream (3.2.0: 0.951 and 0.938, the local drop tracked as a possible regression), and a published pipeline-vs-component decomposition([`benchmarks/`](benchmarks/README.md)). Others publish numbers too (Mem0,Zep, Hindsight, EverOS, MemPalace, agentmemory), mostly end-to-end QA withtheir own reader and judge models. The edge is reproducibility; we claim no