Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(benchmarks): publish the 3.2.0 LongMemEval-S release run

AkitaOnRails394fd873b974
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md
index 99cb93bb..35d17045 100644
--- a/docs/benchmarks/README.md
+++ b/docs/benchmarks/README.md
@@ -13,6 +13,8 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| date | dataset | mode | overall hit@5 | file |
|---|---|---|---|---|
+| **2026-10-22** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.2.0 release)** | **0.951** | [longmemeval-s-3.2.0-release.md](longmemeval-s-3.2.0-release.md) |
+| **2026-10-22** | LongMemEval-S (v1) | **local embeddings + observation stream (3.2.0 release)** | **0.938** | [longmemeval-s-3.2.0-release.md](longmemeval-s-3.2.0-release.md) |
| **2026-10-20** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.1.2 release)** | **0.951** | [longmemeval-s-3.1.2-release.md](longmemeval-s-3.1.2-release.md) |
| **2026-10-20** | LongMemEval-S (v1) | **local embeddings + observation stream (3.1.2 release)** | **0.947** | [longmemeval-s-3.1.2-release.md](longmemeval-s-3.1.2-release.md) |
| **2026-10-16** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.1 release)** | **0.951** | [longmemeval-s-3.1.0-release.md](longmemeval-s-3.1.0-release.md) |
@@ -27,6 +29,13 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |
| 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |
+The 2026-10-22 minor-release run found FTS identical to 3.1.2 on every
+question. Local embeddings fell from 0.947 to 0.938 `hit@5` (five questions
+lost, one gained), with higher query latency and 75% longer ingest. That is
+outside the +/-0.005 noise band and is tracked as a possible regression for
+the next release. See the [3.2.0 release report](longmemeval-s-3.2.0-release.md)
+for every slice, ingest performance, the suspected commit and provenance.
+
The 2026-10-20 patch-release run found no retrieval regression against 3.1.
FTS is identical at every accuracy cutoff. Local embeddings improved from
0.940 to 0.947 `hit@5`, with stable latency and context size. See the
diff --git a/docs/benchmarks/longmemeval-s-3.2.0-release.md b/docs/benchmarks/longmemeval-s-3.2.0-release.md
new file mode 100644
index 00000000..c2a795ff
--- /dev/null
+++ b/docs/benchmarks/longmemeval-s-3.2.0-release.md
@@ -0,0 +1,104 @@
+# LongMemEval-S release benchmark: 3.2.0
+
+The 3.2.0 release benchmark runs all 470 answerable LongMemEval-S questions
+through the shipped hook capture pipeline. It measures the supported default
+retrieval paths: zero-LLM FTS and the in-process local embedder, both with the
+production-default observation stream enabled.
+
+## Provenance
+
+- code: both modes used `1c86aebc14addccc7ac875913d565c681bfa4a50`, the
+ `v3.2.0` tag. The benchmark ran after the release was published, and this
+ report is the only change since.
+- dataset: `longmemeval_s`, sha256
+ `08d8dad4be43ee2049a22ff5674eb86725d0ce5ff434cde2627e5e8e7e117894`.
+- hardware: AMD Ryzen 9 7950X3D (32 threads). No build or test workload ran
+ during either measurement. Static code review and a few single-threaded
+ shell timings in throwaway containers overlapped the FTS run and the first
+ minutes of the local run; the FTS result matches 3.1.2 exactly and the local
+ slowdown below lasts the whole run, so that work does not explain either.
+- commands, after `cargo clean` and building `ai-memory` and `ai-memory-eval`
+ in release mode:
+
+ ```bash
+ ai-memory-eval retrieval \
+ --server-bin target/release/ai-memory \
+ --concurrency 6 --http-timeout-secs 900 \
+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true
+
+ ai-memory-eval retrieval \
+ --server-bin target/release/ai-memory \
+ --concurrency 6 --http-timeout-secs 900 \
+ --embeddings local \
+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true
+ ```
+
+- run dirs: `evals/runs/2026-10-22T02-00-49-545842361Z-retrieval` (FTS)
+ and `evals/runs/2026-10-22T03-05-33-423353129Z-retrieval` (local).
+
+## Results
+
+FTS is identical to 3.1.2 on every question at every cutoff. Local embeddings
+lost five questions at `hit@5` and gained one, moving from 0.947 to 0.938, and
+query latency and ingest time rose. That change is outside the
+pre-registered +/-0.005 overall noise band and is tracked as a possible
+regression for the next release.
+
+| mode | hit@1 | hit@5 | hit@10 | p50 | p95 | mean context tokens |
+|---|---:|---:|---:|---:|---:|---:|
+| FTS 3.1.2 | 0.857 | 0.951 | 0.972 | 12 ms | 103 ms | 528 |
+| **FTS 3.2.0** | **0.857** | **0.951** | **0.972** | **12 ms** | **97 ms** | **528** |
+| local 3.1.2 | 0.768 | 0.947 | 0.977 | 98 ms | 225 ms | 528 |
+| **local 3.2.0** | **0.757** | **0.938** | **0.974** | **118 ms** | **295 ms** | **528** |
+
+Recall moved with hit rate: local `recall@5` went from 0.879 to 0.873 and
+`recall@10` from 0.947 to 0.945. FTS recall is unchanged at every cutoff.
+
+Per-slice `hit@5`:
+
+| slice | n | FTS 3.1.2 | FTS 3.2.0 | local 3.1.2 | local 3.2.0 |
+|---|---:|---:|---:|---:|---:|
+| knowledge-update | 72 | 1.000 | 1.000 | 1.000 | 1.000 |
+| multi-session | 121 | 0.967 | 0.967 | 0.975 | 0.975 |
+| temporal-reasoning | 127 | 0.921 | 0.921 | 0.921 | 0.890 |
+| single-session-assistant | 56 | 1.000 | 1.000 | 0.893 | 0.875 |
+| single-session-preference | 30 | 0.700 | 0.700 | 0.900 | 0.900 |
+| single-session-user | 64 | 1.000 | 1.000 | 0.953 | 0.969 |
+
+The local losses are four temporal-reasoning questions (`71017276`,
+`gpt4_68e94288`, `gpt4_b5700ca0`, `gpt4_e061b84f`) and one
+single-session-assistant question (`5809eb10`); `001be529`
+(single-session-user) was gained. The temporal-reasoning drop of 0.031 sits at
+the upper edge of the +/-0.02 to 0.03 small-slice band, and the other two
+slice moves are one question each.
+
+Local query latency rose on every slice, from 98 to 118 ms at p50 and from
+225 to 295 ms at p95. Returned context is unchanged.
+
+## Ingest
+
+| mode | items | requests | throughput | wall clock | median ingest per question | fallback answers |
+|---|---:|---:|---:|---:|---:|---:|
+| FTS 3.1.2 | 278,946 | 1,403 | 12 items/s per question stream | 3,837.0 s | 48.7 s | 0/470 |
+| FTS 3.2.0 | 278,946 | 1,403 | 12 items/s per question stream | 3,882.6 s | 48.5 s | 0/470 |
+| local 3.1.2 | 278,946 | 1,403 | 10 items/s per question stream | 4,899.8 s | 60.1 s | 0/470 |
+| local 3.2.0 | 278,946 | 1,403 | 5 items/s per question stream | 8,590.6 s | 101.3 s | 0/470 |
+
+FTS ingest is unchanged, so the CJK search index added in 3.2.0 does not
+account for the local slowdown. Local ingest took 75% longer end to end.
+
+No document exceeded the 16 KiB capture bound. The raw-observation fallback
+was not needed, so both results measure indexed retrieval.
+
+## Suspected cause
+
+3.2.0 reserved local-embedding capacity for interactive queries (`107ada8d`,
+"Reserved local-embedding capacity for interactive search queries" in the
+3.2.0 changelog). The local embedder is now wrapped in a two-slot limit: one
+slot for background document embedding and one held back for queries. Page
+writes therefore embed one document at a time per server, which matches the
+halved ingest throughput. Whether the slower embedding also explains the
+accuracy loss, for example by leaving some pages without a vector when their
+question is asked, has not been measured. The follow-up issue for the next
+release ([#1580](https://github.com/akitaonrails/ai-memory/issues/1580)) records the deltas, run dirs and this commit so the trade can be
+measured and decided there.
diff --git a/docs/comparison.md b/docs/comparison.md
index 1eaa7986..eeee8dfd 100644
--- a/docs/comparison.md
+++ b/docs/comparison.md
@@ -55,7 +55,8 @@ LongMemEval-S hit@5 **0.666** zero-LLM and **0.815** with the default local
embeddings in the published 2.6 line. The observation stream in 3.0
(on by default) raises zero-LLM to **0.951** and local embeddings
to **0.945** on all 470 questions, measured on one build with the stream off
-and on. With capture skipped and each raw session indexed whole (component
+and on. The latest release run (3.2.0) measures **0.951** zero-LLM and
+**0.938** local ([report](benchmarks/longmemeval-s-3.2.0-release.md)). With capture skipped and each raw session indexed whole (component
mode, ranking only), zero-LLM FTS scores 0.972; the
[decomposition report](benchmarks/longmemeval-s-decomposition-2026-10-12.md)
explains the difference.
@@ -441,7 +442,8 @@ ships something ai-memory does not, it is named.
card showing the user which memories were injected.
- **Retrieval through capture, and no local reranker.** Through the shipped
pipeline, LongMemEval-S hit@5 is 0.815 with local embeddings and 0.666
- zero-LLM (2.6); the 3.0 observation stream raises them to 0.945 and 0.951. Ranking
+ zero-LLM (2.6); the 3.0 observation stream raises them to 0.945 and 0.951
+ (3.2.0: 0.938 and 0.951). Ranking
the same raw sessions whole scores 0.972 zero-LLM. The
[decomposition](benchmarks/longmemeval-s-decomposition-2026-10-12.md) traced
the gap to capture, and within capture not to the 2 KB privacy cap (pages holding exactly the captured text
@@ -462,7 +464,7 @@ ships something ai-memory does not, it is named.
| Tool | Number | Kind | Setup |
|---|---|---|---|
- | ai-memory 3.0 | LongMemEval-S hit@5 0.951 FTS / 0.945 local (2.6: 0.666 / 0.815) | Retrieval through capture | In-repo harness, all 470 questions; component mode FTS 0.972 |
+ | ai-memory 3.2 | LongMemEval-S hit@5 0.951 FTS / 0.938 local (3.0: 0.951 / 0.945; 2.6: 0.666 / 0.815) | Retrieval through capture | In-repo harness, all 470 questions; component mode FTS 0.972 |
| MemPalace | 96.6% R@5 raw, 98.4% held-out (450 q) | Retrieval only | Different harness |
| agentmemory | 95.2% `recall_any@5`; 2026-10-09 baseline 96.2% any-hit, 84.68% strict recall-all@5 | Retrieval, component | Whole raw sessions, reranker disabled; the authors say the baseline supports no competitive claim |
| Mem0 | LongMemEval 94.4, LoCoMo 92.5, BEAM 64.1 (1M) / 48.6 (10M) | End-to-end QA, LLM-judged | Managed platform, top-200 budget; harness defaults to gpt-4o reader and judge; headline models not stated |
diff --git a/docs/competitive-parity.md b/docs/competitive-parity.md
index 04af326a..93d30cd8 100644
--- a/docs/competitive-parity.md
+++ b/docs/competitive-parity.md
@@ -47,7 +47,7 @@ None of the competitors below has more than one or two of these; none has all.
#16), self-hosted, with no paid tier.
- **Reproducible retrieval benchmarks**: LongMemEval-S hit@5 0.815 local and
0.666 zero-LLM through the shipped 2.6 pipeline, 0.951 zero-LLM and 0.945 local with the 3.0
- observation stream, and a published pipeline-vs-component decomposition
+ observation stream (3.2.0: 0.951 and 0.938, the local drop tracked as a possible regression), and a published pipeline-vs-component decomposition
([`benchmarks/`](benchmarks/README.md)). Others publish numbers too (Mem0,
Zep, Hindsight, EverOS, MemPalace, agentmemory), mostly end-to-end QA with
their own reader and judge models. The edge is reproducibility; we claim no