Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(benchmarks): publish the 3.2.1 LongMemEval-S release run

AkitaOnRailse52abcb6bf2a
diff --git a/docs/benchmarks/longmemeval-s-3.2.1-release.md b/docs/benchmarks/longmemeval-s-3.2.1-release.md
new file mode 100644
index 00000000..deb7a4b7
--- /dev/null
+++ b/docs/benchmarks/longmemeval-s-3.2.1-release.md
@@ -0,0 +1,112 @@
+# LongMemEval-S release benchmark: 3.2.1
+
+The 3.2.1 release benchmark runs all 470 answerable LongMemEval-S questions
+through the shipped hook capture pipeline, measuring the supported default
+retrieval paths: zero-LLM FTS and the in-process local embedder, both with the
+production-default observation stream enabled. It is the first release run on
+the fast procedure (wiki git checkpoints skipped in the eval server; see
+[Running it fast](README.md#running-it-fast)): accuracy, latency and
+context-token numbers stay comparable with earlier releases, and only ingest
+wall clock differs, because runs before October 2026 paid for the commits.
+
+## Provenance
+
+- code: both modes used `7566d039fe902341ffc885325a31b836ceedcedc`, the
+ release candidate that became `v3.2.1`. It contains the #1580 fix
+ (embedding capacity sized from the CPU, bounded query wait) and no other
+ retrieval-relevant change since 3.2.0.
+- dataset: `longmemeval_s`, sha256
+ `08d8dad4be43ee2049a22ff5674eb86725d0ce5ff434cde2627e5e8e7e117894`.
+- hardware: AMD Ryzen 9 7950X3D (32 threads). Unlike the 3.1.2 and 3.2.0
+ runs, this one ran while several agent sessions shared the machine. The
+ accuracy numbers are load-insensitive; the query latencies below are not,
+ and the load context is stated with them.
+- commands, after building `ai-memory` and `ai-memory-eval` in release mode:
+
+ ```bash
+ ai-memory-eval retrieval \
+ --server-bin target/release/ai-memory \
+ --concurrency 6 --http-timeout-secs 900 \
+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true \
+ --server-env AI_MEMORY_EVAL_SKIP_WIKI_CHECKPOINTS=1
+
+ ai-memory-eval retrieval \
+ --server-bin target/release/ai-memory \
+ --concurrency 6 --http-timeout-secs 900 \
+ --embeddings local \
+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true \
+ --server-env AI_MEMORY_EVAL_SKIP_WIKI_CHECKPOINTS=1
+ ```
+
+- run dirs: `evals/runs/2026-10-22T17-11-44-812925488Z-retrieval` (FTS)
+ and `evals/runs/2026-10-22T17-12-55-885591885Z-retrieval` (local).
+
+## Results
+
+FTS matches 3.2.0 (and 3.1.2) on every question at every cutoff, which also
+re-confirms that skipping the eval server's wiki checkpoints changes no
+published number. Local embeddings recovered the accuracy 3.2.0 lost: overall
+`hit@5` is 0.949, above both 3.2.0's 0.938 and 3.1.2's 0.947.
+
+| mode | hit@1 | hit@5 | hit@10 | p50 | p95 | mean context tokens |
+|---|---:|---:|---:|---:|---:|---:|
+| FTS 3.2.0 | 0.857 | 0.951 | 0.972 | 12 ms | 97 ms | 528 |
+| **FTS 3.2.1** | **0.857** | **0.951** | **0.972** | **12 ms** | **84 ms** | **528** |
+| local 3.1.2 | 0.768 | 0.947 | 0.977 | 98 ms | 225 ms | 528 |
+| local 3.2.0 | 0.757 | 0.938 | 0.974 | 118 ms | 295 ms | 528 |
+| **local 3.2.1** | **0.760** | **0.949** | **0.968** | **171 ms** | **389 ms** | **528** |
+
+Local `recall@5` is 0.882 (3.1.2: 0.879, 3.2.0: 0.873) and `recall@10` 0.936
+(0.947, 0.945). `hit@1` sits between the two earlier releases and `hit@10`
+half a point under both: the same boundary questions that 3.2.0 pushed past
+rank 5 came back, some landing inside the top 5 and others between 6 and 10.
+
+Per-slice `hit@5`:
+
+| slice | n | local 3.1.2 | local 3.2.0 | local 3.2.1 |
+|---|---:|---:|---:|---:|
+| knowledge-update | 72 | 1.000 | 1.000 | 1.000 |
+| multi-session | 121 | 0.975 | 0.975 | 0.975 |
+| single-session-assistant | 56 | 0.893 | 0.875 | **0.893** |
+| single-session-preference | 30 | 0.900 | 0.900 | 0.900 |
+| single-session-user | 64 | 0.953 | 0.969 | **0.953** |
+| temporal-reasoning | 127 | 0.921 | 0.890 | **0.929** |
+
+The two slices 3.2.0 moved both came back: temporal-reasoning recovered the
+four lost questions and gained a fifth (0.929, its best value across the
+three releases), and single-session-assistant returned its lost question
+exactly. single-session-user gave back the one question 3.2.0 had gained.
+That is the run-to-run picture #1580 documented: these particular questions
+sit at the rank-5 boundary and flip between identical builds, so single-
+question deltas in this band carry no signal, and the slice-to-slice
+round trip is what recovery looks like.
+
+## Latency and ingest
+
+The local p50/p95 above (171/389 ms) are not comparable to the earlier rows:
+this run shared the machine with other agent workloads, and query latency is
+CPU-sensitive. Two comparisons that are:
+
+- A controlled subset run (the five lost plus one gained question and ten
+ temporal controls, 16 questions, quiet machine, same build) measured
+ **66/87 ms** p50/p95 against 3.2.0's full-run 118/295: the capacity fix
+ removed the contention 3.2.0 added, and the reserved query slot keeps
+ latency under 3.1.2's 98/225 as well.
+- FTS latency, which the embedding scheduler cannot touch, measured 12/84 ms
+ on the same loaded machine — further evidence the local numbers above are
+ load, not regression.
+
+Ingest wall clock is no longer comparable across releases: the fast procedure
+skips wiki git commits the earlier runs paid for. FTS ingested 278,946 items
+in 61 s; local finished in 1,841 s with embedding 87% of server-side ingest
+time, the known remaining ceiling. Run-to-run wall-clock spread under ambient
+load is roughly ±15%.
+
+## Verdict for #1580
+
+The 3.2.0 accuracy loss was boundary-question noise, not a ranking change:
+with the capacity fix landed, the same questions flip between identical
+builds, and this release run recovered both moved slices. The real 3.2.0
+regressions — serialized document embedding, silently degraded queries,
+higher latency under load — are fixed and measured. No follow-up retrieval
+work is required from #1580.
diff --git a/docs/comparison.md b/docs/comparison.md
index eeee8dfd..696ded37 100644
--- a/docs/comparison.md
+++ b/docs/comparison.md
@@ -55,8 +55,8 @@ LongMemEval-S hit@5 **0.666** zero-LLM and **0.815** with the default local
embeddings in the published 2.6 line. The observation stream in 3.0
(on by default) raises zero-LLM to **0.951** and local embeddings
to **0.945** on all 470 questions, measured on one build with the stream off
-and on. The latest release run (3.2.0) measures **0.951** zero-LLM and
-**0.938** local ([report](benchmarks/longmemeval-s-3.2.0-release.md)). With capture skipped and each raw session indexed whole (component
+and on. The latest release run (3.2.1) measures **0.951** zero-LLM and
+**0.949** local ([report](benchmarks/longmemeval-s-3.2.1-release.md)). With capture skipped and each raw session indexed whole (component
mode, ranking only), zero-LLM FTS scores 0.972; the
[decomposition report](benchmarks/longmemeval-s-decomposition-2026-10-12.md)
explains the difference.
@@ -443,7 +443,7 @@ ships something ai-memory does not, it is named.
- **Retrieval through capture, and no local reranker.** Through the shipped
pipeline, LongMemEval-S hit@5 is 0.815 with local embeddings and 0.666
zero-LLM (2.6); the 3.0 observation stream raises them to 0.945 and 0.951
- (3.2.0: 0.938 and 0.951). Ranking
+ (3.2.1: 0.949 and 0.951). Ranking
the same raw sessions whole scores 0.972 zero-LLM. The
[decomposition](benchmarks/longmemeval-s-decomposition-2026-10-12.md) traced
the gap to capture, and within capture not to the 2 KB privacy cap (pages holding exactly the captured text
@@ -464,7 +464,7 @@ ships something ai-memory does not, it is named.
| Tool | Number | Kind | Setup |
|---|---|---|---|
- | ai-memory 3.2 | LongMemEval-S hit@5 0.951 FTS / 0.938 local (3.0: 0.951 / 0.945; 2.6: 0.666 / 0.815) | Retrieval through capture | In-repo harness, all 470 questions; component mode FTS 0.972 |
+ | ai-memory 3.2 | LongMemEval-S hit@5 0.951 FTS / 0.949 local (3.0: 0.951 / 0.945; 2.6: 0.666 / 0.815) | Retrieval through capture | In-repo harness, all 470 questions; component mode FTS 0.972 |
| MemPalace | 96.6% R@5 raw, 98.4% held-out (450 q) | Retrieval only | Different harness |
| agentmemory | 95.2% `recall_any@5`; 2026-10-09 baseline 96.2% any-hit, 84.68% strict recall-all@5 | Retrieval, component | Whole raw sessions, reranker disabled; the authors say the baseline supports no competitive claim |
| Mem0 | LongMemEval 94.4, LoCoMo 92.5, BEAM 64.1 (1M) / 48.6 (10M) | End-to-end QA, LLM-judged | Managed platform, top-200 budget; harness defaults to gpt-4o reader and judge; headline models not stated |
diff --git a/docs/competitive-parity.md b/docs/competitive-parity.md
index 93d30cd8..6e3acaf8 100644
--- a/docs/competitive-parity.md
+++ b/docs/competitive-parity.md
@@ -47,7 +47,7 @@ None of the competitors below has more than one or two of these; none has all.
#16), self-hosted, with no paid tier.
- **Reproducible retrieval benchmarks**: LongMemEval-S hit@5 0.815 local and
0.666 zero-LLM through the shipped 2.6 pipeline, 0.951 zero-LLM and 0.945 local with the 3.0
- observation stream (3.2.0: 0.951 and 0.938, the local drop tracked as a possible regression), and a published pipeline-vs-component decomposition
+ observation stream (3.2.1: 0.951 and 0.949, the 3.2.0 local drop recovered — it was boundary-question noise, see the 3.2.1 report), and a published pipeline-vs-component decomposition
([`benchmarks/`](benchmarks/README.md)). Others publish numbers too (Mem0,
Zep, Hindsight, EverOS, MemPalace, agentmemory), mostly end-to-end QA with
their own reader and judge models. The edge is reproducibility; we claim no