Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(benchmarks): publish 3.2.2 release run

AkitaOnRails799d53f73664
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md
index 2f7753c3..69438f5e 100644
--- a/docs/benchmarks/README.md
+++ b/docs/benchmarks/README.md
@@ -13,6 +13,10 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| date | dataset | mode | overall hit@5 | file |
|---|---|---|---|---|
+| **2026-10-22** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.2.2 release)** | **0.951** | [longmemeval-s-3.2.2-release.md](longmemeval-s-3.2.2-release.md) |
+| **2026-10-22** | LongMemEval-S (v1) | **local embeddings + observation stream (3.2.2 release)** | **0.945** | [longmemeval-s-3.2.2-release.md](longmemeval-s-3.2.2-release.md) |
+| **2026-10-22** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.2.1 release)** | **0.951** | [longmemeval-s-3.2.1-release.md](longmemeval-s-3.2.1-release.md) |
+| **2026-10-22** | LongMemEval-S (v1) | **local embeddings + observation stream (3.2.1 release)** | **0.949** | [longmemeval-s-3.2.1-release.md](longmemeval-s-3.2.1-release.md) |
| **2026-10-22** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.2.0 release)** | **0.951** | [longmemeval-s-3.2.0-release.md](longmemeval-s-3.2.0-release.md) |
| **2026-10-22** | LongMemEval-S (v1) | **local embeddings + observation stream (3.2.0 release)** | **0.938** | [longmemeval-s-3.2.0-release.md](longmemeval-s-3.2.0-release.md) |
| **2026-10-20** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.1.2 release)** | **0.951** | [longmemeval-s-3.1.2-release.md](longmemeval-s-3.1.2-release.md) |
@@ -29,6 +33,18 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |
| 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |
+The 3.2.2 patch-release run found FTS identical to 3.2.1 on every question.
+Local embeddings moved from 0.949 to 0.945 `hit@5`, inside the pre-registered
+overall noise band; the two changed questions remain within the top ten, and
+`hit@10` improved from 0.968 to 0.974. See the
+[3.2.2 release report](longmemeval-s-3.2.2-release.md) for provenance, every
+slice, and the explicit threshold assessment.
+
+The 3.2.1 patch-release run recovered 3.2.0's local embedding accuracy and
+documented the remaining boundary-question variation. See the
+[3.2.1 release report](longmemeval-s-3.2.1-release.md) for the capacity-fix
+measurement and controlled latency follow-up.
+
The 2026-10-22 minor-release run found FTS identical to 3.1.2 on every
question. Local embeddings fell from 0.947 to 0.938 `hit@5` (five questions
lost, one gained), with higher query latency and 75% longer ingest. That is
diff --git a/docs/benchmarks/longmemeval-s-3.2.2-release.md b/docs/benchmarks/longmemeval-s-3.2.2-release.md
new file mode 100644
index 00000000..03f7f0aa
--- /dev/null
+++ b/docs/benchmarks/longmemeval-s-3.2.2-release.md
@@ -0,0 +1,88 @@
+# LongMemEval-S release benchmark: 3.2.2
+
+The 3.2.2 release benchmark runs all 470 answerable LongMemEval-S questions
+through the shipped hook capture pipeline. It measures the default zero-LLM
+FTS path and the in-process local embedder, both with the production-default
+observation stream enabled. The run was pre-registered in
+[#1620](https://github.com/akitaonrails/ai-memory/issues/1620): exact FTS
+parity, local overall `hit@5` within +/-0.005, and slice deltas within
+0.02-0.03 were the release thresholds.
+
+## Provenance
+
+- code: both modes used `a736b4b74eb954e9a76ade38a6f839d80b11ff6b`,
+ the frozen 3.2.2 release candidate.
+- dataset: `longmemeval_s`, sha256
+ `08d8dad4be43ee2049a22ff5674eb86725d0ce5ff434cde2627e5e8e7e117894`.
+- hardware: AMD Ryzen 9 7950X3D (32 threads), with other agent workloads
+ sharing the machine. Accuracy is load-insensitive; latency is not.
+- commands, after building `ai-memory` and `ai-memory-eval` in release mode:
+
+ ```bash
+ ai-memory-eval retrieval \
+ --server-bin target/release/ai-memory \
+ --concurrency 6 --http-timeout-secs 900 \
+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true \
+ --server-env AI_MEMORY_EVAL_SKIP_WIKI_CHECKPOINTS=1
+
+ ai-memory-eval retrieval \
+ --server-bin target/release/ai-memory \
+ --concurrency 6 --http-timeout-secs 900 \
+ --embeddings local \
+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true \
+ --server-env AI_MEMORY_EVAL_SKIP_WIKI_CHECKPOINTS=1
+ ```
+
+- run dirs: `evals/runs/2026-10-22T22-32-08-613671222Z-retrieval` (FTS)
+ and `evals/runs/2026-10-22T22-33-15-405163139Z-retrieval` (local).
+
+## Results
+
+FTS matches 3.2.1 exactly at every accuracy cutoff. Local `hit@5` moved from
+0.949 to 0.945, a two-question change and within the pre-registered overall
+noise band. Both questions moved only from ranks 1-5 to ranks 6-10;
+`hit@10` improved from 0.968 to 0.974.
+
+| mode | hit@1 | hit@3 | hit@5 | hit@10 | recall@5 | recall@10 | p50 | p95 | mean context tokens |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| FTS 3.2.1 | 0.857 | 0.930 | 0.951 | 0.972 | 0.893 | 0.937 | 12 ms | 84 ms | 527.8 |
+| **FTS 3.2.2** | **0.857** | **0.930** | **0.951** | **0.972** | **0.893** | **0.937** | **12 ms** | **73 ms** | **534.8** |
+| local 3.2.1 | 0.760 | 0.904 | 0.949 | 0.968 | 0.882 | 0.936 | 171 ms | 389 ms | 527.8 |
+| **local 3.2.2** | **0.755** | **0.900** | **0.945** | **0.974** | **0.880** | **0.940** | **188 ms** | **468 ms** | **535.2** |
+
+Per-slice local `hit@5`:
+
+| slice | n | 3.2.1 | 3.2.2 | delta |
+|---|---:|---:|---:|---:|
+| knowledge-update | 72 | 1.000 | 1.000 | 0.000 |
+| multi-session | 121 | 0.975 | 0.975 | 0.000 |
+| single-session-assistant | 56 | 0.893 | 0.893 | 0.000 |
+| single-session-preference | 30 | 0.900 | 0.867 | -0.033 |
+| single-session-user | 64 | 0.953 | 0.953 | 0.000 |
+| temporal-reasoning | 127 | 0.929 | 0.921 | -0.008 |
+
+The preference slice exceeds its nominal 0.03 threshold by 0.003, but that is
+one question in a 30-question slice. The temporal delta is also one question.
+The affected questions (`57f827a0` and `gpt4_7ddcf75f`) remain hits at rank
+10. This is the same rank-boundary behavior documented by the 3.2.1 run, not
+a broad ranking shift. No question fell out of the top ten on either slice.
+
+## Latency and ingest
+
+The local latency changed from 171/389 ms to 188/468 ms p50/p95 while the
+machine was under varying shared load, so it is not evidence of a product
+regression. FTS p50 stayed at 12 ms and p95 improved to 73 ms. Both modes
+retrieved zero raw-observation fallback hits across all 470 questions, which
+also confirms that the fallback-related fixes in this patch were inactive in
+the measured ranking path.
+
+FTS ingested 278,946 items in 61.9 seconds. Local embeddings ingested the same
+items in 1,335.1 seconds, down from 1,841.1 seconds in the loaded 3.2.1 run.
+Ingest wall time remains load-sensitive and is not used as a release gate.
+
+## Verdict
+
+There is no material retrieval regression. FTS is exact, local overall
+`hit@5` is inside the pre-registered tolerance, the two top-five changes stay
+inside the top ten, and local `hit@10` and `recall@10` both improved. The
+3.2.2 release gate passes.