Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs: publish 3.3 retrieval benchmark
AkitaOnRails251b4d80b9a1
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.mdindex 0efc68e8..f6d35303 100644--- a/docs/benchmarks/README.md+++ b/docs/benchmarks/README.md@@ -13,6 +13,8 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch| date | dataset | mode | overall hit@5 | file ||---|---|---|---|---|+| **2026-10-29** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.3.0 release)** | **0.951** | [longmemeval-s-3.3.0-release.md](longmemeval-s-3.3.0-release.md) |+| **2026-10-29** | LongMemEval-S (v1) | **local embeddings + observation stream (3.3.0 release)** | **0.945** | [longmemeval-s-3.3.0-release.md](longmemeval-s-3.3.0-release.md) || **2026-10-26** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.2.6 release)** | **0.951** | [longmemeval-s-3.2.6-release.md](longmemeval-s-3.2.6-release.md) || **2026-10-26** | LongMemEval-S (v1) | **local embeddings + observation stream (3.2.6 release)** | **0.945** | [longmemeval-s-3.2.6-release.md](longmemeval-s-3.2.6-release.md) || **2026-10-25** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.2.4 release)** | **0.951** | [longmemeval-s-3.2.4-release.md](longmemeval-s-3.2.4-release.md) |@@ -37,6 +39,13 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) || 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |+The 3.3.0 minor-release run found FTS identical to 3.2.6 at every accuracy+and recall cutoff and kept local `hit@5` exact. Local `hit@10` and+`recall@10` each moved by 0.007, slightly outside the documented overall+noise band, and are tracked in #1854. See the+[3.3.0 release report](longmemeval-s-3.3.0-release.md) for provenance, every+slice, and the explicit threshold assessment.+The 3.2.6 patch-release run found FTS identical to 3.2.4 at every accuracyand recall cutoff. Local embeddings kept the same `hit@5` and `hit@10`overall, while `recall@10` improved. See thediff --git a/docs/benchmarks/longmemeval-s-3.3.0-release.md b/docs/benchmarks/longmemeval-s-3.3.0-release.mdnew file mode 100644index 00000000..e02a5318--- /dev/null+++ b/docs/benchmarks/longmemeval-s-3.3.0-release.md@@ -0,0 +1,75 @@+# LongMemEval-S release benchmark: 3.3.0++The 3.3.0 release benchmark runs all 470 answerable LongMemEval-S questions+through the shipped hook capture pipeline. It measures the default zero-LLM+FTS path and the in-process local embedder, both with the production-default+observation stream enabled.++## Provenance++- code: both modes used `c92047ed3b5f639aab615ae4e33f3310c52c0843`,+ the frozen 3.3.0 release candidate before this report.+- dataset: `longmemeval_s`, sha256+ `08d8dad4be43ee2049a22ff5674eb86725d0ce5ff434cde2627e5e8e7e117894`.+- hardware: AMD Ryzen 9 7950X3D (32 threads). The two modes ran serially;+ no build or test workload overlapped the local-embedding run.+- commands, after building `ai-memory` and `ai-memory-eval` in release mode:++ ```bash+ ai-memory-eval retrieval \+ --server-bin target/release/ai-memory \+ --concurrency 6 --http-timeout-secs 900 \+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true \+ --server-env AI_MEMORY_EVAL_SKIP_WIKI_CHECKPOINTS=1++ ai-memory-eval retrieval \+ --server-bin target/release/ai-memory \+ --concurrency 6 --http-timeout-secs 900 \+ --embeddings local \+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true \+ --server-env AI_MEMORY_EVAL_SKIP_WIKI_CHECKPOINTS=1+ ```++- run dirs: `evals/runs/2026-10-29T21-20-20-570928323Z-retrieval` (FTS)+ and `evals/runs/2026-10-29T21-21-17-842507936Z-retrieval` (local).++## Results++FTS matches 3.2.6 exactly at every accuracy and recall cutoff. Local+embeddings keep the primary `hit@5` metric exact. Local `hit@10` and+`recall@10` each moved by 0.007, roughly three questions and just outside the+documented +/-0.005 overall noise band. The result is published rather than+re-run until favorable; #1854 tracks the per-question investigation.++| mode | hit@1 | hit@3 | hit@5 | hit@10 | recall@5 | recall@10 | p50 | p95 | mean context tokens |+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|+| FTS 3.2.6 | 0.857 | 0.930 | 0.951 | 0.972 | 0.893 | 0.937 | 11 ms | 73 ms | 534.8 |+| **FTS 3.3.0** | **0.857** | **0.930** | **0.951** | **0.972** | **0.893** | **0.937** | **11 ms** | **73 ms** | **534.8** |+| local 3.2.6 | 0.766 | 0.904 | 0.945 | 0.977 | 0.879 | 0.947 | 157 ms | 363 ms | 535.2 |+| **local 3.3.0** | **0.762** | **0.909** | **0.945** | **0.970** | **0.876** | **0.940** | **141 ms** | **330 ms** | **534.5** |++Local `hit@5` moved on two small slices while remaining exact overall:++| slice | n | 3.2.6 | 3.3.0 | delta |+|---|---:|---:|---:|---:|+| knowledge-update | 72 | 1.000 | 1.000 | 0.000 |+| multi-session | 121 | 0.975 | 0.975 | 0.000 |+| single-session-assistant | 56 | 0.893 | 0.893 | 0.000 |+| single-session-preference | 30 | 0.867 | 0.900 | +0.033 |+| single-session-user | 64 | 0.969 | 0.953 | -0.016 |+| temporal-reasoning | 127 | 0.913 | 0.913 | 0.000 |++## Latency and ingest++FTS query latency stayed at 11/73 ms p50/p95. Local query latency moved from+157/363 ms to 141/330 ms. FTS ingested 278,946 items in 55.2 seconds. Local+embeddings ingested the same items in 1,624.1 seconds. Ingest wall time is+load-sensitive and is not a release gate. Both runs returned zero+raw-observation fallback hits.++## Verdict++The primary retrieval gate passes: FTS is exact and local `hit@5` is exact.+The top-ten movement is small but outside the pre-registered overall noise+band, so it is published here and tracked in #1854 instead of hidden or used+to tune this release.