Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs: publish 3.1.2 retrieval benchmark

AkitaOnRailsed6aa09ed390
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md
index 609874f7..3b45d158 100644
--- a/docs/benchmarks/README.md
+++ b/docs/benchmarks/README.md
@@ -13,6 +13,8 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| date | dataset | mode | overall hit@5 | file |
|---|---|---|---|---|
+| **2026-10-20** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.1.2 release)** | **0.951** | [longmemeval-s-3.1.2-release.md](longmemeval-s-3.1.2-release.md) |
+| **2026-10-20** | LongMemEval-S (v1) | **local embeddings + observation stream (3.1.2 release)** | **0.947** | [longmemeval-s-3.1.2-release.md](longmemeval-s-3.1.2-release.md) |
| **2026-10-16** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.1 release)** | **0.951** | [longmemeval-s-3.1.0-release.md](longmemeval-s-3.1.0-release.md) |
| **2026-10-16** | LongMemEval-S (v1) | **local embeddings + observation stream (3.1 release)** | **0.940** | [longmemeval-s-3.1.0-release.md](longmemeval-s-3.1.0-release.md) |
| **2026-10-15** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (3.0.1 release)** | **0.951** | [longmemeval-s-3.0.1-release.md](longmemeval-s-3.0.1-release.md) |
@@ -25,6 +27,12 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |
| 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |
+The 2026-10-20 patch-release run found no retrieval regression against 3.1.
+FTS is identical at every accuracy cutoff. Local embeddings improved from
+0.940 to 0.947 `hit@5`, with stable latency and context size. See the
+[3.1.2 release report](longmemeval-s-3.1.2-release.md) for every slice,
+ingest performance, commands, and exact provenance.
+
The 2026-10-16 minor-release run found no material retrieval regression
against 3.0.1. FTS is identical at every overall cutoff. Local embeddings
moved from 0.943 to 0.940 `hit@5`, inside the pre-registered +/-0.005 noise
diff --git a/docs/benchmarks/longmemeval-s-3.1.2-release.md b/docs/benchmarks/longmemeval-s-3.1.2-release.md
new file mode 100644
index 00000000..d4366516
--- /dev/null
+++ b/docs/benchmarks/longmemeval-s-3.1.2-release.md
@@ -0,0 +1,78 @@
+# LongMemEval-S release benchmark: 3.1.2
+
+The 3.1.2 release benchmark runs all 470 answerable LongMemEval-S questions
+through the shipped hook capture pipeline. It measures the supported default
+retrieval paths: zero-LLM FTS and the in-process local embedder, both with the
+production-default observation stream enabled.
+
+## Provenance
+
+- code: both modes used `80133202cf7298d1b7ebf6c7248072cf2f702ba4`.
+ The only changes allowed after measurement are this benchmark report and
+ release metadata. Neither changes retrieval, capture, model weights, or
+ ranking.
+- dataset: `longmemeval_s`, sha256
+ `08d8dad4be43ee2049a22ff5674eb86725d0ce5ff434cde2627e5e8e7e117894`.
+- hardware: AMD Ryzen 9 7950X3D (32 threads). No build or test workload ran
+ concurrently with either measurement.
+- commands, after building `ai-memory` and `ai-memory-eval` in release mode:
+
+ ```bash
+ ai-memory-eval retrieval \
+ --server-bin target/release/ai-memory \
+ --concurrency 6 --http-timeout-secs 900 \
+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true
+
+ ai-memory-eval retrieval \
+ --server-bin target/release/ai-memory \
+ --concurrency 6 --http-timeout-secs 900 \
+ --embeddings local \
+ --server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true
+ ```
+
+- run dirs: `evals/runs/2026-10-20T16-31-20-838890843Z-retrieval` (FTS)
+ and `evals/runs/2026-10-20T17-35-22-137201437Z-retrieval` (local).
+
+## Results
+
+FTS accuracy is identical to 3.1 at every overall cutoff. Local embeddings
+improved by three questions at `hit@5`, from 0.940 to 0.947. Query latency and
+returned context remained stable.
+
+| mode | hit@1 | hit@5 | hit@10 | p50 | p95 | mean context tokens |
+|---|---:|---:|---:|---:|---:|---:|
+| FTS 3.1 | 0.857 | 0.951 | 0.972 | 12 ms | 101 ms | 528 |
+| **FTS 3.1.2** | **0.857** | **0.951** | **0.972** | **12 ms** | **103 ms** | **528** |
+| local 3.1 | 0.755 | 0.940 | 0.972 | 99 ms | 225 ms | 528 |
+| **local 3.1.2** | **0.768** | **0.947** | **0.977** | **98 ms** | **225 ms** | **528** |
+
+The local `hit@5` gain is +0.007, just beyond the pre-registered +/-0.005
+run-to-run band and in the favorable direction. Its `hit@1` returned to the
+3.0.1 value of 0.768. FTS p95 changed by 2 ms. These measurements do not show
+a quality, latency, or context-size regression.
+
+Per-slice `hit@5`:
+
+| slice | n | FTS 3.1 | FTS 3.1.2 | local 3.1 | local 3.1.2 |
+|---|---:|---:|---:|---:|---:|
+| knowledge-update | 72 | 1.000 | 1.000 | 1.000 | 1.000 |
+| multi-session | 121 | 0.967 | 0.967 | 0.975 | 0.975 |
+| temporal-reasoning | 127 | 0.921 | 0.921 | 0.898 | 0.921 |
+| single-session-assistant | 56 | 1.000 | 1.000 | 0.893 | 0.893 |
+| single-session-preference | 30 | 0.700 | 0.700 | 0.900 | 0.900 |
+| single-session-user | 64 | 1.000 | 1.000 | 0.953 | 0.953 |
+
+All FTS slices are exact. Five local slices are also exact; temporal reasoning
+gained three questions, moving from 0.898 to 0.921. The +0.023 change is
+within the pre-registered +/-0.02 to 0.03 slice band and agrees with the
+overall result.
+
+## Ingest
+
+| mode | items | requests | throughput | wall clock | fallback answers |
+|---|---:|---:|---:|---:|---:|
+| FTS | 278,946 | 1,403 | 12 items/s per question stream | 3,837.0 s | 0/470 |
+| local | 278,946 | 1,403 | 10 items/s per question stream | 4,899.8 s | 0/470 |
+
+No document exceeded the 16 KiB capture bound. The raw-observation fallback
+was not needed, so both results measure indexed retrieval.