Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs: publish the final observation-stream numbers and the harness correction
AkitaOnRails94a80d5d086c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex dbb065d1..4d2be259 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -362,7 +362,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Published `docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md`, whichsplits the LongMemEval-S retrieval number into capture cost, ranking qualityand harness artifacts, and records what the observation stream recovered- (zero-LLM FTS `hit@5` 0.662 → 0.855 on all 470 questions). The benchmark,+ (zero-LLM FTS `hit@5` 0.662 → 0.951 and local embeddings 0.796 → 0.936 on+ all 470 questions). The benchmark,evals and comparison docs no longer blame the 2 KB excerpt cap for the gap(pages holding exactly the captured text score 0.968 against 0.972 for wholesessions) and no longer attribute agentmemory's 0.967 to LongMemEval-S: it isdiff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 427aee8a..145e1885 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -1429,9 +1429,9 @@ the server URL is loopback and no `default` server profile is registered.[`docs/benchmarks/`](benchmarks/README.md)), next to the recall-evalframework([`crates/ai-memory-consolidate/tests/suite/recall_eval.rs`](../crates/ai-memory-consolidate/tests/suite/recall_eval.rs)).- Still unmeasured: a full-set local-embeddings run with the observation- stream, and `belief_authority_weight`, which needs an LLM-consolidated- corpus because zero-LLM session pages carry no `page_evidence`.+ Still unmeasured: `belief_authority_weight`, which needs an+ LLM-consolidated corpus because zero-LLM session pages carry no+ `page_evidence`.## Reading orderdiff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.mdindex cd198eb6..4f6981df 100644--- a/docs/benchmarks/README.md+++ b/docs/benchmarks/README.md@@ -13,24 +13,25 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch| date | dataset | mode | overall hit@5 | file ||---|---|---|---|---|-| **2026-10-13** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (2.7, unreleased; on by default)** | **0.855** | [longmemeval-s-decomposition-2026-10-12.md](longmemeval-s-decomposition-2026-10-12.md) |-| **2026-09-21** | LongMemEval-S (v1) | **local embeddings (2.4 RC, current)** | **0.815** | [retrieval-ab-r2.md](retrieval-ab-r2.md) |+| **2026-10-13** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (2.7, unreleased; on by default)** | **0.951** | [longmemeval-s-decomposition-2026-10-12.md](longmemeval-s-decomposition-2026-10-12.md) |+| **2026-10-13** | LongMemEval-S (v1) | **local embeddings + observation stream (2.7, unreleased; on by default)** | **0.936** | [longmemeval-s-decomposition-2026-10-12.md](longmemeval-s-decomposition-2026-10-12.md) |+| 2026-09-21 | LongMemEval-S (v1) | local embeddings (2.4 RC, current release) | 0.815 | [retrieval-ab-r2.md](retrieval-ab-r2.md) || 2026-09-21 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (2.4 RC) | 0.666 | [retrieval-ab-r2.md](retrieval-ab-r2.md) || 2026-09-01 | LongMemEval-S (v1) | local embeddings (2.0 default, prior) | 0.823 | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) || 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) || 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |-The 2026-10-13 row is the 2.7 observation stream measured on all 470-questions, stream off against stream on on the same build: zero-LLM FTS-`hit@5` 0.662 → 0.855 (hit@1 0.511 → 0.611, hit@10 0.687 → 0.934; 91 questions-gained, none lost). Local embeddings with the stream went 0.800 → 0.867 on a-90-question stratified sample; a full-set local run with the stream has not-been done yet, so 0.815 stays the latest full-set local number. Context-tokens for the top five hits rose 7–18%, and p95 latency rose 13 ms on the-full set (84 → 97 ms). These numbers come from the harness corrected on-2026-10-13, which replays each question's sessions under their own session-ids; the first published figures (0.666 → 0.868) came from a harness that-reused them across questions and left a few sessions without a page. Details and commands are in the+The 2026-10-13 rows are the 2.7 observation stream measured on all 470+questions, comparing the stream off and on with the same build. Zero-LLM FTS+`hit@5` went 0.662 → 0.951 (hit@1 0.511 → 0.857, hit@10 0.687 → 0.972; 139+questions gained, 3 lost) and local embeddings 0.796 → 0.936 (hit@1 0.536 →+0.757; 72 gained, 6 lost). Context tokens for the top five hits rose 3.8%+(FTS) and 17.0% (local). Against the last published release numbers (2.4/2.6:+0.666 FTS, 0.815 local), that is +0.285 and +0.121. These numbers come from+the harness corrected on 2026-10-13, which replays each question's sessions+under their own session ids; the first published stream figures (0.666 →+0.868) came from a harness that reused them across questions and left a few+sessions without a page. Details and commands are in thedecomposition report below.The 2026-09-21 (2.4 RC) re-run shows no default-ranking regression: local@@ -59,7 +60,7 @@ holding exactly the text capture keeps score 0.968. It was the zero-LLMsession page, which indexed only 80-character turn titles and no assistanttext. The harness's per-turn date prefix used about half of each title, whichis an artifact of the harness, not of production capture. The observation-stream recovered about two thirds of that FTS gap.+stream recovered nearly all of that FTS gap (0.662 → 0.951).## A/B comparisons (R2)diff --git a/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md b/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.mdindex c8d9df69..1f25b20a 100644--- a/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md+++ b/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md@@ -44,11 +44,13 @@ pipeline-vs-whole-session A/B, on `release/2.7` at `85eb1723` with`AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false` on every arm (see "Harnesscorrection").-Observation-stream runs (R1, re-run 2026-10-13 on the corrected harness):+Observation-stream runs (2026-10-13, corrected harness):-- build: `release/2.7` at `85eb1723`, where the stream is on by default; the- baseline side sets `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false` and the- candidate side `=true`+- build: the shipped ranking, `228d72f0` on top of `e124b123`, for the+ full-set FTS and local A/Bs; `release/2.7` at `85eb1723` for the first+ version. The stream is on by default in both, so the baseline side sets+ `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false` and the candidate side+ `=true`- harness: `85eb1723`, which replays each question's sessions under their ownsession ids (see "Harness correction")- dataset, hardware and model as above (original `longmemeval_s` file)@@ -188,13 +190,53 @@ The fix keeps the session page as it is and adds the observation bodies toranking instead. With `[retrieval] observation_stream` on, project and`scopes` searches also run full-text search over the project's observations,and each matching session votes once, through its best observation, for its-own `sessions/<id>.md` page as one more RRF stream. Hits are still pages, and-nothing new is written to the wiki. It is on by default in 2.7; setting it to-`false` restores the previous ranking exactly.+own `sessions/<id>.md` page as one more RRF stream. A page the stream voted+for takes its lexical evidence from the stream alone: page FTS over its+title-only body still reports a rank but adds nothing, so a session that only+shares topic words in its titles cannot outvote the session holding the+answer. Hits are still pages, and nothing new is written to the wiki. It is on+by default in 2.7; setting it to `false` restores the previous ranking+exactly.++Stream off → on, same build (`release/2.7` at `228d72f0` on top of+`e124b123`, the shipped ranking), same questions, all 470, corrected harness:++| run | hit@5 | hit@1 | hit@10 | recall@5 | p95 ms | flips |+|---|---|---|---|---|---|---|+| FTS | 0.662 → **0.951** | 0.511 → 0.857 | 0.687 → 0.972 | 0.533 → 0.893 | 90 → 112 | +139/−3 |+| Local embeddings | 0.796 → **0.936** | 0.536 → 0.757 | 0.881 → 0.970 | 0.657 → 0.871 | 264 → 234 | +72/−6 |++FTS `hit@5` per slice, off → on: knowledge-update 0.764 → 1.000,+multi-session 0.562 → 0.967, single-session-assistant 0.804 → 1.000,+single-session-preference 0.533 → 0.700, single-session-user 0.688 → 1.000,+temporal-reasoning 0.654 → 0.921. The FTS and local runs ran at the same time+on one host, so their latencies are only indicative.++Context tokens rise because the candidate returns more hits: with the stream+off, FTS returned fewer than five hits for 219 of 470 questions (5.6 on+average), and with it on, 9.9. The cost of the top five hits rises much less:-Baseline is the stream off, candidate is the stream on, same build, same-questions. These are the corrected-harness runs (see "Harness correction"-below):+| run | mean context tokens, all hits | top five hits only |+|---|---|---|+| FTS | 366.7 → 528.1 | 256.3 → 266.1 (+3.8%) |+| Local embeddings | 451.4 → 528.3 | 225.7 → 264.1 (+17.0%) |++"Top five hits only" scales each question's estimate by `min(returned, 5) /+returned`.++The pipeline now ranks an evidence session into the top five for 95.1% of+questions with FTS alone, against 96.8% for pages holding exactly the captured+text and 97.2% for whole raw sessions. With the stream on, local embeddings+score slightly below FTS alone (0.936 against 0.951), mostly on+single-session-assistant questions (0.875 against 1.000): the session page's+vector, like its FTS row, sees only the title-only body.++### First version++In its first version the stream's vote was added on top of page FTS, so a+session page whose title list shared topic words with the question still+collected both. That version, stream off → on, corrected harness, on+`release/2.7` at `85eb1723`:| run | questions | hit@5 | hit@1 | hit@10 | recall@5 | p95 ms | flips ||---|---|---|---|---|---|---|---|@@ -206,28 +248,14 @@ FTS `hit@5` per slice on the full set, off → on: knowledge-update 0.764 →0.944, multi-session 0.562 → 0.868, single-session-assistant 0.804 → 0.839,single-session-preference 0.533 → 0.667, single-session-user 0.688 → 0.875,temporal-reasoning 0.654 → 0.835. No slice dropped, and no question lost its-top-5 hit. The three runs shared the host one after another, never with each-other. An earlier build whose observation scan was not yet bounded to the+top-5 hit. The three runs ran one after another on one host. An earlier build whose observation scan was not yet bounded to theproject measured a candidate p95 of 1,373 ms; the bounded scan is whatshipped.-Context tokens rise. The candidate returns more hits: with the stream off,-FTS returned fewer than five hits for 219 of 470 questions (5.6 hits on-average), and with it on, 9.9. So the raw mean rises more than the cost of the-top five hits does:--| run | mean context tokens, all hits | top five hits only |-|---|---|---|-| FTS, 470 questions | 366.7 → 548.8 | 256.3 → 276.4 (+7.9%) |-| FTS, 180 questions | 414.9 → 565.7 | 266.7 → 285.4 (+7.0%) |-| Local, 90 questions | 476.6 → 563.2 | 238.3 → 281.6 (+18.2%) |--"Top five hits only" scales each question's estimate by `min(returned, 5) /-returned`.--The stream recovered about two thirds of the FTS headroom the decomposition-found (0.662 → 0.855, against 0.968 for captured-text pages). A full-set-local run with the stream on has not been done yet.+Its top-five context cost rose 7.9% (FTS, 470), 7.0% (FTS, 180) and 18.2%+(local, 90). Letting the stream alone rank a page it voted for, the shipped+version above, took FTS `hit@5` from 0.855 to 0.951 and lowered the top-five+token cost.## Harness correction (2026-10-13)@@ -254,8 +282,8 @@ stream, which had partly been scoring sessions whose page was missing:| Local pipeline, strat-230 | 0.809 / 0.804 | 0.787 || FTS date-prefix ablation, strat-120 | 0.658 → 0.792 | 0.658 → 0.792 || Local date-prefix ablation, strat-120 | 0.800 → 0.900 | 0.800 → 0.892 |-| FTS stream off → on, 470 | 0.666 → 0.868 | 0.662 → 0.855 |-| Local stream off → on, 90 | 0.800 → 0.867 | 0.800 → 0.867 |+| FTS stream off → on, 470 (first version) | 0.666 → 0.868 | 0.662 → 0.855 |+| Local stream off → on, 90 (first version) | 0.800 → 0.867 | 0.800 → 0.867 |Component, grep and captured-session runs write pages directly or read thedataset, never replaying session ids through hooks, so they are unchanged.@@ -336,16 +364,16 @@ $E retrieval --stratify 20 --embeddings local --date-prefix turn --candidate-dat$E retrieval --stratify 20 --embeddings local --candidate-server-env AI_MEMORY_RETRIEVAL__BELIEF_AUTHORITY_WEIGHT=0.3```-Observation stream (corrected harness, stream on by default):+Observation stream, shipped ranking (corrected harness, stream on by+default): the full-set FTS and local A/Bs. The first version's runs used the+same commands on `85eb1723`, plus `--stratify 30` (FTS) and `--embeddings+local --stratify 15`:```bashB=target/releaseAB="--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false--candidate-server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true"-# full set, then FTS on 180 and local on 90$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 900 $AB$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 900 \- --stratify 30 $AB-$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 900 \- --embeddings local --stratify 15 $AB+ --embeddings local $AB```diff --git a/docs/benchmarks/retrieval-ab-r2.md b/docs/benchmarks/retrieval-ab-r2.mdindex 1e37bce4..caa48748 100644--- a/docs/benchmarks/retrieval-ab-r2.md+++ b/docs/benchmarks/retrieval-ab-r2.md@@ -102,9 +102,9 @@ cargo run --release -p ai-memory-eval -- retrieval --candidateThe full-set A/B for the 2.7 observation stream (stream off → on, zero-LLMFTS, 470 scored) is published with its provenance in[`longmemeval-s-decomposition-2026-10-12.md`](longmemeval-s-decomposition-2026-10-12.md#what-the-observation-stream-recovered):-`hit@5` 0.662 → 0.855, hit@1 0.511 → 0.611, hit@10 0.687 → 0.934, 91 questions-gained and none lost (corrected harness, 2026-10-13). The local-embeddings A/B ran on a 90-question stratified-sample (0.800 → 0.867). The stream is on by default in 2.7, so a baseline run+`hit@5` 0.662 → 0.951, hit@1 0.511 → 0.857, hit@10 0.687 → 0.972, 139+questions gained and 3 lost; local embeddings 0.796 → 0.936 on the full set+(corrected harness, 2026-10-13). The stream is on by default in 2.7, so a baseline runof the 2.4 configuration on a 2.7 build needs`--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false`.diff --git a/docs/comparison.md b/docs/comparison.mdindex 46300fe2..f971db1a 100644--- a/docs/comparison.md+++ b/docs/comparison.md@@ -42,9 +42,10 @@ What sets it apart:It also publishes numbers, produced by the in-repo harness with fullprovenance and measured end to end through the shipped capture pipeline:**LongMemEval-S hit@5 about 0.82** with the default local embeddings (0.823 and-0.815 in the two published full-set runs) and 0.666 zero-LLM. The observation stream in 2.7-(unreleased, on by default) raises zero-LLM from 0.662 to **0.855** on all 470-questions, measured on one build with the stream off and on.+0.815 in the two published full-set runs) and 0.666 zero-LLM in 2.6. The+observation stream in 2.7 (unreleased, on by default) raises zero-LLM to+**0.951** and local embeddings to **0.936** on all 470 questions, measured on+one build with the stream off and on.With capture skipped and each raw session indexed whole (component mode,ranking only), zero-LLM FTS scores 0.972; the[decomposition report](benchmarks/longmemeval-s-decomposition-2026-10-12.md)@@ -296,7 +297,7 @@ operator turns it on.- **Retrieval through capture, and no local reranker.** Through the shippedpipeline, LongMemEval-S hit@5 is about 0.82 with local embeddings and 0.666- zero-LLM (2.6); the 2.7 observation stream raises zero-LLM to 0.855. Ranking+ zero-LLM (2.6); the 2.7 observation stream raises them to 0.936 and 0.951. Rankingthe same raw sessions whole scores 0.972 zero-LLM. The[decomposition](benchmarks/longmemeval-s-decomposition-2026-10-12.md) tracedthe gap to capture, and within capture not to the 2 KB privacy cap (pages holding exactly the captured textdiff --git a/docs/competitive-parity.md b/docs/competitive-parity.mdindex df8f2bd1..d38b7770 100644--- a/docs/competitive-parity.md+++ b/docs/competitive-parity.md@@ -43,7 +43,7 @@ None of the competitors below has more than one or two of these; none has all.- **Multi-user shared-within-project** ("pages shared, batons owned", invariant#16), self-hosted, with no paid tier.- **Honest, reproducible benchmarks**: LongMemEval-S hit@5 about 0.82 local and- 0.666 zero-LLM through the shipped pipeline, 0.855 zero-LLM with the 2.7+ 0.666 zero-LLM through the shipped pipeline, 0.951 zero-LLM and 0.936 local with the 2.7observation stream, and a published pipeline-vs-component decomposition([`benchmarks/`](benchmarks/README.md)).@@ -119,7 +119,7 @@ implemented without a separate decision.** Ordered by leverage.([`benchmarks/longmemeval-s-decomposition-2026-10-12.md`](benchmarks/longmemeval-s-decomposition-2026-10-12.md))split the pipeline number into capture cost, ranking and harnessartifacts, which led to the 2.7 observation stream (zero-LLM FTS hit@5- 0.662 → 0.855). What is still unrun is the dormant substrate below: the+ 0.662 → 0.951, local 0.796 → 0.936). What is still unrun is the dormant substrate below: thebelief-strength weight cannot move a zero-LLM corpus (every page'sconfidence is 0.0 without consolidation evidence, so the measured deltawas 0.000 FTS), so it needs an LLM-consolidated corpus; typed-edge rank@@ -258,10 +258,11 @@ implemented without a separate decision.** Ordered by leverage.*self-inflicted, not copied*: the decomposition showed the zero-LLM sessionpage indexed only 80-character turn titles and no assistant text, whileranking whole sessions scored 0.972. The 2.7 observation stream, on by- default after a measured R2 delta, recovered about two thirds- of that (0.662 → 0.855 zero-LLM FTS on the full set; 0.800 → 0.867 local on- a 90-question sample). Still open: no local zero-LLM reranker, a full-set- local run with the stream, belief-strength (`page_evidence`) wired but+ default after a measured R2 delta, recovered nearly all of+ that (0.662 → 0.951 zero-LLM FTS and 0.796 → 0.936 local on the full set).+ Still open: no local zero-LLM reranker, local embeddings scoring slightly+ below FTS with the stream on (the session page's vector sees only its+ title-only body), belief-strength (`page_evidence`) wired but**default-off** and unmeasurable without an LLM-consolidated corpus, andtyped-edge **RANK** weighting not built.- **Migration answer:** for the self-hosted, multi-harness, team, zero-LLM,