Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs: publish the final observation-stream numbers and the harness correction

AkitaOnRails94a80d5d086c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index dbb065d1..4d2be259 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -362,7 +362,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Published `docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md`, which
splits the LongMemEval-S retrieval number into capture cost, ranking quality
and harness artifacts, and records what the observation stream recovered
- (zero-LLM FTS `hit@5` 0.662 → 0.855 on all 470 questions). The benchmark,
+ (zero-LLM FTS `hit@5` 0.662 → 0.951 and local embeddings 0.796 → 0.936 on
+ all 470 questions). The benchmark,
evals and comparison docs no longer blame the 2 KB excerpt cap for the gap
(pages holding exactly the captured text score 0.968 against 0.972 for whole
sessions) and no longer attribute agentmemory's 0.967 to LongMemEval-S: it is
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 427aee8a..145e1885 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -1429,9 +1429,9 @@ the server URL is loopback and no `default` server profile is registered.
[`docs/benchmarks/`](benchmarks/README.md)), next to the recall-eval
framework
([`crates/ai-memory-consolidate/tests/suite/recall_eval.rs`](../crates/ai-memory-consolidate/tests/suite/recall_eval.rs)).
- Still unmeasured: a full-set local-embeddings run with the observation
- stream, and `belief_authority_weight`, which needs an LLM-consolidated
- corpus because zero-LLM session pages carry no `page_evidence`.
+ Still unmeasured: `belief_authority_weight`, which needs an
+ LLM-consolidated corpus because zero-LLM session pages carry no
+ `page_evidence`.
## Reading order
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md
index cd198eb6..4f6981df 100644
--- a/docs/benchmarks/README.md
+++ b/docs/benchmarks/README.md
@@ -13,24 +13,25 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch
| date | dataset | mode | overall hit@5 | file |
|---|---|---|---|---|
-| **2026-10-13** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (2.7, unreleased; on by default)** | **0.855** | [longmemeval-s-decomposition-2026-10-12.md](longmemeval-s-decomposition-2026-10-12.md) |
-| **2026-09-21** | LongMemEval-S (v1) | **local embeddings (2.4 RC, current)** | **0.815** | [retrieval-ab-r2.md](retrieval-ab-r2.md) |
+| **2026-10-13** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (2.7, unreleased; on by default)** | **0.951** | [longmemeval-s-decomposition-2026-10-12.md](longmemeval-s-decomposition-2026-10-12.md) |
+| **2026-10-13** | LongMemEval-S (v1) | **local embeddings + observation stream (2.7, unreleased; on by default)** | **0.936** | [longmemeval-s-decomposition-2026-10-12.md](longmemeval-s-decomposition-2026-10-12.md) |
+| 2026-09-21 | LongMemEval-S (v1) | local embeddings (2.4 RC, current release) | 0.815 | [retrieval-ab-r2.md](retrieval-ab-r2.md) |
| 2026-09-21 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (2.4 RC) | 0.666 | [retrieval-ab-r2.md](retrieval-ab-r2.md) |
| 2026-09-01 | LongMemEval-S (v1) | local embeddings (2.0 default, prior) | 0.823 | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) |
| 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) |
| 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |
-The 2026-10-13 row is the 2.7 observation stream measured on all 470
-questions, stream off against stream on on the same build: zero-LLM FTS
-`hit@5` 0.662 → 0.855 (hit@1 0.511 → 0.611, hit@10 0.687 → 0.934; 91 questions
-gained, none lost). Local embeddings with the stream went 0.800 → 0.867 on a
-90-question stratified sample; a full-set local run with the stream has not
-been done yet, so 0.815 stays the latest full-set local number. Context
-tokens for the top five hits rose 7–18%, and p95 latency rose 13 ms on the
-full set (84 → 97 ms). These numbers come from the harness corrected on
-2026-10-13, which replays each question's sessions under their own session
-ids; the first published figures (0.666 → 0.868) came from a harness that
-reused them across questions and left a few sessions without a page. Details and commands are in the
+The 2026-10-13 rows are the 2.7 observation stream measured on all 470
+questions, comparing the stream off and on with the same build. Zero-LLM FTS
+`hit@5` went 0.662 → 0.951 (hit@1 0.511 → 0.857, hit@10 0.687 → 0.972; 139
+questions gained, 3 lost) and local embeddings 0.796 → 0.936 (hit@1 0.536 →
+0.757; 72 gained, 6 lost). Context tokens for the top five hits rose 3.8%
+(FTS) and 17.0% (local). Against the last published release numbers (2.4/2.6:
+0.666 FTS, 0.815 local), that is +0.285 and +0.121. These numbers come from
+the harness corrected on 2026-10-13, which replays each question's sessions
+under their own session ids; the first published stream figures (0.666 →
+0.868) came from a harness that reused them across questions and left a few
+sessions without a page. Details and commands are in the
decomposition report below.
The 2026-09-21 (2.4 RC) re-run shows no default-ranking regression: local
@@ -59,7 +60,7 @@ holding exactly the text capture keeps score 0.968. It was the zero-LLM
session page, which indexed only 80-character turn titles and no assistant
text. The harness's per-turn date prefix used about half of each title, which
is an artifact of the harness, not of production capture. The observation
-stream recovered about two thirds of that FTS gap.
+stream recovered nearly all of that FTS gap (0.662 → 0.951).
## A/B comparisons (R2)
diff --git a/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md b/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md
index c8d9df69..1f25b20a 100644
--- a/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md
+++ b/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md
@@ -44,11 +44,13 @@ pipeline-vs-whole-session A/B, on `release/2.7` at `85eb1723` with
`AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false` on every arm (see "Harness
correction").
-Observation-stream runs (R1, re-run 2026-10-13 on the corrected harness):
+Observation-stream runs (2026-10-13, corrected harness):
-- build: `release/2.7` at `85eb1723`, where the stream is on by default; the
- baseline side sets `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false` and the
- candidate side `=true`
+- build: the shipped ranking, `228d72f0` on top of `e124b123`, for the
+ full-set FTS and local A/Bs; `release/2.7` at `85eb1723` for the first
+ version. The stream is on by default in both, so the baseline side sets
+ `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false` and the candidate side
+ `=true`
- harness: `85eb1723`, which replays each question's sessions under their own
session ids (see "Harness correction")
- dataset, hardware and model as above (original `longmemeval_s` file)
@@ -188,13 +190,53 @@ The fix keeps the session page as it is and adds the observation bodies to
ranking instead. With `[retrieval] observation_stream` on, project and
`scopes` searches also run full-text search over the project's observations,
and each matching session votes once, through its best observation, for its
-own `sessions/<id>.md` page as one more RRF stream. Hits are still pages, and
-nothing new is written to the wiki. It is on by default in 2.7; setting it to
-`false` restores the previous ranking exactly.
+own `sessions/<id>.md` page as one more RRF stream. A page the stream voted
+for takes its lexical evidence from the stream alone: page FTS over its
+title-only body still reports a rank but adds nothing, so a session that only
+shares topic words in its titles cannot outvote the session holding the
+answer. Hits are still pages, and nothing new is written to the wiki. It is on
+by default in 2.7; setting it to `false` restores the previous ranking
+exactly.
+
+Stream off → on, same build (`release/2.7` at `228d72f0` on top of
+`e124b123`, the shipped ranking), same questions, all 470, corrected harness:
+
+| run | hit@5 | hit@1 | hit@10 | recall@5 | p95 ms | flips |
+|---|---|---|---|---|---|---|
+| FTS | 0.662 → **0.951** | 0.511 → 0.857 | 0.687 → 0.972 | 0.533 → 0.893 | 90 → 112 | +139/−3 |
+| Local embeddings | 0.796 → **0.936** | 0.536 → 0.757 | 0.881 → 0.970 | 0.657 → 0.871 | 264 → 234 | +72/−6 |
+
+FTS `hit@5` per slice, off → on: knowledge-update 0.764 → 1.000,
+multi-session 0.562 → 0.967, single-session-assistant 0.804 → 1.000,
+single-session-preference 0.533 → 0.700, single-session-user 0.688 → 1.000,
+temporal-reasoning 0.654 → 0.921. The FTS and local runs ran at the same time
+on one host, so their latencies are only indicative.
+
+Context tokens rise because the candidate returns more hits: with the stream
+off, FTS returned fewer than five hits for 219 of 470 questions (5.6 on
+average), and with it on, 9.9. The cost of the top five hits rises much less:
-Baseline is the stream off, candidate is the stream on, same build, same
-questions. These are the corrected-harness runs (see "Harness correction"
-below):
+| run | mean context tokens, all hits | top five hits only |
+|---|---|---|
+| FTS | 366.7 → 528.1 | 256.3 → 266.1 (+3.8%) |
+| Local embeddings | 451.4 → 528.3 | 225.7 → 264.1 (+17.0%) |
+
+"Top five hits only" scales each question's estimate by `min(returned, 5) /
+returned`.
+
+The pipeline now ranks an evidence session into the top five for 95.1% of
+questions with FTS alone, against 96.8% for pages holding exactly the captured
+text and 97.2% for whole raw sessions. With the stream on, local embeddings
+score slightly below FTS alone (0.936 against 0.951), mostly on
+single-session-assistant questions (0.875 against 1.000): the session page's
+vector, like its FTS row, sees only the title-only body.
+
+### First version
+
+In its first version the stream's vote was added on top of page FTS, so a
+session page whose title list shared topic words with the question still
+collected both. That version, stream off → on, corrected harness, on
+`release/2.7` at `85eb1723`:
| run | questions | hit@5 | hit@1 | hit@10 | recall@5 | p95 ms | flips |
|---|---|---|---|---|---|---|---|
@@ -206,28 +248,14 @@ FTS `hit@5` per slice on the full set, off → on: knowledge-update 0.764 →
0.944, multi-session 0.562 → 0.868, single-session-assistant 0.804 → 0.839,
single-session-preference 0.533 → 0.667, single-session-user 0.688 → 0.875,
temporal-reasoning 0.654 → 0.835. No slice dropped, and no question lost its
-top-5 hit. The three runs shared the host one after another, never with each
-other. An earlier build whose observation scan was not yet bounded to the
+top-5 hit. The three runs ran one after another on one host. An earlier build whose observation scan was not yet bounded to the
project measured a candidate p95 of 1,373 ms; the bounded scan is what
shipped.
-Context tokens rise. The candidate returns more hits: with the stream off,
-FTS returned fewer than five hits for 219 of 470 questions (5.6 hits on
-average), and with it on, 9.9. So the raw mean rises more than the cost of the
-top five hits does:
-
-| run | mean context tokens, all hits | top five hits only |
-|---|---|---|
-| FTS, 470 questions | 366.7 → 548.8 | 256.3 → 276.4 (+7.9%) |
-| FTS, 180 questions | 414.9 → 565.7 | 266.7 → 285.4 (+7.0%) |
-| Local, 90 questions | 476.6 → 563.2 | 238.3 → 281.6 (+18.2%) |
-
-"Top five hits only" scales each question's estimate by `min(returned, 5) /
-returned`.
-
-The stream recovered about two thirds of the FTS headroom the decomposition
-found (0.662 → 0.855, against 0.968 for captured-text pages). A full-set
-local run with the stream on has not been done yet.
+Its top-five context cost rose 7.9% (FTS, 470), 7.0% (FTS, 180) and 18.2%
+(local, 90). Letting the stream alone rank a page it voted for, the shipped
+version above, took FTS `hit@5` from 0.855 to 0.951 and lowered the top-five
+token cost.
## Harness correction (2026-10-13)
@@ -254,8 +282,8 @@ stream, which had partly been scoring sessions whose page was missing:
| Local pipeline, strat-230 | 0.809 / 0.804 | 0.787 |
| FTS date-prefix ablation, strat-120 | 0.658 → 0.792 | 0.658 → 0.792 |
| Local date-prefix ablation, strat-120 | 0.800 → 0.900 | 0.800 → 0.892 |
-| FTS stream off → on, 470 | 0.666 → 0.868 | 0.662 → 0.855 |
-| Local stream off → on, 90 | 0.800 → 0.867 | 0.800 → 0.867 |
+| FTS stream off → on, 470 (first version) | 0.666 → 0.868 | 0.662 → 0.855 |
+| Local stream off → on, 90 (first version) | 0.800 → 0.867 | 0.800 → 0.867 |
Component, grep and captured-session runs write pages directly or read the
dataset, never replaying session ids through hooks, so they are unchanged.
@@ -336,16 +364,16 @@ $E retrieval --stratify 20 --embeddings local --date-prefix turn --candidate-dat
$E retrieval --stratify 20 --embeddings local --candidate-server-env AI_MEMORY_RETRIEVAL__BELIEF_AUTHORITY_WEIGHT=0.3
```
-Observation stream (corrected harness, stream on by default):
+Observation stream, shipped ranking (corrected harness, stream on by
+default): the full-set FTS and local A/Bs. The first version's runs used the
+same commands on `85eb1723`, plus `--stratify 30` (FTS) and `--embeddings
+local --stratify 15`:
```bash
B=target/release
AB="--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false
--candidate-server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true"
-# full set, then FTS on 180 and local on 90
$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 900 $AB
$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 900 \
- --stratify 30 $AB
-$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 900 \
- --embeddings local --stratify 15 $AB
+ --embeddings local $AB
```
diff --git a/docs/benchmarks/retrieval-ab-r2.md b/docs/benchmarks/retrieval-ab-r2.md
index 1e37bce4..caa48748 100644
--- a/docs/benchmarks/retrieval-ab-r2.md
+++ b/docs/benchmarks/retrieval-ab-r2.md
@@ -102,9 +102,9 @@ cargo run --release -p ai-memory-eval -- retrieval --candidate
The full-set A/B for the 2.7 observation stream (stream off → on, zero-LLM
FTS, 470 scored) is published with its provenance in
[`longmemeval-s-decomposition-2026-10-12.md`](longmemeval-s-decomposition-2026-10-12.md#what-the-observation-stream-recovered):
-`hit@5` 0.662 → 0.855, hit@1 0.511 → 0.611, hit@10 0.687 → 0.934, 91 questions
-gained and none lost (corrected harness, 2026-10-13). The local-embeddings A/B ran on a 90-question stratified
-sample (0.800 → 0.867). The stream is on by default in 2.7, so a baseline run
+`hit@5` 0.662 → 0.951, hit@1 0.511 → 0.857, hit@10 0.687 → 0.972, 139
+questions gained and 3 lost; local embeddings 0.796 → 0.936 on the full set
+(corrected harness, 2026-10-13). The stream is on by default in 2.7, so a baseline run
of the 2.4 configuration on a 2.7 build needs
`--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false`.
diff --git a/docs/comparison.md b/docs/comparison.md
index 46300fe2..f971db1a 100644
--- a/docs/comparison.md
+++ b/docs/comparison.md
@@ -42,9 +42,10 @@ What sets it apart:
It also publishes numbers, produced by the in-repo harness with full
provenance and measured end to end through the shipped capture pipeline:
**LongMemEval-S hit@5 about 0.82** with the default local embeddings (0.823 and
-0.815 in the two published full-set runs) and 0.666 zero-LLM. The observation stream in 2.7
-(unreleased, on by default) raises zero-LLM from 0.662 to **0.855** on all 470
-questions, measured on one build with the stream off and on.
+0.815 in the two published full-set runs) and 0.666 zero-LLM in 2.6. The
+observation stream in 2.7 (unreleased, on by default) raises zero-LLM to
+**0.951** and local embeddings to **0.936** on all 470 questions, measured on
+one build with the stream off and on.
With capture skipped and each raw session indexed whole (component mode,
ranking only), zero-LLM FTS scores 0.972; the
[decomposition report](benchmarks/longmemeval-s-decomposition-2026-10-12.md)
@@ -296,7 +297,7 @@ operator turns it on.
- **Retrieval through capture, and no local reranker.** Through the shipped
pipeline, LongMemEval-S hit@5 is about 0.82 with local embeddings and 0.666
- zero-LLM (2.6); the 2.7 observation stream raises zero-LLM to 0.855. Ranking
+ zero-LLM (2.6); the 2.7 observation stream raises them to 0.936 and 0.951. Ranking
the same raw sessions whole scores 0.972 zero-LLM. The
[decomposition](benchmarks/longmemeval-s-decomposition-2026-10-12.md) traced
the gap to capture, and within capture not to the 2 KB privacy cap (pages holding exactly the captured text
diff --git a/docs/competitive-parity.md b/docs/competitive-parity.md
index df8f2bd1..d38b7770 100644
--- a/docs/competitive-parity.md
+++ b/docs/competitive-parity.md
@@ -43,7 +43,7 @@ None of the competitors below has more than one or two of these; none has all.
- **Multi-user shared-within-project** ("pages shared, batons owned", invariant
#16), self-hosted, with no paid tier.
- **Honest, reproducible benchmarks**: LongMemEval-S hit@5 about 0.82 local and
- 0.666 zero-LLM through the shipped pipeline, 0.855 zero-LLM with the 2.7
+ 0.666 zero-LLM through the shipped pipeline, 0.951 zero-LLM and 0.936 local with the 2.7
observation stream, and a published pipeline-vs-component decomposition
([`benchmarks/`](benchmarks/README.md)).
@@ -119,7 +119,7 @@ implemented without a separate decision.** Ordered by leverage.
([`benchmarks/longmemeval-s-decomposition-2026-10-12.md`](benchmarks/longmemeval-s-decomposition-2026-10-12.md))
split the pipeline number into capture cost, ranking and harness
artifacts, which led to the 2.7 observation stream (zero-LLM FTS hit@5
- 0.662 → 0.855). What is still unrun is the dormant substrate below: the
+ 0.662 → 0.951, local 0.796 → 0.936). What is still unrun is the dormant substrate below: the
belief-strength weight cannot move a zero-LLM corpus (every page's
confidence is 0.0 without consolidation evidence, so the measured delta
was 0.000 FTS), so it needs an LLM-consolidated corpus; typed-edge rank
@@ -258,10 +258,11 @@ implemented without a separate decision.** Ordered by leverage.
*self-inflicted, not copied*: the decomposition showed the zero-LLM session
page indexed only 80-character turn titles and no assistant text, while
ranking whole sessions scored 0.972. The 2.7 observation stream, on by
- default after a measured R2 delta, recovered about two thirds
- of that (0.662 → 0.855 zero-LLM FTS on the full set; 0.800 → 0.867 local on
- a 90-question sample). Still open: no local zero-LLM reranker, a full-set
- local run with the stream, belief-strength (`page_evidence`) wired but
+ default after a measured R2 delta, recovered nearly all of
+ that (0.662 → 0.951 zero-LLM FTS and 0.796 → 0.936 local on the full set).
+ Still open: no local zero-LLM reranker, local embeddings scoring slightly
+ below FTS with the stream on (the session page's vector sees only its
+ title-only body), belief-strength (`page_evidence`) wired but
**default-off** and unmeasurable without an LLM-consolidated corpus, and
typed-edge **RANK** weighting not built.
- **Migration answer:** for the self-hosted, multi-harness, team, zero-LLM,