Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs: publish the LongMemEval-S decomposition and sync the comparison docs
AkitaOnRails3664238dd80e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b22196b5..83893b95 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -318,6 +318,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0in `docs/support-matrix.md` as `MCP-only (documented)`: each recipe followsthe client's own documentation at a named version and has not yet been runend to end. Aider is not listed because it has no MCP support.+- Published `docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md`, which+ splits the LongMemEval-S retrieval number into capture cost, ranking quality+ and harness artifacts, and records what the observation stream recovered+ (zero-LLM FTS `hit@5` 0.666 → 0.868 on all 470 questions). The benchmark,+ evals and comparison docs no longer blame the 2 KB excerpt cap for the gap+ (pages holding exactly the captured text score 0.968 against 0.972 for whole+ sessions) and no longer attribute agentmemory's 0.967 to LongMemEval-S: it is+ the grep baseline on agentmemory's in-house corpus, and its LongMemEval-S+ figure is 95.2% from a component harness with the reranker disabled, not+ like-for-like with ai-memory's pipeline numbers. `docs/comparison.md` gains a+ "From agentmemory" migration note.- Changed `memory_query` ranking to include the observation stream bydefault, after it raised hit@5 on the retrieval eval for both zero-LLM andlocal-embedding search while staying inside its latency and context-tokendiff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 31f3fda4..7ca03ca5 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -1422,9 +1422,14 @@ the server URL is loopback and no `default` server profile is registered.every row at its path. Better *reading*(graph exploration) is open. See[`docs/frontend-api.md`](frontend-api.md#10-known-gaps-and-deliberate-non-goals).-* **Real LongMemEval-S harness.** The recall-eval framework exists- ([`crates/ai-memory-consolidate/tests/suite/recall_eval.rs`](../crates/ai-memory-consolidate/tests/suite/recall_eval.rs));- porting LongMemEval-S itself requires the dataset.+* **Retrieval measurement gaps.** The LongMemEval-S harness ships in+ `evals/` (pipeline, component and decomposition modes; results in+ [`docs/benchmarks/`](benchmarks/README.md)), next to the recall-eval+ framework+ ([`crates/ai-memory-consolidate/tests/suite/recall_eval.rs`](../crates/ai-memory-consolidate/tests/suite/recall_eval.rs)).+ Still unmeasured: a full-set local-embeddings run with the observation+ stream, and `belief_authority_weight`, which needs an LLM-consolidated+ corpus because zero-LLM session pages carry no `page_evidence`.## Reading orderdiff --git a/docs/ROADMAP-2.0.md b/docs/ROADMAP-2.0.mdindex 17e3409b..104a67b3 100644--- a/docs/ROADMAP-2.0.md+++ b/docs/ROADMAP-2.0.md@@ -46,8 +46,10 @@ audits published numbers (mempalace). We publish none until this exists.HuggingFace (`xiaowu0162/longmemeval-v2`) with an official harness,but its trajectories are WebArena/ServiceNow-style *web-agent*histories, not coding sessions. The numbers competitors publish- (agentmemory 0.967 R@5, mcp-memory-service 0.804/0.860) are against- **v1** chat histories. Therefore: target **v1 (S variant) for+ (agentmemory 95.2% R@5, mcp-memory-service 0.804/0.860) are against+ **v1** chat histories. (Corrected 2026-10-12: this line used to say+ agentmemory 0.967, which is the grep baseline on its in-house+ coding-agent-life-v1 corpus, not a LongMemEval number.) Therefore: target **v1 (S variant) forcross-project comparability**, and adopt **V2's five-ability taxonomy**(static state recall, dynamic state tracking, workflow knowledge,environment gotchas, premise awareness) as the rubric for adiff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.mdindex 719854cc..3150215a 100644--- a/docs/benchmarks/README.md+++ b/docs/benchmarks/README.md@@ -13,12 +13,22 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch| date | dataset | mode | overall hit@5 | file ||---|---|---|---|---|+| **2026-10-12** | LongMemEval-S (v1) | **zero-llm FTS + observation stream (2.7, unreleased; on by default)** | **0.868** | [longmemeval-s-decomposition-2026-10-12.md](longmemeval-s-decomposition-2026-10-12.md) || **2026-09-21** | LongMemEval-S (v1) | **local embeddings (2.4 RC, current)** | **0.815** | [retrieval-ab-r2.md](retrieval-ab-r2.md) || 2026-09-21 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (2.4 RC) | 0.666 | [retrieval-ab-r2.md](retrieval-ab-r2.md) || 2026-09-01 | LongMemEval-S (v1) | local embeddings (2.0 default, prior) | 0.823 | [longmemeval-s-2026-09-01-local.md](longmemeval-s-2026-09-01-local.md) || 2026-09-01 | LongMemEval-S (v1) | zero-llm, stopword-filtered FTS (prior) | 0.668 | [longmemeval-s-2026-09-01-fts.md](longmemeval-s-2026-09-01-fts.md) || 2026-09-01 | LongMemEval-S (v1) | zero-llm (pre-2.0 FTS) | 0.617 | [longmemeval-s-2026-09-01.md](longmemeval-s-2026-09-01.md) |+The 2026-10-12 row is the 2.7 observation stream measured on all 470+questions: zero-LLM FTS `hit@5` 0.666 → 0.868 (hit@1 0.530 → 0.619, hit@10+0.694 → 0.936). Local embeddings with the stream went 0.800 → 0.867 on a+90-question stratified sample; a full-set local run with the stream has not+been done yet, so 0.815 stays the latest full-set local number. Context+tokens for the top five hits rose 7–19% in the final A/B runs, and p95+latency stayed within the stream's budget. Details and commands are in the+decomposition report below.+The 2026-09-21 (2.4 RC) re-run shows no default-ranking regression: local`hit@5` 0.823 → 0.815 and FTS `hit@5` 0.668 → 0.666 are identical withinrun-to-run noise. 2.4's aging/retrieval features are opt-in and off by default,@@ -32,9 +42,20 @@ measured steps: dropping stopwords from bare-query FTS OR-joinscorrect masked-mean pooling. The pooling fix alone was worth ~6.6points over a padded-attention implementation, which thecalibration tests caught. Every intermediate number was measured on this-harness before the next change landed. For context, published-embedding-based numbers on this dataset: agentmemory 0.967 R@5-(hybrid + reranking), doobidoo/mcp-memory-service 0.804 R@5.+harness before the next change landed.++## Decomposition: pipeline vs component++[longmemeval-s-decomposition-2026-10-12.md](longmemeval-s-decomposition-2026-10-12.md)+splits the pipeline number into capture cost, ranking quality and harness+artifacts. With each raw session indexed whole (component mode, ranking+only), zero-LLM FTS scores 0.972 `hit@5` on all 470 questions, against 0.666+through the shipped capture path. The gap was not the 2 KB excerpt cap: pages+holding exactly the text capture keeps score 0.968. It was the zero-LLM+session page, which indexed only 80-character turn titles and no assistant+text. The harness's per-turn date prefix used about half of each title, which+is an artifact of the harness, not of production capture. The observation+stream recovered about two thirds of that FTS gap.## A/B comparisons (R2)@@ -53,14 +74,18 @@ baseline-vs-baseline determinism check is clean (all accuracy/token deltas `0.00graph. **mode: zero-llm** is the deterministic floor (`embedding_provider= "none"`, or any host where the model cannot load): FTS5 +entity/graph only.-- **Comparability.** Published numbers from other systems on this dataset- (agentmemory 0.967 R@5, doobidoo/mcp-memory-service 0.804 R@5) are- embedding-based retrieval over raw chat logs. Our `hit@5` is the- comparable statistic, but our pipeline additionally pays for- production-shaped capture: excerpts are bounded at the 2 KB privacy- boundary, so evidence deep inside one long turn is out of reach of the- index. The cost is deliberate, because the benchmark measures the shipped- system rather than an idealised retriever.+- **Comparability.** Our `hit@5` is the statistic other systems publish as+ R@5 (any evidence session in the top 5). doobidoo/mcp-memory-service+ publishes 0.804 R@5. agentmemory publishes 95.2% `recall_any@5` (April+ 2026), measured by indexing each whole raw session directly (BM25 plus+ vectors, reranker disabled), which is a component measurement. Our+ pipeline numbers also pay for capture, so they are not like-for-like with+ a component number, and even our component numbers use a+ different index and document construction. Our grep engine reproduces+ agentmemory's grep adapter on all 470 questions, so the question set and+ metric match; a joint evaluation of the two systems is pending, and no+ ranking between them is claimed. The decomposition report has the+ details.- **Regression gate.** Roadmap items 2-6 re-run this benchmark; a changethat lowers a slice materially is a regression to fix, not a note topublish.diff --git a/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md b/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.mdnew file mode 100644index 00000000..0ee61bc6--- /dev/null+++ b/docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md@@ -0,0 +1,320 @@+# LongMemEval-S retrieval decomposition (2026-10-12)++This report splits ai-memory's LongMemEval-S `hit@5` into its parts:+evidence the index never sees, ranking on evidence it does see, and+artifacts of the harness itself. It then records what the observation+stream, shipped in 2.7 (unreleased), recovered.++It publishes two kinds of number, and they answer different questions:++- **Pipeline** numbers run the shipped system end to end. Each haystack+ replays through `POST /hook/batch` at production cadence, the server builds+ its zero-LLM session pages, and `memory_query` ranks them. This is what a+ user of ai-memory gets.+- **Component** numbers skip capture. Each raw session is written as one page+ through `POST /admin/write-page`, so ranking sees the full text. This+ measures the ranker alone.++Every number below comes from the in-repo harness (`evals/`, see+[`evals/README.md`](../../evals/README.md)), with the flags shown.++## Provenance++Decomposition runs (M1, 2026-10-10 to 2026-10-11):++- server: `release/2.7` at `7dd6b2c2` (the binary reports 2.6.3), before the+ observation stream existed+- harness: `420a8556` (the instruments) and `31f515dc` (adds+ `--http-timeout-secs`); `release/2.7` carries the same `evals/` tree as+ `014e123b` and `b7ac5a70`+- dataset: `longmemeval_s` (sha256 `08d8dad4be43ee2049a22ff5674eb86725d0ce5ff434cde2627e5e8e7e117894`);+ where noted, `longmemeval_s_cleaned` (Hugging Face+ `xiaowu0162/longmemeval-cleaned` at `98d7416c24c778c2fee6e6f3006e7a073259d48f`,+ sha256 `d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442`)+- hardware: AMD Ryzen 9 7950X3D 16-Core (32 threads), about 90 GB RAM,+ Linux 7.2.5, rustc 1.95.0+- local model: all-MiniLM-L6-v2+- questions: the 470 answerable questions (30 abstention questions excluded),+ or a stratified sample: `--stratify N` takes the first N questions of each+ of the six types (single-session-preference has only 30)++Observation-stream runs (R1, 2026-10-12):++- final build: `release/2.7` at `8d8c7780`, stream switched on with+ `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true` (it became the default in+ `5b550300`)+- full-set A/B: run twice, on observation-stream work not yet committed on+ top of `b7ac5a70` (the reports record that commit); the final build's+ full-set run reproduces the candidate's accuracy exactly+- dataset, hardware and model as above (original `longmemeval_s` file)++Latency and ingest throughput were measured while two or three runs shared+the host. Accuracy is unaffected, because each run starts its own server on+its own data dir.++## Pipeline and component numbers (hit@5)++| What is measured | FTS (zero-LLM) | Local embeddings |+|---|---|---|+| Grep over whole raw sessions, no server (`--engine grep`) | 0.887 (470; 417/470 on both dataset files) | n/a |+| **Pipeline**, shipped 2.6 capture and ranking (`--ingest hooks`) | **0.666** (470) | 0.809 and 0.804 (strat-230, two runs); 0.800 (strat-120, three runs) |+| Pipeline with the per-turn date prefix removed (strat-120) | 0.658 → 0.792 | 0.800 → 0.900 |+| Pages holding only the text capture keeps (`--ingest captured-session`) | 0.968 (470) | 0.804 → 0.943 (strat-230) |+| **Component**, whole raw sessions (`--ingest whole-session`) | **0.972** (470, both dataset files) | 0.809 → 0.957 (strat-230) |++Strat-230 is `--stratify 40` and strat-120 is `--stratify 20`. The published+full-set local pipeline number is 0.815+([`retrieval-ab-r2.md`](retrieval-ab-r2.md)); a full local A/B takes about+3.5 hours on this host, so the local column uses samples.++Other FTS metrics on all 470 questions:++| | pipeline | captured-session pages | whole-session pages |+|---|---|---|---|+| hit@1 | 0.530 | 0.887 | 0.885 |+| hit@10 | 0.694 | 0.979 | 0.977 |+| recall@5 | 0.536 | 0.918 | 0.922 |+| context tokens, mean | 351 | 502 | 505 |+| query p50 / p95 ms | 14 / 99 | 27 / 99 | 28 / 93 |++FTS `hit@5` per slice, pipeline → whole-session: knowledge-update 0.764 →+1.000, multi-session 0.579 → 0.975, single-session-assistant 0.804 → 1.000,+single-session-preference 0.533 → 0.900, single-session-user 0.688 → 1.000,+temporal-reasoning 0.654 → 0.945. Whole-session pages gained 145 questions+and lost 1.++Confirmation runs on strat-120 reproduce each effect: whole vs pipeline FTS++0.308 (+38/−1) against +0.306 on the full set; captured vs pipeline FTS++0.300 (+38/−2); local whole vs pipeline +0.142 against +0.148 on strat-230.+FTS returned identical per-question results across repeated runs.++With the evidence in the index, the same FTS ranks an evidence session into+the top 5 for 97.2% of questions; through capture it managed 66.6%. Most of+the gap is therefore in capture.++## Reachability: the evidence was not on the page++`--reachability` checks, for each answer-bearing turn (the dataset's+`has_answer` flag, 886 turns after removing repeated session ids), whether it+survives into each surface that ranking reads. A turn counts as reachable when+at least half of its distinctive tokens are on the surface. Distinctive tokens+are content words, lowercased, of three or more characters or containing a+digit, minus stopwords, that occur in at most 3 sessions of the question's+haystack. The ceiling is the share of questions with at least one reachable+evidence turn.++Pipeline, FTS, all 470 questions:++| slice | evidence turns | session page: turn / ceiling | observations: turn / ceiling |+|---|---|---|---|+| overall | 886 | 0.056 / **0.102** | 1.000 / **1.000** |+| knowledge-update | 144 | 0.083 / 0.153 | 1.000 / 1.000 |+| multi-session | 323 | 0.034 / 0.083 | 1.000 / 1.000 |+| single-session-assistant | 56 | 0.018 / 0.018 | 1.000 / 1.000 |+| single-session-preference | 44 | 0.250 / 0.367 | 1.000 / 1.000 |+| single-session-user | 66 | 0.167 / 0.172 | 1.000 / 1.000 |+| temporal-reasoning | 253 | 0.016 / 0.031 | 1.000 / 1.000 |++The cause is the shape of the zero-LLM session page `sessions/<id>.md`. Its+indexed body lists only each observation's title+(`crates/ai-memory-hooks/src/synth.rs`, `render_body` and+`render_raw_observation`), and titles are cut to 80 characters+(`truncate_for_title` in `crates/ai-memory-core/src/sanitize.rs`). Assistant+turns (`stop` observations) are titled just `stop`, so no assistant text+reaches the page at all. The full observation bodies were indexed in+`observations_fts` the whole time, but `memory_query` read them only when page+search found nothing. With about 50 session pages per question that almost+never happened: 50 of 470 questions took that fallback, and those hit at+46/50.++The pipeline still reached 0.666 because sessions were often found through+topic words at the start of a prompt rather than through the answer. Only 39+of its 313 hits had page-reachable evidence. Of its 157 misses, 148 had none,+and whole-session pages recover 145 of the 157.++The local embedder reads the page body cut to 6,144 bytes and then to 512+tokens. On these short pipeline pages that window holds the whole page. On+full-text pages it covers 0.712 of evidence turns and a 0.791 question+ceiling (whole sessions), or 0.703 and 0.785 (captured text).++## A harness artifact: the per-turn date prefix++A replayed history has to carry its own dates, so the harness prepends+`[session date: YYYY/MM/DD (Day) HH:MM] ` to every turn (`build_items` in+`evals/src/retrieval/ingest.rs`). That is 39 characters, about half of the+80-character title budget. Production capture adds no such prefix.++Moving the date to the session start (`--candidate-date-prefix+session-start`) or dropping it (`none`) on strat-120:++- FTS `hit@5` 0.658 → 0.792 (+0.133; 21 gained, 5 lost). Two independent runs+ gave identical per-question results, and `none` gives the same number.+- Local `hit@5` 0.800 → 0.900 (+0.100; 14 gained, 2 lost).+- Session-page turn reachability rises from 0.080 to 0.267, and the page+ ceiling from 0.142 to 0.408.++So the published pipeline numbers understate shipped behaviour on this+content. The harness keeps `turn` as the default so earlier baselines stay+reproducible.++## The privacy caps are not the cause++Earlier versions of these docs said the 2 KB cap on captured assistant+excerpts explained most of the gap to raw-log retrievers. The measurement+says otherwise. `--ingest captured-session` writes one page per session+holding exactly the text capture keeps: user turns up to 16 KiB and assistant+excerpts cut at 2 KB. Those pages score 0.968 FTS `hit@5` on all 470+questions, against 0.972 for whole raw sessions. The caps cost 0.004. The+step from those pages to the shipped session page, which indexes only the+titles, cost 0.302.++The 16 KiB durable-body backstop does not truncate page writes either: it+applies to observation bodies (`OBSERVATION_BODY_MAX_BYTES`). In the+whole-session run, 3,050 of 23,566 pages were over 16 KiB, and all were+stored whole.++## What the observation stream recovered++The fix keeps the session page as it is and adds the observation bodies to+ranking instead. With `[retrieval] observation_stream` on, project and+`scopes` searches also run full-text search over the project's observations,+and each matching session votes once, through its best observation, for its+own `sessions/<id>.md` page as one more RRF stream. Hits are still pages, and+nothing new is written to the wiki. It is on by default in 2.7; setting it to+`false` restores the previous ranking exactly.++Baseline is the stream off, candidate is the stream on, same build, same+questions:++| run | questions | hit@5 | hit@1 | hit@10 | recall@5 | p95 ms | flips |+|---|---|---|---|---|---|---|---|+| FTS, full set | 470 | 0.666 → **0.868** | 0.530 → 0.619 | 0.694 → 0.936 | 0.536 → 0.767 | 103 → 98 | +96/−1 |+| FTS, final build | 180 (`--stratify 30`) | 0.656 → **0.850** | 0.494 → 0.589 | 0.689 → 0.928 | 0.558 → 0.741 | 32 → 43 | +36/−1 |+| Local, final build | 90 (`--stratify 15`) | 0.800 → **0.867** | 0.489 → 0.533 | 0.867 → 0.933 | 0.677 → 0.773 | 77 → 84 | +7/−1 |++FTS `hit@5` per slice on the full set, off → on: knowledge-update 0.764 →+0.944, multi-session 0.579 → 0.893, single-session-assistant 0.804 → 0.857,+single-session-preference 0.533 → 0.667, single-session-user 0.688 → 0.891,+temporal-reasoning 0.654 → 0.843. No slice dropped. The two full-set A/B runs+gave identical accuracy, and the final build's single-config full-set run+gave the same 0.868, with a p95 of 95 ms. The full-set p95 in the table is+from the second A/B run. The first ran on a build whose observation scan was+not yet bounded to the project, and measured a baseline p95 of 76 ms and a+candidate p95 of 1,373 ms; the bounded scan is what shipped.++Context tokens rise. The candidate returns more hits: with the stream off,+FTS returned fewer than five hits for 242 of 470 questions (5.2 hits on+average), and with it on, 9.8. So the raw mean rises more than the cost of the+top five hits does:++| run | mean context tokens, all hits | top five hits only |+|---|---|---|+| FTS, 180 questions | 411.2 → 561.7 | 264.0 → 283.6 (+7.4%) |+| Local, 90 questions | 473.6 → 562.4 | 236.8 → 281.2 (+18.8%) |+| FTS, 470 questions | 350.2 → 542.1 | 249.6 → 273.6 (+9.6%) |++"Top five hits only" scales each question's estimate by `min(returned, 5) /+returned`. In the 470-question row the "off" side is from the `b7ac5a70` A/B+and the "on" side is the final build; the final build trimmed the excerpt a+hit carries, which affects only the "on" side.++The stream recovered about two thirds of the FTS headroom the decomposition+found (0.666 → 0.868, against 0.968 for captured-text pages). A full-set+local run with the stream on has not been done yet.++## Comparability with other published numbers++- **Harness agreement.** `--engine grep` reimplements agentmemory's grep+ adapter (tokens `[a-z0-9_]+` longer than two characters, substring match+ over the dated session text, stable sort, first occurrence of a repeated+ session). It agrees with agentmemory's archived per-question grep results on+ all 470 questions, for hit, recall@5 and recall-all@5, on both dataset+ files. Both harnesses score the same 470 answerable questions, and our+ `hit@5` is the same statistic as their any-hit@5 over distinct sessions.+ That shows the question set and the metric line up. It is not a comparison+ of the two systems.+- **agentmemory's published figure** on LongMemEval-S is 95.2%+ `recall_any@5`, committed in April 2026. Their benchmark indexes each whole+ raw session as one document directly in in-memory components, with BM25+ plus vectors and the reranker disabled (`rerankEnabled=false`). That is a+ component measurement. ai-memory's pipeline numbers include capture, and+ even its component numbers use a different index, document construction and+ dataset file. **The two are not like-for-like, and no ranking between the+ tools is claimed here.** A joint evaluation on one harness is pending.+- The cleaned dataset file changes the haystacks of 454 questions but keeps+ every question and evidence label. Grep and whole-session FTS score the same+ on both files.++## Caveats++- `belief_authority_weight` could not be measured here. Zero-LLM session pages+ have no `page_evidence` rows, so every page's confidence is 0.0: with+ `AI_MEMORY_RETRIEVAL__BELIEF_AUTHORITY_WEIGHT=0.3` the FTS delta was exactly+ 0.000 and the local delta −0.008 (one flip, within local run-to-run noise).+ It needs an LLM-consolidated corpus.+- With full text in the index, local embeddings scored below FTS alone on the+ same 230 questions (0.943 against 0.974 on captured-text pages; 10 lost,+ 3 gained). The page vector covers only the first 512 tokens.+- 8 of 845 evidence sessions had no session page after hook ingest, affecting+ at most 5 questions. The cause was not investigated.+- Hook ingest slows as the store grows: in the full pipeline run, completed+ questions per minute fell from about 22 to about 5 (about 23,000 sessions+ across 470 projects, contended host). The cause was not profiled.++## Reproduce++Build once, from a `release/2.7` checkout:++```bash+cargo build --release -p ai-memory-cli -p ai-memory-eval+E=target/release/ai-memory-eval+```++Decomposition (server at `7dd6b2c2`, before the observation stream). On a+build where the stream is on by default, add+`--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false` to reproduce+these:++```bash+# grep calibration, both dataset files+$E retrieval --engine grep+$E retrieval --engine grep --dataset-variant s-cleaned+# pipeline vs component, full set, with reachability+$E retrieval --reachability --ingest hooks --candidate-ingest whole-session+# captured-text pages, full set+$E retrieval --http-timeout-secs 900 --reachability --ingest captured-session+# component on the cleaned file+$E retrieval --http-timeout-secs 900 --dataset-variant s-cleaned --ingest whole-session+# date prefix and belief-weight ablations, confirmations (strat-120)+$E retrieval --stratify 20 --reachability --date-prefix turn --candidate-date-prefix session-start+$E retrieval --stratify 20 --date-prefix turn --candidate-date-prefix none+$E retrieval --stratify 20 --candidate-server-env AI_MEMORY_RETRIEVAL__BELIEF_AUTHORITY_WEIGHT=0.3+$E retrieval --stratify 20 --ingest hooks --candidate-ingest whole-session+$E retrieval --stratify 20 --ingest hooks --candidate-ingest captured-session+# local embeddings (strat-230 and strat-120)+$E retrieval --stratify 40 --embeddings local --ingest hooks --candidate-ingest whole-session+$E retrieval --stratify 40 --embeddings local --ingest hooks --candidate-ingest captured-session+$E retrieval --stratify 20 --embeddings local --ingest hooks --candidate-ingest whole-session+$E retrieval --stratify 20 --embeddings local --date-prefix turn --candidate-date-prefix session-start+$E retrieval --stratify 20 --embeddings local --candidate-server-env AI_MEMORY_RETRIEVAL__BELIEF_AUTHORITY_WEIGHT=0.3+```++Observation stream. These ran against builds where the stream was off by+default; on a build where it defaults on, pass+`--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false` for the baseline+side:++```bash+B=target/release+FLAG=AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true+# full set, A/B+$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 300 \+ --candidate-server-env $FLAG+# final build: FTS on 180, local on 90, then the full set with the stream on+$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 300 \+ --stratify 30 --candidate-server-env $FLAG+$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 600 \+ --embeddings local --stratify 15 --candidate-server-env $FLAG+$B/ai-memory-eval retrieval --server-bin $B/ai-memory --concurrency 6 --http-timeout-secs 300 \+ --server-env $FLAG+```diff --git a/docs/benchmarks/retrieval-ab-r2.md b/docs/benchmarks/retrieval-ab-r2.mdindex 8d2b3464..4f61d2b5 100644--- a/docs/benchmarks/retrieval-ab-r2.md+++ b/docs/benchmarks/retrieval-ab-r2.md@@ -97,12 +97,23 @@ cargo run --release -p ai-memory-eval -- retrieval --candidate-embeddings localcargo run --release -p ai-memory-eval -- retrieval --candidate```+### Observation stream (2.7, unreleased)++The full-set A/B for the 2.7 observation stream (stream off → on, zero-LLM+FTS, 470 scored) is published with its provenance in+[`longmemeval-s-decomposition-2026-10-12.md`](longmemeval-s-decomposition-2026-10-12.md#what-the-observation-stream-recovered):+`hit@5` 0.666 → 0.868, hit@1 0.530 → 0.619, hit@10 0.694 → 0.936, 96 questions+gained and 1 lost. The local-embeddings A/B ran on a 90-question stratified+sample (0.800 → 0.867). The stream is on by default in 2.7, so a baseline run+of the 2.4 configuration on a 2.7 build needs+`--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false`.+### Illustrative sample-20 run (triple + QA-accuracy, NOT a baseline)Recorded to show the triple + QA columns on a real run. **`--sample 20`, 20LongMemEval-S questions: illustrative, NOT the full-500 baseline.** The-published baseline stays 0.823 hit@5 (local) / 0.668 zero-LLM in-[README.md](README.md); do not cite the numbers below as a baseline.+published baselines are in [README.md](README.md); do not cite the numbers+below as a baseline.Retrieval (zero-LLM):diff --git a/docs/comparison.md b/docs/comparison.mdindex fe1935df..cf209f3f 100644--- a/docs/comparison.md+++ b/docs/comparison.md@@ -23,7 +23,9 @@ What sets it apart:(Claude Code, Codex, Cursor, Gemini, OpenCode, Kimi, …) feed and read thesame memory. Quit one agent, open another in the same repo, and you get ahandoff. Handoffs are a typed protocol (owned, claimed exactly once) instead- of a note file.+ of a note file. Replication between servers is not automatic; in 2.7+ (unreleased) the `ai-memory-wikisync` companion adds two-way,+ conflict-aware sync between a server's wiki and a git repository.- **Zero-LLM default.** Capture, FTS5 + entity + graph retrieval, localembeddings, and rule-based summaries all work with **no API key**. An LLM isopt-in for richer consolidation; nothing requires one.@@ -37,13 +39,20 @@ What sets it apart:- **One self-contained binary.** Bundled SQLite, vendored libgit2; no externalservices to stand up. It runs on a laptop, a homelab box, or a LAN server.-It also publishes numbers: **LongMemEval-S hit@5 0.823** (local-embeddings-default; 0.668 zero-LLM), produced by the in-repo harness with full provenance.-For context on the same dataset, mcp-memory-service-reports 0.804 R@5 and agentmemory 0.967 R@5 (hybrid + reranking). ai-memory is-comparable to the former and below the latter, and it documents *why*-(the pipeline pays a real 2 KB privacy-capture cost the raw-log retrievers do-not). See [where we're behind](#where-were-behind-or-different-by-choice).+It also publishes numbers, produced by the in-repo harness with full+provenance and measured end to end through the shipped capture pipeline:+**LongMemEval-S hit@5 about 0.82** with the default local embeddings (0.823 and+0.815 in the two published full-set runs) and 0.666 zero-LLM. The observation stream in 2.7+(unreleased, on by default) raises zero-LLM to **0.868** on all 470 questions.+With capture skipped and each raw session indexed whole (component mode,+ranking only), zero-LLM FTS scores 0.972; the+[decomposition report](benchmarks/longmemeval-s-decomposition-2026-10-12.md)+explains the difference. For context on the same dataset, mcp-memory-service+reports 0.804 R@5, and agentmemory publishes 95.2% `recall_any@5` from a+component harness (whole raw sessions, BM25 plus vectors, reranker disabled).+Pipeline and component numbers are not like-for-like, and a joint evaluation+with agentmemory is pending, so this page claims no ranking between them. See+[where we're behind](#where-were-behind-or-different-by-choice).## By camp@@ -58,7 +67,7 @@ not). See [where we're behind](#where-were-behind-or-different-by-choice).| Paper-backed pages | **Hindsight** (Vectorize) | "Mental models" = living markdown pages an agent boots from; belief-strength consolidation; a preprint | Postgres/pgvector-primary, LLM-required; strict per-bank isolation (no team-sharing within a project). ai-memory now ships belief-strength `confidence` + an opt-in LLM "dream" rewrite, both zero-LLM-default-preserving, off by default, and R2-gated before default-on || Closest sibling (fact-row twin) | doobidoo/mcp-memory-service | SQLite(+vec), local ONNX, hook capture, typed edges, honest numbers | What ai-memory would be if it chose fact-rows over wiki **pages** || Platform-native | Claude Code auto-memory | Zero setup, on by default | Machine-local, **no sync**, single-agent, repo-scoped, no tool-lifecycle capture, no team |-| **File-first wiki (ai-memory)** | ai-memory, basic-memory, OKF | Human-editable markdown truth + derived index; cross-agent; zero-LLM default; multi-user | Below the reranking leaders on raw R@5; LLM-optional means no VLM fact-extraction sophistication |+| **File-first wiki (ai-memory)** | ai-memory, basic-memory, OKF | Human-editable markdown truth + derived index; cross-agent; zero-LLM default; multi-user | No local reranker yet; LLM-optional means no VLM fact-extraction sophistication || File-first wiki (LLM-leaning) | **memU**, **EverOS** | Markdown source of truth + derived vector/SQLite(+LanceDB) indexes; offline consolidation; skill distillation from session history | LLM-required for the core capture/distill flow (no zero-LLM default); Python multi-service + a hosted-cloud option, vs one self-contained zero-LLM binary; memU patches host instruction files rather than exposing a git-diffable wiki || Self-hosted coding-agent store | **Engram** | Single Go binary, SQLite+FTS5, MCP stdio, project-scoped, `mem_session_summary` handoffs, git-sync across machines | Structured rows the agent saves through MCP tools (SQLite is the truth, git-syncs compressed chunks), not a hand-editable git-markdown wiki captured automatically from lifecycle hooks || Governed multi-agent / fleet memory | **Caura** | Shared fleet memory: agent/team/org visibility scopes, four trust tiers, audit log, PII flagging, contradiction/supersession, auto knowledge graph | Fleet-governance-first on Postgres+pgvector (LLM extraction), hosted option; ai-memory optimizes one project's git-backed wiki with `pages shared, batons owned`, not org-scale fleet governance |@@ -69,7 +78,8 @@ not). See [where we're behind](#where-were-behind-or-different-by-choice).This is a crowded, fast-moving field, and **every toolcompared here is actively maintained** (as of 2026-09-18, all had commits withinthe last ~10 days, and none was stale or archived; the five new-entrant rows below-were verified 2026-09-22). Raw GitHub popularity, though,+were verified 2026-09-22, and the agentmemory row was re-checked 2026-10-10).+Raw GitHub popularity, though,tracks funding and app-developer reach more than coding-agent fitness. Thestar leaders are the app-personalization and hosted-context players (a differentbuyer), while the tools closest to ai-memory's file-first, self-hosted,@@ -82,7 +92,7 @@ coding-continuity niche are smaller by design.| Zep/Graphiti | 31.0k | 2026-09-08 | active || Cognee | 30.8k | 2026-09-15 | active || Supermemory | 30.1k | 2026-08-17 | active |-| agentmemory | 28.6k | 2026-08-16 | active |+| agentmemory | 29.3k | v0.9.30 (2026-10-06) | active || TencentDB Agent Memory | 27.2k | v2.0.1 (2026-08-25) | active || Letta | 24.8k | 2026-05-14 | active (releases lag code) || Hindsight | 23.9k | 2026-09-14 | active |@@ -167,8 +177,10 @@ ai-memory's own zero-LLM, file-first terms (design of record:Two of these, belief-strength folded into ranking and the dream pass, changebehavior only where a provider is configured and the operator opts in,and both are **off by default and gated on a recall eval (R2) before they may-default on**. No such eval has been run yet, so ai-memory claims *parity of-mechanism, not a measured quality win*: the code ships, switched off.+default on**. No eval has shown a gain yet (the zero-LLM benchmark corpus+has no consolidation evidence for belief-strength to act on), so ai-memory+claims *parity of mechanism, not a measured quality win*: the code ships,+switched off.Everything else here is zero-LLM, reversible, and off by default until anoperator turns it on.@@ -243,6 +255,31 @@ operator turns it on.file-first, zero-LLM, single-binary project memory, use ai-memory. (Caura'strust-tier governance is a real capability we do not match; see[`competitive-parity.md`](competitive-parity.md).)+- **From agentmemory:** the closest sibling in intent: automatic hook capture+ for coding agents, keyless by default. You gain markdown in git as the+ source of truth (agentmemory keeps its state in iii-engine's KV store,+ file-backed or Redis, with a one-way Obsidian export), session summaries and+ handoffs without an LLM key (agentmemory's summarize step does nothing+ without one), one process instead of a Node worker plus the iii-engine+ binary, workspace/project isolation with adversarial tests, per-user auth+ with DB tokens and OIDC, claim-once handoffs and cross-project messaging,+ guarded backup/restore/purge, and no runtime dependency under the Elastic+ License (iii-engine's engine is ELv2). You give up editing and forgetting+ memories in a browser (the 2.7 web UI adds page history with diffs, a+ session timeline, a recall trace and a health panel, but stays read-only by+ design), the iii console's OpenTelemetry traces (ai-memory 2.7 has+ per-stream timings in `explain=true` and a root-only Prometheus+ `/admin/metrics` instead), the coordination tools (leases, actions,+ routines, signals, sentinels), mesh sync (the nearest equivalent is the+ git-based `ai-memory-wikisync` companion), image memory, the two-way Claude+ `MEMORY.md` bridge, PaaS deploy templates, the `npx` one-line installer, and+ the opt-in local cross-encoder reranker. The benchmark numbers are not+ like-for-like (component vs pipeline, see above). To move your data, the+ `ai-memory-importer` companion (2.7) reads agentmemory's Obsidian vault+ (`agentmemory-vault`) or its `ExportData` JSON (`agentmemory-export`), and+ `ai-memory doctor` flags agentmemory MCP entries and rules-file guideline+ blocks left behind; see+ [Coming from agentmemory](cookbook.md#coming-from-agentmemory).- **From TencentDB Agent Memory (proxy fan-in server):** you trade a Nodethree-service proxy that intercepts each harness's API base URL (distillingChat Memory / Skill / Wiki / CodeGraph assets) for a file-first binary that@@ -254,11 +291,16 @@ operator turns it on.## Where we're behind, or different by choice-- **Raw retrieval score, and no local reranker.** 0.823 hit@5 on LongMemEval-S- is comparable to mcp-memory-service and below agentmemory's 0.967 (hybrid +- reranking). Part of the gap is deliberate: a 2 KB privacy cap on captured excerpts puts- evidence deep inside one long turn out of the index's reach, and the benchmark- measures the *shipped, sanitized* system instead of an idealized retriever. The only+- **Retrieval through capture, and no local reranker.** Through the shipped+ pipeline, LongMemEval-S hit@5 is about 0.82 with local embeddings and 0.666+ zero-LLM (2.6); the 2.7 observation stream raises zero-LLM to 0.868. Ranking+ the same raw sessions whole scores 0.972 zero-LLM. The+ [decomposition](benchmarks/longmemeval-s-decomposition-2026-10-12.md) traced+ the gap to capture, and within capture not to the 2 KB privacy cap (pages holding exactly the captured text+ score 0.968) but the zero-LLM session page, which indexed only short turn+ titles. agentmemory's published 95.2% is a component number with its+ reranker disabled, so it is not like-for-like with ours; a joint evaluation+ is pending. The onlyreranker is LLM-as-judge, so the zero-LLM default path has none; the opt-in`answer=true` dialectic synthesis (2.4) is likewise LLM-required, off bydefault, and its end-to-end QA-accuracy result is early and small-sample (see@@ -268,7 +310,7 @@ operator turns it on.The 2.4 memory-aging set includes the two pieces that could move retrieval*quality*, folding belief-strength `confidence` into ranking authority andthe LLM "dream" rewrite of cold clusters, but both are **off by default and- gated on a recall eval (R2) that has not been run**. We claim parity of+ gated on a recall eval (R2) that has not shown a gain yet**. We claim parity of*mechanism* with Hindsight/mcp-memory-service here, not a measured recall orQA improvement. Until R2 shows a positive delta the accurate statement is"shipped, opt-in, unproven," and the default path is unchanged.diff --git a/docs/competitive-parity.md b/docs/competitive-parity.mdindex 2f3d17c7..c928b018 100644--- a/docs/competitive-parity.md+++ b/docs/competitive-parity.md@@ -42,7 +42,10 @@ None of the competitors below has more than one or two of these; none has all."both agents read the same files."- **Multi-user shared-within-project** ("pages shared, batons owned", invariant#16), self-hosted, with no paid tier.-- **A single honest, reproducible benchmark** (0.823 hit@5 local; 0.668 zero-LLM).+- **Honest, reproducible benchmarks**: LongMemEval-S hit@5 about 0.82 local and+ 0.666 zero-LLM through the shipped pipeline, 0.868 zero-LLM with the 2.7+ observation stream, and a published pipeline-vs-component decomposition+ ([`benchmarks/`](benchmarks/README.md)).## Migration verdicts, by competitor@@ -50,7 +53,7 @@ None of the competitors below has more than one or two of these; none has all.|---|---|---|---|| **mcp-memory-service** (fact-row twin) | **Yes** if you liked hook capture + typed edges + honest numbers | Files-as-truth (not opaque fact rows), handoffs, cross-project messaging, `as_of` temporal, a marginally better + reproducible number, and (new on 2.4) its aging set at parity done zero-LLM/reversible/off-by-default (per-tier decay curves, extractive compression, DBSCAN cold-cluster dedup, access boosts, 0.4–0.75-band contradiction flagging) | Cloud/multi-backend replication (Cloudflare/Milvus), graph viz. (Belief-strength + clustering consolidation are no longer a give-up: ai-memory now ships both, clustering as a zero-LLM default-off pass, belief-strength off/R2-gated) || **basic-memory** (file-first sibling) | **Yes** for multi-harness coding continuity; **no** for a personal Obsidian KB with live team co-edit | Ambient capture (no `write_note` ceremony), full lifecycle, handoffs/messaging, git-versioned truth, published numbers | Local zero-cost reranking, real-time collaborative editing, Postgres backend, `build_context` graph-walk, hosted mobile/web UX |-| **agentmemory** (ideological sibling) | **Yes** for operability + data ownership | Self-contained binary (no `iii-engine` sidecar), files-as-truth, SQL indexes committed in-txn, typed handoffs, Windows parity, fuller auth | ~13 pts raw R@5 (they rerank), a P2P mesh-sync primitive, ~31 tools of (mostly speculative) surface |+| **agentmemory** (ideological sibling) | **Yes** for operability + data ownership | Self-contained binary (no `iii-engine` sidecar, no ELv2 runtime), files-as-truth, SQL indexes committed in-txn, zero-LLM summaries and handoffs, typed claim-once handoffs, a multi-user auth ladder (DB tokens, OIDC), native Windows without a manual sidecar download (ours is still "Experimental"), and a 2.7 importer for its vault and `ExportData`. Both now default to auth on for new installs (theirs since 0.9.30, ours in 2.7, unreleased) | Browser edit/forget (the 2.7 web UI adds page history/diff, a session timeline, a recall trace and a health panel, but stays read-only by design), the iii console's OTEL traces (we ship explain timings + Prometheus `/admin/metrics` in 2.7), a P2P mesh-sync primitive (`ai-memory-wikisync` is our git-based answer), the opt-in local cross-encoder reranker, the `npx` installer, ~31 more tools (54, all visible by default; mostly speculative). Benchmarks are not like-for-like: their 95.2% LongMemEval-S is a component measurement with the reranker disabled, ours is end-to-end; a joint evaluation is pending || **Claude Code built-in memory** | **Yes** for teams / multi-machine / multi-harness; **not obviously** for a solo dev on one machine in Claude only | Cross-machine, cross-harness, team sharing, tool-lifecycle capture, real search | Zero setup: it's already on with no server to run (our one honest structural disadvantage for the solo case) || **Hindsight** (pages-over-facts) | **Yes** for self-hosted/offline/team; **no** if you need their published accuracy + are fine with an LLM-required cloud | Zero-LLM, files you own, one binary, in-project team sharing (vs strict per-bank isolation), and (new on 2.4) a belief-strength `confidence` + an opt-in "dream" rewrite that borrow their model but never delete a source and keep the zero-LLM default | Their published headline accuracy, VLM/LLM extraction depth, and belief-strength that *actually moves ranking by default* (ours ships but is off/R2-gated until an eval proves it) || **Zep/Graphiti** (temporal KG) | **Yes** for self-hosting coders avoiding Neo4j+LLM (Community Edition is discontinued); **no** for enterprise graph-query/world-time | One binary on SQLite, zero-LLM, files, coding-harness native, a supported self-host | True bi-temporal (world vs observed time), Cypher/BFS graph queries, custom entity/edge types, `minRating` |@@ -92,10 +95,13 @@ remaining risk is that borrowed ideas sit **dormant** until the eval harness (R2| Contradiction detection (0.4–0.75 band) | mcp-memory-service | **SHIPPED (parity), advisory-only** | 2.4 (#814) surfaces likely conflicts in `memory_lint` from existing embeddings; zero-LLM, never edits/deletes/persists an edge || Surprisal-first + idle/cancel dream scheduling | Honcho (Dreamer) / Supermemory | **SHIPPED (parity), opt-in** | 2.4 (#816) idle-scheduled, cancels on activity, orders most-novel-first; off by default, observable `DreamReport` per run || Git-as-snapshot | agentmemory | **BETTER** | The wiki files *are* the truth (diffable), not a `state.json` dump |+| Viewer: version diff, session timeline/replay, health | agentmemory | **SHIPPED (2.7), read-only by design** | Page history with server-side diffs (superseded versions included, so a concurrent write's loser stays visible), a paged session timeline, and a root-only health panel with the same fix lines as `ai-memory status`. Edit/forget in the browser is declined (#482); a `clippy.toml` forbids `Wiki` writes in the web crate |+| Recall tracing / metrics | agentmemory (via the iii console's OTEL) | **SHIPPED (2.7), different shape** | No OTEL exporter: `explain=true` reports per-stream timings, `/web/search` shows a recall trace with per-stream ranks and RRF contributions, and root-only `/admin/metrics` exposes Prometheus counters and latency histograms with no content or scope in any label |+| Auth on by default | agentmemory (0.9.30) | **SHIPPED (2.7), new installs only** | `init` on a fresh data dir creates an owner-only root token and the server requires it; existing installs keep working and opt in with `ai-memory auth enable`, which rewires hooks and MCP clients without writing the token into a harness config || Pages-over-facts / living pages | Hindsight | **PARITY** (native) | Same architecture, file-first vs their Postgres-first; `settled_first` briefing shipped || Cross-session experience/abstraction pass | Letta/Hindsight/surveys | **REAL, ~parity** | ≥2-session evidence guard is real, but the pass is LLM-required and **off by default** || Typed relation edges | graph camp | **PARTLY SHIPPED** | Link-neighbours are now a first-class retrieval surface: `memory_read_page include_related` walks the `page_links` graph outward (bounded BFS, #775). Retrieval **RANK weighting by edge type is still not done** (edge kind stays explain-only in `memory_query`) |-| Belief-strength consolidation | Hindsight / mcp-memory-service | **SHIPPED, off by default (parity of mechanism, unproven)** | 2.4 (#815) computes a read-time, zero-LLM `confidence` over `page_evidence` (distinct-session breadth, recency, live `contradicts`, capped `[0,0.95]`), exposed inertly in `explain`/`status`; foldable into ranking authority behind a **default-0.0, R2-gated** weight. Our improvement over Hindsight's documented **entrenchment** failure: breadth-not-raw-count weighting, recency shading, a confidence cap, and **supersession always wins** regardless of evidence. Not yet default-on; no R2 delta run |+| Belief-strength consolidation | Hindsight / mcp-memory-service | **SHIPPED, off by default (parity of mechanism, unproven)** | 2.4 (#815) computes a read-time, zero-LLM `confidence` over `page_evidence` (distinct-session breadth, recency, live `contradicts`, capped `[0,0.95]`), exposed inertly in `explain`/`status`; foldable into ranking authority behind a **default-0.0, R2-gated** weight. Our improvement over Hindsight's documented **entrenchment** failure: breadth-not-raw-count weighting, recency shading, a confidence cap, and **supersession always wins** regardless of evidence. Not yet default-on; the R2 run on the zero-LLM corpus measured 0.000, because only consolidation writes the evidence it reads || LLM merge/rewrite "dream" pass | Hindsight (Dreamer) / Supermemory ("dreaming") | **SHIPPED, opt-in/off (parity of mechanism, unproven)** | 2.4 (#816) rewrites cold clusters into one page via the gated `apply_batch` path, `dry_run` first, JSON-schema output; **never deletes a source** (superseded with a merge-note stub, `restore-page` recovers), off by default, provider-less stores keep the zero-LLM A3 path. Off until an R2 quality delta is shown || bi-temporal-lite | Zep/Graphiti | **BEHIND by design** | Ingestion-time only; world-time validity deferred (because dating facts needs an LLM and breaks the zero-LLM path) |@@ -104,20 +110,29 @@ remaining risk is that borrowed ideas sit **dormant** until the eval harness (R2These are documented recommendations only. **Nothing here is scheduled or to beimplemented without a separate decision.** Ordered by leverage.-1. **R2 reproducible eval harness, now built (#771/#772).** It was the+1. **R2 reproducible eval harness, built (#771/#772) and run.** It was themeta-blocker: belief-strength ranking, typed-edge rank weighting, and apublished accuracy/latency/context-tokens *triple* were all gated on "prove it- with a number." The harness now measures that triple **and** end-to-end- QA-accuracy (LLM-as-judge), so the gate is open; what remains is running the- full-500 matrix and then activating the dormant substrate below against it.- Illustrative sample-20 numbers are in- [`benchmarks/retrieval-ab-r2.md`](benchmarks/retrieval-ab-r2.md); the- published full-500 triple is still pending.-2. **A zero-LLM / local reranker (candle cross-encoder).** The most-cited- weakness: our only reranker is LLM-as-judge, so the zero-LLM default path,- which is our headline, has *no* reranking, while basic-memory ships a local one and- agentmemory/Hindsight's score lead is largely reranking (0.823 vs 0.952 same- dataset). This would close a gap on our own keyless turf. Gate on R2.+ with a number." The full-set FTS→local triple was published on 2026-09-21+ ([`benchmarks/retrieval-ab-r2.md`](benchmarks/retrieval-ab-r2.md)), and the+ 2026-10-12 decomposition+ ([`benchmarks/longmemeval-s-decomposition-2026-10-12.md`](benchmarks/longmemeval-s-decomposition-2026-10-12.md))+ split the pipeline number into capture cost, ranking and harness+ artifacts, which led to the 2.7 observation stream (zero-LLM FTS hit@5+ 0.666 → 0.868). What is still unrun is the dormant substrate below: the+ belief-strength weight cannot move a zero-LLM corpus (every page's+ confidence is 0.0 without consolidation evidence, so the measured delta+ was 0.000 FTS), so it needs an LLM-consolidated corpus; typed-edge rank+ weighting is not built.+2. **A zero-LLM / local reranker (candle cross-encoder).** Our only reranker is+ LLM-as-judge, so the zero-LLM default path has *no* reranking, while+ basic-memory ships a local one and agentmemory ships one opt-in (off by+ default, and disabled in its published LongMemEval-S run). The+ decomposition narrowed the case: FTS already ranks whole raw sessions at+ 0.972 hit@5, so a reranker's headroom on hit@5 is small; what is left is+ hit@1 (0.885 on whole sessions) and recall on multi-evidence questions.+ Re-scope any reranker work to those metrics, measured with the observation+ stream on.3. **Activate the dormant substrate: belief-strength now wired, off bydefault (2.4, #815).** `page_evidence` confidence is now computed at readtime (zero-LLM) and can be folded into `PageAuthority` behind a clamped,@@ -126,17 +141,21 @@ implemented without a separate decision.** Ordered by leverage.the anti-entrenchment guards built in. It is exposed inertly in`memory_query explain` and `memory_status` today; **it does not move rankinguntil an operator sets the weight, and turning it on is gated on a positive- R2 delta that has not been run.** Still behind: **typed-edge rank+ R2 delta, which the zero-LLM benchmark corpus cannot produce (no+ consolidation evidence, so the measured delta was 0.000).** Still behind: **typed-edge rankweighting** (edge kind stays explain-only) and a `contradicts` *authority*cap; A5 (#814) surfaces contradictions in `memory_lint` but does not yetfeed rank.-4. **Sharpen the cross-machine story.** `comparison.md`'s "cross-machine *by- construction*" is overstated: it's one shared server + manual git/rsync, with- **no automatic replication** (agentmemory ships a P2P primitive; mcp-memory-- service has Cloudflare/Milvus). Either reframe as "one server, many machines"- + an explicit git/rsync note, or design a bounded `ai-memory sync` over the git- wiki (push/pull the source-of-truth dir, rebuild the index), which would be- a cleaner, file-first P2P story than an opaque KV LWW.+4. **Sharpen the cross-machine story: partly done in 2.7.** `comparison.md`'s+ "cross-machine *by construction*" meant one shared server + manual git/rsync,+ with **no automatic replication** (agentmemory ships a P2P primitive; mcp-memory-+ service has Cloudflare/Milvus). 2.7 (unreleased) adds the file-first answer:+ the `ai-memory-wikisync` companion syncs a server's wiki with a git+ repository both ways through the public MCP write path, with conditional+ writes (`expected_page_id`/`create_only`, #986), conflicts that need+ `--prefer`, opt-in delete propagation, and a `sync --check` CI gate.+ `comparison.md` now says replication between servers is not automatic and+ names the companion.5. **Close the setup-friction gap vs the built-in.** This is our one structuralweakness for the solo user. Much is already done (`run` auto-wire, `doctor`,`backfill`, `install-self-routing`); surface it as *the* onboarding path in the@@ -178,10 +197,15 @@ implemented without a separate decision.** Ordered by leverage.preferences page already covers the useful user-prefs slice).7. **Fix doc staleness the audit surfaced** (accuracy, not features):`research-basic-memory.md` understates basic-memory's shipped cross-encoder- reranking + Teams tier; `research-agentmemory.md` says 53 tools / 124 endpoints- (now 54 / 130); `comparison.md` attributes agentmemory's `0.967` to- LongMemEval-S when it's their in-house `coding-agent-life-v1` (the like-for-like- LongMemEval-S is 0.952 vs our 0.823).+ reranking + Teams tier. The agentmemory items were corrected on 2026-10-12:+ `comparison.md`, `benchmarks/README.md` and `evals/README.md` no longer+ attribute the `0.967` (the grep baseline on their 15-query+ `coding-agent-life-v1` corpus) to LongMemEval-S, where they publish 95.2%+ `recall_any@5` without a reranker; and `research-agentmemory.md` /+ `issues-agentmemory.md` now carry a dated-snapshot header (54 tools, all+ visible by default; 138 REST endpoints, per their README as of 2026-10-10).+ Their 95.2% is a component measurement and our 0.82 a pipeline one, so+ neither pairing is like-for-like; a joint evaluation is pending.8. **Fleet-scale trust governance: a real gap vs Caura (issue #810).** Ourmulti-user story is per-project (`pages shared, batons owned`, invariant #16)with a root→DB-user→OIDC auth ladder and an audit log, and it is deliberately@@ -212,16 +236,24 @@ implemented without a separate decision.** Ordered by leverage.**file-first**, and **R2/opt-in discipline** on anything that touches rankingor calls an LLM. The R2 harness that gates the quality work now exists andmeasures the accuracy+latency+context-tokens triple plus end-to-end- QA-accuracy (illustrative sample-20 only so far).-- **The real weakness is still retrieval quality**, and it remains- *self-inflicted by discipline, not by copying*: no local zero-LLM reranker- exists, belief-strength (`page_evidence`) is now wired but **default-off**,- typed-edge **RANK** weighting is still not built, and the published- *full-dataset* triple is still pending (only illustrative sample-20 numbers so- far). No R2 delta has been run, so nothing quality-affecting defaults on. The- gate (R2) is now- open, so the next step is to **activate what's already there against a- full-500 run.**+ QA-accuracy (the QA column is still illustrative sample-20 only).+- **2.7 (unreleased) closes several agentmemory-specific gaps** the same way:+ read-only page history/diff, a session timeline, a recall trace and a+ root-only health panel in the web UI; explain timings and a content-free+ `/admin/metrics`; auth on by default for new installs with `auth enable`+ for existing ones; MCP recipes for seven more clients; an importer for+ agentmemory's vault and export plus a `doctor` check for its leftovers; and+ git-based wiki sync (`ai-memory-wikisync`).+- **The real weakness was retrieval through capture**, and it was+ *self-inflicted, not copied*: the decomposition showed the zero-LLM session+ page indexed only 80-character turn titles and no assistant text, while+ ranking whole sessions scored 0.972. The 2.7 observation stream, on by+ default after a measured R2 delta, recovered about two thirds+ of that (0.666 → 0.868 zero-LLM FTS on the full set; 0.800 → 0.867 local on+ a 90-question sample). Still open: no local zero-LLM reranker, a full-set+ local run with the stream, belief-strength (`page_evidence`) wired but+ **default-off** and unmeasurable without an LLM-consolidated corpus, and+ typed-edge **RANK** weighting not built.- **Migration answer:** for the self-hosted, multi-harness, team, zero-LLM,own-your-data coding-memory user, yes: we do the basics and add what theycan't get elsewhere. For app personalization, enterprise graph queries, agentdiff --git a/docs/issues-agentmemory.md b/docs/issues-agentmemory.mdindex 630c3b1a..e9e45585 100644--- a/docs/issues-agentmemory.md+++ b/docs/issues-agentmemory.md@@ -1,5 +1,15 @@# agentmemory - Issue & PR Pain-Point Synthesis+> **Snapshot as of 2026-05-21; superseded in part.** This records agentmemory+> as it was then and is kept as history: the invariant citations in+> `ARCHITECTURE.md` and `design-decisions.md` point at the bugs listed here.+> As of 2026-10-10 (v0.9.30): the iii engine is a public repository+> (Apache-2.0 and Elastic License 2.0; the engine directory is ELv2), #303,+> #309 and #519 are closed, the two-config-paths bug was fixed in 0.9.29, and+> idle sessions now end. Current state:+> [`research-2026-landscape.md`](research-2026-landscape.md) §2 and+> [`competitive-parity.md`](competitive-parity.md).+> Source: GitHub `rohitg00/agentmemory`, captured 2026-05-21.> Repo health: 15.7k stars, very active. ~50 merged PRs in last week.> Architecture: TypeScript MCP server over a native Rust `iii-engine` KV.diff --git a/docs/research-2026-landscape.md b/docs/research-2026-landscape.mdindex 02c1a30b..4472566b 100644--- a/docs/research-2026-landscape.md+++ b/docs/research-2026-landscape.md@@ -38,13 +38,23 @@ rather than either alone.## 2. How the original research subjects evolved**agentmemory (rohitg00)** is actively maintained and maturing along the-trajectory the May report predicted. Notable since then: a real-workload-benchmark ("coding-agent-life-v1", 635 requests / 888K tokens / 35 hours,-R@5 = 0.967 for the hybrid retriever), `AGENT_ID` multi-agent isolation-with an opt-in isolated recall scope, and runtime cost warnings when a-premium model is configured for compression. Still on the iii-engine +-JSON-in-KV substrate the May report identified as its structural-constraint. The 53-tool surface did not shrink.+trajectory the May report predicted. Notable since then: an in-house+benchmark, "coding-agent-life-v1" (15 sessions and 15 queries; hybrid R@5+1.000 after a 2026-06-03 correction, grep 0.967), `AGENT_ID` multi-agent+isolation with an opt-in isolated recall scope, and runtime cost warnings+when a premium model is configured for compression. Its LongMemEval-S figure+is still 95.2% `recall_any@5`, measured on whole raw sessions in a component+harness with the reranker disabled; a 2026-10-09 baseline adds a stricter+metric and archived receipts. Still on the iii-engine + JSON-in-KV+substrate the May report identified as its structural constraint, now with+an opt-in Redis backend, vectors kept in buckets with a pending log, and the+BM25 index rebuilt at every boot. The tool surface grew from 53 to 54, and+all 54 are now visible by default. Version 0.9.30 (2026-10-06) turned auth+on by default with a generated secret and added a durable capture inbox+with an offline hook spool. (Corrected 2026-10-10: this entry used to give+the benchmark as 635 requests / 888K tokens / 35 hours with hybrid R@5 =+0.967. Those workload figures could not be found in agentmemory's+repository, and 0.967 is now the grep baseline.)**basic-memory** is alive, with the same positioning (structured markdown bothhumans and LLMs edit; local files). No architectural surprises. Its@@ -166,7 +176,7 @@ rewrite of cold clusters that is **off by default and never deletes asource**. We designed *against* Hindsight's own documented failure,belief *entrenchment* (raw proof-count pins a wrong page), withbreadth-not-count weighting, a confidence cap, and supersession always-winning over evidence. Honest caveat: no R2 delta has been run, so this is+winning over evidence. Honest caveat: no R2 delta has shown a gain yet, so this isparity of *mechanism*, not a measured accuracy win. Full analysis in[`research-hindsight.md`](research-hindsight.md).@@ -437,7 +447,7 @@ research). This is a healthy, crowded, fast-moving field: "we picked a deadspace" is not a claim we can make, and none of these can be dismissed asunmaintained. (The five issue-#810 new-entrant rows, TencentDB Agent Memory,memU, EverOS, Engram and Caura, were verified against the GitHub metadata endpoint-on 2026-09-22.)+on 2026-09-22, and the agentmemory row was re-checked on 2026-10-10.)| Project | Stars (~) | Last commit | Latest release | Status ||---|---|---|---|---|@@ -446,7 +456,7 @@ on 2026-09-22.)| Zep/Graphiti | 31.0k | 2026-09-17 | v0.30.2 (2026-09-08) | ACTIVE || Cognee | 30.8k | 2026-09-18 | v1.5.4rc1 (2026-09-15) | ACTIVE || Supermemory | 30.1k | 2026-09-18 | server-v0.0.8 (2026-08-17) | ACTIVE |-| agentmemory | 28.6k | 2026-09-14 | v0.9.29 (2026-08-16) | ACTIVE |+| agentmemory | 29.3k | 2026-10-10 | v0.9.30 (2026-10-06) | ACTIVE || TencentDB Agent Memory | 27.2k | 2026-09-22 | v2.0.1 (2026-08-25) | ACTIVE || Letta | 24.8k | 2026-09-10 | 0.16.8 (2026-05-14) | ACTIVE (releases lag code) || Hindsight | 23.9k | 2026-09-18 | v0.10.0 (2026-09-14) | ACTIVE |@@ -554,8 +564,8 @@ accuracy, agentmemory/mcp-memory-service quote R@5; pick one and stateit) so our result is comparable, not just present. **Now load-bearing on2.4:** the two aging features that can move quality, belief-strength foldedinto ranking (#815) and the LLM "dream" pass (#816), ship **off by default-and gated on this harness before they may default on**; no R2 delta has been-run yet, so both remain opt-in and unproven (see+and gated on this harness before they may default on**; no R2 delta has+shown a gain yet, so both remain opt-in and unproven (see[`design-memory-aging.md`](design-memory-aging.md) R2 acceptance).**R3: Typed relation edges (small-medium).** `causes` / `fixes` /@@ -661,7 +671,15 @@ benchmark number before R2 exists; chasing agentmemory's tool-countsharing"; document360.com and mindstudio.ai explainers (spec v0.1,2026-06-12).- agentmemory: github.com/rohitg00/agentmemory (README, CHANGELOG,- docs/benchmarks/2026-05-20-coding-agent-life-v1.md).+ docs/benchmarks/2026-05-20-coding-agent-life-v1.md). Re-checked+ 2026-10-10 at commit `7ee1628`: README, CHANGELOG (0.9.27–0.9.30),+ `benchmark/longmemeval-bench.ts` and `benchmark/LONGMEMEVAL.md` (whole+ sessions, `rerankEnabled=false`), `docs/benchmarks/2026-10-09-evaluation-baseline.md`,+ `eval/README.md` and `eval/runner/adapters/`, `src/state/hybrid-search.ts`+ and `src/state/reranker.ts` (reranker opt-in via `RERANK_ENABLED`),+ `src/mcp/tools-registry.ts` (54 tools), the git history of the+ coding-agent-life-v1 report (correction in `e5c9dc6`), and the GitHub+ repository and release metadata.- mcp-memory-service: github.com/doobidoo/mcp-memory-service (README,Wiki, v11.10.0 release notes, issue #27).- mempalace: github.com/MemPalace/mempalace (BENCHMARKS.md, issue #29);diff --git a/docs/research-agentmemory.md b/docs/research-agentmemory.mdindex 6e7f2ec7..f8f7e178 100644--- a/docs/research-agentmemory.md+++ b/docs/research-agentmemory.md@@ -1,5 +1,12 @@# agentmemory - Research Report+> **Snapshot as of 2026-05-21; superseded in part.** Kept as history. As of+> 2026-10-10 (v0.9.30) agentmemory has 54 MCP tools, all visible by default,+> and 138 REST endpoints; auth is on by default; the iii engine is a public+> repository under Apache-2.0 and Elastic License 2.0. Its LongMemEval-S+> figure (95.2% `recall_any@5`) is measured on whole raw sessions with the+> reranker disabled.+>> **Follow-up:** the September 2026 landscape refresh (how this project> evolved, what the field looks like now, and what we concluded) is in> [`research-2026-landscape.md`](research-2026-landscape.md).diff --git a/evals/README.md b/evals/README.mdindex d1843c85..e91c8c4d 100644--- a/evals/README.md+++ b/evals/README.md@@ -292,7 +292,11 @@ cargo run -p ai-memory-eval -- retrieval --sample 3 \These flags split a hit@k number into evidence the index never saw andranking on evidence it did. All are opt-in; the defaults reproduce the-published baselines.+published baselines. The results are in+[`docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md`](../docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md).+On a 2.7 server the observation stream is on by default, so reproducing a+2.6-era baseline also needs+`--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false`.- `--reachability` reads every evidence session back after the query andreports, per slice, how many answer-bearing turns (the dataset's@@ -334,12 +338,24 @@ cargo run --release -p ai-memory-eval -- retrieval --stratify 5 \Honest-numbers notes:- capture is production-shaped: excerpts are bounded at the 2 KB- privacy boundary, so evidence buried deep inside one long turn is- genuinely out of reach — that is the system being measured, not a- harness artifact;+ privacy boundary. The decomposition measured what that costs: pages+ holding exactly the captured text score 0.968 FTS `hit@5` against+ 0.972 for whole raw sessions. The larger cost was the zero-LLM session+ page, which indexed only 80-character turn titles; the observation+ stream (on by default in 2.7) puts the observation bodies back into+ ranking;- each session's original date is prepended to its turn text, since areplayed history must carry its own timestamps (the official- harness exposes the same information to retrievers);-- competitors' published numbers (e.g. agentmemory 0.967 R@5,- mcp-memory-service 0.804 R@5) are answer-evidence Recall@5 on this- same v1 dataset, comparable to our `hit@5`.+ harness exposes the same information to retrievers). The prefix is 39+ characters, about half of a turn title, and production capture adds+ none, so the default `--date-prefix turn` understates the pipeline on+ this content (+0.133 FTS `hit@5` on a 120-question sample with the date+ moved to session start). It stays the default so earlier baselines+ remain reproducible;+- other systems' published numbers on this dataset (e.g. agentmemory's+ 95.2% `recall_any@5`, mcp-memory-service 0.804 R@5) use the same+ statistic as our `hit@5`. agentmemory's indexes whole raw sessions+ directly, a component measurement rather than a pipeline one, so it is+ not like-for-like with the default `--ingest hooks` numbers. Its+ 0.967 is a different benchmark (the grep baseline on its 15-query+ coding-agent-life-v1 corpus), not LongMemEval-S.