Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(retrieval): turn the observation stream on by default
AkitaOnRailsa389873a3385
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 58f5c056..d2ee7896 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -29,9 +29,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`ai-memory-web` crate now has a `clippy.toml` that forbids the `Wiki` writeand destructive methods, so a write path in the web UI or `/api/v1` fails`cargo clippy -D warnings`.-- Added an opt-in observation stream to `memory_query` ranking:- `[retrieval] observation_stream = true` (env- `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true`, default off). A zero-LLM+- Added an observation stream to `memory_query` ranking:+ `[retrieval] observation_stream` (env+ `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM`, on by default). A zero-LLMsession page keeps only a short excerpt of each turn, so the detail aquestion asks about often lives only in the session's observations. With theflag on, project and `scopes` searches also run full-text search over@@ -211,6 +211,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0in `docs/support-matrix.md` as `MCP-only (documented)`: each recipe followsthe client's own documentation at a named version and has not yet been runend to end. Aider is not listed because it has no MCP support.+- Changed `memory_query` ranking to include the observation stream by+ default, after it raised hit@5 on the retrieval eval for both zero-LLM and+ local-embedding search while staying inside its latency and context-token+ budgets. `explain=true` lists `observation` in+ `streams_active` whenever it runs. To keep the previous ranking, set+ `[retrieval] observation_stream = false` (or+ `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false`); ranking is then identical+ to before.- Changed `ai-memory-wikisync` files to carry a small frontmatter (`title`,plus `tags`, `pinned` and a non-default `tier` when set) above the body,so metadata survives a round trip. The first run after upgrading rewritesdiff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex e0f2f45c..9e81d792 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -1636,13 +1636,14 @@ impl DreamSettings {}}-/// `[retrieval]` opt-in ranking signals layered on the RRF fusion in-/// `memory_query`. Every default leaves ranking byte-identical to a store+/// `[retrieval]` ranking signals layered on the RRF fusion in+/// `memory_query`. Only `observation_stream` is on by default; with it+/// turned off, every other default leaves ranking byte-identical to a store/// that never heard of this section.////// Env form: `AI_MEMORY_RETRIEVAL__QUERY_INTENT=true`,/// `AI_MEMORY_RETRIEVAL__ABSTRACT_VECTORS=true`,-/// `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true`.+/// `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false`.#[derive(Debug, Clone, Copy, Serialize, Deserialize)]#[serde(default)]pub struct RetrievalSettings {@@ -1667,7 +1668,7 @@ pub struct RetrievalSettings {/// Add the observation-evidence stream (R1) to the RRF fusion: matching/// observation bodies vote, once per session, for that session's latest/// `sessions/<id>.md` page in the same project. Read-only; the wiki is- /// not enlarged. OFF by default.+ /// not enlarged. ON by default; `false` restores the previous ranking.pub observation_stream: bool,}@@ -3315,15 +3316,19 @@ mod tests {}#[test]- fn retrieval_observation_stream_defaults_off_and_loads_from_toml() {- assert!(!Config::default().retrieval.tuning().observation_stream);+ fn retrieval_observation_stream_defaults_on_and_loads_from_toml() {+ assert!(Config::default().retrieval.tuning().observation_stream);let tmp = TempDir::new().unwrap();let config_path = tmp.path().join("config.toml");- std::fs::write(&config_path, "[retrieval]\nobservation_stream = true\n").unwrap();+ std::fs::write(&config_path, "[retrieval]\nobservation_stream = false\n").unwrap();let config = Config::load(Some(&config_path), Some(tmp.path().to_path_buf())).unwrap();- assert!(config.retrieval.tuning().observation_stream);+ assert!(!config.retrieval.tuning().observation_stream);// The other [retrieval] signals keep their defaults.assert!(!config.retrieval.tuning().abstract_vectors);+ // A config that never names the key gets the default.+ std::fs::write(&config_path, "[retrieval]\nabstract_vectors = false\n").unwrap();+ let config = Config::load(Some(&config_path), Some(tmp.path().to_path_buf())).unwrap();+ assert!(config.retrieval.tuning().observation_stream);}#[test]@@ -3332,9 +3337,9 @@ mod tests {if std::env::var_os(CHILD_MARKER).is_some() {let tmp = TempDir::new().unwrap();let config_path = tmp.path().join("config.toml");- std::fs::write(&config_path, "[retrieval]\nobservation_stream = false\n").unwrap();+ std::fs::write(&config_path, "[retrieval]\nobservation_stream = true\n").unwrap();let config = Config::load(Some(&config_path), Some(tmp.path().to_path_buf())).unwrap();- assert!(config.retrieval.tuning().observation_stream);+ assert!(!config.retrieval.tuning().observation_stream);println!("observation-stream-env-applied");return;}@@ -3345,7 +3350,7 @@ mod tests {.arg("--test-threads=1").arg("--nocapture").env(CHILD_MARKER, "1")- .env("AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM", "true")+ .env("AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM", "false").output().unwrap();assert!(diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rsindex 9ed2b5b0..5189d35e 100644--- a/crates/ai-memory-mcp/src/server.rs+++ b/crates/ai-memory-mcp/src/server.rs@@ -848,7 +848,9 @@ struct MemoryQueryResponse {/// Present only when `explain=true`: which retrieval streams ran for/// the primary search. Project/scopes retrieval always runs `fts` and/// `entity`, and `graph`; `vector` is present only when an embedder- /// produced a query vector. Cross-project `global=true` retrieval is FTS-only.+ /// produced a query vector, and `observation` while `[retrieval]+ /// observation_stream` is on (the default). Cross-project `global=true`+ /// retrieval is FTS-only./// An `as_of` query runs `entity` plus version-filtered `fts`.#[serde(skip_serializing_if = "Option::is_none")]streams_active: Option<Vec<&'static str>>,@@ -10424,7 +10426,7 @@ mod tests {let value: serde_json::Value = serde_json::from_str(&text.text).unwrap();assert_eq!(value["streams_active"],- serde_json::json!(["fts", "entity", "graph"])+ serde_json::json!(["fts", "entity", "graph", "observation"]));let details = &value["hits"][0]["score_details"];assert_eq!(details["fts_rank"], 1);@@ -10545,7 +10547,7 @@ mod tests {);assert_eq!(entity_hit["streams_active"],- serde_json::json!(["fts", "entity", "graph"])+ serde_json::json!(["fts", "entity", "graph", "observation"]));let graph_hit = query("graphseed").await;@@ -12848,7 +12850,7 @@ mod tests {let value: serde_json::Value = serde_json::from_str(&text).unwrap();assert_eq!(value["streams_active"],- serde_json::json!(["fts", "entity", "graph"])+ serde_json::json!(["fts", "entity", "graph", "observation"]));assert_eq!(value["hits"]diff --git a/crates/ai-memory-mcp/tests/suite/query_observation_stream.rs b/crates/ai-memory-mcp/tests/suite/query_observation_stream.rsindex bbe0f260..ee7008ec 100644--- a/crates/ai-memory-mcp/tests/suite/query_observation_stream.rs+++ b/crates/ai-memory-mcp/tests/suite/query_observation_stream.rs@@ -7,6 +7,7 @@//! the session page (hits stay pages), explains the vote with an//! `observation` stream entry, and shows the matching observation excerpt.//! With it off, the same query finds no page and falls back to `raw_hits`.+//! The stream is on by default.use ai_memory_core::{AgentKind, NewObservation, NewPage, NewSession, ObservationKind, PagePath, ProjectId,@@ -34,18 +35,21 @@ struct Seeded {}async fn seeded(observation_stream: bool) -> Seeded {- seeded_with(observation_stream, true).await+ seeded_with(Some(observation_stream), true).await}-async fn seeded_with(observation_stream: bool, session_page: bool) -> Seeded {+/// `observation_stream: None` keeps the store's default tuning.+async fn seeded_with(observation_stream: Option<bool>, session_page: bool) -> Seeded {let tmp = TempDir::new().expect("tempdir");let mut store = Store::open(tmp.path()).expect("store");// The server reads its tuning once at construction, as `serve` does from// `Config::load`.- store.reader.set_retrieval_tuning(RetrievalTuning {- observation_stream,- ..RetrievalTuning::default()- });+ if let Some(observation_stream) = observation_stream {+ store.reader.set_retrieval_tuning(RetrievalTuning {+ observation_stream,+ ..RetrievalTuning::default()+ });+ }let ws = store.writer.get_or_create_workspace(WS).await.expect("ws");let proj = store.writer@@ -240,6 +244,23 @@ async fn memory_query_finds_a_session_page_by_its_observation_evidence() {}}+#[tokio::test]+async fn memory_query_ranks_with_the_observation_stream_by_default() {+ let seeded = seeded_with(None, true).await;+ let payload = query(&seeded.router, args()).await;++ let session_path = format!("sessions/{}.md", seeded.session_id);+ let hit = payload["hits"]+ .as_array()+ .expect("hits")+ .iter()+ .find(|h| h["path"] == session_path.as_str())+ .unwrap_or_else(|| panic!("session page missing: {payload}"));+ assert_eq!(hit["score_details"]["observation_rank"], 1, "{payload}");+ let streams = payload["streams_active"].as_array().expect("streams");+ assert!(streams.contains(&json!("observation")), "{streams:?}");+}+#[tokio::test]async fn memory_query_without_the_stream_never_reports_an_observation_vote() {let seeded = seeded(false).await;@@ -262,7 +283,7 @@ async fn memory_query_without_the_stream_never_reports_an_observation_vote() {#[tokio::test]async fn a_live_session_without_a_page_still_reaches_the_raw_fallback() {- let seeded = seeded_with(true, false).await;+ let seeded = seeded_with(Some(true), false).await;let payload = query(&seeded.router,json!({diff --git a/crates/ai-memory-store/src/retrieval_tuning.rs b/crates/ai-memory-store/src/retrieval_tuning.rsindex b0d6871c..8b2a523f 100644--- a/crates/ai-memory-store/src/retrieval_tuning.rs+++ b/crates/ai-memory-store/src/retrieval_tuning.rs@@ -1,7 +1,8 @@-//! Opt-in post-fusion ranking signals for `hybrid_search`.+//! Post-fusion ranking signals for `hybrid_search`.//!-//! Bounded, independently switchable signals, all off by default so a-//! store that never configures them ranks exactly as before:+//! Bounded, independently switchable signals. All but the observation stream+//! are off by default; with every one of them off a store ranks exactly as it+//! did before any of them existed://!//! * **Session-recall routing** — a zero-LLM lexical check that recognises//! queries phrased as "find a past session / what did we do back then"@@ -24,14 +25,16 @@//! the evidence a question needs often lives only in the session's//! sanitized observation bodies. Each session's best-ranked matching//! observation votes once for that session's latest page; a session with-//! no page contributes nothing (the raw fallback still covers it).+//! no page contributes nothing (the raw fallback still covers it). On by+//! default since it met its measured success criteria.//!//! Every signal is reported in `SearchExplain`, so an `explain=true`//! query can account for the returned rank.use serde::{Deserialize, Serialize};-/// Tunables for the opt-in ranking signals. `Default` disables all of them.+/// Tunables for the ranking signals. `Default` enables only the observation+/// stream.#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]#[serde(default)]pub struct RetrievalTuning {@@ -59,8 +62,9 @@ pub struct RetrievalTuning {/// Add the observation-evidence stream to the RRF fusion (R1): search/// `observations_fts` in the same project and let each matching/// observation vote, once per session, for its session's latest- /// `sessions/<id>.md` page. Nothing is written. Ships OFF: it changes- /// ranking, and the maintainer decides default-on from measurements.+ /// `sessions/<id>.md` page. Nothing is written. ON by default: on the+ /// retrieval eval it lifted zero-LLM hit@5 without breaking the latency or+ /// token budgets. `false` restores the previous ranking byte for byte.pub observation_stream: bool,}@@ -79,7 +83,7 @@ impl Default for RetrievalTuning {// OFF by default: folding belief confidence into ranking is// R2-gated (see the field doc).belief_authority_weight: 0.0,- observation_stream: false,+ observation_stream: true,}}}@@ -180,12 +184,12 @@ mod tests {use super::*;#[test]- fn default_tuning_is_inert() {+ fn default_tuning_enables_only_the_observation_stream() {let t = RetrievalTuning::default();assert!(!t.session_recall_routing);assert!(!t.abstract_vectors);assert_eq!(t.belief_authority_weight, 0.0);- assert!(!t.observation_stream);+ assert!(t.observation_stream);}#[test]diff --git a/crates/ai-memory-store/tests/suite/retrieval_observation_stream.rs b/crates/ai-memory-store/tests/suite/retrieval_observation_stream.rsindex 5931e6be..8f75b330 100644--- a/crates/ai-memory-store/tests/suite/retrieval_observation_stream.rs+++ b/crates/ai-memory-store/tests/suite/retrieval_observation_stream.rs@@ -1,6 +1,7 @@//! `[retrieval] observation_stream` (R1): matching observation bodies vote,//! once per session, for that session's latest `sessions/<id>.md` page as one-//! more RRF stream. Off by default and byte-identical when off.+//! more RRF stream. On by default, and byte-identical to the pre-stream+//! ranking when turned off.use ai_memory_core::{AgentKind, NewObservation, NewPage, NewSession, ObservationId, ObservationKind, PagePath,@@ -15,6 +16,18 @@ pub(crate) fn stream_on() -> RetrievalTuning {}}+pub(crate) fn stream_off() -> RetrievalTuning {+ RetrievalTuning {+ observation_stream: false,+ ..RetrievalTuning::default()+ }+}++#[test]+fn the_stream_is_on_by_default() {+ assert_eq!(RetrievalTuning::default(), stream_on());+}+pub(crate) async fn scope(store: &Store, ws: &str, proj: &str) -> (WorkspaceId, ProjectId) {let ws = store.writer@@ -161,6 +174,7 @@ async fn an_observation_hit_lifts_its_own_session_page() {observe(&store, ws, proj, other, "we talked about gardening instead").await;// Off: the evidence is unreachable from pages.+ store.reader.set_retrieval_tuning(stream_off());assert!(search(&store, ws, proj, "quasar telescope").await@@ -311,7 +325,8 @@ async fn a_superseded_or_expired_session_page_takes_no_vote() {async fn stream_off_ranks_byte_identically_to_a_store_without_observations() {async fn seeded(with_observations: bool) -> (tempfile::TempDir, Store, WorkspaceId, ProjectId) {let tmp = tempfile::tempdir().unwrap();- let store = Store::open(tmp.path()).unwrap();+ let mut store = Store::open(tmp.path()).unwrap();+ store.reader.set_retrieval_tuning(stream_off());let (ws, proj) = scope(&store, "default", "app").await;for (i, body) in ["kestrel nest under the bridge",diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 8e05adf8..e3c9daca 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -157,13 +157,14 @@ from hook paths.fusion; it is limited to one call per query and four calls in flight, and anyinvalid, failed, timed-out, or saturated attempt preserves the local order.Global and supplemental global-preference results do not take this path.- With the opt-in `[retrieval] observation_stream`, project and `scopes`+ With `[retrieval] observation_stream` (on by default), project and `scopes`searches add one more RRF stream: full-text search over `observations_fts`in the same `(workspace, project)`, where each matching session votes once,through its best-ranked observation, for its own latest unexpired`sessions/<id>.md` page in that scope. The vote never crosses projects (boththe observation and the page are scope-filtered), a session without a page- contributes nothing, and nothing is written. If+ contributes nothing, and nothing is written. Setting it to `false` restores+ the earlier ranking byte for byte. Ifcompiled wiki pages miss entirely in default, explicit project, or explicit`scopes` mode, bounded raw observation FTS returns fallback `raw_hits`;`global=true` searches compiled wiki pages across projects only. Page hits@@ -434,7 +435,7 @@ prunable rather than only after it ages in place.| `pages_fts` | FTS5 virtual table over `(title, body)`, auto-synced by triggers. || `sessions`, `observations` | Sanitized, bounded lifecycle-hook projections. `sessions.ended_observation_count` is the stable generation watermark for resumed-session re-end eligibility; wall clocks are not used for that decision. They are an operational audit trail, not a complete native transcript. || `session_consolidation_jobs` | Durable, observation-generation-idempotent queue for the *automatic* SessionEnd LLM consolidation worker. One bounded server worker leases jobs, retries provider failures with backoff, and recovers expired leases after restart. A manual `memory_consolidate` writes its page out-of-band and then reconciles this table (flipping a `failed`/`pending`/`superseded` row for the session to `completed`, never touching a live `running` lease), so the operator does not see a `failed` job for a session that is in fact consolidated. |-| `observations_fts` | FTS5 virtual table over raw observation `(title, body)`. Feeds the bounded `raw_hits` fallback and, behind the opt-in `[retrieval] observation_stream`, the observation-evidence RRF stream that votes for session pages. |+| `observations_fts` | FTS5 virtual table over raw observation `(title, body)`. Feeds the bounded `raw_hits` fallback and, through `[retrieval] observation_stream` (on by default), the observation-evidence RRF stream that votes for session pages. || `workstreams`, `managed_runs`, `workstream_native_sessions` | Optional lease state plus per-harness native source and delivery cursors for `ai-memory run`. || `workstream_events`, `workstream_events_fts` | Append-only normalized visible transcript events and full-text search; immutable sanitized source batches also live under `raw/workstreams/`. || `links` | Wikilink / markdown cross-references. `to_page_id` (a global PageId) is nullable for unresolved forward links. `to_workspace` / `to_project` carry a cross-project scope (NULL = the source page's own project). |@@ -591,7 +592,7 @@ does not need to ride on the public protocol change.| Tool | Hint | Purpose ||---|---|---|-| `memory_query` | read-only | FTS5 + entity-match + graph RRF + optional vector RRF search, followed by bounded kind/tier/pinned/tag authority adjustment and raw fallback. Bumps access counters for page hits. Defaults to the current project; single-project calls (project implicit or named with `workspace`+`project`) also union the reserved `_global` preferences scope as `global_scope_hits`, and only an explicit multi-`scopes` set opts out (#930); `scopes` searches named sibling projects; `global=true` searches every project at once (each hit annotated with its workspace + project). With `AI_MEMORY_RERANKER=llm`, project/scopes candidate pools are fused before at most one final LLM relevance pass; query/title/snippet data is bounded and JSON-encoded, and any timeout, provider error, invalid/incomplete score set, or four-call concurrency saturation preserves the adjusted order. The distinct `global=true` FTS-only ranker and supplemental global-preference hits are not reranked. `explain=true` attaches per-hit `score_details` (per-stream ranks, matched entities, raw FTS/cosine/entity inverse-frequency scores, the matched observation ids when the opt-in observation stream voted, RRF contributions, graph provenance including the typed edge kind (`causes`/`fixes`/`contradicts`) a neighbour was reached by, the page's evidence count, authority multiplier, and optional rerank score) to project/scopes hits plus a top-level `streams_active` list. The global FTS-only ranker reports its active stream without per-hit details. `include_expired=true` also returns TTL-expired pages. `include_superseded=true` also returns superseded (non-latest) page versions across the FTS/entity/vector/graph streams, each hit labelled `superseded: true` (the current version is never marked); default-off is byte-identical to the latest-only behaviour, and `global=true` / `as_of` are unaffected. `pin_first=true` prepends the project's bounded pinned latest pages (`ReaderPool::list_pinned_pages`, cap 10) ahead of the fused hits, deduped by page id (a pinned page that also matches appears once, marked `pinned: true`) and re-truncated to the requested limit; it applies to single-project searches (default or `workspace`+`project`), is ignored on `scopes`/`global`/`as_of`, and default-off is byte-identical. `answer=true` (opt-in, off by default) additionally synthesizes a cited natural-language answer over the top hits via the configured LLM provider (`complete_structured`, JSON-schema `{ answer, citations }`), attached as `answer: { text, citations }`; with no provider configured it returns the hits plus an `answer_unavailable` note instead of erroring, and with `answer` unset/`false` no provider is accessed and the response is byte-identical (invariant #13). It applies to the normal single-project/`scopes` path; `global`/`as_of` ignore it. Answer quality is not yet eval-validated. An optional `reasoning` tier (`minimal` (default) / `low` / `medium` / `high` / `max`) tunes the synthesis effort: `ChatRequest` carries no per-request reasoning field (the provider-level `reasoning_effort` is fixed at construction from config), so the tier maps to a per-tier max-token budget scaled off the path's base (answer base 2 000; `minimal` = 1x = byte-identical, `low` 1.5x, `medium` 2x, `high` 3x, `max` 4x). The tier is inert unless the `answer` LLM path runs (invariant #13); an unknown value is rejected by the schema (invariant #7). |+| `memory_query` | read-only | FTS5 + entity-match + graph RRF + optional vector RRF search, followed by bounded kind/tier/pinned/tag authority adjustment and raw fallback. Bumps access counters for page hits. Defaults to the current project; single-project calls (project implicit or named with `workspace`+`project`) also union the reserved `_global` preferences scope as `global_scope_hits`, and only an explicit multi-`scopes` set opts out (#930); `scopes` searches named sibling projects; `global=true` searches every project at once (each hit annotated with its workspace + project). With `AI_MEMORY_RERANKER=llm`, project/scopes candidate pools are fused before at most one final LLM relevance pass; query/title/snippet data is bounded and JSON-encoded, and any timeout, provider error, invalid/incomplete score set, or four-call concurrency saturation preserves the adjusted order. The distinct `global=true` FTS-only ranker and supplemental global-preference hits are not reranked. `explain=true` attaches per-hit `score_details` (per-stream ranks, matched entities, raw FTS/cosine/entity inverse-frequency scores, the matched observation ids when the observation stream voted, RRF contributions, graph provenance including the typed edge kind (`causes`/`fixes`/`contradicts`) a neighbour was reached by, the page's evidence count, authority multiplier, and optional rerank score) to project/scopes hits plus a top-level `streams_active` list. The global FTS-only ranker reports its active stream without per-hit details. `include_expired=true` also returns TTL-expired pages. `include_superseded=true` also returns superseded (non-latest) page versions across the FTS/entity/vector/graph streams, each hit labelled `superseded: true` (the current version is never marked); default-off is byte-identical to the latest-only behaviour, and `global=true` / `as_of` are unaffected. `pin_first=true` prepends the project's bounded pinned latest pages (`ReaderPool::list_pinned_pages`, cap 10) ahead of the fused hits, deduped by page id (a pinned page that also matches appears once, marked `pinned: true`) and re-truncated to the requested limit; it applies to single-project searches (default or `workspace`+`project`), is ignored on `scopes`/`global`/`as_of`, and default-off is byte-identical. `answer=true` (opt-in, off by default) additionally synthesizes a cited natural-language answer over the top hits via the configured LLM provider (`complete_structured`, JSON-schema `{ answer, citations }`), attached as `answer: { text, citations }`; with no provider configured it returns the hits plus an `answer_unavailable` note instead of erroring, and with `answer` unset/`false` no provider is accessed and the response is byte-identical (invariant #13). It applies to the normal single-project/`scopes` path; `global`/`as_of` ignore it. Answer quality is not yet eval-validated. An optional `reasoning` tier (`minimal` (default) / `low` / `medium` / `high` / `max`) tunes the synthesis effort: `ChatRequest` carries no per-request reasoning field (the provider-level `reasoning_effort` is fixed at construction from config), so the tier maps to a per-tier max-token budget scaled off the path's base (answer base 2 000; `minimal` = 1x = byte-identical, `low` 1.5x, `medium` 2x, `high` 3x, `max` 4x). The tier is inert unless the `answer` LLM path runs (invariant #13); an unknown value is rejected by the schema (invariant #7). || `memory_recent` | read-only | Most-recently-updated `is_latest=1` pages. || `memory_read_page` | read-only | Fetch the FULL body of a single wiki page by `path` or by top FTS5 hit for a `query`; optional `workspace` + `project` targets a named sibling workspace/project. Use when an agent needs more than the 24-word snippets from `memory_query`. `include_related=true` also walks the link graph outward from the page (bounded BFS reusing the `page_links` primitive per node: default 1 hop, hard cap 3, global visited set for dedup/cycle-safety, total-node cap 50, cross-project aware) and returns a `related` array of reachable pages, each labelled with its `depth` (hop distance) and `direction` (`link`/`backlink`); default-off is byte-identical (no `related` field). || `memory_read_session_observations` | read-only | Page through ONE session's raw hook observations (`ObservationRecord` with full sanitized body, capped per row by `body_max_chars`), restricted to the rows that landed in the resolved scope and to sessions the caller may see; `total` and `elided_other_scope` report the in-scope count and the rows the session left in another project. `session_id` omitted reads the latest completed visible session. |@@ -977,7 +978,7 @@ enabled = false # true: skip low-information session pages from# max_repetition_ratio = 0.7 # 1 - distinct/total tokens above this ⇒ skip# repetition_min_tokens = 6 # repetition check applies only above this-[retrieval] # opt-in ranking signals; all off by default+[retrieval] # ranking signals; only observation_stream is on by defaultquery_intent = false # lexical session-recall routing: queries phrased as# "上次 / …的会话 / last time / yesterday / última sessão"# hand session pages back their default kind/tier authority penalty@@ -993,15 +994,16 @@ belief_authority_weight = 0.0 # fold read-time belief-strength confidence (P# no belief query runs. confidence/evidence_count are still# exposed in explain regardless (inert). DEFAULT OFF,# R2-gated: do not default on without a positive R2 delta.-observation_stream = false # R1 observation-evidence stream: FTS over observations_fts+observation_stream = true # R1 observation-evidence stream: FTS over observations_fts# in the queried project; each matching session votes ONCE# (best observation) for its own latest sessions/<id>.md# page as one more RRF stream. Its excerpt is the snippet# only when page FTS missed that page; explain reports# `observation_rank` + `observation_ids`. Read-only;# sessions without a page keep- # the raw_hits fallback; global=true unchanged. DEFAULT OFF.- # Env: AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true+ # the raw_hits fallback; global=true unchanged. DEFAULT ON;+ # false restores the previous ranking byte for byte.+ # Env: AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false[search.fts] # FTS5 query-preparation tuning (contrast with [retrieval]:# this is not a ranking signal). Omit the whole section fordiff --git a/docs/security-boundaries.md b/docs/security-boundaries.mdindex fa4656ea..2ed58885 100644--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@ -22,7 +22,7 @@ means the boundary is not built yet.| # | Boundary | Enforcing code | Adversarial test(s) | Coverage ||---|----------|----------------|---------------------|----------|-| 1 | Per-project isolation (3-tuple) | `ai-memory-store/src/scope.rs` `ScopeResolver::resolve_read_args`/`resolve_write_args`, no-create `lookup_existing_scope`; reader queries filter by `(workspace_id, project_id)`; `reader.rs` `sessions_for_scope_filtered` joins only consolidation jobs in the requested workspace and project; `reader.rs` `incremental_pages` filters workspace and project before its keyset and limit; `reader.rs` `session_counts_by_agent` filters mixed-source observations by the requested workspace and project; `reader.rs` `observation_stream_votes` (opt-in `[retrieval] observation_stream`) filters both the matching observations and the `sessions/<id>.md` page they vote for to the queried workspace and project; `reader.rs` `page_versions` / `page_version_body` (web page history) read by `(workspace_id, project_id, path)` and take a caller's version id only inside that triple; `ai-memory-web` `routes/session.rs` takes a raw session id only through `session_summary_scoped` (row or observation in the project, owner filter) and `session_observations_scoped` | `store` `scope.rs` read-resolution table tests; `tests/suite/multi_session.rs`; `session_observations.rs` `consolidation_summary_latest_generation_is_scope_and_owner_bound` (a foreign-scope job is absent; the own latest generation and owner controls pass), `incremental_pages_equal_timestamps_cutoff_expiry_and_foreign_scope` (foreign scope absent with a same-scope control), `mixed_capture_count_scopes_observations_and_preserves_owner_and_window` (foreign-scope observations cannot turn an own session into a mixed one; same-scope control can); handoff receiver-session scope: see row 4, `multi_session.rs` `handoff_accept_rejects_foreign_or_missing_receiver_session` (cross-project and cross-workspace receivers refused); `multi_session.rs` `observation_stream_never_crosses_a_project_or_workspace` (a home observation cannot vote for a same-path session page in another project or workspace, and a foreign project's observation cannot lift a home page; control: home's own page receives the vote), fails with either the observation or the page scope filter removed; `multi_session.rs` `page_history_keeps_the_overwritten_version_and_stays_in_scope` (a sibling project's or another workspace's version id, or a home id under another path, reads as `None`; control: both home versions); `ai-memory-web` `history_replay.rs` `history_refuses_foreign_project_workspace_and_version_ids` (another project's or workspace's path, and a foreign version id as `from` or `to`, answer 404 on `/web/w/…/h/*path` and `/api/v1/…/history/*path` without the foreign text; control: the home history lists only home's rows at a path a sibling also uses, and an explicit home pair diffs) and `session_timeline_refuses_foreign_session_ids` (a sibling project's or another workspace's session id and an unknown id → 404, another operator's owned session → 404; controls: the home session, and the owner reading theirs); bite: dropping the scope filter from `page_version_body` or the project filter from `page_versions`, or replacing the `session_summary_scoped` check or its owner filter, fails them | STRONG |+| 1 | Per-project isolation (3-tuple) | `ai-memory-store/src/scope.rs` `ScopeResolver::resolve_read_args`/`resolve_write_args`, no-create `lookup_existing_scope`; reader queries filter by `(workspace_id, project_id)`; `reader.rs` `sessions_for_scope_filtered` joins only consolidation jobs in the requested workspace and project; `reader.rs` `incremental_pages` filters workspace and project before its keyset and limit; `reader.rs` `session_counts_by_agent` filters mixed-source observations by the requested workspace and project; `reader.rs` `observation_stream_votes` (`[retrieval] observation_stream`, on by default) filters both the matching observations and the `sessions/<id>.md` page they vote for to the queried workspace and project; `reader.rs` `page_versions` / `page_version_body` (web page history) read by `(workspace_id, project_id, path)` and take a caller's version id only inside that triple; `ai-memory-web` `routes/session.rs` takes a raw session id only through `session_summary_scoped` (row or observation in the project, owner filter) and `session_observations_scoped` | `store` `scope.rs` read-resolution table tests; `tests/suite/multi_session.rs`; `session_observations.rs` `consolidation_summary_latest_generation_is_scope_and_owner_bound` (a foreign-scope job is absent; the own latest generation and owner controls pass), `incremental_pages_equal_timestamps_cutoff_expiry_and_foreign_scope` (foreign scope absent with a same-scope control), `mixed_capture_count_scopes_observations_and_preserves_owner_and_window` (foreign-scope observations cannot turn an own session into a mixed one; same-scope control can); handoff receiver-session scope: see row 4, `multi_session.rs` `handoff_accept_rejects_foreign_or_missing_receiver_session` (cross-project and cross-workspace receivers refused); `multi_session.rs` `observation_stream_never_crosses_a_project_or_workspace` (a home observation cannot vote for a same-path session page in another project or workspace, and a foreign project's observation cannot lift a home page; control: home's own page receives the vote), fails with either the observation or the page scope filter removed; `multi_session.rs` `page_history_keeps_the_overwritten_version_and_stays_in_scope` (a sibling project's or another workspace's version id, or a home id under another path, reads as `None`; control: both home versions); `ai-memory-web` `history_replay.rs` `history_refuses_foreign_project_workspace_and_version_ids` (another project's or workspace's path, and a foreign version id as `from` or `to`, answer 404 on `/web/w/…/h/*path` and `/api/v1/…/history/*path` without the foreign text; control: the home history lists only home's rows at a path a sibling also uses, and an explicit home pair diffs) and `session_timeline_refuses_foreign_session_ids` (a sibling project's or another workspace's session id and an unknown id → 404, another operator's owned session → 404; controls: the home session, and the owner reading theirs); bite: dropping the scope filter from `page_version_body` or the project filter from `page_versions`, or replacing the `session_summary_scoped` check or its owner filter, fails them | STRONG || 1b | Reserved `_global` scope union never leaks a foreign project (#930) | `ai-memory-mcp/src/server.rs` `memory_query` — the union runs for single-project queries (`scopes` empty) and searches **only** `lookup_global_scope`'s reserved `(workspace_id, project_id)`, never an arbitrary project; the double-search guard resolves the *queried* project (named or active) so it can't be tricked into skipping | `server.rs` `global_union_never_leaks_a_foreign_projects_pages` (a third real project's page must not surface via the union; reserved page must), `single_project_query_unions_global_scope_and_multi_scope_skips_it` | STRONG || 1c | Reserved `_global` scope is read-open, write-gated (GHSA-7qj3-7wqw-m5w6) | `ai-memory-store/src/project_authz.rs` `ProjectAuthz::authorize` (a write to `default/_global` needs root or a `write` grant in either access mode; creator ignored) fed by `resolve_project_authz`/`read_project_row` (`reserved_global`, grant read for the global scope regardless of mode); `ai-memory-mcp/src/server.rs` `memory_write_page` resolves `scope: "global"` through `write_target_ids_with_actor` like an explicit `default/_global` | `project_authz.rs` `the_global_scope_admits_writes_only_from_root_or_a_write_grant`; `server.rs` `a_user_without_a_global_grant_cannot_change_the_global_scope` (both spellings and delete refused for an ungranted user and for `_global`'s creator; page survives; reads stay open), `a_global_write_grant_admits_writes_and_the_union_stays_readable` (legitimate control: write grant admits both spellings, another user's restricted-project query still unions the page); bite-checked with the authz rule removed and with the raw `create_global_scope` path restored | STRONG || 1d | A private cross-project profile is readable and writable by its operator alone (`[profile] share = "user"`) | `ai-memory-store/src/ops.rs` `initial_access_mode` (a `_profile.<user id>` project is always created `restricted`, creator recorded in the same transaction); `ai-memory-store/src/scope.rs` `lookup_profile_scope` / `create_profile_scope` (the project name derives from the authenticated viewer's id; no database user, no private profile); `ai-memory-mcp/src/server.rs` `profile_target_ids` and the `memory_query` profile union (resolved per viewer through the per-project choke point); `scope.rs` `refuse_foreign_profile_name` (called by `ScopeResolver` explicit writes and `create_explicit_scope_guarded`: a database user may name only their own `default/_profile.<id>`, so nobody can create another operator's private profile first and lock its owner out); `ops.rs` `rename_project` (no rename onto or away from a profile name); `admin.rs` `handle_project_access` (a profile project's mode is fixed) and `move_project_core` (a private profile cannot leave `default`) | store `integration::profile` `a_private_profile_refuses_every_other_operator` (another operator's read and write by name refused; owner control admitted), `reserved_profile_projects_get_their_fixed_access_modes`; `server.rs` `a_private_profile_is_private_to_its_operator` (no union leak; read, write and delete by name refused; owner and own-profile controls); `multi_session.rs` `two_harnesses_of_one_operator_share_one_private_profile`; both fail with the restricted-mode guard removed; `server.rs` `a_user_cannot_squat_another_operators_private_profile` (another user's explicit write naming the owner's private profile refused and creates nothing; owner's profile and explicit own-name controls; fails with the name guard removed); `ops.rs` `rename_project_refuses_profile_names_both_ways`; `admin.rs` `project_access_sets_the_mode_and_refuses_what_it_must`; `admin_move.rs` `move_project_refuses_a_private_profile` | STRONG |