Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(retrieval): turn the observation stream on by default

AkitaOnRailsa389873a3385
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 58f5c056..d2ee7896 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -29,9 +29,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`ai-memory-web` crate now has a `clippy.toml` that forbids the `Wiki` write
and destructive methods, so a write path in the web UI or `/api/v1` fails
`cargo clippy -D warnings`.
-- Added an opt-in observation stream to `memory_query` ranking:
- `[retrieval] observation_stream = true` (env
- `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true`, default off). A zero-LLM
+- Added an observation stream to `memory_query` ranking:
+ `[retrieval] observation_stream` (env
+ `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM`, on by default). A zero-LLM
session page keeps only a short excerpt of each turn, so the detail a
question asks about often lives only in the session's observations. With the
flag on, project and `scopes` searches also run full-text search over
@@ -211,6 +211,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
in `docs/support-matrix.md` as `MCP-only (documented)`: each recipe follows
the client's own documentation at a named version and has not yet been run
end to end. Aider is not listed because it has no MCP support.
+- Changed `memory_query` ranking to include the observation stream by
+ default, after it raised hit@5 on the retrieval eval for both zero-LLM and
+ local-embedding search while staying inside its latency and context-token
+ budgets. `explain=true` lists `observation` in
+ `streams_active` whenever it runs. To keep the previous ranking, set
+ `[retrieval] observation_stream = false` (or
+ `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false`); ranking is then identical
+ to before.
- Changed `ai-memory-wikisync` files to carry a small frontmatter (`title`,
plus `tags`, `pinned` and a non-default `tier` when set) above the body,
so metadata survives a round trip. The first run after upgrading rewrites
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index e0f2f45c..9e81d792 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -1636,13 +1636,14 @@ impl DreamSettings {
}
}
-/// `[retrieval]` opt-in ranking signals layered on the RRF fusion in
-/// `memory_query`. Every default leaves ranking byte-identical to a store
+/// `[retrieval]` ranking signals layered on the RRF fusion in
+/// `memory_query`. Only `observation_stream` is on by default; with it
+/// turned off, every other default leaves ranking byte-identical to a store
/// that never heard of this section.
///
/// Env form: `AI_MEMORY_RETRIEVAL__QUERY_INTENT=true`,
/// `AI_MEMORY_RETRIEVAL__ABSTRACT_VECTORS=true`,
-/// `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true`.
+/// `AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false`.
#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
#[serde(default)]
pub struct RetrievalSettings {
@@ -1667,7 +1668,7 @@ pub struct RetrievalSettings {
/// Add the observation-evidence stream (R1) to the RRF fusion: matching
/// observation bodies vote, once per session, for that session's latest
/// `sessions/<id>.md` page in the same project. Read-only; the wiki is
- /// not enlarged. OFF by default.
+ /// not enlarged. ON by default; `false` restores the previous ranking.
pub observation_stream: bool,
}
@@ -3315,15 +3316,19 @@ mod tests {
}
#[test]
- fn retrieval_observation_stream_defaults_off_and_loads_from_toml() {
- assert!(!Config::default().retrieval.tuning().observation_stream);
+ fn retrieval_observation_stream_defaults_on_and_loads_from_toml() {
+ assert!(Config::default().retrieval.tuning().observation_stream);
let tmp = TempDir::new().unwrap();
let config_path = tmp.path().join("config.toml");
- std::fs::write(&config_path, "[retrieval]\nobservation_stream = true\n").unwrap();
+ std::fs::write(&config_path, "[retrieval]\nobservation_stream = false\n").unwrap();
let config = Config::load(Some(&config_path), Some(tmp.path().to_path_buf())).unwrap();
- assert!(config.retrieval.tuning().observation_stream);
+ assert!(!config.retrieval.tuning().observation_stream);
// The other [retrieval] signals keep their defaults.
assert!(!config.retrieval.tuning().abstract_vectors);
+ // A config that never names the key gets the default.
+ std::fs::write(&config_path, "[retrieval]\nabstract_vectors = false\n").unwrap();
+ let config = Config::load(Some(&config_path), Some(tmp.path().to_path_buf())).unwrap();
+ assert!(config.retrieval.tuning().observation_stream);
}
#[test]
@@ -3332,9 +3337,9 @@ mod tests {
if std::env::var_os(CHILD_MARKER).is_some() {
let tmp = TempDir::new().unwrap();
let config_path = tmp.path().join("config.toml");
- std::fs::write(&config_path, "[retrieval]\nobservation_stream = false\n").unwrap();
+ std::fs::write(&config_path, "[retrieval]\nobservation_stream = true\n").unwrap();
let config = Config::load(Some(&config_path), Some(tmp.path().to_path_buf())).unwrap();
- assert!(config.retrieval.tuning().observation_stream);
+ assert!(!config.retrieval.tuning().observation_stream);
println!("observation-stream-env-applied");
return;
}
@@ -3345,7 +3350,7 @@ mod tests {
.arg("--test-threads=1")
.arg("--nocapture")
.env(CHILD_MARKER, "1")
- .env("AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM", "true")
+ .env("AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM", "false")
.output()
.unwrap();
assert!(
diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rs
index 9ed2b5b0..5189d35e 100644
--- a/crates/ai-memory-mcp/src/server.rs
+++ b/crates/ai-memory-mcp/src/server.rs
@@ -848,7 +848,9 @@ struct MemoryQueryResponse {
/// Present only when `explain=true`: which retrieval streams ran for
/// the primary search. Project/scopes retrieval always runs `fts` and
/// `entity`, and `graph`; `vector` is present only when an embedder
- /// produced a query vector. Cross-project `global=true` retrieval is FTS-only.
+ /// produced a query vector, and `observation` while `[retrieval]
+ /// observation_stream` is on (the default). Cross-project `global=true`
+ /// retrieval is FTS-only.
/// An `as_of` query runs `entity` plus version-filtered `fts`.
#[serde(skip_serializing_if = "Option::is_none")]
streams_active: Option<Vec<&'static str>>,
@@ -10424,7 +10426,7 @@ mod tests {
let value: serde_json::Value = serde_json::from_str(&text.text).unwrap();
assert_eq!(
value["streams_active"],
- serde_json::json!(["fts", "entity", "graph"])
+ serde_json::json!(["fts", "entity", "graph", "observation"])
);
let details = &value["hits"][0]["score_details"];
assert_eq!(details["fts_rank"], 1);
@@ -10545,7 +10547,7 @@ mod tests {
);
assert_eq!(
entity_hit["streams_active"],
- serde_json::json!(["fts", "entity", "graph"])
+ serde_json::json!(["fts", "entity", "graph", "observation"])
);
let graph_hit = query("graphseed").await;
@@ -12848,7 +12850,7 @@ mod tests {
let value: serde_json::Value = serde_json::from_str(&text).unwrap();
assert_eq!(
value["streams_active"],
- serde_json::json!(["fts", "entity", "graph"])
+ serde_json::json!(["fts", "entity", "graph", "observation"])
);
assert_eq!(
value["hits"]
diff --git a/crates/ai-memory-mcp/tests/suite/query_observation_stream.rs b/crates/ai-memory-mcp/tests/suite/query_observation_stream.rs
index bbe0f260..ee7008ec 100644
--- a/crates/ai-memory-mcp/tests/suite/query_observation_stream.rs
+++ b/crates/ai-memory-mcp/tests/suite/query_observation_stream.rs
@@ -7,6 +7,7 @@
//! the session page (hits stay pages), explains the vote with an
//! `observation` stream entry, and shows the matching observation excerpt.
//! With it off, the same query finds no page and falls back to `raw_hits`.
+//! The stream is on by default.
use ai_memory_core::{
AgentKind, NewObservation, NewPage, NewSession, ObservationKind, PagePath, ProjectId,
@@ -34,18 +35,21 @@ struct Seeded {
}
async fn seeded(observation_stream: bool) -> Seeded {
- seeded_with(observation_stream, true).await
+ seeded_with(Some(observation_stream), true).await
}
-async fn seeded_with(observation_stream: bool, session_page: bool) -> Seeded {
+/// `observation_stream: None` keeps the store's default tuning.
+async fn seeded_with(observation_stream: Option<bool>, session_page: bool) -> Seeded {
let tmp = TempDir::new().expect("tempdir");
let mut store = Store::open(tmp.path()).expect("store");
// The server reads its tuning once at construction, as `serve` does from
// `Config::load`.
- store.reader.set_retrieval_tuning(RetrievalTuning {
- observation_stream,
- ..RetrievalTuning::default()
- });
+ if let Some(observation_stream) = observation_stream {
+ store.reader.set_retrieval_tuning(RetrievalTuning {
+ observation_stream,
+ ..RetrievalTuning::default()
+ });
+ }
let ws = store.writer.get_or_create_workspace(WS).await.expect("ws");
let proj = store
.writer
@@ -240,6 +244,23 @@ async fn memory_query_finds_a_session_page_by_its_observation_evidence() {
}
}
+#[tokio::test]
+async fn memory_query_ranks_with_the_observation_stream_by_default() {
+ let seeded = seeded_with(None, true).await;
+ let payload = query(&seeded.router, args()).await;
+
+ let session_path = format!("sessions/{}.md", seeded.session_id);
+ let hit = payload["hits"]
+ .as_array()
+ .expect("hits")
+ .iter()
+ .find(|h| h["path"] == session_path.as_str())
+ .unwrap_or_else(|| panic!("session page missing: {payload}"));
+ assert_eq!(hit["score_details"]["observation_rank"], 1, "{payload}");
+ let streams = payload["streams_active"].as_array().expect("streams");
+ assert!(streams.contains(&json!("observation")), "{streams:?}");
+}
+
#[tokio::test]
async fn memory_query_without_the_stream_never_reports_an_observation_vote() {
let seeded = seeded(false).await;
@@ -262,7 +283,7 @@ async fn memory_query_without_the_stream_never_reports_an_observation_vote() {
#[tokio::test]
async fn a_live_session_without_a_page_still_reaches_the_raw_fallback() {
- let seeded = seeded_with(true, false).await;
+ let seeded = seeded_with(Some(true), false).await;
let payload = query(
&seeded.router,
json!({
diff --git a/crates/ai-memory-store/src/retrieval_tuning.rs b/crates/ai-memory-store/src/retrieval_tuning.rs
index b0d6871c..8b2a523f 100644
--- a/crates/ai-memory-store/src/retrieval_tuning.rs
+++ b/crates/ai-memory-store/src/retrieval_tuning.rs
@@ -1,7 +1,8 @@
-//! Opt-in post-fusion ranking signals for `hybrid_search`.
+//! Post-fusion ranking signals for `hybrid_search`.
//!
-//! Bounded, independently switchable signals, all off by default so a
-//! store that never configures them ranks exactly as before:
+//! Bounded, independently switchable signals. All but the observation stream
+//! are off by default; with every one of them off a store ranks exactly as it
+//! did before any of them existed:
//!
//! * **Session-recall routing** — a zero-LLM lexical check that recognises
//! queries phrased as "find a past session / what did we do back then"
@@ -24,14 +25,16 @@
//! the evidence a question needs often lives only in the session's
//! sanitized observation bodies. Each session's best-ranked matching
//! observation votes once for that session's latest page; a session with
-//! no page contributes nothing (the raw fallback still covers it).
+//! no page contributes nothing (the raw fallback still covers it). On by
+//! default since it met its measured success criteria.
//!
//! Every signal is reported in `SearchExplain`, so an `explain=true`
//! query can account for the returned rank.
use serde::{Deserialize, Serialize};
-/// Tunables for the opt-in ranking signals. `Default` disables all of them.
+/// Tunables for the ranking signals. `Default` enables only the observation
+/// stream.
#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
#[serde(default)]
pub struct RetrievalTuning {
@@ -59,8 +62,9 @@ pub struct RetrievalTuning {
/// Add the observation-evidence stream to the RRF fusion (R1): search
/// `observations_fts` in the same project and let each matching
/// observation vote, once per session, for its session's latest
- /// `sessions/<id>.md` page. Nothing is written. Ships OFF: it changes
- /// ranking, and the maintainer decides default-on from measurements.
+ /// `sessions/<id>.md` page. Nothing is written. ON by default: on the
+ /// retrieval eval it lifted zero-LLM hit@5 without breaking the latency or
+ /// token budgets. `false` restores the previous ranking byte for byte.
pub observation_stream: bool,
}
@@ -79,7 +83,7 @@ impl Default for RetrievalTuning {
// OFF by default: folding belief confidence into ranking is
// R2-gated (see the field doc).
belief_authority_weight: 0.0,
- observation_stream: false,
+ observation_stream: true,
}
}
}
@@ -180,12 +184,12 @@ mod tests {
use super::*;
#[test]
- fn default_tuning_is_inert() {
+ fn default_tuning_enables_only_the_observation_stream() {
let t = RetrievalTuning::default();
assert!(!t.session_recall_routing);
assert!(!t.abstract_vectors);
assert_eq!(t.belief_authority_weight, 0.0);
- assert!(!t.observation_stream);
+ assert!(t.observation_stream);
}
#[test]
diff --git a/crates/ai-memory-store/tests/suite/retrieval_observation_stream.rs b/crates/ai-memory-store/tests/suite/retrieval_observation_stream.rs
index 5931e6be..8f75b330 100644
--- a/crates/ai-memory-store/tests/suite/retrieval_observation_stream.rs
+++ b/crates/ai-memory-store/tests/suite/retrieval_observation_stream.rs
@@ -1,6 +1,7 @@
//! `[retrieval] observation_stream` (R1): matching observation bodies vote,
//! once per session, for that session's latest `sessions/<id>.md` page as one
-//! more RRF stream. Off by default and byte-identical when off.
+//! more RRF stream. On by default, and byte-identical to the pre-stream
+//! ranking when turned off.
use ai_memory_core::{
AgentKind, NewObservation, NewPage, NewSession, ObservationId, ObservationKind, PagePath,
@@ -15,6 +16,18 @@ pub(crate) fn stream_on() -> RetrievalTuning {
}
}
+pub(crate) fn stream_off() -> RetrievalTuning {
+ RetrievalTuning {
+ observation_stream: false,
+ ..RetrievalTuning::default()
+ }
+}
+
+#[test]
+fn the_stream_is_on_by_default() {
+ assert_eq!(RetrievalTuning::default(), stream_on());
+}
+
pub(crate) async fn scope(store: &Store, ws: &str, proj: &str) -> (WorkspaceId, ProjectId) {
let ws = store
.writer
@@ -161,6 +174,7 @@ async fn an_observation_hit_lifts_its_own_session_page() {
observe(&store, ws, proj, other, "we talked about gardening instead").await;
// Off: the evidence is unreachable from pages.
+ store.reader.set_retrieval_tuning(stream_off());
assert!(
search(&store, ws, proj, "quasar telescope")
.await
@@ -311,7 +325,8 @@ async fn a_superseded_or_expired_session_page_takes_no_vote() {
async fn stream_off_ranks_byte_identically_to_a_store_without_observations() {
async fn seeded(with_observations: bool) -> (tempfile::TempDir, Store, WorkspaceId, ProjectId) {
let tmp = tempfile::tempdir().unwrap();
- let store = Store::open(tmp.path()).unwrap();
+ let mut store = Store::open(tmp.path()).unwrap();
+ store.reader.set_retrieval_tuning(stream_off());
let (ws, proj) = scope(&store, "default", "app").await;
for (i, body) in [
"kestrel nest under the bridge",
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 8e05adf8..e3c9daca 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -157,13 +157,14 @@ from hook paths.
fusion; it is limited to one call per query and four calls in flight, and any
invalid, failed, timed-out, or saturated attempt preserves the local order.
Global and supplemental global-preference results do not take this path.
- With the opt-in `[retrieval] observation_stream`, project and `scopes`
+ With `[retrieval] observation_stream` (on by default), project and `scopes`
searches add one more RRF stream: full-text search over `observations_fts`
in the same `(workspace, project)`, where each matching session votes once,
through its best-ranked observation, for its own latest unexpired
`sessions/<id>.md` page in that scope. The vote never crosses projects (both
the observation and the page are scope-filtered), a session without a page
- contributes nothing, and nothing is written. If
+ contributes nothing, and nothing is written. Setting it to `false` restores
+ the earlier ranking byte for byte. If
compiled wiki pages miss entirely in default, explicit project, or explicit
`scopes` mode, bounded raw observation FTS returns fallback `raw_hits`;
`global=true` searches compiled wiki pages across projects only. Page hits
@@ -434,7 +435,7 @@ prunable rather than only after it ages in place.
| `pages_fts` | FTS5 virtual table over `(title, body)`, auto-synced by triggers. |
| `sessions`, `observations` | Sanitized, bounded lifecycle-hook projections. `sessions.ended_observation_count` is the stable generation watermark for resumed-session re-end eligibility; wall clocks are not used for that decision. They are an operational audit trail, not a complete native transcript. |
| `session_consolidation_jobs` | Durable, observation-generation-idempotent queue for the *automatic* SessionEnd LLM consolidation worker. One bounded server worker leases jobs, retries provider failures with backoff, and recovers expired leases after restart. A manual `memory_consolidate` writes its page out-of-band and then reconciles this table (flipping a `failed`/`pending`/`superseded` row for the session to `completed`, never touching a live `running` lease), so the operator does not see a `failed` job for a session that is in fact consolidated. |
-| `observations_fts` | FTS5 virtual table over raw observation `(title, body)`. Feeds the bounded `raw_hits` fallback and, behind the opt-in `[retrieval] observation_stream`, the observation-evidence RRF stream that votes for session pages. |
+| `observations_fts` | FTS5 virtual table over raw observation `(title, body)`. Feeds the bounded `raw_hits` fallback and, through `[retrieval] observation_stream` (on by default), the observation-evidence RRF stream that votes for session pages. |
| `workstreams`, `managed_runs`, `workstream_native_sessions` | Optional lease state plus per-harness native source and delivery cursors for `ai-memory run`. |
| `workstream_events`, `workstream_events_fts` | Append-only normalized visible transcript events and full-text search; immutable sanitized source batches also live under `raw/workstreams/`. |
| `links` | Wikilink / markdown cross-references. `to_page_id` (a global PageId) is nullable for unresolved forward links. `to_workspace` / `to_project` carry a cross-project scope (NULL = the source page's own project). |
@@ -591,7 +592,7 @@ does not need to ride on the public protocol change.
| Tool | Hint | Purpose |
|---|---|---|
-| `memory_query` | read-only | FTS5 + entity-match + graph RRF + optional vector RRF search, followed by bounded kind/tier/pinned/tag authority adjustment and raw fallback. Bumps access counters for page hits. Defaults to the current project; single-project calls (project implicit or named with `workspace`+`project`) also union the reserved `_global` preferences scope as `global_scope_hits`, and only an explicit multi-`scopes` set opts out (#930); `scopes` searches named sibling projects; `global=true` searches every project at once (each hit annotated with its workspace + project). With `AI_MEMORY_RERANKER=llm`, project/scopes candidate pools are fused before at most one final LLM relevance pass; query/title/snippet data is bounded and JSON-encoded, and any timeout, provider error, invalid/incomplete score set, or four-call concurrency saturation preserves the adjusted order. The distinct `global=true` FTS-only ranker and supplemental global-preference hits are not reranked. `explain=true` attaches per-hit `score_details` (per-stream ranks, matched entities, raw FTS/cosine/entity inverse-frequency scores, the matched observation ids when the opt-in observation stream voted, RRF contributions, graph provenance including the typed edge kind (`causes`/`fixes`/`contradicts`) a neighbour was reached by, the page's evidence count, authority multiplier, and optional rerank score) to project/scopes hits plus a top-level `streams_active` list. The global FTS-only ranker reports its active stream without per-hit details. `include_expired=true` also returns TTL-expired pages. `include_superseded=true` also returns superseded (non-latest) page versions across the FTS/entity/vector/graph streams, each hit labelled `superseded: true` (the current version is never marked); default-off is byte-identical to the latest-only behaviour, and `global=true` / `as_of` are unaffected. `pin_first=true` prepends the project's bounded pinned latest pages (`ReaderPool::list_pinned_pages`, cap 10) ahead of the fused hits, deduped by page id (a pinned page that also matches appears once, marked `pinned: true`) and re-truncated to the requested limit; it applies to single-project searches (default or `workspace`+`project`), is ignored on `scopes`/`global`/`as_of`, and default-off is byte-identical. `answer=true` (opt-in, off by default) additionally synthesizes a cited natural-language answer over the top hits via the configured LLM provider (`complete_structured`, JSON-schema `{ answer, citations }`), attached as `answer: { text, citations }`; with no provider configured it returns the hits plus an `answer_unavailable` note instead of erroring, and with `answer` unset/`false` no provider is accessed and the response is byte-identical (invariant #13). It applies to the normal single-project/`scopes` path; `global`/`as_of` ignore it. Answer quality is not yet eval-validated. An optional `reasoning` tier (`minimal` (default) / `low` / `medium` / `high` / `max`) tunes the synthesis effort: `ChatRequest` carries no per-request reasoning field (the provider-level `reasoning_effort` is fixed at construction from config), so the tier maps to a per-tier max-token budget scaled off the path's base (answer base 2 000; `minimal` = 1x = byte-identical, `low` 1.5x, `medium` 2x, `high` 3x, `max` 4x). The tier is inert unless the `answer` LLM path runs (invariant #13); an unknown value is rejected by the schema (invariant #7). |
+| `memory_query` | read-only | FTS5 + entity-match + graph RRF + optional vector RRF search, followed by bounded kind/tier/pinned/tag authority adjustment and raw fallback. Bumps access counters for page hits. Defaults to the current project; single-project calls (project implicit or named with `workspace`+`project`) also union the reserved `_global` preferences scope as `global_scope_hits`, and only an explicit multi-`scopes` set opts out (#930); `scopes` searches named sibling projects; `global=true` searches every project at once (each hit annotated with its workspace + project). With `AI_MEMORY_RERANKER=llm`, project/scopes candidate pools are fused before at most one final LLM relevance pass; query/title/snippet data is bounded and JSON-encoded, and any timeout, provider error, invalid/incomplete score set, or four-call concurrency saturation preserves the adjusted order. The distinct `global=true` FTS-only ranker and supplemental global-preference hits are not reranked. `explain=true` attaches per-hit `score_details` (per-stream ranks, matched entities, raw FTS/cosine/entity inverse-frequency scores, the matched observation ids when the observation stream voted, RRF contributions, graph provenance including the typed edge kind (`causes`/`fixes`/`contradicts`) a neighbour was reached by, the page's evidence count, authority multiplier, and optional rerank score) to project/scopes hits plus a top-level `streams_active` list. The global FTS-only ranker reports its active stream without per-hit details. `include_expired=true` also returns TTL-expired pages. `include_superseded=true` also returns superseded (non-latest) page versions across the FTS/entity/vector/graph streams, each hit labelled `superseded: true` (the current version is never marked); default-off is byte-identical to the latest-only behaviour, and `global=true` / `as_of` are unaffected. `pin_first=true` prepends the project's bounded pinned latest pages (`ReaderPool::list_pinned_pages`, cap 10) ahead of the fused hits, deduped by page id (a pinned page that also matches appears once, marked `pinned: true`) and re-truncated to the requested limit; it applies to single-project searches (default or `workspace`+`project`), is ignored on `scopes`/`global`/`as_of`, and default-off is byte-identical. `answer=true` (opt-in, off by default) additionally synthesizes a cited natural-language answer over the top hits via the configured LLM provider (`complete_structured`, JSON-schema `{ answer, citations }`), attached as `answer: { text, citations }`; with no provider configured it returns the hits plus an `answer_unavailable` note instead of erroring, and with `answer` unset/`false` no provider is accessed and the response is byte-identical (invariant #13). It applies to the normal single-project/`scopes` path; `global`/`as_of` ignore it. Answer quality is not yet eval-validated. An optional `reasoning` tier (`minimal` (default) / `low` / `medium` / `high` / `max`) tunes the synthesis effort: `ChatRequest` carries no per-request reasoning field (the provider-level `reasoning_effort` is fixed at construction from config), so the tier maps to a per-tier max-token budget scaled off the path's base (answer base 2 000; `minimal` = 1x = byte-identical, `low` 1.5x, `medium` 2x, `high` 3x, `max` 4x). The tier is inert unless the `answer` LLM path runs (invariant #13); an unknown value is rejected by the schema (invariant #7). |
| `memory_recent` | read-only | Most-recently-updated `is_latest=1` pages. |
| `memory_read_page` | read-only | Fetch the FULL body of a single wiki page by `path` or by top FTS5 hit for a `query`; optional `workspace` + `project` targets a named sibling workspace/project. Use when an agent needs more than the 24-word snippets from `memory_query`. `include_related=true` also walks the link graph outward from the page (bounded BFS reusing the `page_links` primitive per node: default 1 hop, hard cap 3, global visited set for dedup/cycle-safety, total-node cap 50, cross-project aware) and returns a `related` array of reachable pages, each labelled with its `depth` (hop distance) and `direction` (`link`/`backlink`); default-off is byte-identical (no `related` field). |
| `memory_read_session_observations` | read-only | Page through ONE session's raw hook observations (`ObservationRecord` with full sanitized body, capped per row by `body_max_chars`), restricted to the rows that landed in the resolved scope and to sessions the caller may see; `total` and `elided_other_scope` report the in-scope count and the rows the session left in another project. `session_id` omitted reads the latest completed visible session. |
@@ -977,7 +978,7 @@ enabled = false # true: skip low-information session pages from
# max_repetition_ratio = 0.7 # 1 - distinct/total tokens above this ⇒ skip
# repetition_min_tokens = 6 # repetition check applies only above this
-[retrieval] # opt-in ranking signals; all off by default
+[retrieval] # ranking signals; only observation_stream is on by default
query_intent = false # lexical session-recall routing: queries phrased as
# "上次 / …的会话 / last time / yesterday / última sessão"
# hand session pages back their default kind/tier authority penalty
@@ -993,15 +994,16 @@ belief_authority_weight = 0.0 # fold read-time belief-strength confidence (P
# no belief query runs. confidence/evidence_count are still
# exposed in explain regardless (inert). DEFAULT OFF,
# R2-gated: do not default on without a positive R2 delta.
-observation_stream = false # R1 observation-evidence stream: FTS over observations_fts
+observation_stream = true # R1 observation-evidence stream: FTS over observations_fts
# in the queried project; each matching session votes ONCE
# (best observation) for its own latest sessions/<id>.md
# page as one more RRF stream. Its excerpt is the snippet
# only when page FTS missed that page; explain reports
# `observation_rank` + `observation_ids`. Read-only;
# sessions without a page keep
- # the raw_hits fallback; global=true unchanged. DEFAULT OFF.
- # Env: AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=true
+ # the raw_hits fallback; global=true unchanged. DEFAULT ON;
+ # false restores the previous ranking byte for byte.
+ # Env: AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false
[search.fts] # FTS5 query-preparation tuning (contrast with [retrieval]:
# this is not a ranking signal). Omit the whole section for
diff --git a/docs/security-boundaries.md b/docs/security-boundaries.md
index fa4656ea..2ed58885 100644
--- a/docs/security-boundaries.md
+++ b/docs/security-boundaries.md
@@ -22,7 +22,7 @@ means the boundary is not built yet.
| # | Boundary | Enforcing code | Adversarial test(s) | Coverage |
|---|----------|----------------|---------------------|----------|
-| 1 | Per-project isolation (3-tuple) | `ai-memory-store/src/scope.rs` `ScopeResolver::resolve_read_args`/`resolve_write_args`, no-create `lookup_existing_scope`; reader queries filter by `(workspace_id, project_id)`; `reader.rs` `sessions_for_scope_filtered` joins only consolidation jobs in the requested workspace and project; `reader.rs` `incremental_pages` filters workspace and project before its keyset and limit; `reader.rs` `session_counts_by_agent` filters mixed-source observations by the requested workspace and project; `reader.rs` `observation_stream_votes` (opt-in `[retrieval] observation_stream`) filters both the matching observations and the `sessions/<id>.md` page they vote for to the queried workspace and project; `reader.rs` `page_versions` / `page_version_body` (web page history) read by `(workspace_id, project_id, path)` and take a caller's version id only inside that triple; `ai-memory-web` `routes/session.rs` takes a raw session id only through `session_summary_scoped` (row or observation in the project, owner filter) and `session_observations_scoped` | `store` `scope.rs` read-resolution table tests; `tests/suite/multi_session.rs`; `session_observations.rs` `consolidation_summary_latest_generation_is_scope_and_owner_bound` (a foreign-scope job is absent; the own latest generation and owner controls pass), `incremental_pages_equal_timestamps_cutoff_expiry_and_foreign_scope` (foreign scope absent with a same-scope control), `mixed_capture_count_scopes_observations_and_preserves_owner_and_window` (foreign-scope observations cannot turn an own session into a mixed one; same-scope control can); handoff receiver-session scope: see row 4, `multi_session.rs` `handoff_accept_rejects_foreign_or_missing_receiver_session` (cross-project and cross-workspace receivers refused); `multi_session.rs` `observation_stream_never_crosses_a_project_or_workspace` (a home observation cannot vote for a same-path session page in another project or workspace, and a foreign project's observation cannot lift a home page; control: home's own page receives the vote), fails with either the observation or the page scope filter removed; `multi_session.rs` `page_history_keeps_the_overwritten_version_and_stays_in_scope` (a sibling project's or another workspace's version id, or a home id under another path, reads as `None`; control: both home versions); `ai-memory-web` `history_replay.rs` `history_refuses_foreign_project_workspace_and_version_ids` (another project's or workspace's path, and a foreign version id as `from` or `to`, answer 404 on `/web/w/…/h/*path` and `/api/v1/…/history/*path` without the foreign text; control: the home history lists only home's rows at a path a sibling also uses, and an explicit home pair diffs) and `session_timeline_refuses_foreign_session_ids` (a sibling project's or another workspace's session id and an unknown id → 404, another operator's owned session → 404; controls: the home session, and the owner reading theirs); bite: dropping the scope filter from `page_version_body` or the project filter from `page_versions`, or replacing the `session_summary_scoped` check or its owner filter, fails them | STRONG |
+| 1 | Per-project isolation (3-tuple) | `ai-memory-store/src/scope.rs` `ScopeResolver::resolve_read_args`/`resolve_write_args`, no-create `lookup_existing_scope`; reader queries filter by `(workspace_id, project_id)`; `reader.rs` `sessions_for_scope_filtered` joins only consolidation jobs in the requested workspace and project; `reader.rs` `incremental_pages` filters workspace and project before its keyset and limit; `reader.rs` `session_counts_by_agent` filters mixed-source observations by the requested workspace and project; `reader.rs` `observation_stream_votes` (`[retrieval] observation_stream`, on by default) filters both the matching observations and the `sessions/<id>.md` page they vote for to the queried workspace and project; `reader.rs` `page_versions` / `page_version_body` (web page history) read by `(workspace_id, project_id, path)` and take a caller's version id only inside that triple; `ai-memory-web` `routes/session.rs` takes a raw session id only through `session_summary_scoped` (row or observation in the project, owner filter) and `session_observations_scoped` | `store` `scope.rs` read-resolution table tests; `tests/suite/multi_session.rs`; `session_observations.rs` `consolidation_summary_latest_generation_is_scope_and_owner_bound` (a foreign-scope job is absent; the own latest generation and owner controls pass), `incremental_pages_equal_timestamps_cutoff_expiry_and_foreign_scope` (foreign scope absent with a same-scope control), `mixed_capture_count_scopes_observations_and_preserves_owner_and_window` (foreign-scope observations cannot turn an own session into a mixed one; same-scope control can); handoff receiver-session scope: see row 4, `multi_session.rs` `handoff_accept_rejects_foreign_or_missing_receiver_session` (cross-project and cross-workspace receivers refused); `multi_session.rs` `observation_stream_never_crosses_a_project_or_workspace` (a home observation cannot vote for a same-path session page in another project or workspace, and a foreign project's observation cannot lift a home page; control: home's own page receives the vote), fails with either the observation or the page scope filter removed; `multi_session.rs` `page_history_keeps_the_overwritten_version_and_stays_in_scope` (a sibling project's or another workspace's version id, or a home id under another path, reads as `None`; control: both home versions); `ai-memory-web` `history_replay.rs` `history_refuses_foreign_project_workspace_and_version_ids` (another project's or workspace's path, and a foreign version id as `from` or `to`, answer 404 on `/web/w/…/h/*path` and `/api/v1/…/history/*path` without the foreign text; control: the home history lists only home's rows at a path a sibling also uses, and an explicit home pair diffs) and `session_timeline_refuses_foreign_session_ids` (a sibling project's or another workspace's session id and an unknown id → 404, another operator's owned session → 404; controls: the home session, and the owner reading theirs); bite: dropping the scope filter from `page_version_body` or the project filter from `page_versions`, or replacing the `session_summary_scoped` check or its owner filter, fails them | STRONG |
| 1b | Reserved `_global` scope union never leaks a foreign project (#930) | `ai-memory-mcp/src/server.rs` `memory_query` — the union runs for single-project queries (`scopes` empty) and searches **only** `lookup_global_scope`'s reserved `(workspace_id, project_id)`, never an arbitrary project; the double-search guard resolves the *queried* project (named or active) so it can't be tricked into skipping | `server.rs` `global_union_never_leaks_a_foreign_projects_pages` (a third real project's page must not surface via the union; reserved page must), `single_project_query_unions_global_scope_and_multi_scope_skips_it` | STRONG |
| 1c | Reserved `_global` scope is read-open, write-gated (GHSA-7qj3-7wqw-m5w6) | `ai-memory-store/src/project_authz.rs` `ProjectAuthz::authorize` (a write to `default/_global` needs root or a `write` grant in either access mode; creator ignored) fed by `resolve_project_authz`/`read_project_row` (`reserved_global`, grant read for the global scope regardless of mode); `ai-memory-mcp/src/server.rs` `memory_write_page` resolves `scope: "global"` through `write_target_ids_with_actor` like an explicit `default/_global` | `project_authz.rs` `the_global_scope_admits_writes_only_from_root_or_a_write_grant`; `server.rs` `a_user_without_a_global_grant_cannot_change_the_global_scope` (both spellings and delete refused for an ungranted user and for `_global`'s creator; page survives; reads stay open), `a_global_write_grant_admits_writes_and_the_union_stays_readable` (legitimate control: write grant admits both spellings, another user's restricted-project query still unions the page); bite-checked with the authz rule removed and with the raw `create_global_scope` path restored | STRONG |
| 1d | A private cross-project profile is readable and writable by its operator alone (`[profile] share = "user"`) | `ai-memory-store/src/ops.rs` `initial_access_mode` (a `_profile.<user id>` project is always created `restricted`, creator recorded in the same transaction); `ai-memory-store/src/scope.rs` `lookup_profile_scope` / `create_profile_scope` (the project name derives from the authenticated viewer's id; no database user, no private profile); `ai-memory-mcp/src/server.rs` `profile_target_ids` and the `memory_query` profile union (resolved per viewer through the per-project choke point); `scope.rs` `refuse_foreign_profile_name` (called by `ScopeResolver` explicit writes and `create_explicit_scope_guarded`: a database user may name only their own `default/_profile.<id>`, so nobody can create another operator's private profile first and lock its owner out); `ops.rs` `rename_project` (no rename onto or away from a profile name); `admin.rs` `handle_project_access` (a profile project's mode is fixed) and `move_project_core` (a private profile cannot leave `default`) | store `integration::profile` `a_private_profile_refuses_every_other_operator` (another operator's read and write by name refused; owner control admitted), `reserved_profile_projects_get_their_fixed_access_modes`; `server.rs` `a_private_profile_is_private_to_its_operator` (no union leak; read, write and delete by name refused; owner and own-profile controls); `multi_session.rs` `two_harnesses_of_one_operator_share_one_private_profile`; both fail with the restricted-mode guard removed; `server.rs` `a_user_cannot_squat_another_operators_private_profile` (another user's explicit write naming the owner's private profile refused and creates nothing; owner's profile and explicit own-name controls; fails with the name guard removed); `ops.rs` `rename_project_refuses_profile_names_both_ways`; `admin.rs` `project_access_sets_the_mode_and_refuses_what_it_must`; `admin_move.rs` `move_project_refuses_a_private_profile` | STRONG |