Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(web): filter search by kind and tier inside the ranker
Peterson Salme1d58eb02c54c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bbc933d8..de9acff3 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,12 +14,13 @@ and [0.x](docs/changelog/0.x.md).### Added- Added kind and tier filters to the web search. The search page in `/web`has "Kind" and "Tier" selects, and `/web/search` accepts `kind` and `tier`- for both a project search and a cross-project one. The filter runs after- ranking, so the hits that remain keep the order `memory_query` gives them,- and a filtered search looks through its top 500 matches rather than the- first page of results, saying so when the ranker had more. The kind match- ignores case and a nonstandard kind is flagged on an empty result. An- unknown `tier` is a 400. (#1248)+ for both a project search and a cross-project one. A filtered search looks+ through up to 500 candidates per stream rather than the first page of+ results and says so when the ranker had more, so a rare kind is not crowded+ out. A project search filters inside the ranker, before the reranker, so+ the reranker scores only pages the filter keeps. The kind match ignores+ case and a nonstandard kind is flagged on an empty result. An unknown+ `tier` is a 400. (#1248)- Added backlinks, outgoing links and an outline to the web page view. A pagein `/web` now has a side column (below the text on narrow screens) with "Onthis page", the h2 and h3 headings linked to their anchors (shown from twodiff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rsindex 69510a13..39c48e3b 100644--- a/crates/ai-memory-mcp/src/server.rs+++ b/crates/ai-memory-mcp/src/server.rs@@ -2426,7 +2426,8 @@ impl AiMemoryServer {// the `expires_at > cutoff` guard, i.e. expired pages stay// searchable when the caller opted in.let expiry_cutoff = options.include_expired.then_some(i64::MIN);- self.reader+ let found = self+ .reader.hybrid_search_scopes(scopes,ai_memory_store::HybridSearch {@@ -2437,9 +2438,11 @@ impl AiMemoryServer {include_superseded: options.include_superseded,explain: options.explain,owner: options.owner,+ filter: ai_memory_store::PageFilter::default(),},)- .await+ .await?;+ Ok((found.hits, found.timings))}fn rerank_fetch_limit(&self, limit: usize) -> usize {diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex 8f7e357f..91304d16 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -85,18 +85,18 @@ pub use reader::{ActivityWindow, AgentSessionCount, AuditEvent, AuditLogFilter, AutoImproveCandidateSession,AutoImproveParkedClaim, BriefPageBody, BriefingPage, BriefingSnapshot, ClientActivity,ContaminationFinding, ContaminationReport, ContaminationSummary, ContradictionEdge,- DecayCandidate, DecayTombstone, DerivedIndexStatus, EmbeddingTripleCount, FeedbackFinding,- GraphVia, HealthDetail, HealthPage, HybridSearch, InboxCount, InboxEntry, ObservationHit,- ObservationOrder, ObservationPage, ObservationPageResult, ObservationRecord, OpenSession,- PAGE_KINDS, PAGE_VERSIONS_MAX_LIMIT, PageAuthor, PageHit, PageHitWithMeta, PageLinks,- PageLinksPreview, PageMeta, PageSummary, PageVersion, PageVersionBody, PageVersions,- ProjectCoordinateCollisionReason, ProjectCoordinateDiagnostic, ProjectCoordinateStatus,- ProjectSummary, RELATED_WALK_MAX_DEPTH, RELATED_WALK_MAX_NODES, ReaderPool,- ReindexTargetStatus, RelatedNode, RelatedPage, RrfContributions, ScopeRow, SearchExplain,- SessionConsolidationQueue, SessionConsolidationSummary, SessionDependentRows,- SessionEndDisposition, SessionSummary, SettledPage, StatusCounts, StorageStatus,- StoredEmbedding, StoredPageBody, StreamTimings, VectorQuery, WorkspaceScopeRow,- WorkspaceSummary, f32_vec_to_bytes,+ DecayCandidate, DecayTombstone, DerivedIndexStatus, EmbeddingTripleCount,+ FILTER_CANDIDATE_POOL, FeedbackFinding, GraphVia, HealthDetail, HealthPage, HybridHits,+ HybridSearch, InboxCount, InboxEntry, ObservationHit, ObservationOrder, ObservationPage,+ ObservationPageResult, ObservationRecord, OpenSession, PAGE_KINDS, PAGE_VERSIONS_MAX_LIMIT,+ PageAuthor, PageFilter, PageHit, PageHitWithMeta, PageLinks, PageLinksPreview, PageMeta,+ PageSummary, PageVersion, PageVersionBody, PageVersions, ProjectCoordinateCollisionReason,+ ProjectCoordinateDiagnostic, ProjectCoordinateStatus, ProjectSummary, RELATED_WALK_MAX_DEPTH,+ RELATED_WALK_MAX_NODES, ReaderPool, ReindexTargetStatus, RelatedNode, RelatedPage,+ RrfContributions, ScopeRow, SearchExplain, SessionConsolidationQueue,+ SessionConsolidationSummary, SessionDependentRows, SessionEndDisposition, SessionSummary,+ SettledPage, StatusCounts, StorageStatus, StoredEmbedding, StoredPageBody, StreamTimings,+ VectorQuery, WorkspaceScopeRow, WorkspaceSummary, f32_vec_to_bytes,};pub use retrieval_tuning::{RetrievalTuning, is_session_recall_query};pub use scope::{diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rsindex 581787cb..795f8d38 100644--- a/crates/ai-memory-store/src/reader.rs+++ b/crates/ai-memory-store/src/reader.rs@@ -438,12 +438,16 @@ pub(crate) const OBSERVATION_ROWID_SPAN_SQL: &str = "SELECT \(SELECT MAX(rowid) FROM observations INDEXED BY idx_observations_project \WHERE project_id = ?1)";-#[derive(Debug, Clone, Copy)]+#[derive(Debug, Clone)]struct PageAuthority {factor: f64,/// Kind/tier penalty this page carries *because* it is a session page/// (`0.0` for every other kind). A `session_recall` query gives it back.session_penalty: f64,+ /// Semantic kind and tier the factor was derived from, kept so a+ /// [`PageFilter`] needs no second read of the same rows.+ kind: String,+ tier: String,}impl PageAuthority {@@ -516,6 +520,8 @@ impl PageAuthority {Self {factor: factor.clamp(0.55, 1.50),session_penalty,+ kind: kind.to_owned(),+ tier: tier.to_owned(),}}@@ -869,6 +875,51 @@ pub struct VectorQuery<'a> {pub dim: u32,}+/// Candidates each stream is asked for when a [`PageFilter`] is active.+///+/// The streams have no kind or tier predicate, so a filter can only keep what+/// they return. A pool this much wider than an unfiltered search's keeps a+/// rare kind from being crowded out by pages of other kinds that outrank it.+pub const FILTER_CANDIDATE_POOL: usize = 500;++/// Narrows a [`HybridSearch`] to pages of one kind and/or tier.+///+/// Applied after fusion and authority, before the result is cut to+/// `limit`, so `limit` counts hits that passed it.+#[derive(Debug, Clone, Copy, Default)]+pub struct PageFilter<'a> {+ /// Semantic kind to keep, compared ASCII case-insensitively; `None`+ /// keeps every kind.+ pub kind: Option<&'a str>,+ /// Memory tier to keep; `None` keeps every tier.+ pub tier: Option<&'a str>,+}++impl PageFilter<'_> {+ fn is_active(&self) -> bool {+ self.kind.is_some() || self.tier.is_some()+ }++ /// Whether a page of this kind and tier passes the filter.+ #[must_use]+ pub fn keeps(&self, kind: &str, tier: &str) -> bool {+ self.kind.is_none_or(|k| k.eq_ignore_ascii_case(kind))+ && self.tier.is_none_or(|t| t == tier)+ }+}++/// What [`ReaderPool::hybrid_search_scopes`] returns.+#[derive(Debug)]+pub struct HybridHits {+ /// Hits best first, each with its explain details when asked for.+ pub hits: Vec<(PageHit, Option<SearchExplain>)>,+ /// Per-stream wall time; `explain` only.+ pub timings: Option<StreamTimings>,+ /// A [`PageFilter`] was active and a stream returned its whole+ /// candidate pool, so matches ranked past it were never looked at.+ pub filter_pool_full: bool,+}+/// One [`ReaderPool::hybrid_search_scopes`] request.#[derive(Debug, Clone, Copy)]pub struct HybridSearch<'a> {@@ -887,6 +938,8 @@ pub struct HybridSearch<'a> {/// Who is asking, as the session timeline sees them: an observation/// excerpt is attached only from a session this filter admits.pub owner: &'a OwnerFilter,+ /// Keep only pages of this kind and/or tier; the default keeps all.+ pub filter: PageFilter<'a>,}/// Per-query wall time of each retrieval stream, in milliseconds, for an@@ -1299,7 +1352,8 @@ pub struct PageHitWithMeta {/// Relevance rank after the bounded authority adjustment (lower is better).pub rank: f64,/// Semantic kind, so the web search can filter without a second lookup.- /// Not serialized: the MCP `global_hits` shape stays as it was.+ /// Empty on a recency listing, which nothing filters. Not serialized: the+ /// MCP `global_hits` shape stays as it was.#[serde(skip)]pub kind: String,/// Memory tier; see `kind`.@@ -2970,7 +3024,7 @@ impl ReaderPool {.await?;let mut authorities: std::collections::HashMap<PageId, PageAuthority> = fts_candidates.iter()- .map(|(hit, authority)| (hit.id, *authority))+ .map(|(hit, authority)| (hit.id, authority.clone())).collect();// Entity-only hits never passed through an FTS candidate row;// read their authority off the version row itself (no latest@@ -3570,8 +3624,7 @@ impl ReaderPool {let sql = format!("SELECT workspaces.name, projects.name, pages.path, pages.title, \{descriptor} AS snip, \- CAST(pages.updated_at AS REAL) AS rank, \- {kind_expr} AS kind, pages.tier \+ CAST(pages.updated_at AS REAL) AS rank \FROM pages \JOIN projects ON projects.id = pages.project_id \JOIN workspaces ON workspaces.id = pages.workspace_id \@@ -3579,7 +3632,6 @@ impl ReaderPool {ORDER BY pages.updated_at DESC \LIMIT ?1",descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json"),- kind_expr = page_kind_expr("pages.path", "pages.frontmatter_json"),not_expired = not_expired("pages", "?2"),);let mut stmt = conn.prepare_cached(&sql)?;@@ -3593,22 +3645,11 @@ impl ReaderPool {let title: String = row.get(3)?;let snippet = page_descriptor(&row.get::<_, String>(4)?, &title);let rank: f64 = row.get(5)?;- let kind: String = row.get(6)?;- let tier: String = row.get(7)?;- Ok((- workspace_name,- project_name,- path,- title,- snippet,- rank,- kind,- tier,- ))+ Ok((workspace_name, project_name, path, title, snippet, rank))})?;let mut hits = Vec::new();for row in rows {- let (workspace_name, project_name, path, title, snippet, rank, kind, tier) = row?;+ let (workspace_name, project_name, path, title, snippet, rank) = row?;hits.push(PageHitWithMeta {workspace_name,project_name,@@ -3616,8 +3657,8 @@ impl ReaderPool {title,snippet,rank,- kind,- tier,+ kind: String::new(),+ tier: String::new(),});}Ok(hits)@@ -6156,6 +6197,7 @@ impl ReaderPool {false,include_superseded,&OwnerFilter::Any,+ PageFilter::default(),).await?.0@@ -6220,7 +6262,7 @@ impl ReaderPool {expiry_cutoff_us: Option<i64>,include_superseded: bool,) -> StoreResult<(Vec<(PageHit, SearchExplain)>, StreamTimings)> {- let (hits, timings) = self+ let (hits, timings, _) = self.hybrid_search_inner(workspace_id,project_id,@@ -6234,6 +6276,7 @@ impl ReaderPool {true,include_superseded,&OwnerFilter::Any,+ PageFilter::default(),).await?;// Belief-strength fields (`evidence_count`, `confidence`,@@ -6268,7 +6311,7 @@ impl ReaderPool {&self,scopes: &[(WorkspaceId, ProjectId)],search: HybridSearch<'_>,- ) -> StoreResult<(Vec<(PageHit, Option<SearchExplain>)>, Option<StreamTimings>)> {+ ) -> StoreResult<HybridHits> {// Provider/model/dim only select which stored vectors are eligible;// with no query vector the vector stream never runs, so the empty// triple is inert rather than a fake identity.@@ -6282,9 +6325,10 @@ impl ReaderPool {None => (None, String::new(), String::new(), 0),};let mut timings: Option<StreamTimings> = None;+ let mut filter_pool_full = false;let mut per_scope = Vec::with_capacity(scopes.len());for &(workspace_id, project_id) in scopes {- let (hits, scope_timings) = self+ let (hits, scope_timings, pool_full) = self.hybrid_search_inner(workspace_id,project_id,@@ -6298,8 +6342,10 @@ impl ReaderPool {search.explain,search.include_superseded,search.owner,+ search.filter,).await?;+ filter_pool_full |= pool_full;if search.explain {timings.get_or_insert_with(Default::default)@@ -6345,7 +6391,11 @@ impl ReaderPool {} else {hits};- Ok((hits, timings))+ Ok(HybridHits {+ hits,+ timings,+ filter_pool_full,+ })}/// The streams [`Self::hybrid_search_scopes`] runs for a query, in the@@ -6381,7 +6431,8 @@ impl ReaderPool {explain: bool,include_superseded: bool,owner_filter: &OwnerFilter,- ) -> StoreResult<(Vec<(PageHit, Option<SearchExplain>)>, StreamTimings)> {+ filter: PageFilter<'_>,+ ) -> StoreResult<(Vec<(PageHit, Option<SearchExplain>)>, StreamTimings, bool)> {let cutoff = expiry_cutoff_us.unwrap_or_else(now_us);// Stream timings exist only for explain: `explain.then(Instant::now)`// reads no clock on the normal path.@@ -6394,7 +6445,13 @@ impl ReaderPool {// bounded candidate window used by FTS-only search. A `limit * 2`// window was too narrow at small limits for a canonical page to enter// the fused pool and receive its bounded promotion.- let candidate_limit = authority_candidate_limit(limit);+ // A filter can only keep what the streams return, so they are asked+ // for a wide pool and `limit` is applied to what survives.+ let candidate_limit = authority_candidate_limit(if filter.is_active() {+ limit.max(FILTER_CANDIDATE_POOL)+ } else {+ limit+ });// Tokenize the borrowed query before the FTS candidate call consumes// it. This avoids cloning a caller-controlled query on the hot path.// The default path uses the latest-only entry; the opt-in path reaches@@ -6442,7 +6499,7 @@ impl ReaderPool {timings.fts_ms = elapsed_ms(started);let mut authorities: std::collections::HashMap<PageId, PageAuthority> = fts_candidates.iter()- .map(|(hit, authority)| (hit.id, *authority))+ .map(|(hit, authority)| (hit.id, authority.clone())).collect();let fts_hits: Vec<PageHit> = fts_candidates.into_iter()@@ -6543,6 +6600,11 @@ impl ReaderPool {Vec::new()};+ let filter_pool_full = filter.is_active()+ && [fts_hits.len(), vec_hits.len(), entity_hits.len()]+ .into_iter()+ .any(|returned| returned >= candidate_limit);+// RRF fuse: score(d) = Σ 1/(k + rank_i(d)) over rankers.let k = 60.0_f64;struct Fused {@@ -6852,6 +6914,13 @@ impl ReaderPool {// back to HashMap iteration order..then_with(|| a.0.path.as_str().cmp(b.0.path.as_str()))});+ if filter.is_active() {+ out.retain(|(hit, _)| {+ authorities+ .get(&hit.id)+ .is_some_and(|a| filter.keeps(&a.kind, &a.tier))+ });+ }out.truncate(limit);// Label the superseded versions among the final hits so the caller can// tell historical versions from the current one. Only the opt-in path@@ -6866,7 +6935,7 @@ impl ReaderPool {hit.superseded = superseded.contains(&hit.id);}}- Ok((out, timings))+ Ok((out, timings, filter_pool_full))}/// The subset of `page_ids` that are superseded (non-latest) versions in@@ -8526,52 +8595,6 @@ impl ReaderPool {.await}- /// Semantic kind and tier of each latest page in `page_ids`, in one query.- ///- /// Ids that name no latest page are absent from the map. The caller has- /// already scoped the ids (they come from a search the viewer was allowed- /// to run), so this reads no repository the viewer could not.- ///- /// # Errors- /// Propagates any SQL or pool error.- pub async fn page_kinds_tiers_by_ids(- &self,- page_ids: &[PageId],- ) -> StoreResult<std::collections::HashMap<PageId, (String, String)>> {- if page_ids.is_empty() {- return Ok(std::collections::HashMap::new());- }- let id_blobs: Vec<Value> = page_ids- .iter()- .map(|id| Value::Blob(id.as_bytes().to_vec()))- .collect();- self.with_conn(move |conn| {- let placeholders = std::iter::repeat_n("?", id_blobs.len())- .collect::<Vec<_>>()- .join(", ");- let sql = format!(- "SELECT pg.id, {kind_expr}, pg.tier FROM pages pg \- WHERE pg.is_latest = 1 AND pg.id IN ({placeholders})",- kind_expr = page_kind_expr("pg.path", "pg.frontmatter_json"),- );- let mut stmt = conn.prepare(&sql)?;- let rows = stmt.query_map(params_from_iter(id_blobs.iter()), |row| {- Ok((- row.get::<_, Vec<u8>>(0)?,- row.get::<_, String>(1)?,- row.get::<_, String>(2)?,- ))- })?;- let mut out = std::collections::HashMap::new();- for row in rows {- let (id, kind, tier) = row?;- out.insert(PageId::from_slice(&id)?, (kind, tier));- }- Ok(out)- })- .await- }-/// Look up a page's workspace and project names by its path (across all/// workspaces and projects). Returns the first `is_latest = 1` match.///@@ -12922,11 +12945,12 @@ mod tests {explain,false,&OwnerFilter::Any,+ super::PageFilter::default(),)};- let (_, off) = search(false).await.unwrap();+ let (_, off, _) = search(false).await.unwrap();assert_eq!(off, super::StreamTimings::default());- let (_, on) = search(true).await.unwrap();+ let (_, on, _) = search(true).await.unwrap();assert!(on.fts_ms.is_some() && on.entity_ms.is_some() && on.graph_ms.is_some());assert!(on.observation_ms.is_some(), "the stream is on by default");assert!(on.vector_ms.is_none(), "no query vector, no vector scan");diff --git a/crates/ai-memory-store/tests/suite/multi_session.rs b/crates/ai-memory-store/tests/suite/multi_session.rsindex 5a5d3816..44704e60 100644--- a/crates/ai-memory-store/tests/suite/multi_session.rs+++ b/crates/ai-memory-store/tests/suite/multi_session.rs@@ -3550,11 +3550,12 @@ async fn explained_search_as(include_superseded: false,explain: true,owner,+ filter: ai_memory_store::PageFilter::default(),},).await.unwrap()- .0+ .hits.into_iter().map(|(hit, details)| (hit, details.unwrap())).collect()diff --git a/crates/ai-memory-web/src/recall.rs b/crates/ai-memory-web/src/recall.rsindex e84e34a4..1e3ca002 100644--- a/crates/ai-memory-web/src/recall.rs+++ b/crates/ai-memory-web/src/recall.rs@@ -19,7 +19,7 @@ use std::sync::Arc;use ai_memory_core::{OwnerFilter, ProjectId, WorkspaceId};use ai_memory_llm::{Embedder, QueryVector, RerankCandidate, RerankStage};use ai_memory_store::{- HybridSearch, PageHit, SearchExplain, StoreResult, StreamTimings, VectorQuery,+ HybridSearch, PageFilter, PageHit, SearchExplain, StoreResult, StreamTimings, VectorQuery,};use crate::state::WebState;@@ -43,12 +43,17 @@ pub(crate) struct RecallResult {pub timings: Option<StreamTimings>,/// The streams that ran, in `memory_query`'s `streams_active` order.pub streams_active: Vec<&'static str>,+ /// A filter was set and a stream returned its whole candidate pool, so+ /// matches ranked past it were not looked at.+ pub filter_pool_full: bool,}-/// Rank `query` over `scopes` exactly as a `memory_query` project/scopes-/// search does (latest versions only, expired pages hidden). `owner` is the-/// viewer's session-timeline filter: an observation excerpt is shown only-/// from a session it admits, as in `memory_query`.+/// Rank `query` over `scopes` as a `memory_query` project/scopes search does+/// (latest versions only, expired pages hidden), keeping only pages that pass+/// `filter`. The filter runs in the ranker, before the reranker, so the+/// reranker scores only pages the caller can see. `owner` is the viewer's+/// session-timeline filter: an observation excerpt is shown only from a+/// session it admits, as in `memory_query`.pub(crate) async fn search_scopes(state: &WebState,scopes: &[(WorkspaceId, ProjectId)],@@ -56,6 +61,7 @@ pub(crate) async fn search_scopes(limit: usize,explain: bool,owner: &OwnerFilter,+ filter: PageFilter<'_>,) -> StoreResult<RecallResult> {let query_vector: Option<QueryVector> = match &state.recall.embedder {Some(embedder) => ai_memory_llm::embed_search_query(embedder.as_ref(), query).await,@@ -66,7 +72,7 @@ pub(crate) async fn search_scopes(.rerank.as_ref().map_or(limit, |stage| stage.fetch_limit(limit));- let (hits, mut timings) = state+ let found = state.reader.hybrid_search_scopes(scopes,@@ -83,9 +89,11 @@ pub(crate) async fn search_scopes(include_superseded: false,explain,owner,+ filter,},).await?;+ let (hits, mut timings) = (found.hits, found.timings);let hits = match &state.recall.rerank {Some(stage) => {let started = explain.then(std::time::Instant::now);@@ -122,5 +130,6 @@ pub(crate) async fn search_scopes(hits,timings,streams_active: state.reader.active_streams(query_vector.is_some()),+ filter_pool_full: found.filter_pool_full,})}diff --git a/crates/ai-memory-web/src/routes/api.rs b/crates/ai-memory-web/src/routes/api.rsindex ffed748f..3734a981 100644--- a/crates/ai-memory-web/src/routes/api.rs+++ b/crates/ai-memory-web/src/routes/api.rs@@ -406,7 +406,8 @@ async fn search_with_request(.iter().map(|scope| (scope.workspace_id, scope.project_id)).collect();- crate::recall::search_scopes(state, &scopes, &term, limit, false, owner)+ let filter = ai_memory_store::PageFilter::default();+ crate::recall::search_scopes(state, &scopes, &term, limit, false, owner, filter).await.map(|result| result.hits.into_iter().map(|(hit, _)| hit).collect())}diff --git a/crates/ai-memory-web/src/routes/search.rs b/crates/ai-memory-web/src/routes/search.rsindex acc494e1..edd3c1bf 100644--- a/crates/ai-memory-web/src/routes/search.rs+++ b/crates/ai-memory-web/src/routes/search.rs@@ -12,15 +12,19 @@//! viewer may read, as `memory_query(global=true)` does. Its trace is the//! FTS time alone; there are no per-hit streams to show.//!-//! `kind` and `tier` narrow either mode *after* ranking, so the surviving-//! hits keep the order `memory_query` gives them.+//! `kind` and `tier` narrow either mode after fusion, inside the ranker. The+//! streams are asked for a wider candidate pool so a rare kind is not crowded+//! out, which means the order of the survivors can differ slightly from the+//! unfiltered top hits of the same query.use std::collections::BTreeMap;use std::str::FromStr;use std::sync::Arc;use ai_memory_core::Tier;-use ai_memory_store::{PAGE_KINDS, SearchExplain, StreamTimings};+use ai_memory_store::{+ FILTER_CANDIDATE_POOL, PAGE_KINDS, PageFilter, SearchExplain, StreamTimings,+};use askama::Template;use axum::extract::{Query, State};use axum::http::StatusCode;@@ -40,9 +44,6 @@ const GLOBAL_DEFAULT_LIMIT: usize = 50;const SCOPED_DEFAULT_LIMIT: usize = 20;/// Upper bound on `limit`, the same clamp `memory_query` applies.const MAX_LIMIT: usize = 100;-/// Matches a filtered search ranks before it filters. The ranker has no-/// kind or tier predicate, so a filter only sees this many top hits.-const FILTER_POOL: usize = 500;/// Memory tiers the filter offers, in `Tier` declaration order.const TIERS: [&str; 4] = ["working", "episodic", "semantic", "procedural"];@@ -138,14 +139,19 @@ pub(crate) async fn handler(None => None,};let filtered = kind.is_some() || tier.is_some();- let keeps = |hit_kind: &str, hit_tier: &str| {- kind.as_deref()- .is_none_or(|k| k.eq_ignore_ascii_case(hit_kind))- && tier.as_deref().is_none_or(|t| t == hit_tier)+ let filter = PageFilter {+ kind: kind.as_deref(),+ tier: tier.as_deref(),+ };+ // The global search has no ranker-side filter, so a filtered one is asked+ // for a wide pool and the limit is applied to what survives.+ let fetch_limit = |limit: usize| {+ if filtered {+ FILTER_CANDIDATE_POOL+ } else {+ limit + 1+ }};- // A filter runs after ranking, so the ranker is asked for a wide pool- // and the limit is applied to what survives.- let fetch_limit = |limit: usize| if filtered { FILTER_POOL } else { limit + 1 };let internal = |error: ai_memory_store::StoreError| {tracing::error!(%error, "web search failed");StatusCode::INTERNAL_SERVER_ERROR.into_response()@@ -177,9 +183,10 @@ pub(crate) async fn handler(&state,&[(resolved.workspace_id, resolved.project_id)],&query,- fetch_limit(limit),+ limit + 1,explain,&super::api::owner_filter_for(actor),+ filter,).await{@@ -188,23 +195,10 @@ pub(crate) async fn handler(};timings = result.timings;streams_active = result.streams_active;- // The trace position is the rank before filtering.- let mut ranked: Vec<_> = result.hits.into_iter().enumerate().collect();- pool_capped = filtered && ranked.len() >= FILTER_POOL;- if filtered {- let ids: Vec<_> = ranked.iter().map(|(_, (hit, _))| hit.id).collect();- let facets = match state.reader.page_kinds_tiers_by_ids(&ids).await {- Ok(facets) => facets,- Err(error) => return internal(error),- };- ranked.retain(|(_, (hit, _))| {- facets- .get(&hit.id)- .is_some_and(|(hit_kind, hit_tier)| keeps(hit_kind, hit_tier))- });- }- truncated = ranked.len() > limit;- for (position, (hit, details)) in ranked.into_iter().take(limit) {+ pool_capped = result.filter_pool_full;+ truncated = result.hits.len() > limit;+ // The trace position is the rank among the hits shown.+ for (position, (hit, details)) in result.hits.into_iter().take(limit).enumerate() {let path = hit.path.as_str().to_owned();hits.push(SearchHit {href: page_href(workspace, project, &path),@@ -242,10 +236,10 @@ pub(crate) async fn handler(});streams_active = vec!["fts"];}- pool_capped = filtered && raw.len() >= FILTER_POOL;+ pool_capped = filtered && raw.len() >= FILTER_CANDIDATE_POOL;let raw: Vec<_> = raw.into_iter()- .filter(|h| keeps(&h.kind, &h.tier))+ .filter(|h| filter.keeps(&h.kind, &h.tier)).collect();truncated = raw.len() > limit;for h in raw.into_iter().take(limit) {@@ -313,7 +307,7 @@ pub(crate) async fn handler(hit_count,filtered,pool_capped,- filter_pool: FILTER_POOL,+ filter_pool: FILTER_CANDIDATE_POOL,unknown_kind,kind_options,tier_options,diff --git a/crates/ai-memory-web/tests/suite/recall_trace.rs b/crates/ai-memory-web/tests/suite/recall_trace.rsindex c46ef7b1..05ddbbfb 100644--- a/crates/ai-memory-web/tests/suite/recall_trace.rs+++ b/crates/ai-memory-web/tests/suite/recall_trace.rs@@ -8,13 +8,15 @@//! partial scope rejected, and no other project's pages in the result or the//! trace.-use std::sync::Arc;+use std::sync::{Arc, Mutex};use ai_memory_core::{ActorContext, AuthorizedViewer, NewUser, PagePath, ProjectId, Tier, UserId, UserRole,WorkspaceId,};-use ai_memory_llm::{Embedder, SyntheticEmbedder};+use ai_memory_llm::{+ Embedder, RerankCandidate, RerankScore, RerankStage, Reranker, SyntheticEmbedder,+};use ai_memory_store::{GrantLevel, Store};use ai_memory_web::{Recall, WebMountSpec, split_web_routers};use ai_memory_wiki::{Wiki, WritePageRequest};@@ -713,3 +715,138 @@ fn tier_filter_offers_every_tier() {assert_eq!(name.parse::<Tier>().unwrap(), tier);}}++/// Scores every candidate equally and remembers which pages it was shown.+#[derive(Default)]+struct RecordingReranker {+ seen: Mutex<Vec<String>>,+}++#[async_trait::async_trait]+impl Reranker for RecordingReranker {+ fn name(&self) -> &'static str {+ "recording"+ }++ fn model(&self) -> &str {+ "test"+ }++ async fn rerank(+ &self,+ _query: &str,+ candidates: &[RerankCandidate],+ ) -> ai_memory_llm::LlmResult<Vec<RerankScore>> {+ self.seen+ .lock()+ .unwrap()+ .extend(candidates.iter().map(|c| c.title.clone()));+ Ok(candidates+ .iter()+ .map(|c| RerankScore {+ id: c.id.clone(),+ relevance: 0.5,+ })+ .collect())+ }+}++/// The filter runs in the ranker, before the reranker: pages the filter+/// drops are never sent to the (paid, slow) reranker.+#[tokio::test]+async fn reranker_is_shown_only_the_pages_the_filter_keeps() {+ let f = fixture(false).await;+ add_procedure(&f).await;+ // The stage leaves a single candidate alone, so the filter needs two.+ let ws = f+ .store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ f.wiki+ .write_page(WritePageRequest {+ frontmatter: serde_json::json!({"kind": "procedure"}),+ tier: Tier::Procedural,+ ..req(+ ws,+ f.alpha,+ "procedures/audit.md",+ "# Audit\n\nAudit every token rotation after the script ran.",+ )+ })+ .await+ .unwrap();+ let reranker = Arc::new(RecordingReranker::default());+ let recall = Recall {+ embedder: None,+ rerank: Some(RerankStage::new(reranker.clone())),+ };+ let app = web(&f, recall);+ let base = "/web/search?q=token%20rotation&workspace=default&project=alpha";++ // Control: unfiltered, the reranker sees more than the procedure.+ let (status, body) = get(&app, base, None).await;+ assert_eq!(status, StatusCode::OK, "{body}");+ let unfiltered = std::mem::take(&mut *reranker.seen.lock().unwrap());+ assert!(unfiltered.len() > 2, "{unfiltered:?}");++ let (status, body) = get(&app, &format!("{base}&kind=procedure"), None).await;+ assert_eq!(status, StatusCode::OK, "{body}");+ let mut shown = hit_paths(&body);+ shown.sort();+ assert_eq!(+ shown,+ ["procedures/audit.md", "procedures/rotate.md"],+ "{body}"+ );+ let filtered = reranker.seen.lock().unwrap().clone();+ assert_eq!(filtered.len(), 2, "{filtered:?}");+}++/// A rare kind that the unfiltered result ranks out of view is still found:+/// the filter is applied to a candidate pool far wider than one page.+#[tokio::test]+async fn kind_filter_finds_a_rare_kind_the_unfiltered_page_ranks_out() {+ let f = fixture(false).await;+ let ws = f+ .store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ for n in 0..60 {+ f.wiki+ .write_page(req(+ ws,+ f.alpha,+ &format!("notes/crowd-{n}.md"),+ "# Crowd\n\ncrowdword crowdword crowdword crowdword crowdword.",+ ))+ .await+ .unwrap();+ }+ f.wiki+ .write_page(WritePageRequest {+ frontmatter: serde_json::json!({"kind": "gotcha"}),+ ..req(+ ws,+ f.alpha,+ "gotchas/quiet.md",+ "# Quiet\n\nOne mention of crowdword among many other unrelated words \+ about cats, rivers, mountains, bridges and old maps.",+ )+ })+ .await+ .unwrap();+ let app = web(&f, Recall::default());+ let base = "/web/search?q=crowdword&workspace=default&project=alpha&limit=5";++ let (_, body) = get(&app, base, None).await;+ assert!(+ !hit_paths(&body).contains(&"gotchas/quiet.md".to_owned()),+ "control: the gotcha must be ranked out of the first page: {body}"+ );+ let (_, body) = get(&app, &format!("{base}&kind=gotcha"), None).await;+ assert_eq!(hit_paths(&body), ["gotchas/quiet.md"], "{body}");+}diff --git a/docs/security-boundaries.md b/docs/security-boundaries.mdindex 131e4613..5fa0c071 100644--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@ -105,7 +105,7 @@ means the boundary is not built yet.| 14d | `include_related` and the web page's backlinks show and walk only readable projects (#708) | `ai-memory-store/src/reader.rs` `related_walk(.., viewer)` — both the outgoing and incoming hop queries carry `readable_repository_predicate("pg.project_id", viewer)`, the same far-end filter `page_links` and `page_links_preview` apply (the preview's visible totals are counted under it too), so a page in a project the viewer cannot read is neither returned nor expanded; `ai-memory-mcp/src/server.rs` `memory_read_page` passes `viewer_from_parts`, `ai-memory-web/src/routes/page.rs` passes the request's viewer to `page_links_preview`, and the access bump only touches the filtered nodes | `related_walk.rs` `the_walk_hides_and_does_not_cross_projects_the_viewer_cannot_read` (an outgoing and an incoming neighbour in an ungranted restricted project are absent, and so is a granted page reachable only through one; control: a granted restricted project is shown, and no viewer walks the whole graph) — fails with the predicate removed; `ai-memory-web` `tests/suite/routes.rs` `page_backlink_from_an_unreadable_project_is_hidden` (the `/web` page's "Cited by" omits a page in an ungranted restricted project, with no title, path or project name; controls: a reader of both projects and an unrestricted viewer see it) — fails with the page handler passing no viewer to `page_links_preview`; `related_walk.rs` `page_links_preview_caps_in_sql_and_counts_only_readable_neighbours` (back-links from an ungranted restricted project are neither listed nor counted in the total behind "and N more"; control: no viewer counts them all) — fails with the predicate removed; `ai-memory-mcp` `server.rs` `bob_does_not_see_restricted_pages_through_the_related_walk` (an ungranted user's `memory_read_page{include_related}` omits the restricted page; control: no viewer still reaches it) — fails with `memory_read_page` passing `None` instead of `viewer_from_parts` | STRONG || 14e | `memory_write_page` session evidence cites only a session of the target project (#708) | `ai-memory-mcp/src/server.rs` `memory_write_page` — a `session_id` is accepted only when `require_session_in_scope` resolves the session (where its observations landed, else its session row, as `Consolidator::session_target` does) to the same `(workspace_id, project_id)` the write grant was checked against; unknown and foreign ids get one refusal, so the error is no existence oracle; `scope: "global"` with a `session_id` is refused; `refuse_pinned_page_overwrite` keeps a write that cites a session off a pinned page; only a write to `sessions/<id>.md` reconciles the session's queued SessionEnd job | `server.rs` `write_page_refuses_a_session_from_another_project` (a foreign and an unknown session refused, no evidence row written, the foreign job still queued; control: the project's own session accepted), `session_evidence_write_leaves_a_pinned_page_alone` (control: the same write without `session_id` edits the page), `session_evidence_on_another_path_leaves_the_job_queued`, `write_page_rejects_a_malformed_or_global_session_id` — each of the scope check, the pinned check and the session-page-only reconcile proven to fail its test when removed | STRONG || 14f | Consolidation apply path validates cross-scope wikilinks against the store (#1218) | `ai-memory-consolidate/src/link_guard.rs` `unresolved_cross_scope_links` — every `[[project:path]]` / `[[workspace/project:path]]` / `[[_global:path]]` target in a proposed body is resolved with the no-create `lookup_existing_scope` (never creates a scope) and batch-checked via `reader.rs` `existing_latest_paths`; a missing workspace/project, an ambiguous project name, or a missing target page de-links the wikilink, while a genuine store failure aborts the consolidation before any write. Typed `relations:` frontmatter targets go through the same check, and `prune_relations` drops an unresolved entry (an emptied relation and an emptied map with it). `delink_body` → `ai-memory-wiki/src/markdown.rs` `delink_wikilinks` drops the brackets and keeps the label, skipping code fences and inline code exactly as `extract_links` does. Wired into all three apply paths (`consolidator.rs` single-page and batch, `bootstrap.rs`) after the pinned/foreign-slot/portable guards and before the atomic `Wiki::apply_batch` / `write_page`, so an invented cross-project dependency can no longer be written and flagged by the lint pass later. Same-project `[[target]]` links stay untouched by design (the target may be written later). The probe runs with no viewer, so it reports nothing for a `restricted` project (the global preferences project excepted): a link into one is always de-linked, and whether a link survived says nothing about which paths exist there | `consolidator.rs` `batch_delinks_invented_cross_scope_wikilinks` (invented `[[_global:nope]]`, `[[otherproj:notes/missing.md]]`, `[[ghostproj:x.md]]` de-linked to plain text; controls: a cross-scope link to an existing page and a same-project pending link survive byte-identical; `dangling_cross_project_links` left empty), `single_page_delinks_invented_cross_scope_wikilinks`, `single_page_drops_unresolved_cross_scope_relations` (unresolved `relations:` entries dropped; controls: a live cross-scope and a same-project target stay), `link_guard.rs` `prune_relations_drops_only_unresolved_cross_scope_entries`, `bootstrap.rs` `bootstrap_delinks_invented_cross_scope_wikilinks` — each proven to fail with the guard neutralized; `markdown.rs` `delink_wikilinks_strips_only_listed_cross_scope_links`, `delink_wikilinks_without_label_keeps_the_target_text`, `delink_wikilinks_skips_code_and_leaves_empty_set_untouched`; store `multi_session.rs` `existing_latest_paths_reports_exactly_the_live_paths` (exact live-path set, superseded path still resolves, foreign scope sees nothing), `existing_latest_paths_reports_nothing_for_a_restricted_project` (real and absent paths answer alike while restricted; control: reopened, the page is seen) | STRONG |-| 14g | Web project search and its recall trace honour `memory_query`'s scope rules (#708) and never describe another project | `ai-memory-web/src/routes/search.rs` resolves `workspace`+`project` through `resolve_read_scope` (`lookup_existing_scope_guarded`, no-create, `ProjectAccess::Read`) before any ranking, and refuses partial scope; scoped `/api/v1/search` keeps `lookup_project` / `resolve_many_existing_scopes_guarded`; `ai-memory-web/src/recall.rs` ranks only the resolved scopes through `ReaderPool::hybrid_search_scopes`, whose streams and graph expansion are bound to each `(workspace_id, project_id)`; a search with no scope stays `search_pages_with_meta(.., viewer)` (readable repositories only) | `ai-memory-web` `tests/suite/recall_trace.rs` — `project_search_refuses_a_restricted_project_without_a_grant` (bob, with no grant, gets 403 from `/web/search` with and without `explain=1` and from scoped `/api/v1/search`, with no path, body text or trace in the response, and a `scopes` POST naming the restricted project next to an open one is refused; alice's grant and an unrestricted viewer are the controls), `project_search_fails_closed_on_unknown_and_partial_scope` (unknown project or workspace 404 with nothing created, partial scope 400), `explain_renders_stream_ranks_rrf_authority_and_timings` (a better-matching page in a sibling project never reaches the hits or the trace), `search_scope_selector_lists_only_readable_projects` (the search page's "Search in" selector lists only projects the viewer may read, and a hand-typed `scope=` for an ungranted project is a 403 that leaks nothing; controls: the granted reader and an unrestricted viewer; passing no viewer to `list_projects_with_stats` in the handler fails it), `routes.rs` `search_page_scope_selector_round_trips_and_rejects_malformed_values` (a malformed `scope` is a 400); `recall_trace.rs` `kind_filter_does_not_widen_what_a_viewer_can_read` (a `kind`/`tier` filter only narrows: bob gets no hits and a 403 on a scoped search; alice and an unrestricted viewer are the controls), `search_filters_a_project_by_kind_and_tier_keeping_rank_order` (the filter runs after ranking, so survivors keep `memory_query`'s order); `ai-memory-mcp` `tests/suite/web_search_parity.rs` (same top-k order as `memory_query` from `/web` and `/api/v1`, with a deterministic embedder and with none, explain on and off, and merged `scopes`; controls show a web search without the embedder, or the old FTS-only scoped search, would fail it) | STRONG. Bite checks: passing no viewer to `resolve_read_scope` in the search handler, to `lookup_project` or to `resolve_scopes` in the API each fails the restricted-project test; dropping the query vector in `recall.rs` fails the embedder parity tests |+| 14g | Web project search and its recall trace honour `memory_query`'s scope rules (#708) and never describe another project | `ai-memory-web/src/routes/search.rs` resolves `workspace`+`project` through `resolve_read_scope` (`lookup_existing_scope_guarded`, no-create, `ProjectAccess::Read`) before any ranking, and refuses partial scope; scoped `/api/v1/search` keeps `lookup_project` / `resolve_many_existing_scopes_guarded`; `ai-memory-web/src/recall.rs` ranks only the resolved scopes through `ReaderPool::hybrid_search_scopes`, whose streams and graph expansion are bound to each `(workspace_id, project_id)`; a search with no scope stays `search_pages_with_meta(.., viewer)` (readable repositories only) | `ai-memory-web` `tests/suite/recall_trace.rs` — `project_search_refuses_a_restricted_project_without_a_grant` (bob, with no grant, gets 403 from `/web/search` with and without `explain=1` and from scoped `/api/v1/search`, with no path, body text or trace in the response, and a `scopes` POST naming the restricted project next to an open one is refused; alice's grant and an unrestricted viewer are the controls), `project_search_fails_closed_on_unknown_and_partial_scope` (unknown project or workspace 404 with nothing created, partial scope 400), `explain_renders_stream_ranks_rrf_authority_and_timings` (a better-matching page in a sibling project never reaches the hits or the trace), `search_scope_selector_lists_only_readable_projects` (the search page's "Search in" selector lists only projects the viewer may read, and a hand-typed `scope=` for an ungranted project is a 403 that leaks nothing; controls: the granted reader and an unrestricted viewer; passing no viewer to `list_projects_with_stats` in the handler fails it), `routes.rs` `search_page_scope_selector_round_trips_and_rejects_malformed_values` (a malformed `scope` is a 400); `recall_trace.rs` `kind_filter_does_not_widen_what_a_viewer_can_read` (a `kind`/`tier` filter only narrows: bob gets no hits and a 403 on a scoped search; alice and an unrestricted viewer are the controls), `search_filters_a_project_by_kind_and_tier_keeping_rank_order` (survivors of a filter keep the unfiltered order), `reranker_is_shown_only_the_pages_the_filter_keeps` (a dropped page never reaches the reranker), `kind_filter_finds_a_rare_kind_the_unfiltered_page_ranks_out` (the filter sees past the first page of results); `ai-memory-mcp` `tests/suite/web_search_parity.rs` (same top-k order as `memory_query` from `/web` and `/api/v1`, with a deterministic embedder and with none, explain on and off, and merged `scopes`; controls show a web search without the embedder, or the old FTS-only scoped search, would fail it) | STRONG. Bite checks: passing no viewer to `resolve_read_scope` in the search handler, to `lookup_project` or to `resolve_scopes` in the API each fails the restricted-project test; dropping the query vector in `recall.rs` fails the embedder parity tests || 15 | Agent-archive restore path confinement, resource bounds, and consistency | `ai-memory-cli/src/commands/restore_agents.rs` stages only bounded regular UTF-8 entries; requires a one-to-one manifest/body/checksum/target mapping; allowlists destinations by scope; rejects symlinked descendants; writes each file atomically and rolls back earlier writes after a later failure. `backup_agents.rs` skips direct and directory-root source symlinks, bounds individual assets, and atomically replaces a private archive only after gzip completion | `backup_agents.rs` `restore_agents_rejects_path_traversal`, `restore_agents_rejects_unauthorized_destination_path`, `restore_agents_verifies_sha256_integrity`, `restore_agents_requires_a_complete_one_to_one_manifest`, `restore_agents_rejects_oversized_entries_from_the_header`, `restore_agents_rejects_a_symlinked_destination_parent`, `backup_agents_does_not_follow_config_or_asset_root_symlinks`, and `failed_backup_leaves_the_previous_archive_intact`; `backup_and_restore_agents_round_trip` is the valid-path control; `restore_agents::tests::rollback_removes_new_files_and_restores_overwrites` covers recovery | STRONG || 16 | Relay queue identity, migration, and private filesystem state | `companions/ai-memory-relay/src/queue.rs` `Queue::open_inner` performs read-only identity preflight, rechecks under `IMMEDIATE`, commits schema and metadata together; `configure` checks the presence of user objects and rechecks `identity`/`schema_version` metadata on the same connection before every bounded BUSY/LOCKED retry; `fsguard.rs` checks existing paths and creates owner-only files before SQLite opens them | `tests/relay.rs` `a_foreign_database_is_refused_byte_for_byte`, `unknown_schemas_refuse_without_mutation`, `a_foreign_database_with_a_writer_is_refused_before_write_coordination`, `a_foreign_identity_after_preflight_is_refused_without_mutation`, `a_foreign_identity_during_journal_retry_is_refused_without_mutation`, `journal_retry_refuses_a_changed_version_or_empty_database_without_mutation`; controls: `journal_transition_retries_sqlite_lock_without_losing_v1_values`, `fresh_queue_init_and_v1_reopens_work_across_processes`; recovery: `interrupted_and_failed_migrations_roll_back_and_recover`, `journal_reclassification_busy_preserves_committed_migration_and_retries_same_queue`; diagnostics: `classification_read_failures_display_the_sqlite_cause_once_and_preserve_foreign_bytes`; Unix privacy: `the_queue_database_and_its_sidecars_are_owner_only` and the contended migration control | Guard-removal proof covers preflight, locked recheck, and journal retry identity/version checks; full table structure is outside this classification. Windows path rejection is preserved; Windows ACL hardening remains operator-owned |