Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(web): filter search by kind and tier inside the ranker

Peterson Salme1d58eb02c54c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bbc933d8..de9acff3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,12 +14,13 @@ and [0.x](docs/changelog/0.x.md).
### Added
- Added kind and tier filters to the web search. The search page in `/web`
has "Kind" and "Tier" selects, and `/web/search` accepts `kind` and `tier`
- for both a project search and a cross-project one. The filter runs after
- ranking, so the hits that remain keep the order `memory_query` gives them,
- and a filtered search looks through its top 500 matches rather than the
- first page of results, saying so when the ranker had more. The kind match
- ignores case and a nonstandard kind is flagged on an empty result. An
- unknown `tier` is a 400. (#1248)
+ for both a project search and a cross-project one. A filtered search looks
+ through up to 500 candidates per stream rather than the first page of
+ results and says so when the ranker had more, so a rare kind is not crowded
+ out. A project search filters inside the ranker, before the reranker, so
+ the reranker scores only pages the filter keeps. The kind match ignores
+ case and a nonstandard kind is flagged on an empty result. An unknown
+ `tier` is a 400. (#1248)
- Added backlinks, outgoing links and an outline to the web page view. A page
in `/web` now has a side column (below the text on narrow screens) with "On
this page", the h2 and h3 headings linked to their anchors (shown from two
diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rs
index 69510a13..39c48e3b 100644
--- a/crates/ai-memory-mcp/src/server.rs
+++ b/crates/ai-memory-mcp/src/server.rs
@@ -2426,7 +2426,8 @@ impl AiMemoryServer {
// the `expires_at > cutoff` guard, i.e. expired pages stay
// searchable when the caller opted in.
let expiry_cutoff = options.include_expired.then_some(i64::MIN);
- self.reader
+ let found = self
+ .reader
.hybrid_search_scopes(
scopes,
ai_memory_store::HybridSearch {
@@ -2437,9 +2438,11 @@ impl AiMemoryServer {
include_superseded: options.include_superseded,
explain: options.explain,
owner: options.owner,
+ filter: ai_memory_store::PageFilter::default(),
},
)
- .await
+ .await?;
+ Ok((found.hits, found.timings))
}
fn rerank_fetch_limit(&self, limit: usize) -> usize {
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index 8f7e357f..91304d16 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -85,18 +85,18 @@ pub use reader::{
ActivityWindow, AgentSessionCount, AuditEvent, AuditLogFilter, AutoImproveCandidateSession,
AutoImproveParkedClaim, BriefPageBody, BriefingPage, BriefingSnapshot, ClientActivity,
ContaminationFinding, ContaminationReport, ContaminationSummary, ContradictionEdge,
- DecayCandidate, DecayTombstone, DerivedIndexStatus, EmbeddingTripleCount, FeedbackFinding,
- GraphVia, HealthDetail, HealthPage, HybridSearch, InboxCount, InboxEntry, ObservationHit,
- ObservationOrder, ObservationPage, ObservationPageResult, ObservationRecord, OpenSession,
- PAGE_KINDS, PAGE_VERSIONS_MAX_LIMIT, PageAuthor, PageHit, PageHitWithMeta, PageLinks,
- PageLinksPreview, PageMeta, PageSummary, PageVersion, PageVersionBody, PageVersions,
- ProjectCoordinateCollisionReason, ProjectCoordinateDiagnostic, ProjectCoordinateStatus,
- ProjectSummary, RELATED_WALK_MAX_DEPTH, RELATED_WALK_MAX_NODES, ReaderPool,
- ReindexTargetStatus, RelatedNode, RelatedPage, RrfContributions, ScopeRow, SearchExplain,
- SessionConsolidationQueue, SessionConsolidationSummary, SessionDependentRows,
- SessionEndDisposition, SessionSummary, SettledPage, StatusCounts, StorageStatus,
- StoredEmbedding, StoredPageBody, StreamTimings, VectorQuery, WorkspaceScopeRow,
- WorkspaceSummary, f32_vec_to_bytes,
+ DecayCandidate, DecayTombstone, DerivedIndexStatus, EmbeddingTripleCount,
+ FILTER_CANDIDATE_POOL, FeedbackFinding, GraphVia, HealthDetail, HealthPage, HybridHits,
+ HybridSearch, InboxCount, InboxEntry, ObservationHit, ObservationOrder, ObservationPage,
+ ObservationPageResult, ObservationRecord, OpenSession, PAGE_KINDS, PAGE_VERSIONS_MAX_LIMIT,
+ PageAuthor, PageFilter, PageHit, PageHitWithMeta, PageLinks, PageLinksPreview, PageMeta,
+ PageSummary, PageVersion, PageVersionBody, PageVersions, ProjectCoordinateCollisionReason,
+ ProjectCoordinateDiagnostic, ProjectCoordinateStatus, ProjectSummary, RELATED_WALK_MAX_DEPTH,
+ RELATED_WALK_MAX_NODES, ReaderPool, ReindexTargetStatus, RelatedNode, RelatedPage,
+ RrfContributions, ScopeRow, SearchExplain, SessionConsolidationQueue,
+ SessionConsolidationSummary, SessionDependentRows, SessionEndDisposition, SessionSummary,
+ SettledPage, StatusCounts, StorageStatus, StoredEmbedding, StoredPageBody, StreamTimings,
+ VectorQuery, WorkspaceScopeRow, WorkspaceSummary, f32_vec_to_bytes,
};
pub use retrieval_tuning::{RetrievalTuning, is_session_recall_query};
pub use scope::{
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index 581787cb..795f8d38 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -438,12 +438,16 @@ pub(crate) const OBSERVATION_ROWID_SPAN_SQL: &str = "SELECT \
(SELECT MAX(rowid) FROM observations INDEXED BY idx_observations_project \
WHERE project_id = ?1)";
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone)]
struct PageAuthority {
factor: f64,
/// Kind/tier penalty this page carries *because* it is a session page
/// (`0.0` for every other kind). A `session_recall` query gives it back.
session_penalty: f64,
+ /// Semantic kind and tier the factor was derived from, kept so a
+ /// [`PageFilter`] needs no second read of the same rows.
+ kind: String,
+ tier: String,
}
impl PageAuthority {
@@ -516,6 +520,8 @@ impl PageAuthority {
Self {
factor: factor.clamp(0.55, 1.50),
session_penalty,
+ kind: kind.to_owned(),
+ tier: tier.to_owned(),
}
}
@@ -869,6 +875,51 @@ pub struct VectorQuery<'a> {
pub dim: u32,
}
+/// Candidates each stream is asked for when a [`PageFilter`] is active.
+///
+/// The streams have no kind or tier predicate, so a filter can only keep what
+/// they return. A pool this much wider than an unfiltered search's keeps a
+/// rare kind from being crowded out by pages of other kinds that outrank it.
+pub const FILTER_CANDIDATE_POOL: usize = 500;
+
+/// Narrows a [`HybridSearch`] to pages of one kind and/or tier.
+///
+/// Applied after fusion and authority, before the result is cut to
+/// `limit`, so `limit` counts hits that passed it.
+#[derive(Debug, Clone, Copy, Default)]
+pub struct PageFilter<'a> {
+ /// Semantic kind to keep, compared ASCII case-insensitively; `None`
+ /// keeps every kind.
+ pub kind: Option<&'a str>,
+ /// Memory tier to keep; `None` keeps every tier.
+ pub tier: Option<&'a str>,
+}
+
+impl PageFilter<'_> {
+ fn is_active(&self) -> bool {
+ self.kind.is_some() || self.tier.is_some()
+ }
+
+ /// Whether a page of this kind and tier passes the filter.
+ #[must_use]
+ pub fn keeps(&self, kind: &str, tier: &str) -> bool {
+ self.kind.is_none_or(|k| k.eq_ignore_ascii_case(kind))
+ && self.tier.is_none_or(|t| t == tier)
+ }
+}
+
+/// What [`ReaderPool::hybrid_search_scopes`] returns.
+#[derive(Debug)]
+pub struct HybridHits {
+ /// Hits best first, each with its explain details when asked for.
+ pub hits: Vec<(PageHit, Option<SearchExplain>)>,
+ /// Per-stream wall time; `explain` only.
+ pub timings: Option<StreamTimings>,
+ /// A [`PageFilter`] was active and a stream returned its whole
+ /// candidate pool, so matches ranked past it were never looked at.
+ pub filter_pool_full: bool,
+}
+
/// One [`ReaderPool::hybrid_search_scopes`] request.
#[derive(Debug, Clone, Copy)]
pub struct HybridSearch<'a> {
@@ -887,6 +938,8 @@ pub struct HybridSearch<'a> {
/// Who is asking, as the session timeline sees them: an observation
/// excerpt is attached only from a session this filter admits.
pub owner: &'a OwnerFilter,
+ /// Keep only pages of this kind and/or tier; the default keeps all.
+ pub filter: PageFilter<'a>,
}
/// Per-query wall time of each retrieval stream, in milliseconds, for an
@@ -1299,7 +1352,8 @@ pub struct PageHitWithMeta {
/// Relevance rank after the bounded authority adjustment (lower is better).
pub rank: f64,
/// Semantic kind, so the web search can filter without a second lookup.
- /// Not serialized: the MCP `global_hits` shape stays as it was.
+ /// Empty on a recency listing, which nothing filters. Not serialized: the
+ /// MCP `global_hits` shape stays as it was.
#[serde(skip)]
pub kind: String,
/// Memory tier; see `kind`.
@@ -2970,7 +3024,7 @@ impl ReaderPool {
.await?;
let mut authorities: std::collections::HashMap<PageId, PageAuthority> = fts_candidates
.iter()
- .map(|(hit, authority)| (hit.id, *authority))
+ .map(|(hit, authority)| (hit.id, authority.clone()))
.collect();
// Entity-only hits never passed through an FTS candidate row;
// read their authority off the version row itself (no latest
@@ -3570,8 +3624,7 @@ impl ReaderPool {
let sql = format!(
"SELECT workspaces.name, projects.name, pages.path, pages.title, \
{descriptor} AS snip, \
- CAST(pages.updated_at AS REAL) AS rank, \
- {kind_expr} AS kind, pages.tier \
+ CAST(pages.updated_at AS REAL) AS rank \
FROM pages \
JOIN projects ON projects.id = pages.project_id \
JOIN workspaces ON workspaces.id = pages.workspace_id \
@@ -3579,7 +3632,6 @@ impl ReaderPool {
ORDER BY pages.updated_at DESC \
LIMIT ?1",
descriptor = page_descriptor_expr("pages.body", "pages.frontmatter_json"),
- kind_expr = page_kind_expr("pages.path", "pages.frontmatter_json"),
not_expired = not_expired("pages", "?2"),
);
let mut stmt = conn.prepare_cached(&sql)?;
@@ -3593,22 +3645,11 @@ impl ReaderPool {
let title: String = row.get(3)?;
let snippet = page_descriptor(&row.get::<_, String>(4)?, &title);
let rank: f64 = row.get(5)?;
- let kind: String = row.get(6)?;
- let tier: String = row.get(7)?;
- Ok((
- workspace_name,
- project_name,
- path,
- title,
- snippet,
- rank,
- kind,
- tier,
- ))
+ Ok((workspace_name, project_name, path, title, snippet, rank))
})?;
let mut hits = Vec::new();
for row in rows {
- let (workspace_name, project_name, path, title, snippet, rank, kind, tier) = row?;
+ let (workspace_name, project_name, path, title, snippet, rank) = row?;
hits.push(PageHitWithMeta {
workspace_name,
project_name,
@@ -3616,8 +3657,8 @@ impl ReaderPool {
title,
snippet,
rank,
- kind,
- tier,
+ kind: String::new(),
+ tier: String::new(),
});
}
Ok(hits)
@@ -6156,6 +6197,7 @@ impl ReaderPool {
false,
include_superseded,
&OwnerFilter::Any,
+ PageFilter::default(),
)
.await?
.0
@@ -6220,7 +6262,7 @@ impl ReaderPool {
expiry_cutoff_us: Option<i64>,
include_superseded: bool,
) -> StoreResult<(Vec<(PageHit, SearchExplain)>, StreamTimings)> {
- let (hits, timings) = self
+ let (hits, timings, _) = self
.hybrid_search_inner(
workspace_id,
project_id,
@@ -6234,6 +6276,7 @@ impl ReaderPool {
true,
include_superseded,
&OwnerFilter::Any,
+ PageFilter::default(),
)
.await?;
// Belief-strength fields (`evidence_count`, `confidence`,
@@ -6268,7 +6311,7 @@ impl ReaderPool {
&self,
scopes: &[(WorkspaceId, ProjectId)],
search: HybridSearch<'_>,
- ) -> StoreResult<(Vec<(PageHit, Option<SearchExplain>)>, Option<StreamTimings>)> {
+ ) -> StoreResult<HybridHits> {
// Provider/model/dim only select which stored vectors are eligible;
// with no query vector the vector stream never runs, so the empty
// triple is inert rather than a fake identity.
@@ -6282,9 +6325,10 @@ impl ReaderPool {
None => (None, String::new(), String::new(), 0),
};
let mut timings: Option<StreamTimings> = None;
+ let mut filter_pool_full = false;
let mut per_scope = Vec::with_capacity(scopes.len());
for &(workspace_id, project_id) in scopes {
- let (hits, scope_timings) = self
+ let (hits, scope_timings, pool_full) = self
.hybrid_search_inner(
workspace_id,
project_id,
@@ -6298,8 +6342,10 @@ impl ReaderPool {
search.explain,
search.include_superseded,
search.owner,
+ search.filter,
)
.await?;
+ filter_pool_full |= pool_full;
if search.explain {
timings
.get_or_insert_with(Default::default)
@@ -6345,7 +6391,11 @@ impl ReaderPool {
} else {
hits
};
- Ok((hits, timings))
+ Ok(HybridHits {
+ hits,
+ timings,
+ filter_pool_full,
+ })
}
/// The streams [`Self::hybrid_search_scopes`] runs for a query, in the
@@ -6381,7 +6431,8 @@ impl ReaderPool {
explain: bool,
include_superseded: bool,
owner_filter: &OwnerFilter,
- ) -> StoreResult<(Vec<(PageHit, Option<SearchExplain>)>, StreamTimings)> {
+ filter: PageFilter<'_>,
+ ) -> StoreResult<(Vec<(PageHit, Option<SearchExplain>)>, StreamTimings, bool)> {
let cutoff = expiry_cutoff_us.unwrap_or_else(now_us);
// Stream timings exist only for explain: `explain.then(Instant::now)`
// reads no clock on the normal path.
@@ -6394,7 +6445,13 @@ impl ReaderPool {
// bounded candidate window used by FTS-only search. A `limit * 2`
// window was too narrow at small limits for a canonical page to enter
// the fused pool and receive its bounded promotion.
- let candidate_limit = authority_candidate_limit(limit);
+ // A filter can only keep what the streams return, so they are asked
+ // for a wide pool and `limit` is applied to what survives.
+ let candidate_limit = authority_candidate_limit(if filter.is_active() {
+ limit.max(FILTER_CANDIDATE_POOL)
+ } else {
+ limit
+ });
// Tokenize the borrowed query before the FTS candidate call consumes
// it. This avoids cloning a caller-controlled query on the hot path.
// The default path uses the latest-only entry; the opt-in path reaches
@@ -6442,7 +6499,7 @@ impl ReaderPool {
timings.fts_ms = elapsed_ms(started);
let mut authorities: std::collections::HashMap<PageId, PageAuthority> = fts_candidates
.iter()
- .map(|(hit, authority)| (hit.id, *authority))
+ .map(|(hit, authority)| (hit.id, authority.clone()))
.collect();
let fts_hits: Vec<PageHit> = fts_candidates
.into_iter()
@@ -6543,6 +6600,11 @@ impl ReaderPool {
Vec::new()
};
+ let filter_pool_full = filter.is_active()
+ && [fts_hits.len(), vec_hits.len(), entity_hits.len()]
+ .into_iter()
+ .any(|returned| returned >= candidate_limit);
+
// RRF fuse: score(d) = Σ 1/(k + rank_i(d)) over rankers.
let k = 60.0_f64;
struct Fused {
@@ -6852,6 +6914,13 @@ impl ReaderPool {
// back to HashMap iteration order.
.then_with(|| a.0.path.as_str().cmp(b.0.path.as_str()))
});
+ if filter.is_active() {
+ out.retain(|(hit, _)| {
+ authorities
+ .get(&hit.id)
+ .is_some_and(|a| filter.keeps(&a.kind, &a.tier))
+ });
+ }
out.truncate(limit);
// Label the superseded versions among the final hits so the caller can
// tell historical versions from the current one. Only the opt-in path
@@ -6866,7 +6935,7 @@ impl ReaderPool {
hit.superseded = superseded.contains(&hit.id);
}
}
- Ok((out, timings))
+ Ok((out, timings, filter_pool_full))
}
/// The subset of `page_ids` that are superseded (non-latest) versions in
@@ -8526,52 +8595,6 @@ impl ReaderPool {
.await
}
- /// Semantic kind and tier of each latest page in `page_ids`, in one query.
- ///
- /// Ids that name no latest page are absent from the map. The caller has
- /// already scoped the ids (they come from a search the viewer was allowed
- /// to run), so this reads no repository the viewer could not.
- ///
- /// # Errors
- /// Propagates any SQL or pool error.
- pub async fn page_kinds_tiers_by_ids(
- &self,
- page_ids: &[PageId],
- ) -> StoreResult<std::collections::HashMap<PageId, (String, String)>> {
- if page_ids.is_empty() {
- return Ok(std::collections::HashMap::new());
- }
- let id_blobs: Vec<Value> = page_ids
- .iter()
- .map(|id| Value::Blob(id.as_bytes().to_vec()))
- .collect();
- self.with_conn(move |conn| {
- let placeholders = std::iter::repeat_n("?", id_blobs.len())
- .collect::<Vec<_>>()
- .join(", ");
- let sql = format!(
- "SELECT pg.id, {kind_expr}, pg.tier FROM pages pg \
- WHERE pg.is_latest = 1 AND pg.id IN ({placeholders})",
- kind_expr = page_kind_expr("pg.path", "pg.frontmatter_json"),
- );
- let mut stmt = conn.prepare(&sql)?;
- let rows = stmt.query_map(params_from_iter(id_blobs.iter()), |row| {
- Ok((
- row.get::<_, Vec<u8>>(0)?,
- row.get::<_, String>(1)?,
- row.get::<_, String>(2)?,
- ))
- })?;
- let mut out = std::collections::HashMap::new();
- for row in rows {
- let (id, kind, tier) = row?;
- out.insert(PageId::from_slice(&id)?, (kind, tier));
- }
- Ok(out)
- })
- .await
- }
-
/// Look up a page's workspace and project names by its path (across all
/// workspaces and projects). Returns the first `is_latest = 1` match.
///
@@ -12922,11 +12945,12 @@ mod tests {
explain,
false,
&OwnerFilter::Any,
+ super::PageFilter::default(),
)
};
- let (_, off) = search(false).await.unwrap();
+ let (_, off, _) = search(false).await.unwrap();
assert_eq!(off, super::StreamTimings::default());
- let (_, on) = search(true).await.unwrap();
+ let (_, on, _) = search(true).await.unwrap();
assert!(on.fts_ms.is_some() && on.entity_ms.is_some() && on.graph_ms.is_some());
assert!(on.observation_ms.is_some(), "the stream is on by default");
assert!(on.vector_ms.is_none(), "no query vector, no vector scan");
diff --git a/crates/ai-memory-store/tests/suite/multi_session.rs b/crates/ai-memory-store/tests/suite/multi_session.rs
index 5a5d3816..44704e60 100644
--- a/crates/ai-memory-store/tests/suite/multi_session.rs
+++ b/crates/ai-memory-store/tests/suite/multi_session.rs
@@ -3550,11 +3550,12 @@ async fn explained_search_as(
include_superseded: false,
explain: true,
owner,
+ filter: ai_memory_store::PageFilter::default(),
},
)
.await
.unwrap()
- .0
+ .hits
.into_iter()
.map(|(hit, details)| (hit, details.unwrap()))
.collect()
diff --git a/crates/ai-memory-web/src/recall.rs b/crates/ai-memory-web/src/recall.rs
index e84e34a4..1e3ca002 100644
--- a/crates/ai-memory-web/src/recall.rs
+++ b/crates/ai-memory-web/src/recall.rs
@@ -19,7 +19,7 @@ use std::sync::Arc;
use ai_memory_core::{OwnerFilter, ProjectId, WorkspaceId};
use ai_memory_llm::{Embedder, QueryVector, RerankCandidate, RerankStage};
use ai_memory_store::{
- HybridSearch, PageHit, SearchExplain, StoreResult, StreamTimings, VectorQuery,
+ HybridSearch, PageFilter, PageHit, SearchExplain, StoreResult, StreamTimings, VectorQuery,
};
use crate::state::WebState;
@@ -43,12 +43,17 @@ pub(crate) struct RecallResult {
pub timings: Option<StreamTimings>,
/// The streams that ran, in `memory_query`'s `streams_active` order.
pub streams_active: Vec<&'static str>,
+ /// A filter was set and a stream returned its whole candidate pool, so
+ /// matches ranked past it were not looked at.
+ pub filter_pool_full: bool,
}
-/// Rank `query` over `scopes` exactly as a `memory_query` project/scopes
-/// search does (latest versions only, expired pages hidden). `owner` is the
-/// viewer's session-timeline filter: an observation excerpt is shown only
-/// from a session it admits, as in `memory_query`.
+/// Rank `query` over `scopes` as a `memory_query` project/scopes search does
+/// (latest versions only, expired pages hidden), keeping only pages that pass
+/// `filter`. The filter runs in the ranker, before the reranker, so the
+/// reranker scores only pages the caller can see. `owner` is the viewer's
+/// session-timeline filter: an observation excerpt is shown only from a
+/// session it admits, as in `memory_query`.
pub(crate) async fn search_scopes(
state: &WebState,
scopes: &[(WorkspaceId, ProjectId)],
@@ -56,6 +61,7 @@ pub(crate) async fn search_scopes(
limit: usize,
explain: bool,
owner: &OwnerFilter,
+ filter: PageFilter<'_>,
) -> StoreResult<RecallResult> {
let query_vector: Option<QueryVector> = match &state.recall.embedder {
Some(embedder) => ai_memory_llm::embed_search_query(embedder.as_ref(), query).await,
@@ -66,7 +72,7 @@ pub(crate) async fn search_scopes(
.rerank
.as_ref()
.map_or(limit, |stage| stage.fetch_limit(limit));
- let (hits, mut timings) = state
+ let found = state
.reader
.hybrid_search_scopes(
scopes,
@@ -83,9 +89,11 @@ pub(crate) async fn search_scopes(
include_superseded: false,
explain,
owner,
+ filter,
},
)
.await?;
+ let (hits, mut timings) = (found.hits, found.timings);
let hits = match &state.recall.rerank {
Some(stage) => {
let started = explain.then(std::time::Instant::now);
@@ -122,5 +130,6 @@ pub(crate) async fn search_scopes(
hits,
timings,
streams_active: state.reader.active_streams(query_vector.is_some()),
+ filter_pool_full: found.filter_pool_full,
})
}
diff --git a/crates/ai-memory-web/src/routes/api.rs b/crates/ai-memory-web/src/routes/api.rs
index ffed748f..3734a981 100644
--- a/crates/ai-memory-web/src/routes/api.rs
+++ b/crates/ai-memory-web/src/routes/api.rs
@@ -406,7 +406,8 @@ async fn search_with_request(
.iter()
.map(|scope| (scope.workspace_id, scope.project_id))
.collect();
- crate::recall::search_scopes(state, &scopes, &term, limit, false, owner)
+ let filter = ai_memory_store::PageFilter::default();
+ crate::recall::search_scopes(state, &scopes, &term, limit, false, owner, filter)
.await
.map(|result| result.hits.into_iter().map(|(hit, _)| hit).collect())
}
diff --git a/crates/ai-memory-web/src/routes/search.rs b/crates/ai-memory-web/src/routes/search.rs
index acc494e1..edd3c1bf 100644
--- a/crates/ai-memory-web/src/routes/search.rs
+++ b/crates/ai-memory-web/src/routes/search.rs
@@ -12,15 +12,19 @@
//! viewer may read, as `memory_query(global=true)` does. Its trace is the
//! FTS time alone; there are no per-hit streams to show.
//!
-//! `kind` and `tier` narrow either mode *after* ranking, so the surviving
-//! hits keep the order `memory_query` gives them.
+//! `kind` and `tier` narrow either mode after fusion, inside the ranker. The
+//! streams are asked for a wider candidate pool so a rare kind is not crowded
+//! out, which means the order of the survivors can differ slightly from the
+//! unfiltered top hits of the same query.
use std::collections::BTreeMap;
use std::str::FromStr;
use std::sync::Arc;
use ai_memory_core::Tier;
-use ai_memory_store::{PAGE_KINDS, SearchExplain, StreamTimings};
+use ai_memory_store::{
+ FILTER_CANDIDATE_POOL, PAGE_KINDS, PageFilter, SearchExplain, StreamTimings,
+};
use askama::Template;
use axum::extract::{Query, State};
use axum::http::StatusCode;
@@ -40,9 +44,6 @@ const GLOBAL_DEFAULT_LIMIT: usize = 50;
const SCOPED_DEFAULT_LIMIT: usize = 20;
/// Upper bound on `limit`, the same clamp `memory_query` applies.
const MAX_LIMIT: usize = 100;
-/// Matches a filtered search ranks before it filters. The ranker has no
-/// kind or tier predicate, so a filter only sees this many top hits.
-const FILTER_POOL: usize = 500;
/// Memory tiers the filter offers, in `Tier` declaration order.
const TIERS: [&str; 4] = ["working", "episodic", "semantic", "procedural"];
@@ -138,14 +139,19 @@ pub(crate) async fn handler(
None => None,
};
let filtered = kind.is_some() || tier.is_some();
- let keeps = |hit_kind: &str, hit_tier: &str| {
- kind.as_deref()
- .is_none_or(|k| k.eq_ignore_ascii_case(hit_kind))
- && tier.as_deref().is_none_or(|t| t == hit_tier)
+ let filter = PageFilter {
+ kind: kind.as_deref(),
+ tier: tier.as_deref(),
+ };
+ // The global search has no ranker-side filter, so a filtered one is asked
+ // for a wide pool and the limit is applied to what survives.
+ let fetch_limit = |limit: usize| {
+ if filtered {
+ FILTER_CANDIDATE_POOL
+ } else {
+ limit + 1
+ }
};
- // A filter runs after ranking, so the ranker is asked for a wide pool
- // and the limit is applied to what survives.
- let fetch_limit = |limit: usize| if filtered { FILTER_POOL } else { limit + 1 };
let internal = |error: ai_memory_store::StoreError| {
tracing::error!(%error, "web search failed");
StatusCode::INTERNAL_SERVER_ERROR.into_response()
@@ -177,9 +183,10 @@ pub(crate) async fn handler(
&state,
&[(resolved.workspace_id, resolved.project_id)],
&query,
- fetch_limit(limit),
+ limit + 1,
explain,
&super::api::owner_filter_for(actor),
+ filter,
)
.await
{
@@ -188,23 +195,10 @@ pub(crate) async fn handler(
};
timings = result.timings;
streams_active = result.streams_active;
- // The trace position is the rank before filtering.
- let mut ranked: Vec<_> = result.hits.into_iter().enumerate().collect();
- pool_capped = filtered && ranked.len() >= FILTER_POOL;
- if filtered {
- let ids: Vec<_> = ranked.iter().map(|(_, (hit, _))| hit.id).collect();
- let facets = match state.reader.page_kinds_tiers_by_ids(&ids).await {
- Ok(facets) => facets,
- Err(error) => return internal(error),
- };
- ranked.retain(|(_, (hit, _))| {
- facets
- .get(&hit.id)
- .is_some_and(|(hit_kind, hit_tier)| keeps(hit_kind, hit_tier))
- });
- }
- truncated = ranked.len() > limit;
- for (position, (hit, details)) in ranked.into_iter().take(limit) {
+ pool_capped = result.filter_pool_full;
+ truncated = result.hits.len() > limit;
+ // The trace position is the rank among the hits shown.
+ for (position, (hit, details)) in result.hits.into_iter().take(limit).enumerate() {
let path = hit.path.as_str().to_owned();
hits.push(SearchHit {
href: page_href(workspace, project, &path),
@@ -242,10 +236,10 @@ pub(crate) async fn handler(
});
streams_active = vec!["fts"];
}
- pool_capped = filtered && raw.len() >= FILTER_POOL;
+ pool_capped = filtered && raw.len() >= FILTER_CANDIDATE_POOL;
let raw: Vec<_> = raw
.into_iter()
- .filter(|h| keeps(&h.kind, &h.tier))
+ .filter(|h| filter.keeps(&h.kind, &h.tier))
.collect();
truncated = raw.len() > limit;
for h in raw.into_iter().take(limit) {
@@ -313,7 +307,7 @@ pub(crate) async fn handler(
hit_count,
filtered,
pool_capped,
- filter_pool: FILTER_POOL,
+ filter_pool: FILTER_CANDIDATE_POOL,
unknown_kind,
kind_options,
tier_options,
diff --git a/crates/ai-memory-web/tests/suite/recall_trace.rs b/crates/ai-memory-web/tests/suite/recall_trace.rs
index c46ef7b1..05ddbbfb 100644
--- a/crates/ai-memory-web/tests/suite/recall_trace.rs
+++ b/crates/ai-memory-web/tests/suite/recall_trace.rs
@@ -8,13 +8,15 @@
//! partial scope rejected, and no other project's pages in the result or the
//! trace.
-use std::sync::Arc;
+use std::sync::{Arc, Mutex};
use ai_memory_core::{
ActorContext, AuthorizedViewer, NewUser, PagePath, ProjectId, Tier, UserId, UserRole,
WorkspaceId,
};
-use ai_memory_llm::{Embedder, SyntheticEmbedder};
+use ai_memory_llm::{
+ Embedder, RerankCandidate, RerankScore, RerankStage, Reranker, SyntheticEmbedder,
+};
use ai_memory_store::{GrantLevel, Store};
use ai_memory_web::{Recall, WebMountSpec, split_web_routers};
use ai_memory_wiki::{Wiki, WritePageRequest};
@@ -713,3 +715,138 @@ fn tier_filter_offers_every_tier() {
assert_eq!(name.parse::<Tier>().unwrap(), tier);
}
}
+
+/// Scores every candidate equally and remembers which pages it was shown.
+#[derive(Default)]
+struct RecordingReranker {
+ seen: Mutex<Vec<String>>,
+}
+
+#[async_trait::async_trait]
+impl Reranker for RecordingReranker {
+ fn name(&self) -> &'static str {
+ "recording"
+ }
+
+ fn model(&self) -> &str {
+ "test"
+ }
+
+ async fn rerank(
+ &self,
+ _query: &str,
+ candidates: &[RerankCandidate],
+ ) -> ai_memory_llm::LlmResult<Vec<RerankScore>> {
+ self.seen
+ .lock()
+ .unwrap()
+ .extend(candidates.iter().map(|c| c.title.clone()));
+ Ok(candidates
+ .iter()
+ .map(|c| RerankScore {
+ id: c.id.clone(),
+ relevance: 0.5,
+ })
+ .collect())
+ }
+}
+
+/// The filter runs in the ranker, before the reranker: pages the filter
+/// drops are never sent to the (paid, slow) reranker.
+#[tokio::test]
+async fn reranker_is_shown_only_the_pages_the_filter_keeps() {
+ let f = fixture(false).await;
+ add_procedure(&f).await;
+ // The stage leaves a single candidate alone, so the filter needs two.
+ let ws = f
+ .store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ f.wiki
+ .write_page(WritePageRequest {
+ frontmatter: serde_json::json!({"kind": "procedure"}),
+ tier: Tier::Procedural,
+ ..req(
+ ws,
+ f.alpha,
+ "procedures/audit.md",
+ "# Audit\n\nAudit every token rotation after the script ran.",
+ )
+ })
+ .await
+ .unwrap();
+ let reranker = Arc::new(RecordingReranker::default());
+ let recall = Recall {
+ embedder: None,
+ rerank: Some(RerankStage::new(reranker.clone())),
+ };
+ let app = web(&f, recall);
+ let base = "/web/search?q=token%20rotation&workspace=default&project=alpha";
+
+ // Control: unfiltered, the reranker sees more than the procedure.
+ let (status, body) = get(&app, base, None).await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ let unfiltered = std::mem::take(&mut *reranker.seen.lock().unwrap());
+ assert!(unfiltered.len() > 2, "{unfiltered:?}");
+
+ let (status, body) = get(&app, &format!("{base}&kind=procedure"), None).await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ let mut shown = hit_paths(&body);
+ shown.sort();
+ assert_eq!(
+ shown,
+ ["procedures/audit.md", "procedures/rotate.md"],
+ "{body}"
+ );
+ let filtered = reranker.seen.lock().unwrap().clone();
+ assert_eq!(filtered.len(), 2, "{filtered:?}");
+}
+
+/// A rare kind that the unfiltered result ranks out of view is still found:
+/// the filter is applied to a candidate pool far wider than one page.
+#[tokio::test]
+async fn kind_filter_finds_a_rare_kind_the_unfiltered_page_ranks_out() {
+ let f = fixture(false).await;
+ let ws = f
+ .store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ for n in 0..60 {
+ f.wiki
+ .write_page(req(
+ ws,
+ f.alpha,
+ &format!("notes/crowd-{n}.md"),
+ "# Crowd\n\ncrowdword crowdword crowdword crowdword crowdword.",
+ ))
+ .await
+ .unwrap();
+ }
+ f.wiki
+ .write_page(WritePageRequest {
+ frontmatter: serde_json::json!({"kind": "gotcha"}),
+ ..req(
+ ws,
+ f.alpha,
+ "gotchas/quiet.md",
+ "# Quiet\n\nOne mention of crowdword among many other unrelated words \
+ about cats, rivers, mountains, bridges and old maps.",
+ )
+ })
+ .await
+ .unwrap();
+ let app = web(&f, Recall::default());
+ let base = "/web/search?q=crowdword&workspace=default&project=alpha&limit=5";
+
+ let (_, body) = get(&app, base, None).await;
+ assert!(
+ !hit_paths(&body).contains(&"gotchas/quiet.md".to_owned()),
+ "control: the gotcha must be ranked out of the first page: {body}"
+ );
+ let (_, body) = get(&app, &format!("{base}&kind=gotcha"), None).await;
+ assert_eq!(hit_paths(&body), ["gotchas/quiet.md"], "{body}");
+}
diff --git a/docs/security-boundaries.md b/docs/security-boundaries.md
index 131e4613..5fa0c071 100644
--- a/docs/security-boundaries.md
+++ b/docs/security-boundaries.md
@@ -105,7 +105,7 @@ means the boundary is not built yet.
| 14d | `include_related` and the web page's backlinks show and walk only readable projects (#708) | `ai-memory-store/src/reader.rs` `related_walk(.., viewer)` — both the outgoing and incoming hop queries carry `readable_repository_predicate("pg.project_id", viewer)`, the same far-end filter `page_links` and `page_links_preview` apply (the preview's visible totals are counted under it too), so a page in a project the viewer cannot read is neither returned nor expanded; `ai-memory-mcp/src/server.rs` `memory_read_page` passes `viewer_from_parts`, `ai-memory-web/src/routes/page.rs` passes the request's viewer to `page_links_preview`, and the access bump only touches the filtered nodes | `related_walk.rs` `the_walk_hides_and_does_not_cross_projects_the_viewer_cannot_read` (an outgoing and an incoming neighbour in an ungranted restricted project are absent, and so is a granted page reachable only through one; control: a granted restricted project is shown, and no viewer walks the whole graph) — fails with the predicate removed; `ai-memory-web` `tests/suite/routes.rs` `page_backlink_from_an_unreadable_project_is_hidden` (the `/web` page's "Cited by" omits a page in an ungranted restricted project, with no title, path or project name; controls: a reader of both projects and an unrestricted viewer see it) — fails with the page handler passing no viewer to `page_links_preview`; `related_walk.rs` `page_links_preview_caps_in_sql_and_counts_only_readable_neighbours` (back-links from an ungranted restricted project are neither listed nor counted in the total behind "and N more"; control: no viewer counts them all) — fails with the predicate removed; `ai-memory-mcp` `server.rs` `bob_does_not_see_restricted_pages_through_the_related_walk` (an ungranted user's `memory_read_page{include_related}` omits the restricted page; control: no viewer still reaches it) — fails with `memory_read_page` passing `None` instead of `viewer_from_parts` | STRONG |
| 14e | `memory_write_page` session evidence cites only a session of the target project (#708) | `ai-memory-mcp/src/server.rs` `memory_write_page` — a `session_id` is accepted only when `require_session_in_scope` resolves the session (where its observations landed, else its session row, as `Consolidator::session_target` does) to the same `(workspace_id, project_id)` the write grant was checked against; unknown and foreign ids get one refusal, so the error is no existence oracle; `scope: "global"` with a `session_id` is refused; `refuse_pinned_page_overwrite` keeps a write that cites a session off a pinned page; only a write to `sessions/<id>.md` reconciles the session's queued SessionEnd job | `server.rs` `write_page_refuses_a_session_from_another_project` (a foreign and an unknown session refused, no evidence row written, the foreign job still queued; control: the project's own session accepted), `session_evidence_write_leaves_a_pinned_page_alone` (control: the same write without `session_id` edits the page), `session_evidence_on_another_path_leaves_the_job_queued`, `write_page_rejects_a_malformed_or_global_session_id` — each of the scope check, the pinned check and the session-page-only reconcile proven to fail its test when removed | STRONG |
| 14f | Consolidation apply path validates cross-scope wikilinks against the store (#1218) | `ai-memory-consolidate/src/link_guard.rs` `unresolved_cross_scope_links` — every `[[project:path]]` / `[[workspace/project:path]]` / `[[_global:path]]` target in a proposed body is resolved with the no-create `lookup_existing_scope` (never creates a scope) and batch-checked via `reader.rs` `existing_latest_paths`; a missing workspace/project, an ambiguous project name, or a missing target page de-links the wikilink, while a genuine store failure aborts the consolidation before any write. Typed `relations:` frontmatter targets go through the same check, and `prune_relations` drops an unresolved entry (an emptied relation and an emptied map with it). `delink_body` → `ai-memory-wiki/src/markdown.rs` `delink_wikilinks` drops the brackets and keeps the label, skipping code fences and inline code exactly as `extract_links` does. Wired into all three apply paths (`consolidator.rs` single-page and batch, `bootstrap.rs`) after the pinned/foreign-slot/portable guards and before the atomic `Wiki::apply_batch` / `write_page`, so an invented cross-project dependency can no longer be written and flagged by the lint pass later. Same-project `[[target]]` links stay untouched by design (the target may be written later). The probe runs with no viewer, so it reports nothing for a `restricted` project (the global preferences project excepted): a link into one is always de-linked, and whether a link survived says nothing about which paths exist there | `consolidator.rs` `batch_delinks_invented_cross_scope_wikilinks` (invented `[[_global:nope]]`, `[[otherproj:notes/missing.md]]`, `[[ghostproj:x.md]]` de-linked to plain text; controls: a cross-scope link to an existing page and a same-project pending link survive byte-identical; `dangling_cross_project_links` left empty), `single_page_delinks_invented_cross_scope_wikilinks`, `single_page_drops_unresolved_cross_scope_relations` (unresolved `relations:` entries dropped; controls: a live cross-scope and a same-project target stay), `link_guard.rs` `prune_relations_drops_only_unresolved_cross_scope_entries`, `bootstrap.rs` `bootstrap_delinks_invented_cross_scope_wikilinks` — each proven to fail with the guard neutralized; `markdown.rs` `delink_wikilinks_strips_only_listed_cross_scope_links`, `delink_wikilinks_without_label_keeps_the_target_text`, `delink_wikilinks_skips_code_and_leaves_empty_set_untouched`; store `multi_session.rs` `existing_latest_paths_reports_exactly_the_live_paths` (exact live-path set, superseded path still resolves, foreign scope sees nothing), `existing_latest_paths_reports_nothing_for_a_restricted_project` (real and absent paths answer alike while restricted; control: reopened, the page is seen) | STRONG |
-| 14g | Web project search and its recall trace honour `memory_query`'s scope rules (#708) and never describe another project | `ai-memory-web/src/routes/search.rs` resolves `workspace`+`project` through `resolve_read_scope` (`lookup_existing_scope_guarded`, no-create, `ProjectAccess::Read`) before any ranking, and refuses partial scope; scoped `/api/v1/search` keeps `lookup_project` / `resolve_many_existing_scopes_guarded`; `ai-memory-web/src/recall.rs` ranks only the resolved scopes through `ReaderPool::hybrid_search_scopes`, whose streams and graph expansion are bound to each `(workspace_id, project_id)`; a search with no scope stays `search_pages_with_meta(.., viewer)` (readable repositories only) | `ai-memory-web` `tests/suite/recall_trace.rs` — `project_search_refuses_a_restricted_project_without_a_grant` (bob, with no grant, gets 403 from `/web/search` with and without `explain=1` and from scoped `/api/v1/search`, with no path, body text or trace in the response, and a `scopes` POST naming the restricted project next to an open one is refused; alice's grant and an unrestricted viewer are the controls), `project_search_fails_closed_on_unknown_and_partial_scope` (unknown project or workspace 404 with nothing created, partial scope 400), `explain_renders_stream_ranks_rrf_authority_and_timings` (a better-matching page in a sibling project never reaches the hits or the trace), `search_scope_selector_lists_only_readable_projects` (the search page's "Search in" selector lists only projects the viewer may read, and a hand-typed `scope=` for an ungranted project is a 403 that leaks nothing; controls: the granted reader and an unrestricted viewer; passing no viewer to `list_projects_with_stats` in the handler fails it), `routes.rs` `search_page_scope_selector_round_trips_and_rejects_malformed_values` (a malformed `scope` is a 400); `recall_trace.rs` `kind_filter_does_not_widen_what_a_viewer_can_read` (a `kind`/`tier` filter only narrows: bob gets no hits and a 403 on a scoped search; alice and an unrestricted viewer are the controls), `search_filters_a_project_by_kind_and_tier_keeping_rank_order` (the filter runs after ranking, so survivors keep `memory_query`'s order); `ai-memory-mcp` `tests/suite/web_search_parity.rs` (same top-k order as `memory_query` from `/web` and `/api/v1`, with a deterministic embedder and with none, explain on and off, and merged `scopes`; controls show a web search without the embedder, or the old FTS-only scoped search, would fail it) | STRONG. Bite checks: passing no viewer to `resolve_read_scope` in the search handler, to `lookup_project` or to `resolve_scopes` in the API each fails the restricted-project test; dropping the query vector in `recall.rs` fails the embedder parity tests |
+| 14g | Web project search and its recall trace honour `memory_query`'s scope rules (#708) and never describe another project | `ai-memory-web/src/routes/search.rs` resolves `workspace`+`project` through `resolve_read_scope` (`lookup_existing_scope_guarded`, no-create, `ProjectAccess::Read`) before any ranking, and refuses partial scope; scoped `/api/v1/search` keeps `lookup_project` / `resolve_many_existing_scopes_guarded`; `ai-memory-web/src/recall.rs` ranks only the resolved scopes through `ReaderPool::hybrid_search_scopes`, whose streams and graph expansion are bound to each `(workspace_id, project_id)`; a search with no scope stays `search_pages_with_meta(.., viewer)` (readable repositories only) | `ai-memory-web` `tests/suite/recall_trace.rs` — `project_search_refuses_a_restricted_project_without_a_grant` (bob, with no grant, gets 403 from `/web/search` with and without `explain=1` and from scoped `/api/v1/search`, with no path, body text or trace in the response, and a `scopes` POST naming the restricted project next to an open one is refused; alice's grant and an unrestricted viewer are the controls), `project_search_fails_closed_on_unknown_and_partial_scope` (unknown project or workspace 404 with nothing created, partial scope 400), `explain_renders_stream_ranks_rrf_authority_and_timings` (a better-matching page in a sibling project never reaches the hits or the trace), `search_scope_selector_lists_only_readable_projects` (the search page's "Search in" selector lists only projects the viewer may read, and a hand-typed `scope=` for an ungranted project is a 403 that leaks nothing; controls: the granted reader and an unrestricted viewer; passing no viewer to `list_projects_with_stats` in the handler fails it), `routes.rs` `search_page_scope_selector_round_trips_and_rejects_malformed_values` (a malformed `scope` is a 400); `recall_trace.rs` `kind_filter_does_not_widen_what_a_viewer_can_read` (a `kind`/`tier` filter only narrows: bob gets no hits and a 403 on a scoped search; alice and an unrestricted viewer are the controls), `search_filters_a_project_by_kind_and_tier_keeping_rank_order` (survivors of a filter keep the unfiltered order), `reranker_is_shown_only_the_pages_the_filter_keeps` (a dropped page never reaches the reranker), `kind_filter_finds_a_rare_kind_the_unfiltered_page_ranks_out` (the filter sees past the first page of results); `ai-memory-mcp` `tests/suite/web_search_parity.rs` (same top-k order as `memory_query` from `/web` and `/api/v1`, with a deterministic embedder and with none, explain on and off, and merged `scopes`; controls show a web search without the embedder, or the old FTS-only scoped search, would fail it) | STRONG. Bite checks: passing no viewer to `resolve_read_scope` in the search handler, to `lookup_project` or to `resolve_scopes` in the API each fails the restricted-project test; dropping the query vector in `recall.rs` fails the embedder parity tests |
| 15 | Agent-archive restore path confinement, resource bounds, and consistency | `ai-memory-cli/src/commands/restore_agents.rs` stages only bounded regular UTF-8 entries; requires a one-to-one manifest/body/checksum/target mapping; allowlists destinations by scope; rejects symlinked descendants; writes each file atomically and rolls back earlier writes after a later failure. `backup_agents.rs` skips direct and directory-root source symlinks, bounds individual assets, and atomically replaces a private archive only after gzip completion | `backup_agents.rs` `restore_agents_rejects_path_traversal`, `restore_agents_rejects_unauthorized_destination_path`, `restore_agents_verifies_sha256_integrity`, `restore_agents_requires_a_complete_one_to_one_manifest`, `restore_agents_rejects_oversized_entries_from_the_header`, `restore_agents_rejects_a_symlinked_destination_parent`, `backup_agents_does_not_follow_config_or_asset_root_symlinks`, and `failed_backup_leaves_the_previous_archive_intact`; `backup_and_restore_agents_round_trip` is the valid-path control; `restore_agents::tests::rollback_removes_new_files_and_restores_overwrites` covers recovery | STRONG |
| 16 | Relay queue identity, migration, and private filesystem state | `companions/ai-memory-relay/src/queue.rs` `Queue::open_inner` performs read-only identity preflight, rechecks under `IMMEDIATE`, commits schema and metadata together; `configure` checks the presence of user objects and rechecks `identity`/`schema_version` metadata on the same connection before every bounded BUSY/LOCKED retry; `fsguard.rs` checks existing paths and creates owner-only files before SQLite opens them | `tests/relay.rs` `a_foreign_database_is_refused_byte_for_byte`, `unknown_schemas_refuse_without_mutation`, `a_foreign_database_with_a_writer_is_refused_before_write_coordination`, `a_foreign_identity_after_preflight_is_refused_without_mutation`, `a_foreign_identity_during_journal_retry_is_refused_without_mutation`, `journal_retry_refuses_a_changed_version_or_empty_database_without_mutation`; controls: `journal_transition_retries_sqlite_lock_without_losing_v1_values`, `fresh_queue_init_and_v1_reopens_work_across_processes`; recovery: `interrupted_and_failed_migrations_roll_back_and_recover`, `journal_reclassification_busy_preserves_committed_migration_and_retries_same_queue`; diagnostics: `classification_read_failures_display_the_sqlite_cause_once_and_preserve_foreign_bytes`; Unix privacy: `the_queue_database_and_its_sidecars_are_owner_only` and the contended migration control | Guard-removal proof covers preflight, locked recheck, and journal retry identity/version checks; full table structure is outside this classification. Windows path rejection is preserved; Windows ACL hardening remains operator-owned |