Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(store): explain mode for memory_query

Samir Hanna Verza8aafb67a19fb
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d34fe6fd..0ad0ead8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -42,6 +42,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [1.20.0] - 2026-07-30
### Added
+- `memory_query` explain mode: pass `explain: true` and every hit carries
+ `score_details` — its 1-based rank and raw score in each retrieval
+ stream (FTS5 bm25 score, embedding cosine, graph-neighbour provenance
+ naming the seed page and link direction), the per-stream RRF
+ contributions, the fused score, and the bounded page-authority
+ multiplier applied after fusion — plus a top-level `streams_active`
+ list that makes degradation visible (an embedder failure shows up as
+ `["fts"]` instead of `["fts","vector","graph"]`). Reporting the
+ authority factor alongside `fused` is what keeps the surface honest:
+ the returned `rank` is the fused score *after* that multiplier, so
+ `fused` alone would not account for the ordering. Costs nothing extra:
+ the data was already computed and discarded. Hybrid ranking also gained
+ a deterministic path tiebreak for equal adjusted scores, which
+ previously fell back to hash-map iteration order.
- Per-page TTL via a frontmatter `expires_at:` key (RFC3339, or a bare
`YYYY-MM-DD` meaning end of that day UTC), mirrored into a new
`pages.expires_at` column (V36) and settable through a new optional
@@ -333,6 +347,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
ledger claim cannot consume the handoff or suppress retry delivery.
(#235)
+### Fixed
+- `memory_query` no longer skips the hybrid retrieval path when no
+ embedder is configured. The search branched to plain FTS5 whenever a
+ query vector was absent, which silently dropped the graph-neighbour
+ stream from every zero-LLM deployment — the documented default — and
+ would have left the new `explain` output reporting a single active
+ stream. All streams now run through one path; each contributes
+ nothing when its own input is missing, which is what the vector
+ stream already did.
+
## [1.18.0] - 2026-07-23
### Added
diff --git a/README.md b/README.md
index b7af1a8b..61f2c533 100644
--- a/README.md
+++ b/README.md
@@ -187,8 +187,11 @@ priors are at the [bottom](#influences-and-prior-art).
no-stdout behavior and also must call `memory_handoff_accept`.
- **"What did we decide about X six weeks ago?"** Type
`memory_query X` from the agent (or `ai-memory search X` from a
- terminal) - FTS5 over the wiki. Pages are LLM-consolidated, so
- the hit is a coherent decision page, not a raw chat log.
+ terminal) - FTS5 over the wiki, fused with linked-page expansion
+ (plus vector similarity when an embedder is configured). Pages are
+ LLM-consolidated, so the hit is a coherent decision page, not a raw
+ chat log. Pass `explain: true` to see why each hit ranked where it
+ did.
- **"Remember this permanently."** When something is worth keeping
beyond auto-captured session logs - a decision, a convention, a
gotcha - tell the agent "save a permanent note that we standardised
diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rs
index e5224b44..7b3f0e74 100644
--- a/crates/ai-memory-mcp/src/server.rs
+++ b/crates/ai-memory-mcp/src/server.rs
@@ -417,6 +417,12 @@ struct QueryArgs {
/// default; they are deleted by the next forget sweep). Default false.
#[serde(default)]
include_expired: Option<bool>,
+ /// Attach `score_details` to each hit: per-stream ranks (FTS5,
+ /// vector, graph), raw scores, and RRF contributions, plus a
+ /// top-level `streams_active` list. Costs nothing extra — use it to
+ /// understand why a page ranked where it did. Default false.
+ #[serde(default)]
+ explain: Option<bool>,
}
#[derive(Debug, Serialize, Deserialize, schemars::JsonSchema)]
@@ -446,9 +452,29 @@ struct StatusArgs {
workspace: Option<String>,
}
+/// One `memory_query` hit; serializes exactly like a bare
+/// [`ai_memory_store::PageHit`] unless `explain=true` attached
+/// `score_details`.
+#[derive(Debug, Serialize)]
+struct QueryHit {
+ #[serde(flatten)]
+ hit: ai_memory_store::PageHit,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ score_details: Option<ai_memory_store::SearchExplain>,
+}
+
+impl From<ai_memory_store::PageHit> for QueryHit {
+ fn from(hit: ai_memory_store::PageHit) -> Self {
+ Self {
+ hit,
+ score_details: None,
+ }
+ }
+}
+
#[derive(Debug, Serialize)]
struct MemoryQueryResponse {
- hits: Vec<ai_memory_store::PageHit>,
+ hits: Vec<QueryHit>,
#[serde(skip_serializing_if = "Vec::is_empty")]
raw_hits: Vec<ai_memory_store::ObservationHit>,
/// Populated only by a `global=true` query: cross-project hits, each
@@ -462,6 +488,12 @@ struct MemoryQueryResponse {
/// `global=true`).
#[serde(skip_serializing_if = "Vec::is_empty")]
global_scope_hits: Vec<ai_memory_store::PageHit>,
+ /// Present only when `explain=true`: which retrieval streams ran for
+ /// the primary search (`fts` always; `vector` + `graph` only when an
+ /// embedder produced a query vector). Lets callers see degradation —
+ /// e.g. the embedder erroring drops the response to `["fts"]`.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ streams_active: Option<Vec<&'static str>>,
}
/// Response for `memory_recent`. `hits` carries the current project's recent
@@ -1071,36 +1103,35 @@ impl AiMemoryServer {
query_vec: Option<&[f32]>,
limit: usize,
include_expired: bool,
- ) -> ai_memory_store::StoreResult<Vec<PageHit>> {
+ ) -> ai_memory_store::StoreResult<Vec<(PageHit, Option<ai_memory_store::SearchExplain>)>> {
// `i64::MIN` as the expiry cutoff makes every stored TTL pass
// the `expires_at > cutoff` guard, i.e. expired pages stay
// searchable when the caller opted in.
let expiry_cutoff = include_expired.then_some(i64::MIN);
- if let (Some(embedder), Some(qv)) = (&self.embedder, query_vec) {
- return self
- .reader
- .hybrid_search(
- workspace_id,
- project_id,
- query.to_owned(),
- Some(qv.to_vec()),
- embedder.provider().to_string(),
- embedder.model().to_string(),
- embedder.dim(),
- limit,
- expiry_cutoff,
- )
- .await;
- }
- self.reader
- .search_pages_for_project(
+ // Provider/model/dim only select which stored vectors are
+ // eligible; with no query vector the vector stream never runs,
+ // so the empty triple is inert rather than a fake identity.
+ let (provider, model, dim) = match (&self.embedder, query_vec) {
+ (Some(e), Some(_)) => (e.provider().to_string(), e.model().to_string(), e.dim()),
+ _ => (String::new(), String::new(), 0),
+ };
+ Ok(self
+ .reader
+ .hybrid_search_explained(
workspace_id,
project_id,
query.to_owned(),
+ query_vec.map(<[f32]>::to_vec),
+ provider,
+ model,
+ dim,
limit,
expiry_cutoff,
)
- .await
+ .await?
+ .into_iter()
+ .map(|(hit, explain)| (hit, Some(explain)))
+ .collect())
}
/// Override the retention-sweep parameters (typically populated
@@ -1210,6 +1241,7 @@ impl AiMemoryServer {
raw_hits: Vec::new(),
global_hits,
global_scope_hits: Vec::new(),
+ streams_active: None,
});
}
if !args.scopes.is_empty()
@@ -1236,7 +1268,8 @@ impl AiMemoryServer {
Some(self.resolve_query_scopes(&args.scopes).await?)
};
let hits = if let Some(scopes) = &resolved_scopes {
- let mut hits_by_id: HashMap<PageId, PageHit> = HashMap::new();
+ let mut hits_by_id: HashMap<PageId, (PageHit, Option<ai_memory_store::SearchExplain>)> =
+ HashMap::new();
for &(ws, proj) in scopes {
let hits = self
.search_project(
@@ -1251,19 +1284,20 @@ impl AiMemoryServer {
.map_err(|e| McpError::internal_error(e.to_string(), None))?;
for hit in hits {
hits_by_id
- .entry(hit.id)
+ .entry(hit.0.id)
.and_modify(|existing| {
- if hit.rank < existing.rank {
+ if hit.0.rank < existing.0.rank {
*existing = hit.clone();
}
})
.or_insert(hit);
}
}
- let mut hits: Vec<PageHit> = hits_by_id.into_values().collect();
+ let mut hits: Vec<(PageHit, Option<ai_memory_store::SearchExplain>)> =
+ hits_by_id.into_values().collect();
hits.sort_by(|a, b| {
- a.rank
- .partial_cmp(&b.rank)
+ a.0.rank
+ .partial_cmp(&b.0.rank)
.unwrap_or(std::cmp::Ordering::Equal)
});
hits.truncate(limit);
@@ -1287,7 +1321,7 @@ impl AiMemoryServer {
.await
};
let hits = hits.map_err(|e| McpError::internal_error(e.to_string(), None))?;
- self.spawn_access_bump(hits.iter().map(|h| h.id).collect());
+ self.spawn_access_bump(hits.iter().map(|(h, _)| h.id).collect());
// Raw-observation fallback when compiled-page search misses. Works
// for a single resolved project (default / workspace+project) AND
// for explicit `scopes` — the recommended scope-bleed mitigation —
@@ -1362,8 +1396,8 @@ impl AiMemoryServer {
)
.await
.map_err(|e| McpError::internal_error(e.to_string(), None))?;
- self.spawn_access_bump(hits.iter().map(|h| h.id).collect());
- hits
+ self.spawn_access_bump(hits.iter().map(|(h, _)| h.id).collect());
+ hits.into_iter().map(|(h, _)| h).collect()
}
}
Ok(None) => Vec::new(),
@@ -1372,11 +1406,27 @@ impl AiMemoryServer {
} else {
Vec::new()
};
+ let explain = args.explain.unwrap_or(false);
+ let streams_active = explain.then(|| {
+ if query_vec.is_some() {
+ vec!["fts", "vector", "graph"]
+ } else {
+ vec!["fts"]
+ }
+ });
+ let hits = hits
+ .into_iter()
+ .map(|(hit, score_details)| QueryHit {
+ hit,
+ score_details: score_details.filter(|_| explain),
+ })
+ .collect();
let response = MemoryQueryResponse {
hits,
raw_hits,
global_hits: Vec::new(),
global_scope_hits,
+ streams_active,
};
ok_json(&response)
}
@@ -4042,6 +4092,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -4088,6 +4139,7 @@ mod tests {
workspace: workspace.map(str::to_string),
global: None,
include_expired: None,
+ explain: None,
};
let result = server
@@ -4139,6 +4191,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -4264,6 +4317,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -4340,6 +4394,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
test_optional_parts(),
)
@@ -4433,6 +4488,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
test_optional_parts(),
)
@@ -4497,6 +4553,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
test_optional_parts(),
)
@@ -4537,6 +4594,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
test_optional_parts(),
)
@@ -4608,6 +4666,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
test_optional_parts(),
)
@@ -4670,6 +4729,7 @@ mod tests {
workspace: Some("practice".into()),
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -5045,6 +5105,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -5133,6 +5194,7 @@ mod tests {
workspace: None,
global: Some(true),
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -5168,6 +5230,7 @@ mod tests {
workspace: None,
global: Some(true),
include_expired: Some(true),
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -5244,6 +5307,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -5274,6 +5338,7 @@ mod tests {
workspace: Some("ops".into()),
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -5308,6 +5373,7 @@ mod tests {
workspace: None,
global: Some(true),
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -6867,6 +6933,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
@@ -6899,6 +6966,7 @@ mod tests {
workspace: None,
global: None,
include_expired: None,
+ explain: None,
}),
OptionalParts(test_parts_default()),
)
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index 989328d6..57883598 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -45,11 +45,11 @@ pub use ops::{
pub use reader::{
ActivityWindow, AutoImproveCandidateSession, BriefPageBody, BriefingPage, BriefingSnapshot,
ContaminationFinding, ContaminationReport, ContaminationSummary, DecayCandidate,
- DerivedIndexStatus, EmbeddingTripleCount, HealthDetail, HealthPage, ObservationHit,
+ DerivedIndexStatus, EmbeddingTripleCount, GraphVia, HealthDetail, HealthPage, ObservationHit,
OpenSession, PageAuthor, PageHit, PageHitWithMeta, PageLinks, PageMeta, PageSummary,
- ProjectSummary, ReaderPool, ReindexTargetStatus, RelatedPage, ScopeRow, SessionEndDisposition,
- StatusCounts, StoredEmbedding, StoredPageBody, WorkspaceScopeRow, WorkspaceSummary,
- f32_vec_to_bytes,
+ ProjectSummary, ReaderPool, ReindexTargetStatus, RelatedPage, RrfContributions, ScopeRow,
+ SearchExplain, SessionEndDisposition, StatusCounts, StoredEmbedding, StoredPageBody,
+ WorkspaceScopeRow, WorkspaceSummary, f32_vec_to_bytes,
};
pub use scope::{
ResolvedScope, ScopeName, ScopeResolutionError, ScopeResolver, WORKSPACE_PROJECT_PAIR_REQUIRED,
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index 637807c1..8efa0361 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -220,6 +220,80 @@ fn rerank_page_hits_with_meta(
hits
}
+/// A graph-expansion neighbour with provenance (which seed and which
+/// link direction produced it). Internal to hybrid search + explain.
+struct GraphNeighbor {
+ hit: PageHit,
+ /// Index into the seed list handed to the expansion.
+ seed_ord: usize,
+ /// `true` when the neighbour links TO the seed (backlink).
+ incoming: bool,
+}
+
+/// Which seed page pulled a hit in via graph expansion, and the link
+/// direction. Part of [`SearchExplain`].
+#[derive(Debug, Clone, Serialize)]
+pub struct GraphVia {
+ /// Path of the FTS/vector seed page the link was followed from.
+ pub seed_path: String,
+ /// `outgoing` = seed links to the hit; `incoming` = hit links to
+ /// the seed (backlink).
+ pub direction: &'static str,
+}
+
+/// Per-stream RRF contributions (`1/(k+rank)`, k=60) for one hit.
+#[derive(Debug, Clone, Default, Serialize)]
+pub struct RrfContributions {
+ /// FTS5 stream contribution (0.0 when the hit missed that stream).
+ pub fts: f64,
+ /// Vector stream contribution.
+ pub vector: f64,
+ /// Graph-neighbour stream contribution.
+ pub graph: f64,
+}
+
+/// Score transparency for one `memory_query` hit: where the hit ranked
+/// in each retrieval stream, the raw per-stream scores, and how the
+/// RRF fusion combined them. All ranks are 1-based; a `None` rank
+/// means the hit did not surface in that stream.
+///
+/// [`Default`] is "surfaced in no stream": every rank `None`, zero
+/// contributions. Streams fill in their own fields as they contribute.
+#[derive(Debug, Clone, Default, Serialize)]
+pub struct SearchExplain {
+ /// 1-based rank in the FTS5 stream.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub fts_rank: Option<usize>,
+ /// Raw FTS5 (bm25-based) score — lower is better.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub fts_score: Option<f64>,
+ /// 1-based rank in the vector stream.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub vector_rank: Option<usize>,
+ /// Cosine similarity against the query embedding.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub cosine: Option<f32>,
+ /// 1-based rank in the graph-neighbour stream.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub graph_rank: Option<usize>,
+ /// Which seed page and link direction pulled the hit in.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub graph_via: Option<GraphVia>,
+ /// Per-stream RRF contributions.
+ pub rrf: RrfContributions,
+ /// Total fused score (sum of the RRF contributions) — higher is
+ /// better. The returned `rank` is its negation *after* the
+ /// authority multiplier below, so `fused` alone does not reproduce
+ /// the ordering.
+ pub fused: f64,
+ /// Bounded page-authority multiplier applied to the fused rank
+ /// after RRF (canonical kind/tier/`pinned`/frontmatter tags).
+ /// `None` when the page had no authority row to adjust with; `1.0`
+ /// means it was considered and left alone.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub authority: Option<f64>,
+}
+
/// One hit returned by [`ReaderPool::search_pages`].
#[derive(Debug, Clone, Serialize)]
pub struct PageHit {
@@ -2022,8 +2096,7 @@ impl ReaderPool {
.await
}
- /// Return pages linked to or from the seed pages, scoped to latest pages
- /// in the same project.
+ /// `expiry_cutoff_us`: see [`Self::search_pages_for_project`].
///
/// # Errors
/// Propagates any SQL or pool error.
@@ -2035,16 +2108,42 @@ impl ReaderPool {
limit: usize,
expiry_cutoff_us: Option<i64>,
) -> StoreResult<Vec<PageHit>> {
+ Ok(self
+ .graph_neighbors_for_project_explained(
+ workspace_id,
+ project_id,
+ seed_ids,
+ limit,
+ expiry_cutoff_us,
+ )
+ .await?
+ .into_iter()
+ .map(|n| n.hit)
+ .collect())
+ }
+
+ /// [`Self::graph_neighbors_for_project`] plus provenance: which seed
+ /// produced each neighbour (`seed_ord` indexes into the caller's
+ /// `seed_ids`) and in which direction the link points. Feeds the
+ /// `memory_query` explain surface.
+ async fn graph_neighbors_for_project_explained(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ seed_ids: Vec<PageId>,
+ limit: usize,
+ expiry_cutoff_us: Option<i64>,
+ ) -> StoreResult<Vec<GraphNeighbor>> {
if seed_ids.is_empty() || limit == 0 {
return Ok(Vec::new());
}
- let now = expiry_cutoff_us.unwrap_or_else(now_us);
+ let cutoff = expiry_cutoff_us.unwrap_or_else(now_us);
self.with_conn(move |conn| {
let mut seen = std::collections::HashSet::new();
let mut out = Vec::new();
let mut values_clause = String::with_capacity(seed_ids.len() * 8);
- let mut sql_params = Vec::with_capacity(seed_ids.len() * 2 + 4);
+ let mut sql_params = Vec::with_capacity(seed_ids.len() * 2 + 6);
for (idx, seed_id) in seed_ids.iter().enumerate() {
if idx > 0 {
values_clause.push_str(", ");
@@ -2053,6 +2152,7 @@ impl ReaderPool {
sql_params.push(Value::Blob(seed_id.as_bytes().to_vec()));
sql_params.push(Value::Integer(idx as i64));
}
+ let now = cutoff;
sql_params.push(Value::Blob(workspace_id.as_bytes().to_vec()));
sql_params.push(Value::Blob(project_id.as_bytes().to_vec()));
sql_params.push(Value::Integer(now));
@@ -2062,8 +2162,7 @@ impl ReaderPool {
let out_not_expired = not_expired("tp", "?");
let in_not_expired = not_expired("fp", "?");
-
- let mut sql = String::with_capacity(values_clause.len() + 1_400);
+ let mut sql = String::with_capacity(values_clause.len() + 1_500);
write!(
&mut sql,
"WITH seeds(seed_id, seed_ord) AS (VALUES {values_clause}), \
@@ -2084,7 +2183,7 @@ impl ReaderPool {
JOIN pages fp ON fp.id = l.from_page_id \
WHERE fp.workspace_id = ? AND fp.project_id = ? AND fp.is_latest = 1{in_not_expired} \
) \
- SELECT id, path, title, snippet \
+ SELECT id, path, title, snippet, stream_ord \
FROM neighbors \
WHERE NOT EXISTS (SELECT 1 FROM seeds s WHERE s.seed_id = neighbors.id) \
ORDER BY stream_ord ASC, updated_at DESC"
@@ -2097,21 +2196,28 @@ impl ReaderPool {
let path: String = row.get(1)?;
let title: String = row.get(2)?;
let snippet: String = row.get(3)?;
- Ok((id_bytes, path, title, snippet))
+ let stream_ord: i64 = row.get(4)?;
+ Ok((id_bytes, path, title, snippet, stream_ord))
})?;
for row in rows {
- let (id_bytes, path, title, snippet) = row?;
+ let (id_bytes, path, title, snippet, stream_ord) = row?;
let id = PageId::from_slice(&id_bytes)?;
if !seen.insert(id) {
continue;
}
- out.push(PageHit {
- id,
- path: PagePath::new(path)?,
- title,
- snippet,
- rank: 0.0,
+ #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)]
+ let seed_ord = (stream_ord / 2).max(0) as usize;
+ out.push(GraphNeighbor {
+ hit: PageHit {
+ id,
+ path: PagePath::new(path)?,
+ title,
+ snippet,
+ rank: 0.0,
+ },
+ seed_ord,
+ incoming: stream_ord % 2 == 1,
});
if out.len() >= limit {
break;
@@ -2185,14 +2291,14 @@ impl ReaderPool {
/// Applies the bounded page-authority adjustment after fusion and returns
/// the top-`limit` pages by adjusted fused score.
///
- /// When `query_vec` is `None`, the vector stream is skipped but graph
- /// expansion still runs from the FTS seeds.
+ /// Each stream degrades independently to contributing nothing: no
+ /// `query_vec` skips the vector stream, while graph expansion still
+ /// runs from whatever seeds the other streams produced.
///
- /// k=60 is the canonical RRF constant.
+ /// `expiry_cutoff_us`: see [`Self::search_pages_for_project`]. The
+ /// cutoff resolves once here so all streams agree on it.
///
- /// `expiry_cutoff_us`: see [`Self::search_pages_for_project`]. Every
- /// stream shares one cutoff so a page cannot be filtered out of one
- /// ranker while still seeding another.
+ /// k=60 is the canonical RRF constant.
///
/// # Errors
/// Propagates any SQL or pool error.
@@ -2209,6 +2315,44 @@ impl ReaderPool {
limit: usize,
expiry_cutoff_us: Option<i64>,
) -> StoreResult<Vec<PageHit>> {
+ Ok(self
+ .hybrid_search_explained(
+ workspace_id,
+ project_id,
+ query,
+ query_vec,
+ provider,
+ model,
+ dim,
+ limit,
+ expiry_cutoff_us,
+ )
+ .await?
+ .into_iter()
+ .map(|(hit, _)| hit)
+ .collect())
+ }
+
+ /// [`Self::hybrid_search`] plus a [`SearchExplain`] per hit — the
+ /// per-stream ranks, raw scores, and RRF contributions the fusion
+ /// otherwise discards. Computing them is free (the data is already
+ /// in hand); callers that don't need them use `hybrid_search`.
+ ///
+ /// # Errors
+ /// Propagates any SQL or pool error.
+ #[allow(clippy::too_many_arguments)]
+ pub async fn hybrid_search_explained(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ query: String,
+ query_vec: Option<Vec<f32>>,
+ provider: String,
+ model: String,
+ dim: u32,
+ limit: usize,
+ expiry_cutoff_us: Option<i64>,
+ ) -> StoreResult<Vec<(PageHit, SearchExplain)>> {
let cutoff = expiry_cutoff_us.unwrap_or_else(now_us);
// Authority is applied after RRF, so every stream needs the same
// bounded candidate window used by FTS-only search. A `limit * 2`
@@ -2261,8 +2405,25 @@ impl ReaderPool {
seed_ids.push(id);
}
}
+ // seed_ord → seed path, for graph provenance in the explain.
+ let seed_paths: Vec<String> = seed_ids
+ .iter()
+ .map(|id| {
+ fts_hits
+ .iter()
+ .find(|h| h.id == *id)
+ .map(|h| h.path.as_str().to_string())
+ .or_else(|| {
+ vec_hits
+ .iter()
+ .find(|(vid, _, _)| vid == id)
+ .map(|(_, p, _)| p.as_str().to_string())
+ })
+ .unwrap_or_default()
+ })
+ .collect();
let graph_hits = self
- .graph_neighbors_for_project(
+ .graph_neighbors_for_project_explained(
workspace_id,
project_id,
seed_ids,
@@ -2273,70 +2434,97 @@ impl ReaderPool {
// RRF fuse: score(d) = Σ 1/(k + rank_i(d)) over rankers.
let k = 60.0_f64;
- let mut fused: std::collections::HashMap<PageId, (PagePath, String, String, f64, f64)> =
- std::collections::HashMap::new();
+ struct Fused {
+ path: PagePath,
+ title: String,
+ snippet: String,
+ explain: SearchExplain,
+ }
+ let mut fused: std::collections::HashMap<PageId, Fused> = std::collections::HashMap::new();
for (rank, h) in fts_hits.iter().enumerate() {
let contrib = 1.0 / (k + (rank + 1) as f64);
- fused
- .entry(h.id)
- .and_modify(|entry| entry.3 += contrib)
- .or_insert((
- h.path.clone(),
- h.title.clone(),
- h.snippet.clone(),
- contrib,
- h.rank,
- ));
+ let entry = fused.entry(h.id).or_insert_with(|| Fused {
+ path: h.path.clone(),
+ title: h.title.clone(),
+ snippet: h.snippet.clone(),
+ explain: SearchExplain::default(),
+ });
+ entry.explain.fts_rank = Some(rank + 1);
+ entry.explain.fts_score = Some(h.rank);
+ entry.explain.rrf.fts = contrib;
+ entry.explain.fused += contrib;
}
- for (rank, (id, path, _score)) in vec_hits.iter().enumerate() {
+ for (rank, (id, path, cosine)) in vec_hits.iter().enumerate() {
let contrib = 1.0 / (k + (rank + 1) as f64);
- fused
- .entry(*id)
- .and_modify(|entry| entry.3 += contrib)
- .or_insert((path.clone(), String::new(), String::new(), contrib, 0.0));
+ let entry = fused.entry(*id).or_insert_with(|| Fused {
+ path: path.clone(),
+ title: String::new(),
+ snippet: String::new(),
+ explain: SearchExplain::default(),
+ });
+ entry.explain.vector_rank = Some(rank + 1);
+ entry.explain.cosine = Some(*cosine);
+ entry.explain.rrf.vector = contrib;
+ entry.explain.fused += contrib;
}
- for (rank, h) in graph_hits.iter().enumerate() {
+ for (rank, n) in graph_hits.iter().enumerate() {
let contrib = 1.0 / (k + (rank + 1) as f64);
- fused
- .entry(h.id)
- .and_modify(|entry| entry.3 += contrib)
- .or_insert((
- h.path.clone(),
- h.title.clone(),
- h.snippet.clone(),
- contrib,
- h.rank,
- ));
+ let entry = fused.entry(n.hit.id).or_insert_with(|| Fused {
+ path: n.hit.path.clone(),
+ title: n.hit.title.clone(),
+ snippet: n.hit.snippet.clone(),
+ explain: SearchExplain::default(),
+ });
+ entry.explain.graph_rank = Some(rank + 1);
+ entry.explain.graph_via = Some(GraphVia {
+ seed_path: seed_paths.get(n.seed_ord).cloned().unwrap_or_default(),
+ direction: if n.incoming { "incoming" } else { "outgoing" },
+ });
+ entry.explain.rrf.graph = contrib;
+ entry.explain.fused += contrib;
}
- let mut out: Vec<PageHit> = fused
+ let mut out: Vec<(PageHit, SearchExplain)> = fused
.into_iter()
- .map(|(id, (path, title, snippet, fused_rank, _orig))| PageHit {
- id,
- path,
- title,
- snippet,
- rank: -fused_rank, // lower = better (matches FTS5 convention)
+ .map(|(id, entry)| {
+ (
+ PageHit {
+ id,
+ path: entry.path,
+ title: entry.title,
+ snippet: entry.snippet,
+ rank: -entry.explain.fused, // lower = better (matches FTS5 convention)
+ },
+ entry.explain,
+ )
})
.collect();
let missing_authorities: Vec<PageId> = out
.iter()
- .filter_map(|hit| (!authorities.contains_key(&hit.id)).then_some(hit.id))
+ .filter_map(|(hit, _)| (!authorities.contains_key(&hit.id)).then_some(hit.id))
.collect();
authorities.extend(
self.page_authorities_for_project(workspace_id, project_id, missing_authorities)
.await?,
);
- for hit in &mut out {
+ for (hit, explain) in &mut out {
if let Some(authority) = authorities.get(&hit.id) {
hit.rank = authority.adjust_rank(hit.rank);
+ // `fused` stays the raw RRF sum; without the multiplier
+ // beside it the explain could not account for the rank it
+ // returns, which is the whole point of the surface.
+ explain.authority = Some(authority.factor);
}
}
out.sort_by(|a, b| {
- a.rank
- .partial_cmp(&b.rank)
+ a.0.rank
+ .partial_cmp(&b.0.rank)
.unwrap_or(std::cmp::Ordering::Equal)
+ // Deterministic tiebreak: equal fused scores (common when
+ // two streams each contribute a rank-1 hit) previously fell
+ // back to HashMap iteration order.
+ .then_with(|| a.0.path.as_str().cmp(b.0.path.as_str()))
});
out.truncate(limit);
Ok(out)
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 6515e5bb..270da214 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -320,7 +320,7 @@ invariants below.
| Tool | Hint | Purpose |
|---|---|---|
-| `memory_query` | read-only | FTS5 + graph RRF + optional vector RRF search, followed by bounded kind/tier/pinned/tag authority adjustment and raw fallback. Bumps access counters for page hits. Defaults to the current project; default-scoped calls also union the reserved `_global` preferences scope as `global_scope_hits`; `scopes` searches named sibling projects; `global=true` searches every project at once (each hit annotated with its workspace + project). `include_expired=true` also returns TTL-expired pages. |
+| `memory_query` | read-only | FTS5 + graph RRF + optional vector RRF search, followed by bounded kind/tier/pinned/tag authority adjustment and raw fallback. Bumps access counters for page hits. Defaults to the current project; default-scoped calls also union the reserved `_global` preferences scope as `global_scope_hits`; `scopes` searches named sibling projects; `global=true` searches every project at once (each hit annotated with its workspace + project). `explain=true` attaches per-hit `score_details` (per-stream ranks, raw FTS/cosine scores, RRF contributions, graph provenance, and the authority multiplier applied after fusion) plus a top-level `streams_active` list; `include_expired=true` also returns TTL-expired pages. |
| `memory_recent` | read-only | Most-recently-updated `is_latest=1` pages. |
| `memory_read_page` | read-only | Fetch the FULL body of a single wiki page by `path` or by top FTS5 hit for a `query`; optional `workspace` + `project` targets a named sibling workspace/project. Use when an agent needs more than the 24-word snippets from `memory_query`. |
| `memory_status` | read-only | Counts, paths, version. |