Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(store): entity matches as a fourth RRF stream (V38)

Samir Hanna Verza0a02b3ab79ea
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 474b3525..39d47792 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,91 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- Entity-match retrieval as a fourth RRF stream (V38 `entities` +
+ `entity_page_links`). Consolidation now emits an `entities` list per
+ page — the specific technologies, components, and domain nouns the
+ content names — which lands in the page's frontmatter (markdown stays
+ the source of truth; a reindex rebuilds the index) and is matched
+ lexically at query time: exact name, name-prefix, or a word inside a
+ multi-word entity, weighted by inverse entity frequency so a rare noun
+ outranks a ubiquitous one. This finds pages whose *wording* misses the
+ query — a note whose body says "the streaming broker" now answers a
+ query for "jetstream". No spaCy, no NLP dependency, and no LLM call at
+ query time: extraction happens only where an LLM already runs, so a
+ project that never consolidated has an empty table and the stream
+ contributes nothing — the zero-LLM default path is unchanged. With
+ `explain: true` each hit reports its `entity_rank` and
+ `matched_entities`. `entities.id` is deliberately shaped as the stable
+ anchor a future temporal-triples table could reference; that P2 item
+ stays deferred, and no graph database is involved either way.
+- Optional post-RRF reranking for `memory_query`, off by default. Set
+ `AI_MEMORY_RERANKER=llm` (requires `AI_MEMORY_LLM_PROVIDER`) and the
+ query over-fetches `limit * 3` candidates (capped at 30) from the RRF
+ stage, then has the configured LLM provider score each one against the
+ query through JSON-schema structured output — no new provider dialect,
+ and it works across all eight existing providers. Scored candidates
+ sort by relevance and any the model skipped follow in RRF order.
+ Degrades rather than fails: a 20-second timeout, a provider error, or a
+ response covering fewer than half the candidates all fall back to plain
+ RRF order, matching the embedder's contract. With `explain: true` each
+ hit's `score_details` carries its `rerank_score`. An unknown
+ `AI_MEMORY_RERANKER` value, or `llm` without a provider, fails at
+ startup instead of silently disabling the feature.
+- New MCP tool `memory_feedback` (17th tool) — the "finer-grained
+ reinforcement beyond access counts" P2 item. Record how useful a
+ recalled page actually was by exact path: `helpful` / `not_helpful`
+ step the page's new `pages.salience` column (V37, bounded to
+ `[0.25, 2.0]` in 0.25 steps), which now scales the retention formula's
+ time term instead of a single global `salience_default`; `stale` /
+ `wrong` floor the salience AND surface the page as a
+ `feedback_flagged` finding in the next `memory_lint` report. Signals
+ land in a new append-only `page_feedback` table with an optional
+ sanitized reason and full audit-log entry. Nothing is ever deleted by
+ feedback. Signals attach to the page *version* that was read, so
+ rewriting a flagged page retires both its salience and its lint
+ findings — there is no separate dismissal state. Pages without
+ feedback keep `salience = NULL`, which reads as exactly the previous
+ behaviour.
+- `memory_query` explain mode: pass `explain: true` and every hit carries
+ `score_details` — its 1-based rank and raw score in each retrieval
+ stream (FTS5 bm25 score, embedding cosine, graph-neighbour provenance
+ naming the seed page and link direction), the per-stream RRF
+ contributions, the fused score, and the bounded page-authority
+ multiplier applied after fusion — plus a top-level `streams_active`
+ list that makes degradation visible (an embedder failure shows up as
+ `["fts"]` instead of `["fts","vector","graph"]`). Reporting the
+ authority factor alongside `fused` is what keeps the surface honest:
+ the returned `rank` is the fused score *after* that multiplier, so
+ `fused` alone would not account for the ordering. Costs nothing extra:
+ the data was already computed and discarded. Hybrid ranking also gained
+ a deterministic path tiebreak for equal adjusted scores, which
+ previously fell back to hash-map iteration order.
+- Per-page TTL via a frontmatter `expires_at:` key (RFC3339, or a bare
+ `YYYY-MM-DD` meaning end of that day UTC), mirrored into a new
+ `pages.expires_at` column (V36) and settable through a new optional
+ `expires_at` parameter on `memory_write_page`. Expired pages are hidden
+ from `memory_query`/`memory_recent`/briefing/session-brief surfaces —
+ `memory_query` gains `include_expired: true` to still see them — while
+ exact-path reads still return the page, annotated `expired: true`,
+ because an explicit read is not a search. The forget sweep hard-deletes
+ them through the wiki layer, so the markdown file goes too, not just
+ the rows. An explicit TTL outranks `pinned` (a pin means "don't decay
+ this", not "keep it past the date its author set"); `memory_lint`
+ flags pinned+expiring pages so the combination is visible rather than
+ silent ([#309]).
+- New `openai-compat` embedding provider for self-hosted engines
+ (Ollama, LM Studio, vLLM). Set
+ `AI_MEMORY_EMBEDDING_PROVIDER=openai-compat` together with explicit
+ `AI_MEMORY_EMBEDDING_BASE_URL`, `AI_MEMORY_EMBEDDING_MODEL`, and
+ `AI_MEMORY_EMBEDDING_DIM` — there is no safe default model or
+ dimensionality for a self-hosted engine, so each is required rather
+ than guessed. Unlike the other providers it is keyless: a bearer
+ token is sent only when `LLM_API_KEY` is present, for gateways that
+ want one. Embeddings are stored under their own
+ `provider="openai-compat"` identity, so switching an existing
+ `openai`+base-URL setup over changes the stored
+ `{provider, model, dim}` triple — run `ai-memory embed --force` to
+ re-embed. (#300)
- Optional post-RRF reranking for `memory_query`, off by default. Set
`AI_MEMORY_RERANKER=llm` (requires `AI_MEMORY_LLM_PROVIDER`) and the
query over-fetches `limit * 3` candidates (capped at 30) from the RRF
diff --git a/crates/ai-memory-cli/src/commands/reindex.rs b/crates/ai-memory-cli/src/commands/reindex.rs
index d2c7afa8..973e59cf 100644
--- a/crates/ai-memory-cli/src/commands/reindex.rs
+++ b/crates/ai-memory-cli/src/commands/reindex.rs
@@ -121,6 +121,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
diff --git a/crates/ai-memory-consolidate/src/consolidator.rs b/crates/ai-memory-consolidate/src/consolidator.rs
index 9013e966..ea4bf373 100644
--- a/crates/ai-memory-consolidate/src/consolidator.rs
+++ b/crates/ai-memory-consolidate/src/consolidator.rs
@@ -521,6 +521,20 @@ fn build_update(
),
);
}
+ // Entities land in frontmatter (markdown stays the source of truth);
+ // the store derives its index from there, so a reindex rebuilds them.
+ let entities = ai_memory_core::normalize_entities(&upd.entities);
+ if !entities.is_empty() {
+ fm.insert(
+ "entities".into(),
+ serde_json::Value::Array(
+ entities
+ .into_iter()
+ .map(serde_json::Value::String)
+ .collect(),
+ ),
+ );
+ }
if is_slot_path(&path) {
fm.insert(
"slot_kind".into(),
@@ -725,9 +739,22 @@ fn build_batch_request_with_slots(
- \"tier\" (string) required — one of: working | episodic | semantic | procedural\n\
- \"kind\" (string) required — one of: decision | gotcha | rule | fact\n\
- \"tags\" (array of string) required — may be empty `[]`, but the key must be present\n\
+ - \"entities\" (array of string) required — may be empty `[]`, but the key must be present; see below\n\
- \"slot_kind\" (string) optional — ONLY for `_slots/*`; one of \"state\" or \"invariant\"; this is the SLOT WRITE REGIME, NOT a tier value\n\
No other keys except optional `slot_kind` on `_slots/*`. No `body`, no `content`, no `summary`. Field names \
are case-sensitive and the `_markdown` suffix matters.\n\
+ \n## `entities` field — the specific nouns the page is about\n\
+ Up to 10 short names (max 64 chars each), lowercase, taken from \
+ what the page actually names: technologies (`sqlite`, `tokio`), \
+ components (`writer actor`, `hook router`), services, crates, \
+ file or module names, and product/domain nouns. They power a \
+ retrieval stream, so a later query naming one of them finds this \
+ page even when the wording differs.\n\
+ Do NOT include: generic words (`code`, `bug`, `change`, \
+ `refactor`), the tier or kind values, whole sentences, or \
+ restatements of the title. Prefer fewer, more specific entries \
+ over padding the list. `[]` is correct for a page with no \
+ specific nouns.\n\
\n## Output format (read this carefully)\n\
Reply with ONE JSON object matching the ConsolidatedBatch schema, \
and nothing else. NO prose preamble, NO trailing commentary, NO \
@@ -1115,6 +1142,7 @@ mod tests {
body_markdown: "Ship the slot-kind PR.".into(),
tags: Vec::new(),
slot_kind: SlotKind::State,
+ entities: Vec::new(),
};
let (req, _) = build_update(
WorkspaceId::new(),
@@ -1138,6 +1166,7 @@ mod tests {
body_markdown: "body".into(),
tags: Vec::new(),
slot_kind: SlotKind::State,
+ entities: Vec::new(),
};
let actor = ai_memory_core::ActorContext {
user: Some("djalmajr".into()),
@@ -1174,6 +1203,7 @@ mod tests {
body_markdown: "This repo uses a markdown wiki as source of truth.".into(),
tags: Vec::new(),
slot_kind: SlotKind::Invariant,
+ entities: Vec::new(),
};
let (req, _) = build_update(
WorkspaceId::new(),
diff --git a/crates/ai-memory-consolidate/src/curator.rs b/crates/ai-memory-consolidate/src/curator.rs
index 250567ba..990e3896 100644
--- a/crates/ai-memory-consolidate/src/curator.rs
+++ b/crates/ai-memory-consolidate/src/curator.rs
@@ -508,6 +508,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
diff --git a/crates/ai-memory-consolidate/src/lib.rs b/crates/ai-memory-consolidate/src/lib.rs
index 482beef3..d668a9ce 100644
--- a/crates/ai-memory-consolidate/src/lib.rs
+++ b/crates/ai-memory-consolidate/src/lib.rs
@@ -62,7 +62,7 @@ pub use embed::{
EmbedBackfillCounts, EmbedBackfillError, EmbedBackfillOptions, run_embedding_backfill,
};
pub use lint::{LintError, LintFinding, LintReport, run_lint};
-pub use sweep::{EvictedPage, SweepError, SweepReport, run_sweep};
+pub use sweep::{EvictedPage, ExpiredPage, SweepError, SweepReport, run_sweep};
pub use types::{
ConsolidatedBatch, ConsolidatedPage, ConsolidatedPageUpdate, ConsolidationOutcome, PageKind,
SlotKind,
diff --git a/crates/ai-memory-consolidate/src/types.rs b/crates/ai-memory-consolidate/src/types.rs
index 2825741b..74c5fbf2 100644
--- a/crates/ai-memory-consolidate/src/types.rs
+++ b/crates/ai-memory-consolidate/src/types.rs
@@ -125,6 +125,14 @@ pub struct ConsolidatedPageUpdate {
/// current behaviour.
#[serde(default)]
pub slot_kind: SlotKind,
+ /// Salient nouns this page is about — the specific technologies,
+ /// components, services, and files the content names. Indexed as a
+ /// retrieval stream so a query naming one of them finds the page
+ /// even when the wording differs. Normalised and capped
+ /// (`ai_memory_core::normalize_entities`) before storage; defaults
+ /// to empty so older structured outputs still deserialise.
+ #[serde(default)]
+ pub entities: Vec<String>,
}
/// Batch produced by [`ConsolidatorMulti`].
diff --git a/crates/ai-memory-consolidate/tests/recall_eval.rs b/crates/ai-memory-consolidate/tests/recall_eval.rs
index 32ddc9ea..4ff11098 100644
--- a/crates/ai-memory-consolidate/tests/recall_eval.rs
+++ b/crates/ai-memory-consolidate/tests/recall_eval.rs
@@ -250,6 +250,84 @@ async fn graph_neighbor_expansion_recovers_linked_page() {
);
}
+/// The entity stream recovers a probe that BOTH pure FTS and the graph
+/// stream miss: the page's body never carries the query's wording, and it
+/// has no links. The only bridge is the frontmatter `entities:` list the
+/// consolidator writes.
+#[tokio::test]
+async fn entity_stream_recovers_a_probe_fts_and_graph_both_miss() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("open store");
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .expect("ws");
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "eval", None)
+ .await
+ .expect("proj");
+ let wiki = Wiki::new(tmp.path(), store.writer.clone()).expect("wiki");
+
+ wiki.write_page(WritePageRequest {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new("decisions/queue-choice.md").expect("path"),
+ // Body deliberately avoids the probe term and carries no links.
+ frontmatter: serde_json::json!({
+ "title": "Queue choice",
+ "entities": ["nats jetstream", "delivery guarantees"],
+ }),
+ body: "We picked the streaming broker for at-least-once delivery.".into(),
+ tier: Tier::Semantic,
+ pinned: false,
+ title: None,
+ admission_ctx: None,
+ author_id: None,
+ actor: ai_memory_core::ActorContext::anonymous(),
+ })
+ .await
+ .expect("write page");
+
+ let probe = "jetstream";
+ let fts_hits = store
+ .reader
+ .search_pages_for_project(ws, proj, probe.into(), 5, None)
+ .await
+ .expect("fts search");
+ assert!(
+ fts_hits.is_empty(),
+ "pure FTS must miss: the body never says {probe}"
+ );
+
+ let hybrid_hits = store
+ .reader
+ .hybrid_search(
+ ws,
+ proj,
+ probe.into(),
+ None,
+ String::new(),
+ String::new(),
+ 0,
+ 5,
+ None,
+ )
+ .await
+ .expect("hybrid search");
+ assert!(
+ hybrid_hits
+ .iter()
+ .any(|hit| hit.path.as_str() == "decisions/queue-choice.md"),
+ "the entity stream should recover the page: {:?}",
+ hybrid_hits
+ .iter()
+ .map(|h| h.path.as_str())
+ .collect::<Vec<_>>(),
+ );
+}
+
#[tokio::test]
async fn raw_observation_fallback_recovers_detail_when_wiki_misses() {
let tmp = TempDir::new().expect("tempdir");
diff --git a/crates/ai-memory-core/src/lib.rs b/crates/ai-memory-core/src/lib.rs
index 2ec9aaef..40ab32b3 100644
--- a/crates/ai-memory-core/src/lib.rs
+++ b/crates/ai-memory-core/src/lib.rs
@@ -45,7 +45,10 @@ pub use ids::{
PageFeedbackId, PageId, PagePath, ProjectId, SessionId, UserId, WorkspaceId, WorkstreamId,
};
pub use observation::{NewObservation, NewSession, Observation, ObservationKind};
-pub use page::{FeedbackKind, LinkTarget, NewPage, Page, Tier};
+pub use page::{
+ FeedbackKind, LinkTarget, MAX_ENTITIES_PER_PAGE, MAX_ENTITY_LEN, NewPage, Page, Tier,
+ normalize_entities, normalize_entity,
+};
pub use routing_snippet::{MARKER_END, MARKER_START, SNIPPET_BODY, find_marker_line, full_block};
pub use sanitize::{
OBSERVATION_BODY_MAX_BYTES, SanitizeConfig, Sanitized, Sanitizer, truncate_utf8_bytes,
diff --git a/crates/ai-memory-core/src/page.rs b/crates/ai-memory-core/src/page.rs
index d93df0a6..8d4310ef 100644
--- a/crates/ai-memory-core/src/page.rs
+++ b/crates/ai-memory-core/src/page.rs
@@ -74,6 +74,61 @@ pub struct NewPage {
/// search/recent/briefing and hard-deleted by the retention sweep.
#[serde(default)]
pub expires_at: Option<Timestamp>,
+ /// Salient nouns for this page, from the frontmatter `entities:`
+ /// list the consolidator writes. Normalised by
+ /// [`normalize_entity`] before it reaches the store, which indexes
+ /// them as a retrieval stream (V38). Empty for hand-written pages
+ /// that don't declare any.
+ #[serde(default)]
+ pub entities: Vec<String>,
+}
+
+/// Longest accepted entity name; longer values are rejected rather than
+/// truncated (a 100-char "entity" is a sentence, not a noun).
+pub const MAX_ENTITY_LEN: usize = 64;
+/// Cap on entities indexed per page. Beyond this the list stops being a
+/// salience signal and starts being a second copy of the body.
+pub const MAX_ENTITIES_PER_PAGE: usize = 10;
+
+/// Normalise one entity name for storage and matching: trim, collapse
+/// internal whitespace, lowercase. Returns `None` when the result is
+/// empty or longer than [`MAX_ENTITY_LEN`].
+///
+/// Lowercasing is what makes query-time matching lexical rather than a
+/// second LLM call: `Postgres`, `postgres`, and `POSTGRES` all index and
+/// match as one entity.
+#[must_use]
+pub fn normalize_entity(raw: &str) -> Option<String> {
+ let collapsed = raw.split_whitespace().collect::<Vec<_>>().join(" ");
+ let normalized = collapsed.to_lowercase();
+ if normalized.is_empty() || normalized.chars().count() > MAX_ENTITY_LEN {
+ return None;
+ }
+ Some(normalized)
+}
+
+/// Normalise a whole entity list: drop invalid entries, de-duplicate
+/// while preserving order, and cap at [`MAX_ENTITIES_PER_PAGE`].
+#[must_use]
+pub fn normalize_entities<I, S>(raw: I) -> Vec<String>
+where
+ I: IntoIterator<Item = S>,
+ S: AsRef<str>,
+{
+ let mut seen = std::collections::BTreeSet::new();
+ let mut out = Vec::new();
+ for item in raw {
+ let Some(name) = normalize_entity(item.as_ref()) else {
+ continue;
+ };
+ if seen.insert(name.clone()) {
+ out.push(name);
+ if out.len() >= MAX_ENTITIES_PER_PAGE {
+ break;
+ }
+ }
+ }
+ out
}
/// A link target discovered in a page body.
@@ -269,6 +324,7 @@ mod tests {
"\"procedural\""
);
}
+
#[test]
fn feedback_kind_round_trips_and_flags_lint_routing() {
for k in [
@@ -285,4 +341,35 @@ mod tests {
assert!(FeedbackKind::Stale.routes_to_lint());
assert!(FeedbackKind::Wrong.routes_to_lint());
}
+
+ #[test]
+ fn normalize_entity_lowercases_trims_and_collapses() {
+ assert_eq!(
+ normalize_entity(" Postgres ").as_deref(),
+ Some("postgres")
+ );
+ assert_eq!(
+ normalize_entity("Writer\n\tActor").as_deref(),
+ Some("writer actor")
+ );
+ assert_eq!(normalize_entity(" ").as_deref(), None);
+ assert_eq!(normalize_entity("").as_deref(), None);
+ assert_eq!(
+ normalize_entity(&"x".repeat(MAX_ENTITY_LEN)).as_deref(),
+ Some("x".repeat(MAX_ENTITY_LEN).as_str()),
+ );
+ assert_eq!(normalize_entity(&"x".repeat(MAX_ENTITY_LEN + 1)), None);
+ }
+
+ #[test]
+ fn normalize_entities_dedupes_preserves_order_and_caps() {
+ assert_eq!(
+ normalize_entities(["SQLite", "sqlite", " FTS5 ", ""]),
+ vec!["sqlite".to_string(), "fts5".to_string()],
+ );
+ let many: Vec<String> = (0..MAX_ENTITIES_PER_PAGE + 5)
+ .map(|i| format!("entity{i}"))
+ .collect();
+ assert_eq!(normalize_entities(&many).len(), MAX_ENTITIES_PER_PAGE);
+ }
}
diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rs
index 1679c6d0..ee29d966 100644
--- a/crates/ai-memory-hooks/src/router.rs
+++ b/crates/ai-memory-hooks/src/router.rs
@@ -6258,6 +6258,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
}
}
diff --git a/crates/ai-memory-hooks/src/synth.rs b/crates/ai-memory-hooks/src/synth.rs
index d3145a16..ca52447d 100644
--- a/crates/ai-memory-hooks/src/synth.rs
+++ b/crates/ai-memory-hooks/src/synth.rs
@@ -48,6 +48,7 @@ pub fn synthesize_session_page(
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
}
}
diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rs
index bd3c791e..9daa9c0e 100644
--- a/crates/ai-memory-mcp/src/server.rs
+++ b/crates/ai-memory-mcp/src/server.rs
@@ -3377,6 +3377,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -4507,12 +4508,416 @@ mod tests {
assert_eq!(details["fts_rank"], 1);
assert!(details.get("vector_rank").is_none());
assert_eq!(details["rrf"]["vector"], 0.0);
+ // This project never consolidated, so no entity rows exist and the
+ // fourth stream contributes nothing — the same way the vector stream
+ // stays silent without an embedder.
+ assert!(details.get("entity_rank").is_none());
+ assert_eq!(details["rrf"]["entity"], 0.0);
let rank = value["hits"][0]["rank"].as_f64().unwrap();
let fused = details["fused"].as_f64().unwrap();
let authority = details["authority"].as_f64().unwrap();
assert!((rank + fused * authority).abs() < f64::EPSILON);
}
+ /// Regression: with NO embedder configured — the default deployment —
+ /// `memory_query` must still run the entity and graph streams, not
+ /// just FTS. This path used to short-circuit to a bare FTS query,
+ /// which silently disabled both for every zero-LLM install.
+ #[tokio::test]
+ async fn query_without_embedder_still_runs_entity_and_graph_streams() {
+ let (_tmp, store, server, ws, proj) = setup_server().await;
+ assert!(
+ server.embedder.is_none(),
+ "this test is specifically the no-embedder path"
+ );
+
+ // A page findable ONLY via its entities (body avoids the query
+ // word), and a page findable ONLY via a link from an FTS hit.
+ let mut entity_only = NewPage {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new("concepts/broker.md").unwrap(),
+ title: "Broker".into(),
+ body: "The chosen transport gives at-least-once semantics.".into(),
+ tier: Tier::Semantic,
+ frontmatter_json: serde_json::json!({}),
+ pinned: false,
+ links: Vec::new(),
+ author_id: None,
+ expires_at: None,
+ entities: vec!["nats jetstream".into()],
+ };
+ store.writer.upsert_page(entity_only.clone()).await.unwrap();
+ entity_only.path = PagePath::new("concepts/linked.md").unwrap();
+ entity_only.title = "Linked".into();
+ entity_only.body = "Nothing quotable here.".into();
+ entity_only.entities = Vec::new();
+ store.writer.upsert_page(entity_only).await.unwrap();
+ store
+ .writer
+ .upsert_page(NewPage {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new("concepts/seed.md").unwrap(),
+ title: "Seed".into(),
+ body: "graphseed points at [[concepts/linked.md]].".into(),
+ tier: Tier::Semantic,
+ frontmatter_json: serde_json::json!({}),
+ pinned: false,
+ links: vec![PagePath::new("concepts/linked.md").unwrap().into()],
+ author_id: None,
+ expires_at: None,
+ entities: Vec::new(),
+ })
+ .await
+ .unwrap();
+
+ let query = async |q: &str| -> serde_json::Value {
+ let result = server
+ .memory_query(
+ Parameters(QueryArgs {
+ query: q.into(),
+ limit: Some(10),
+ project: None,
+ scopes: Vec::new(),
+ workspace: None,
+ global: None,
+ include_expired: None,
+ explain: Some(true),
+ }),
+ OptionalParts(test_parts_default()),
+ )
+ .await
+ .unwrap();
+ let text = result.content.first().and_then(|c| c.as_text()).unwrap();
+ serde_json::from_str(&text.text).unwrap()
+ };
+
+ let entity_hit = query("jetstream").await;
+ let paths: Vec<&str> = entity_hit["hits"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .map(|h| h["path"].as_str().unwrap())
+ .collect();
+ assert!(
+ paths.contains(&"concepts/broker.md"),
+ "entity stream must run without an embedder: {paths:?}"
+ );
+ assert_eq!(
+ entity_hit["hits"][0]["score_details"]["matched_entities"],
+ serde_json::json!(["nats jetstream"]),
+ "{entity_hit}"
+ );
+
+ let graph_hit = query("graphseed").await;
+ let paths: Vec<&str> = graph_hit["hits"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .map(|h| h["path"].as_str().unwrap())
+ .collect();
+ assert!(
+ paths.contains(&"concepts/linked.md"),
+ "graph stream must run without an embedder: {paths:?}"
+ );
+ }
+
+ /// A configured reranker reorders the RRF result; a failing or
+ /// hanging one degrades to RRF order instead of erroring the query.
+ #[tokio::test]
+ async fn reranker_reorders_hits_and_degrades_on_failure() {
+ use ai_memory_llm::{RerankCandidate, RerankScore, Reranker};
+
+ /// Scores candidates by a fixed path→relevance table; paths not
+ /// in the table are skipped (the "reranker had no opinion" case).
+ struct TableReranker(Vec<(&'static str, f32)>);
+ #[async_trait::async_trait]
+ impl Reranker for TableReranker {
+ fn name(&self) -> &'static str {
+ "table"
+ }
+ fn model(&self) -> &str {
+ "test"
+ }
+ async fn rerank(
+ &self,
+ _query: &str,
+ candidates: &[RerankCandidate],
+ ) -> ai_memory_llm::error::LlmResult<Vec<RerankScore>> {
+ Ok(candidates
+ .iter()
+ .filter_map(|c| {
+ self.0
+ .iter()
+ .find(|(path, _)| *path == c.id)
+ .map(|(_, rel)| RerankScore {
+ id: c.id.clone(),
+ relevance: *rel,
+ })
+ })
+ .collect())
+ }
+ }
+
+ struct FailingReranker;
+ #[async_trait::async_trait]
+ impl Reranker for FailingReranker {
+ fn name(&self) -> &'static str {
+ "failing"
+ }
+ fn model(&self) -> &str {
+ "test"
+ }
+ async fn rerank(
+ &self,
+ _query: &str,
+ _candidates: &[RerankCandidate],
+ ) -> ai_memory_llm::error::LlmResult<Vec<RerankScore>> {
+ Err(ai_memory_llm::LlmError::Provider {
+ status: 500,
+ body: "boom".into(),
+ })
+ }
+ }
+
+ let (_tmp, store, server, ws, proj) = setup_server().await;
+ // Three pages all matching "karpathy"; `foo.md` comes from the
+ // fixture. FTS decides the baseline order.
+ for (path, body) in [
+ ("second.md", "Karpathy wiki second page about compiling"),
+ ("third.md", "Karpathy wiki third page about compiling"),
+ ] {
+ store
+ .writer
+ .upsert_page(NewPage {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new(path).unwrap(),
+ title: path.into(),
+ body: body.into(),
+ tier: Tier::Semantic,
+ frontmatter_json: serde_json::json!({}),
+ pinned: false,
+ links: Vec::new(),
+ author_id: None,
+ expires_at: None,
+ entities: Vec::new(),
+ })
+ .await
+ .unwrap();
+ }
+ let paths_of = |json: &serde_json::Value| -> Vec<String> {
+ json["hits"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .map(|h| h["path"].as_str().unwrap().to_string())
+ .collect()
+ };
+ async fn query(server: &AiMemoryServer) -> serde_json::Value {
+ let args = QueryArgs {
+ query: "karpathy".into(),
+ limit: Some(3),
+ project: None,
+ scopes: Vec::new(),
+ workspace: None,
+ global: None,
+ include_expired: None,
+ explain: Some(true),
+ };
+ let result = server
+ .memory_query(Parameters(args), OptionalParts(test_parts_default()))
+ .await
+ .unwrap();
+ let text = result.content.first().and_then(|c| c.as_text()).unwrap();
+ serde_json::from_str::<serde_json::Value>(&text.text).unwrap()
+ }
+
+ let baseline = paths_of(&query(&server).await);
+ assert_eq!(baseline.len(), 3, "{baseline:?}");
+
+ // Rerank promoting the LAST baseline hit to the front, and
+ // scoring only two of the three (the third keeps its position).
+ let promoted = baseline[2].clone();
+ let demoted = baseline[0].clone();
+ let table: Vec<(&'static str, f32)> = vec![
+ (
+ Box::leak(promoted.clone().into_boxed_str()) as &'static str,
+ 0.95,
+ ),
+ (
+ Box::leak(demoted.clone().into_boxed_str()) as &'static str,
+ 0.10,
+ ),
+ ];
+ let reranked_server =
+ AiMemoryServer::new(store.reader.clone(), store.writer.clone(), ws, proj)
+ .with_reranker(Arc::new(TableReranker(table)));
+ let reranked = query(&reranked_server).await;
+ let order = paths_of(&reranked);
+ assert_eq!(order[0], promoted, "top score must lead: {order:?}");
+ assert_eq!(
+ order[1], demoted,
+ "scored candidates sort by relevance ahead of unjudged ones: {order:?}"
+ );
+ assert_eq!(
+ order[2], baseline[1],
+ "the skipped candidate follows in RRF order: {order:?}"
+ );
+ assert!(
+ (reranked["hits"][0]["score_details"]["rerank_score"]
+ .as_f64()
+ .unwrap()
+ - 0.95)
+ .abs()
+ < 1e-6,
+ "explain must surface the rerank score: {reranked}"
+ );
+ assert!(
+ reranked["hits"][2]["score_details"]
+ .get("rerank_score")
+ .is_none(),
+ "an unscored candidate carries no rerank_score: {reranked}"
+ );
+
+ // Thin coverage (1 of 3 scored) is treated as a degradation.
+ let thin_server = AiMemoryServer::new(store.reader.clone(), store.writer.clone(), ws, proj)
+ .with_reranker(Arc::new(TableReranker(vec![(
+ Box::leak(promoted.clone().into_boxed_str()) as &'static str,
+ 0.99,
+ )])));
+ assert_eq!(
+ paths_of(&query(&thin_server).await),
+ baseline,
+ "a reranker that scored under half the candidates must degrade to RRF order"
+ );
+
+ // A failing reranker must not fail the query.
+ let failing_server =
+ AiMemoryServer::new(store.reader.clone(), store.writer.clone(), ws, proj)
+ .with_reranker(Arc::new(FailingReranker));
+ assert_eq!(
+ paths_of(&query(&failing_server).await),
+ baseline,
+ "a failing reranker must degrade to RRF order"
+ );
+ }
+
+ /// Feedback moves the page's salience, drives the decay formula, and
+ /// routes stale/wrong signals to lint — without deleting anything.
+ #[tokio::test]
+ async fn memory_feedback_moves_salience_and_flags_for_lint() {
+ let (_tmp, store, server, ws, proj) = setup_server().await;
+ let feedback = |signal: &str, reason: Option<&str>| FeedbackArgs {
+ path: "foo.md".into(),
+ signal: signal.into(),
+ reason: reason.map(str::to_string),
+ project: None,
+ workspace: None,
+ };
+ let call = async |args: FeedbackArgs| {
+ server
+ .memory_feedback(Parameters(args), OptionalParts(test_parts_default()))
+ .await
+ };
+ let json_of = |result: &CallToolResult| -> serde_json::Value {
+ let text = result.content.first().and_then(|c| c.as_text()).unwrap();
+ serde_json::from_str(&text.text).unwrap()
+ };
+
+ // Baseline: no feedback yet → NULL salience → decay uses the default.
+ let baseline = store.reader.decay_candidates(ws, proj).await.unwrap();
+ assert_eq!(baseline.len(), 1);
+ assert!(baseline[0].salience.is_none());
+
+ let helpful = json_of(&call(feedback("helpful", None)).await.unwrap());
+ assert_eq!(helpful["recorded"], true);
+ assert_eq!(helpful["signal"], "helpful");
+ assert_eq!(helpful["routed_to_lint"], false);
+ let boosted = helpful["salience"].as_f64().unwrap();
+ assert!(
+ boosted > ai_memory_store::DecayParams::default().salience_default,
+ "helpful must raise salience, got {boosted}",
+ );
+ let after = store.reader.decay_candidates(ws, proj).await.unwrap();
+ assert_eq!(after[0].salience, Some(boosted));
+
+ // stale drops to the floor and flags the page for lint.
+ let stale = json_of(
+ &call(feedback("stale", Some("we moved off Postgres in March")))
+ .await
+ .unwrap(),
+ );
+ assert_eq!(stale["routed_to_lint"], true);
+ assert!(
+ (stale["salience"].as_f64().unwrap() - ai_memory_store::SALIENCE_MIN).abs() < 1e-9,
+ "stale must floor salience: {stale}",
+ );
+ let flagged = store.reader.open_feedback_findings(ws, proj).await.unwrap();
+ assert_eq!(flagged.len(), 1, "{flagged:?}");
+ assert_eq!(flagged[0].path, "foo.md");
+ assert_eq!(flagged[0].kind, "stale");
+ assert_eq!(
+ flagged[0].reason.as_deref(),
+ Some("we moved off Postgres in March")
+ );
+
+ // Nothing was deleted: the page is still searchable.
+ assert!(
+ !store
+ .reader
+ .search_pages_for_project(ws, proj, "karpathy".into(), 5, None)
+ .await
+ .unwrap()
+ .is_empty(),
+ "feedback must never delete or hide a page",
+ );
+
+ // Rewriting the page retires the flag (feedback is per version).
+ store
+ .writer
+ .upsert_page(NewPage {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new("foo.md").unwrap(),
+ title: "Foo".into(),
+ body: "Karpathy says compile, not retrieve. Now on SQLite.".into(),
+ tier: Tier::Semantic,
+ frontmatter_json: serde_json::json!({}),
+ pinned: false,
+ links: Vec::new(),
+ author_id: None,
+ expires_at: None,
+ entities: Vec::new(),
+ })
+ .await
+ .unwrap();
+ assert!(
+ store
+ .reader
+ .open_feedback_findings(ws, proj)
+ .await
+ .unwrap()
+ .is_empty(),
+ "rewriting the page must clear its findings",
+ );
+
+ // Unknown signal and unknown path both fail closed.
+ assert!(call(feedback("meh", None)).await.is_err());
+ assert!(
+ call(FeedbackArgs {
+ path: "does/not/exist.md".into(),
+ signal: "helpful".into(),
+ reason: None,
+ project: None,
+ workspace: None,
+ })
+ .await
+ .is_err(),
+ );
+ }
+
// Issue #154: default-scoped queries union the reserved `_global`
// preferences scope; explicitly scoped queries do not.
#[tokio::test]
@@ -4535,6 +4940,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5066,6 +5472,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5147,6 +5554,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5361,6 +5769,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5480,6 +5889,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5497,6 +5907,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5514,6 +5925,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5606,6 +6018,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5624,6 +6037,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: Some("2020-01-01T23:59:59Z".parse().unwrap()),
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -5739,6 +6153,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -6023,6 +6438,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
diff --git a/crates/ai-memory-mcp/tests/admin_phase3.rs b/crates/ai-memory-mcp/tests/admin_phase3.rs
index a6d30002..6c654e0e 100644
--- a/crates/ai-memory-mcp/tests/admin_phase3.rs
+++ b/crates/ai-memory-mcp/tests/admin_phase3.rs
@@ -497,6 +497,7 @@ async fn reorg_live_graveyards_only_default_workspace_pages() {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -525,6 +526,7 @@ async fn reorg_live_graveyards_only_default_workspace_pages() {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -604,6 +606,7 @@ async fn lint_dry_run_returns_lint_report_shape() {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
diff --git a/crates/ai-memory-mcp/tests/admin_read_page.rs b/crates/ai-memory-mcp/tests/admin_read_page.rs
index f5067d4d..b41dabd4 100644
--- a/crates/ai-memory-mcp/tests/admin_read_page.rs
+++ b/crates/ai-memory-mcp/tests/admin_read_page.rs
@@ -88,6 +88,7 @@ async fn read_page_falls_back_to_db_when_file_missing() {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
diff --git a/crates/ai-memory-mcp/tests/admin_status_search.rs b/crates/ai-memory-mcp/tests/admin_status_search.rs
index 816b0b53..bf3fbb32 100644
--- a/crates/ai-memory-mcp/tests/admin_status_search.rs
+++ b/crates/ai-memory-mcp/tests/admin_status_search.rs
@@ -62,6 +62,7 @@ async fn seed_page(store: &Store, title: &str, path: &str, body: &str) {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
};
store.writer.upsert_page(page).await.unwrap();
}
@@ -152,6 +153,7 @@ async fn list_projects_returns_workspace_project_pairs() {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
diff --git a/crates/ai-memory-mcp/tests/autoscope_multiuser.rs b/crates/ai-memory-mcp/tests/autoscope_multiuser.rs
index bafe9489..8fa9e59e 100644
--- a/crates/ai-memory-mcp/tests/autoscope_multiuser.rs
+++ b/crates/ai-memory-mcp/tests/autoscope_multiuser.rs
@@ -114,6 +114,7 @@ impl MultiUserHarness {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.expect("seed page");
diff --git a/crates/ai-memory-mcp/tests/autoscope_stress.rs b/crates/ai-memory-mcp/tests/autoscope_stress.rs
index 3ae0874b..b35bd7ab 100644
--- a/crates/ai-memory-mcp/tests/autoscope_stress.rs
+++ b/crates/ai-memory-mcp/tests/autoscope_stress.rs
@@ -122,6 +122,7 @@ impl Harness {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.expect("seed page");
diff --git a/crates/ai-memory-store/migrations/V38__entities.sql b/crates/ai-memory-store/migrations/V38__entities.sql
new file mode 100644
index 00000000..fce00b8f
--- /dev/null
+++ b/crates/ai-memory-store/migrations/V38__entities.sql
@@ -0,0 +1,41 @@
+-- Entity index: the noun layer of the wiki, used as a fourth retrieval
+-- stream alongside FTS5, vector cosine, and page-link neighbours.
+--
+-- Entities come from the frontmatter `entities:` list that the
+-- consolidator writes (extraction happens where an LLM already runs, so
+-- the zero-LLM default path is untouched — an empty table simply
+-- contributes nothing, exactly like the vector stream with no embedder).
+-- Markdown stays the source of truth: `reindex` rebuilds both tables
+-- from the files, same contract as `links`.
+--
+-- Query-time matching is lexical (exact + prefix over `name`), never an
+-- LLM call.
+--
+-- Deliberately a plain noun table, not a triple store: `entities.id` is
+-- the stable anchor a future temporal-triples table
+-- (`subject_entity_id`, `predicate`, `object_entity_id`, `valid_from`,
+-- `valid_to`, `source_page_id`) can reference. That P2 item stays
+-- deferred — this builds only the foundation it would need, and no
+-- graph database is involved either way.
+CREATE TABLE entities (
+ id BLOB PRIMARY KEY,
+ workspace_id BLOB NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
+ project_id BLOB NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+ -- Lowercase-normalized surface form; uniqueness is per project so
+ -- two projects can each have their own `postgres` entity.
+ name TEXT NOT NULL,
+ created_at INTEGER NOT NULL,
+ UNIQUE (workspace_id, project_id, name)
+);
+
+CREATE TABLE entity_page_links (
+ entity_id BLOB NOT NULL REFERENCES entities(id) ON DELETE CASCADE,
+ page_id BLOB NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
+ PRIMARY KEY (entity_id, page_id)
+);
+
+-- Reverse lookup for the retrieval stream (page → entities) and for
+-- replacing a page's entity set on rewrite.
+CREATE INDEX idx_entity_page_links_page ON entity_page_links(page_id);
+-- Prefix matching (`name LIKE 'postg%'`) rides this index.
+CREATE INDEX idx_entities_name ON entities(workspace_id, project_id, name);
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index f220dd06..86924b83 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -35,7 +35,10 @@ pub use auto_improve::{
AutoImproveTelemetryCount, FailAutoImproveProposal, NewAutoImproveProposal,
RejectAutoImproveProposal, StageAutoImproveRun, StagedAutoImproveRun, artifact_path_for,
};
-pub use decay::{DecayParams, retention_score};
+pub use decay::{
+ DecayParams, SALIENCE_MAX, SALIENCE_MIN, SALIENCE_STEP, retention_score,
+ salience_after_feedback,
+};
pub use error::{StoreError, StoreResult};
pub use maintenance::MaintenanceJob;
pub use ops::{
@@ -146,6 +149,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
}
}
@@ -3835,6 +3839,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
};
store.writer.upsert_page(page).await.unwrap();
@@ -5386,4 +5391,169 @@ mod tests {
"active"
);
}
+ #[tokio::test]
+ async fn entity_stream_finds_and_weights_pages() {
+ let tmp = TempDir::new().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "entities", None)
+ .await
+ .unwrap();
+
+ let with_entities = |path: &str, body: &str, entities: Vec<&str>| {
+ let mut page = sample_page(ws, proj, path, body);
+ page.entities = entities.into_iter().map(str::to_string).collect();
+ page
+ };
+ // `turbopuffer` is rare (one page); `sqlite` is common (three).
+ store
+ .writer
+ .upsert_page(with_entities(
+ "rare.md",
+ "no query words in this body at all",
+ vec!["turbopuffer", "sqlite"],
+ ))
+ .await
+ .unwrap();
+ for path in ["common1.md", "common2.md"] {
+ store
+ .writer
+ .upsert_page(with_entities(path, "unrelated prose", vec!["sqlite"]))
+ .await
+ .unwrap();
+ }
+
+ // A query naming the entity finds the page whose body lacks the term.
+ let hits = store
+ .reader
+ .entity_hits_for_project(ws, proj, "how do we use Turbopuffer?", 10, None)
+ .await
+ .unwrap();
+ assert_eq!(hits.len(), 1, "{hits:?}");
+ assert_eq!(hits[0].hit.path.as_str(), "rare.md");
+ assert_eq!(hits[0].matched, vec!["turbopuffer".to_string()]);
+
+ // Inverse frequency: the page carrying the rare entity outranks
+ // pages carrying only the common one.
+ let hits = store
+ .reader
+ .entity_hits_for_project(ws, proj, "turbopuffer sqlite", 10, None)
+ .await
+ .unwrap();
+ assert_eq!(hits.len(), 3, "{hits:?}");
+ assert_eq!(hits[0].hit.path.as_str(), "rare.md");
+ assert!(
+ hits[0].weight > hits[1].weight,
+ "rare entity must weigh more: {:?}",
+ hits.iter()
+ .map(|h| (h.hit.path.as_str(), h.weight))
+ .collect::<Vec<_>>(),
+ );
+
+ // Prefix matching, and short tokens are ignored as noise.
+ assert_eq!(
+ store
+ .reader
+ .entity_hits_for_project(ws, proj, "turbopuff", 10, None)
+ .await
+ .unwrap()
+ .len(),
+ 1,
+ "prefix should match the entity",
+ );
+ assert!(
+ store
+ .reader
+ .entity_hits_for_project(ws, proj, "a of", 10, None)
+ .await
+ .unwrap()
+ .is_empty(),
+ "sub-3-char tokens must not match",
+ );
+
+ // Hybrid search surfaces the entity-only hit and explains it.
+ let explained = store
+ .reader
+ .hybrid_search_explained(
+ ws,
+ proj,
+ "turbopuffer".into(),
+ None,
+ String::new(),
+ String::new(),
+ 0,
+ 10,
+ None,
+ )
+ .await
+ .unwrap();
+ let (hit, explain) = explained
+ .iter()
+ .find(|(h, _)| h.path.as_str() == "rare.md")
+ .expect("entity stream must feed hybrid search");
+ assert_eq!(explain.entity_rank, Some(1));
+ assert_eq!(explain.matched_entities, vec!["turbopuffer".to_string()]);
+ assert!(explain.rrf.entity > 0.0);
+ assert!(
+ explain.fts_rank.is_none(),
+ "the body has no query term, so FTS must miss it",
+ );
+ // `rank` is the fused score after the bounded authority multiplier,
+ // so it tracks `fused` only up to that factor — which is exactly why
+ // the explain reports the factor alongside it.
+ let authority = explain.authority.unwrap_or(1.0);
+ assert!((hit.rank + explain.fused * authority).abs() < 1e-12);
+
+ // Rewriting the page replaces its entity set.
+ store
+ .writer
+ .upsert_page(with_entities("rare.md", "rewritten body", vec!["lancedb"]))
+ .await
+ .unwrap();
+ assert!(
+ store
+ .reader
+ .entity_hits_for_project(ws, proj, "turbopuffer", 10, None)
+ .await
+ .unwrap()
+ .is_empty(),
+ "a rewrite must drop the old entity links",
+ );
+ assert_eq!(
+ store
+ .reader
+ .entity_hits_for_project(ws, proj, "lancedb", 10, None)
+ .await
+ .unwrap()
+ .len(),
+ 1,
+ );
+
+ // A project with no declared entities contributes nothing.
+ let bare = store
+ .writer
+ .get_or_create_project(ws, "bare", None)
+ .await
+ .unwrap();
+ store
+ .writer
+ .upsert_page(sample_page(ws, bare, "plain.md", "sqlite and turbopuffer"))
+ .await
+ .unwrap();
+ assert!(
+ store
+ .reader
+ .entity_hits_for_project(ws, bare, "turbopuffer", 10, None)
+ .await
+ .unwrap()
+ .is_empty(),
+ "no entities declared → the stream stays silent",
+ );
+ }
}
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index 7ced4d17..36e221ca 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -588,6 +588,7 @@ pub(crate) fn upsert_page_in_tx(
],
)?;
replace_links_in_tx(tx, &new_id, page)?;
+ replace_entities_in_tx(tx, &new_id, page)?;
refresh_incoming_links_for_path(tx, page, &new_id)?;
audit(
tx,
@@ -626,6 +627,7 @@ pub(crate) fn upsert_page_in_tx(
],
)?;
replace_links_in_tx(tx, &new_id, page)?;
+ replace_entities_in_tx(tx, &new_id, page)?;
refresh_incoming_links_for_path(tx, page, &new_id)?;
audit(
tx,
@@ -641,6 +643,58 @@ pub(crate) fn upsert_page_in_tx(
Ok(new_id)
}
+/// Re-point a page version's entity set (V38). Entity *rows* are shared
+/// per project and intentionally survive: a page dropping its last
+/// mention of `postgres` should not invalidate every other page's link
+/// to the same noun, and the orphan row costs one short string. The
+/// links are what get replaced.
+fn replace_entities_in_tx(
+ tx: &rusqlite::Transaction<'_>,
+ page_id: &PageId,
+ page: &NewPage,
+) -> StoreResult<()> {
+ tx.execute(
+ "DELETE FROM entity_page_links WHERE page_id = ?1",
+ params![page_id.as_bytes()],
+ )?;
+ if page.entities.is_empty() {
+ return Ok(());
+ }
+ let now = Timestamp::now().as_microsecond();
+ // Defence in depth: the wiki layer normalises on the way in, but the
+ // store is the last gate before the UNIQUE(name) constraint.
+ for name in ai_memory_core::normalize_entities(&page.entities) {
+ tx.execute(
+ "INSERT INTO entities (id, workspace_id, project_id, name, created_at) \
+ VALUES (?1, ?2, ?3, ?4, ?5) \
+ ON CONFLICT (workspace_id, project_id, name) DO NOTHING",
+ params![
+ ai_memory_core::PageId::new().as_bytes(),
+ page.workspace_id.as_bytes(),
+ page.project_id.as_bytes(),
+ name,
+ now,
+ ],
+ )?;
+ let entity_id: Vec<u8> = tx.query_row(
+ "SELECT id FROM entities \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND name = ?3",
+ params![
+ page.workspace_id.as_bytes(),
+ page.project_id.as_bytes(),
+ name
+ ],
+ |row| row.get(0),
+ )?;
+ tx.execute(
+ "INSERT INTO entity_page_links (entity_id, page_id) VALUES (?1, ?2) \
+ ON CONFLICT (entity_id, page_id) DO NOTHING",
+ params![entity_id, page_id.as_bytes()],
+ )?;
+ }
+ Ok(())
+}
+
fn replace_links_in_tx(
tx: &rusqlite::Transaction<'_>,
from_page_id: &PageId,
@@ -2620,6 +2674,7 @@ pub(crate) mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
}
}
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index a89acf2a..f3634565 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -236,6 +236,74 @@ pub struct FeedbackFinding {
pub reason: Option<String>,
}
+/// A page matched by the entity stream, with the inverse-frequency
+/// weight that ranked it and the entity names that matched.
+#[derive(Debug, Clone)]
+pub struct EntityHit {
+ /// The matched page.
+ pub hit: PageHit,
+ /// Sum of `1 / pages_carrying_entity` over the matched entities.
+ pub weight: f64,
+ /// Entity names that matched the query.
+ pub matched: Vec<String>,
+}
+
+/// Escape a literal for use inside a SQL `LIKE` pattern with
+/// `ESCAPE '\'`. Entity tokens legitimately contain `_`, which `LIKE`
+/// would otherwise treat as "any single character" — so `foo_bar` would
+/// match `fooxbar`.
+fn like_escape(literal: &str) -> String {
+ literal
+ .replace('\\', "\\\\")
+ .replace('%', "\\%")
+ .replace('_', "\\_")
+}
+
+/// Split a query into entity-match tokens: lowercase, alphanumeric-ish
+/// runs, 3+ characters, de-duplicated, bounded. Short tokens are
+/// dropped because a 1-2 character prefix match is noise, not a signal.
+///
+/// A hyphenated/underscored run yields BOTH the whole token and its
+/// parts — the same both-forms trick [`crate::ops::path_search_text`]
+/// uses on paths, and for the same reason: an entity may be written
+/// `ai-memory`, `ai_memory`, or `ai memory`, and the query may pick
+/// any of them.
+fn entity_query_tokens(query: &str) -> Vec<String> {
+ /// Beyond this the query is prose, not a set of nouns; the FTS
+ /// stream already handles prose better than prefix matching does.
+ const MAX_TOKENS: usize = 12;
+ const MIN_TOKEN_CHARS: usize = 3;
+ let mut seen = std::collections::BTreeSet::new();
+ let mut out = Vec::new();
+ let mut push = |token: String| -> bool {
+ if token.chars().count() < MIN_TOKEN_CHARS {
+ return true;
+ }
+ if seen.insert(token.clone()) {
+ out.push(token);
+ }
+ out.len() < MAX_TOKENS
+ };
+ 'outer: for raw in query.split(|c: char| !(c.is_alphanumeric() || c == '-' || c == '_')) {
+ let token = raw.trim_matches(['-', '_']).to_lowercase();
+ if token.is_empty() {
+ continue;
+ }
+ let compound = token.contains(['-', '_']);
+ if !push(token.clone()) {
+ break;
+ }
+ if compound {
+ for part in token.split(['-', '_']) {
+ if !push(part.to_string()) {
+ break 'outer;
+ }
+ }
+ }
+ }
+ out
+}
+
/// A graph-expansion neighbour with provenance (which seed and which
/// link direction produced it). Internal to hybrid search + explain.
struct GraphNeighbor {
@@ -266,6 +334,8 @@ pub struct RrfContributions {
pub vector: f64,
/// Graph-neighbour stream contribution.
pub graph: f64,
+ /// Entity-match stream contribution.
+ pub entity: f64,
}
/// Score transparency for one `memory_query` hit: where the hit ranked
@@ -295,6 +365,12 @@ pub struct SearchExplain {
/// Which seed page and link direction pulled the hit in.
#[serde(skip_serializing_if = "Option::is_none")]
pub graph_via: Option<GraphVia>,
+ /// 1-based rank in the entity-match stream.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub entity_rank: Option<usize>,
+ /// Entity names on this page that matched the query.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub matched_entities: Vec<String>,
/// Per-stream RRF contributions.
pub rrf: RrfContributions,
/// Total fused score (sum of the RRF contributions) — higher is
@@ -2118,6 +2194,129 @@ impl ReaderPool {
.await
}
+ /// Rank pages by how many of the query's tokens match their indexed
+ /// entities, weighting each match by inverse entity frequency — a
+ /// query token matching an entity that appears on 2 pages is a much
+ /// stronger signal than one matching an entity on 200.
+ ///
+ /// Matching is lexical (exact name, or entity starting with the
+ /// token) so this stays a plain SQL stream with no LLM call. Empty
+ /// result when the project has no entities indexed, which is the
+ /// zero-LLM default: entities only exist where consolidation ran.
+ ///
+ /// `expiry_cutoff_us`: see [`Self::search_pages_for_project`].
+ ///
+ /// # Errors
+ /// Propagates any SQL or pool error.
+ pub async fn entity_hits_for_project(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ query: &str,
+ limit: usize,
+ expiry_cutoff_us: Option<i64>,
+ ) -> StoreResult<Vec<EntityHit>> {
+ let tokens = entity_query_tokens(query);
+ if tokens.is_empty() || limit == 0 {
+ return Ok(Vec::new());
+ }
+ let cutoff = expiry_cutoff_us.unwrap_or_else(now_us);
+ self.with_conn(move |conn| {
+ let mut placeholders = String::with_capacity(tokens.len() * 48);
+ let mut sql_params: Vec<Value> = Vec::with_capacity(tokens.len() * 3 + 4);
+ for (idx, token) in tokens.iter().enumerate() {
+ if idx > 0 {
+ placeholders.push_str(" OR ");
+ }
+ // Three ways a token can match an entity name, in
+ // increasing looseness: exact; the name starts with the
+ // token (`postgres` → `postgresql`); or a *word* inside a
+ // multi-word name starts with the token (`jetstream` →
+ // `nats jetstream`). The last one is why this is a scan
+ // rather than a pure index seek — acceptable because the
+ // table is bounded at 10 entities per page.
+ //
+ // `?` is positional-by-order, same as the graph query.
+ placeholders.push_str(
+ "(e.name = ? \
+ OR e.name LIKE ? || '%' ESCAPE '\\' \
+ OR e.name LIKE '% ' || ? || '%' ESCAPE '\\')",
+ );
+ sql_params.push(Value::Text(token.clone()));
+ let escaped = like_escape(token);
+ sql_params.push(Value::Text(escaped.clone()));
+ sql_params.push(Value::Text(escaped));
+ }
+ sql_params.push(Value::Blob(workspace_id.as_bytes().to_vec()));
+ sql_params.push(Value::Blob(project_id.as_bytes().to_vec()));
+ sql_params.push(Value::Integer(cutoff));
+ #[allow(clippy::cast_possible_wrap)]
+ sql_params.push(Value::Integer(limit as i64));
+
+ let mut sql = String::with_capacity(placeholders.len() + 1_200);
+ write!(
+ &mut sql,
+ // `freq` is how many current pages in this project carry
+ // the entity; 1/freq is the inverse-frequency weight.
+ "WITH matched AS ( \
+ SELECT e.id AS entity_id, e.name AS name \
+ FROM entities e \
+ WHERE ({placeholders}) \
+ ), \
+ freq AS ( \
+ SELECT m.entity_id, m.name, COUNT(*) AS pages \
+ FROM matched m \
+ JOIN entity_page_links l ON l.entity_id = m.entity_id \
+ JOIN pages p ON p.id = l.page_id AND p.is_latest = 1 \
+ GROUP BY m.entity_id, m.name \
+ ) \
+ SELECT pg.id, pg.path, pg.title, substr(pg.body, 1, 240) AS snippet, \
+ SUM(1.0 / f.pages) AS weight, \
+ COUNT(*) AS matches, \
+ group_concat(f.name, ', ') AS names \
+ FROM freq f \
+ JOIN entity_page_links l ON l.entity_id = f.entity_id \
+ JOIN pages pg ON pg.id = l.page_id \
+ WHERE pg.workspace_id = ? AND pg.project_id = ? AND pg.is_latest = 1{not_expired} \
+ GROUP BY pg.id, pg.path, pg.title \
+ ORDER BY weight DESC, matches DESC, pg.path ASC \
+ LIMIT ?",
+ not_expired = not_expired("pg", "?"),
+ )
+ .expect("writing SQL into String cannot fail");
+
+ let mut stmt = conn.prepare(&sql)?;
+ let rows = stmt.query_map(params_from_iter(sql_params.iter()), |row| {
+ let id_bytes: Vec<u8> = row.get(0)?;
+ let path: String = row.get(1)?;
+ let title: String = row.get(2)?;
+ let snippet: String = row.get(3)?;
+ let weight: f64 = row.get(4)?;
+ let names: Option<String> = row.get(6)?;
+ Ok((id_bytes, path, title, snippet, weight, names))
+ })?;
+ let mut out = Vec::new();
+ for row in rows {
+ let (id_bytes, path, title, snippet, weight, names) = row?;
+ out.push(EntityHit {
+ hit: PageHit {
+ id: PageId::from_slice(&id_bytes)?,
+ path: PagePath::new(path)?,
+ title,
+ snippet,
+ rank: 0.0,
+ },
+ weight,
+ matched: names
+ .map(|n| n.split(", ").map(str::to_string).collect())
+ .unwrap_or_default(),
+ });
+ }
+ Ok(out)
+ })
+ .await
+ }
+
/// Return `stale` / `wrong` feedback still attached to a *current*
/// page version in one project, grouped by (page, kind) — the input
/// for the lint pass's `feedback_flagged` findings. Rewriting a
@@ -2175,6 +2374,9 @@ impl ReaderPool {
.await
}
+ /// Return pages linked to or from the seed pages, scoped to latest pages
+ /// in the same project.
+ ///
/// `expiry_cutoff_us`: see [`Self::search_pages_for_project`].
///
/// # Errors
@@ -2366,13 +2568,14 @@ impl ReaderPool {
/// Hybrid search: RRF-fuse FTS5 results with cosine-similarity
/// over the stored embeddings of the matching `(provider, model,
- /// dim)`, then add link-neighbour expansion as a third RRF stream.
- /// Applies the bounded page-authority adjustment after fusion and returns
- /// the top-`limit` pages by adjusted fused score.
+ /// dim)`, entity matches, and link-neighbour expansion — four RRF
+ /// streams. Applies the bounded page-authority adjustment after
+ /// fusion and returns the top-`limit` pages by adjusted fused score.
///
/// Each stream degrades independently to contributing nothing: no
- /// `query_vec` skips the vector stream, while graph expansion still
- /// runs from whatever seeds the other streams produced.
+ /// `query_vec` skips the vector stream, an empty `entities` table
+ /// skips the entity stream, and graph expansion still runs from
+ /// whatever seeds the other streams produced.
///
/// `expiry_cutoff_us`: see [`Self::search_pages_for_project`]. The
/// cutoff resolves once here so all streams agree on it.
@@ -2472,6 +2675,9 @@ impl ReaderPool {
// window was too narrow at small limits for a canonical page to enter
// the fused pool and receive its bounded promotion.
let candidate_limit = authority_candidate_limit(limit);
+ // The FTS candidate call consumes the query; the entity stream
+ // tokenizes the same text separately.
+ let entity_query = query.clone();
// Fetch FTS5 hits first.
let fts_candidates = self
@@ -2491,6 +2697,15 @@ impl ReaderPool {
.into_iter()
.map(|(hit, _authority)| hit)
.collect();
+ let entity_hits = self
+ .entity_hits_for_project(
+ workspace_id,
+ project_id,
+ &entity_query,
+ candidate_limit,
+ Some(cutoff),
+ )
+ .await?;
let mut vec_hits: Vec<(PageId, PagePath, f32)> = Vec::new();
if let Some(qv) = query_vec {
vec_hits = self
@@ -2526,6 +2741,14 @@ impl ReaderPool {
}
}
}
+ for e in &entity_hits {
+ if seed_seen.insert(e.hit.id) {
+ seed_ids.push(e.hit.id);
+ if let Some(paths) = &mut seed_paths {
+ paths.push(e.hit.path.as_str().to_string());
+ }
+ }
+ }
let graph_hits = self
.graph_neighbors_for_project_explained(
workspace_id,
@@ -2581,6 +2804,23 @@ impl ReaderPool {
details.fused += contrib;
}
}
+ for (rank, e) in entity_hits.iter().enumerate() {
+ let contrib = 1.0 / (k + (rank + 1) as f64);
+ let entry = fused.entry(e.hit.id).or_insert_with(|| Fused {
+ path: e.hit.path.clone(),
+ title: e.hit.title.clone(),
+ snippet: e.hit.snippet.clone(),
+ score: 0.0,
+ explain: explain.then(SearchExplain::default),
+ });
+ entry.score += contrib;
+ if let Some(details) = &mut entry.explain {
+ details.entity_rank = Some(rank + 1);
+ details.matched_entities = e.matched.clone();
+ details.rrf.entity = contrib;
+ details.fused += contrib;
+ }
+ }
for (rank, n) in graph_hits.iter().enumerate() {
let contrib = 1.0 / (k + (rank + 1) as f64);
let entry = fused.entry(n.hit.id).or_insert_with(|| Fused {
@@ -5954,6 +6194,7 @@ fn open_read_only(path: &Path) -> StoreResult<Connection> {
#[cfg(test)]
mod tests {
+ use super::{entity_query_tokens, like_escape};
use crate::Store;
use ai_memory_core::{
@@ -6535,4 +6776,43 @@ mod tests {
assert_eq!(matched, None, "legacy trailing separator must not match");
}
+
+ #[test]
+ fn entity_query_tokens_lowercases_filters_short_and_bounds() {
+ // A compound token yields the whole form AND its parts, so a
+ // query can spell an entity `writer-actor` or `writer actor`.
+ assert_eq!(
+ entity_query_tokens("How does the Writer-Actor use SQLite?"),
+ vec![
+ "how".to_string(),
+ "does".to_string(),
+ "the".to_string(),
+ "writer-actor".to_string(),
+ "writer".to_string(),
+ "actor".to_string(),
+ "use".to_string(),
+ "sqlite".to_string(),
+ ],
+ );
+ assert!(
+ entity_query_tokens("a of an X").is_empty(),
+ "sub-3-char tokens are noise for prefix matching",
+ );
+ assert_eq!(
+ entity_query_tokens("dup dup dup"),
+ vec!["dup".to_string()],
+ "tokens de-duplicate",
+ );
+ assert!(entity_query_tokens(&"word ".repeat(50)).len() <= 12);
+ }
+
+ #[test]
+ fn like_escape_neutralises_wildcards() {
+ // `_` is LIKE's single-char wildcard and appears in real entity
+ // names (`writer_actor`), so it must not match `writerxactor`.
+ assert_eq!(like_escape("writer_actor"), "writer\\_actor");
+ assert_eq!(like_escape("100%"), "100\\%");
+ assert_eq!(like_escape("back\\slash"), "back\\\\slash");
+ assert_eq!(like_escape("plain-name"), "plain-name");
+ }
}
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index e62f7415..9a91f534 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -2044,6 +2044,7 @@ mod tests {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
}
}
diff --git a/crates/ai-memory-web/tests/routes.rs b/crates/ai-memory-web/tests/routes.rs
index 7154f022..ba6cee7a 100644
--- a/crates/ai-memory-web/tests/routes.rs
+++ b/crates/ai-memory-web/tests/routes.rs
@@ -39,6 +39,7 @@ fn new_page(
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
}
}
@@ -623,6 +624,7 @@ async fn api_pages_derives_kind_from_path_when_frontmatter_absent() {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
@@ -643,6 +645,7 @@ async fn api_pages_derives_kind_from_path_when_frontmatter_absent() {
links: Vec::new(),
author_id: None,
expires_at: None,
+ entities: Vec::new(),
})
.await
.unwrap();
diff --git a/crates/ai-memory-wiki/src/wiki.rs b/crates/ai-memory-wiki/src/wiki.rs
index e21dca65..1f0b2910 100644
--- a/crates/ai-memory-wiki/src/wiki.rs
+++ b/crates/ai-memory-wiki/src/wiki.rs
@@ -409,6 +409,7 @@ impl Wiki {
links,
author_id: None,
expires_at: meta.expires_at,
+ entities: meta.entities,
})
.await?;
Ok(id)
@@ -843,8 +844,9 @@ impl Wiki {
scrub_frontmatter_strings(&mut markdown.frontmatter, &self.sanitizer);
let title = self.sanitizer.scrub(&detail.summary.title);
let links = extract_links(&markdown.body, &path);
- let emitted = emit(&markdown)?;
let expires_at = parse_expires_at(&path, &markdown.frontmatter)?;
+ let entities = parse_entities(&path, &markdown.frontmatter)?;
+ let emitted = emit(&markdown)?;
let page = NewPage {
workspace_id,
project_id,
@@ -857,6 +859,7 @@ impl Wiki {
links,
author_id,
expires_at,
+ entities,
};
let result = {
@@ -949,8 +952,9 @@ impl Wiki {
frontmatter_json: md.frontmatter,
pinned: meta.pinned,
links,
- expires_at: meta.expires_at,
author_id: None,
+ expires_at: meta.expires_at,
+ entities: meta.entities,
})
.await?;
Ok(id)
@@ -1228,6 +1232,7 @@ impl Wiki {
links: extract_links(&req.body, &req.path),
author_id: req.author_id,
expires_at: parse_expires_at(&req.path, &req.frontmatter)?,
+ entities: parse_entities(&req.path, &req.frontmatter)?,
})
})
.collect::<WikiResult<Vec<_>>>()?;
@@ -1353,6 +1358,7 @@ impl Wiki {
.unwrap_or_else(|| derive_title(&markdown.frontmatter, &markdown.body, &path));
let links = extract_links(&markdown.body, &path);
let expires_at = parse_expires_at(&path, &markdown.frontmatter)?;
+ let entities = parse_entities(&path, &markdown.frontmatter)?;
let Markdown {
frontmatter: final_frontmatter,
@@ -1389,6 +1395,7 @@ impl Wiki {
links,
author_id,
expires_at,
+ entities,
})
.await
{
@@ -1517,10 +1524,12 @@ fn derive_index_metadata(
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expires_at = parse_expires_at(path, frontmatter)?;
+ let entities = parse_entities(path, frontmatter)?;
Ok(IndexMetadata {
tier,
pinned,
expires_at,
+ entities,
})
}
@@ -1530,14 +1539,39 @@ struct IndexMetadata {
tier: Tier,
pinned: bool,
expires_at: Option<jiff::Timestamp>,
+ entities: Vec<String>,
}
-/// Parse the frontmatter `expires_at:` key into a TTL instant.
+/// Parse the optional frontmatter `entities:` list — salient nouns the
+/// consolidator extracted, indexed by the store as a retrieval stream.
///
-/// Accepts RFC3339 (`2026-08-01T12:00:00Z`) or a bare `YYYY-MM-DD`,
-/// which means the *end* of that day in UTC — a date-only TTL that
-/// expired at midnight would surprise anyone who wrote "expires
-/// 2026-08-01" meaning "good through the 1st".
+/// Unlike `expires_at`, malformed entries are *dropped* rather than
+/// rejected: entities are a soft ranking signal, and refusing a whole
+/// page write because one hand-edited entity is 80 characters long
+/// would trade a real page for a marginal signal. A non-array value is
+/// still an error — that's a structural mistake, not a bad item.
+pub(crate) fn parse_entities(
+ path: &PagePath,
+ frontmatter: &serde_json::Value,
+) -> WikiResult<Vec<String>> {
+ let raw = match frontmatter.get("entities") {
+ None | Some(serde_json::Value::Null) => return Ok(Vec::new()),
+ Some(serde_json::Value::Array(items)) => items,
+ Some(_) => {
+ return Err(ai_memory_wiki_error(&format!(
+ "invalid non-array entities in frontmatter for {}",
+ path.as_str()
+ )));
+ }
+ };
+ let strings = raw.iter().filter_map(|v| v.as_str());
+ Ok(ai_memory_core::normalize_entities(strings))
+}
+
+/// Parse the optional frontmatter `expires_at:` key. Accepts RFC3339
+/// (`2026-09-01T12:00:00Z`) or a bare date (`2026-09-01` = end of that
+/// day, UTC). Anything else is rejected in the same style as an
+/// invalid `tier`, so a typo can't silently mean "never expires".
pub(crate) fn parse_expires_at(
path: &PagePath,
frontmatter: &serde_json::Value,
@@ -3574,4 +3608,41 @@ mod tests {
"reindex must reject symlinked manifests, got {err:#}"
);
}
+
+ /// Frontmatter `entities:` round-trips through the write path and is
+ /// normalised on the way into the index. A non-array value is a
+ /// structural error; bad *items* are dropped, because entities are a
+ /// ranking signal and one long hand-typed entry must not cost the page.
+ #[test]
+ fn parse_entities_normalises_and_rejects_non_arrays() {
+ let path = PagePath::new("concepts/x.md").unwrap();
+
+ assert!(
+ parse_entities(&path, &serde_json::json!({}))
+ .unwrap()
+ .is_empty(),
+ "absent key means no entities",
+ );
+ assert_eq!(
+ parse_entities(
+ &path,
+ &serde_json::json!({"entities": ["SQLite", " sqlite ", "Writer\nActor", 42]}),
+ )
+ .unwrap(),
+ vec!["sqlite".to_string(), "writer actor".to_string()],
+ "lowercased, whitespace-collapsed, de-duplicated, non-strings dropped",
+ );
+ assert!(
+ parse_entities(
+ &path,
+ &serde_json::json!({"entities": ["ok", "x".repeat(200)]}),
+ )
+ .unwrap()
+ == vec!["ok".to_string()],
+ "over-long items are dropped, not fatal",
+ );
+ let err = parse_entities(&path, &serde_json::json!({"entities": "sqlite"}))
+ .expect_err("a string instead of a list is a structural error");
+ assert!(err.to_string().contains("non-array entities"), "{err}");
+ }
}
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index b270f6a6..50e61edf 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -110,20 +110,26 @@ from hook paths.
trust boundary and delimiters precede automatically injected handoffs,
project briefs, and managed-workstream packets; current instructions and
checkout state remain authoritative.
-6. `memory_query` answers via FTS5 + link-neighbour RRF; when an
- embedder is configured, vector cosine over `page_embeddings` joins
- the same RRF. Before final truncation, a bounded authority multiplier
- adjusts the relevance score using the canonical page kind, tier,
- `pinned`, and explicit positive/negative frontmatter tags. It favors
- maintained rules, decisions, procedures, and gotchas in close contests
- while keeping episodic, historical, lint, and test evidence searchable.
- No query-intent regex or hard exclusion participates. If compiled wiki pages miss entirely in default,
- explicit project, or explicit `scopes` mode, bounded raw observation
- FTS returns fallback `raw_hits`; `global=true` searches compiled wiki
- pages across projects only. Page hits bump `access_count` +
- `last_accessed_at` - the M8 reinforcement term. That bump is throttled
- to at most once per page per minute, so a burst of overlapping searches
- does not flood the writer actor with redundant reinforcement writes.
+6. `memory_query` answers via FTS5 + entity-match + link-neighbour RRF;
+ when an embedder is configured, vector cosine over `page_embeddings`
+ joins the same RRF. Each stream contributes nothing when its input is
+ absent, so the zero-LLM default is FTS5 + entity + graph. Before final
+ truncation, a bounded authority multiplier adjusts the relevance score
+ using the canonical page kind, tier, `pinned`, and explicit
+ positive/negative frontmatter tags. It favors maintained rules,
+ decisions, procedures, and gotchas in close contests while keeping
+ episodic, historical, lint, and test evidence searchable. No
+ query-intent regex or hard exclusion participates. When
+ `AI_MEMORY_RERANKER=llm` is set the adjusted candidates get a final LLM
+ relevance pass (degrading to the adjusted order on error). If compiled
+ wiki pages miss entirely in default, explicit project, or explicit
+ `scopes` mode, bounded raw observation FTS returns fallback `raw_hits`;
+ `global=true` searches compiled wiki pages across projects only. Page
+ hits bump `access_count` + `last_accessed_at` - the M8 reinforcement
+ term, which `memory_feedback` complements with explicit per-page
+ salience. That bump is throttled to at most once per page per minute,
+ so a burst of overlapping searches does not flood the writer actor
+ with redundant reinforcement writes.
7. The forget sweep runs on demand and on the server's `[maintenance]`
schedule: pages past their frontmatter `expires_at:` TTL are
hard-deleted through the wiki layer (file + rows, pin or not);
@@ -250,6 +256,7 @@ separately gated Claude Code assistant/Stop excerpt remains capped at 2 KB.
| `handoffs` | Typed cross-agent handoff records (open / accepted / expired). |
| `page_embeddings` | Optional vector rows for latest pages, with `(provider, model, dim)` denormalised so hybrid search can ignore stale vectors after an embedding config change and report missing-embedding diagnostics. |
| `page_feedback` | Append-only `memory_feedback` signals (`helpful` / `not_helpful` / `stale` / `wrong`) keyed by page *version*, with an optional reason. Source of truth for the derived `pages.salience`; the lint pass reads unresolved stale/wrong rows joined against `is_latest = 1`, so a rewrite retires the finding. |
+| `entities`, `entity_page_links` | V38 noun index derived from the frontmatter `entities:` list the consolidator writes. Names are lowercase and unique per project; links are per page *version* and replaced on rewrite. Powers the 4th retrieval stream. `entities.id` is deliberately the stable anchor a future temporal-triples table could reference — that P2 item stays deferred. |
| `audit_log` | Every mutation, addressable by `at DESC`. |
**Memory tiers (M8 policy):**
@@ -355,8 +362,7 @@ must re-write its own routing rules into a project's `CLAUDE.md` /
safe default-on learning review through the same approval/write path as
pending writes, and `memory_delete_page` is the exact-path destructive pair
needed by admission-aware mirrors. `memory_handoff_cancel` is the safety valve
-for mistaken handoff creation. The narrow-surface discipline still holds —
-every new tool has to earn its slot — but the v1 count is 16, not 10. `memory_feedback` implements the
+for mistaken handoff creation. `memory_feedback` implements the
"finer-grained reinforcement beyond access counts" P2 item from
`prior-art-implementation-findings.md`: it cannot ride on a read tool
without conflating read and write semantics, and the access counter it
@@ -518,6 +524,19 @@ GitHub token through `/copilot_internal/v2/token`, and calls Copilot Chat with
the `vscode-chat` integration headers. The raw GitHub token is not sent to the
Copilot chat endpoint.
+**Reranker env** (opt-in, requires an LLM provider):
+```
+AI_MEMORY_RERANKER llm (only supported value; unknown values fail at startup)
+```
+`memory_query` then over-fetches (`limit * 3`, capped at 30 candidates)
+from the RRF stage and has the configured LLM provider score each
+candidate against the query via JSON-schema structured output. Scored
+candidates sort by relevance; any the model skipped follow in RRF order.
+On timeout (20 s), provider error, or a response covering fewer than half
+the candidates, the query answers from plain RRF order — the same
+degrade-don't-fail contract as the embedder. Off by default: it puts an
+LLM call on the search hot path.
+
**Embedder env** (opt-in):
```
AI_MEMORY_EMBEDDING_PROVIDER openai | voyage | google | gemini | openai-compat