Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(entities): populate the entity index from tags so as_of works

AkitaOnRails38ded9a58413
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1d12d6fc..fa882713 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
+- `as_of` time-travel queries and the entity retrieval stream now work
+ on real stores. Both read the entity index, which was populated only
+ from an LLM consolidator's `entities:` frontmatter — absent on the
+ bulk of a mature store (bootstrapped pages predate it; stable pages
+ are never re-consolidated), so the index sat empty and `as_of`
+ returned nothing. Entities are now also derived from each page's
+ `tags:` (the nouns a page is already labelled with) at write time, and
+ a one-shot idempotent backfill runs on the first start after upgrading
+ to populate the index for existing pages from their frontmatter —
+ opening each entity-link window at the page version's own `created_at`
+ so historical `as_of` is correct. Broad tags don't distort ranking
+ (the entity stream is inverse-frequency weighted), and a store already
+ written through the current path is a no-op. See `docs/temporal.md`.
- The Windows PowerShell wrapper (`ai-memory.ps1`) no longer crashes with
`git.exe : fatal: not a git repository … NativeCommandError` when run
outside a git repository — which broke `ai-memory status` and every
diff --git a/crates/ai-memory-core/src/lib.rs b/crates/ai-memory-core/src/lib.rs
index 40fde265..f61af500 100644
--- a/crates/ai-memory-core/src/lib.rs
+++ b/crates/ai-memory-core/src/lib.rs
@@ -62,7 +62,7 @@ pub use ids::{
pub use observation::{NewObservation, NewSession, Observation, ObservationKind};
pub use page::{
FeedbackKind, LinkTarget, MAX_ENTITIES_PER_PAGE, MAX_ENTITY_LEN, NewPage, Page, Relation, Tier,
- normalize_entities, normalize_entity,
+ frontmatter_entity_names, normalize_entities, normalize_entity,
};
pub use routing_snippet::{MARKER_END, MARKER_START, SNIPPET_BODY, find_marker_line, full_block};
pub use sanitize::{
diff --git a/crates/ai-memory-core/src/page.rs b/crates/ai-memory-core/src/page.rs
index ded6fe04..d31c652b 100644
--- a/crates/ai-memory-core/src/page.rs
+++ b/crates/ai-memory-core/src/page.rs
@@ -151,6 +151,33 @@ where
out
}
+/// Derive a page's indexed entity names from its frontmatter: the
+/// explicit `entities` list an LLM consolidator may emit, plus the
+/// `tags` list nearly every page carries. Both are arrays of strings
+/// meaning "what this page is about"; merging them populates the entity
+/// retrieval stream and `as_of` timelines deterministically, without
+/// depending on an LLM pass that a mature store's pages never received.
+///
+/// Explicit entities come first so they win the per-page cap. Lenient by
+/// design — a missing or non-array field simply contributes nothing, so
+/// this never fails on hand-edited frontmatter; the strict structural
+/// check for the write path lives in the wiki layer.
+#[must_use]
+pub fn frontmatter_entity_names(frontmatter: &serde_json::Value) -> Vec<String> {
+ fn strings(frontmatter: &serde_json::Value, key: &str) -> Vec<String> {
+ frontmatter
+ .get(key)
+ .and_then(serde_json::Value::as_array)
+ .into_iter()
+ .flatten()
+ .filter_map(|v| v.as_str().map(str::to_owned))
+ .collect()
+ }
+ let mut merged = strings(frontmatter, "entities");
+ merged.extend(strings(frontmatter, "tags"));
+ normalize_entities(merged)
+}
+
/// A link target discovered in a page body.
///
/// A bare `[[path]]` / `[label](path)` resolves within the source page's
@@ -439,4 +466,27 @@ mod tests {
.collect();
assert_eq!(normalize_entities(&many).len(), MAX_ENTITIES_PER_PAGE);
}
+
+ #[test]
+ fn frontmatter_entity_names_merges_entities_and_tags_leniently() {
+ // Explicit entities first (they win the cap), then tags; dupes drop.
+ assert_eq!(
+ frontmatter_entity_names(&serde_json::json!({
+ "entities": ["FTS5"],
+ "tags": ["fts5", "Search"],
+ })),
+ vec!["fts5".to_string(), "search".to_string()],
+ );
+ // Tags alone are enough — the common case on a mature store.
+ assert_eq!(
+ frontmatter_entity_names(&serde_json::json!({"tags": ["Storage"]})),
+ vec!["storage".to_string()],
+ );
+ // Missing or malformed fields contribute nothing, never panic.
+ assert!(frontmatter_entity_names(&serde_json::json!({})).is_empty());
+ assert!(
+ frontmatter_entity_names(&serde_json::json!({"tags": "not-a-list", "entities": 7}))
+ .is_empty(),
+ );
+ }
}
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index 31a90c0f..cfdb0b5c 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -50,9 +50,10 @@ pub use error::{StoreError, StoreResult};
pub use maintenance::MaintenanceJob;
pub use ops::{
AdmittedSession, CompactSummary, Compaction, DeleteWorkspaceSummary, EmbedOutcome,
- EmbeddingWrite, HookSessionAdmission, IngestObservationOutcome, LifecycleOnlyEndOutcome,
- MoveSessionSummary, MoveSummary, ObservationPruneOutcome, OkfMigratedPage, PagesMode,
- PurgeSessionSummary, PurgeSummary, ReorgSummary, purge_session, record_embed_failure,
+ EmbeddingWrite, EntityBackfillSummary, HookSessionAdmission, IngestObservationOutcome,
+ LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary, ObservationPruneOutcome,
+ OkfMigratedPage, PagesMode, PurgeSessionSummary, PurgeSummary, ReorgSummary,
+ backfill_entity_index, purge_session, record_embed_failure,
};
pub use reader::{
ActivityWindow, AgentSessionCount, AuditEvent, AuditLogFilter, AutoImproveCandidateSession,
@@ -132,6 +133,23 @@ impl Store {
migrations::run(&mut conn)?;
conn.pragma_update(None, "foreign_keys", "ON")?;
+ // One-shot, idempotent entity-index backfill (docs/temporal.md).
+ // Populates entities/entity_page_links from existing frontmatter
+ // for pages that predate entity extraction, so the entity
+ // retrieval stream and `as_of` timelines work on an upgraded
+ // store without a manual reindex. A store already populated
+ // through the write path finds no candidate pages and returns
+ // immediately. Runs single-threaded here, before the writer actor
+ // spawns, so it completes before the server accepts traffic.
+ let entity_backfill = ops::backfill_entity_index(&mut conn)?;
+ if entity_backfill.links_added > 0 {
+ tracing::info!(
+ pages = entity_backfill.pages_backfilled,
+ links = entity_backfill.links_added,
+ "entity index backfilled from existing frontmatter"
+ );
+ }
+
let writer = WriterHandle::spawn(conn);
let reader = ReaderPool::new(&db_path, READER_POOL_SOFT_CAP)?;
Ok(Self {
@@ -6404,6 +6422,68 @@ mod tests {
assert!(before.is_empty(), "{before:?}");
}
+ /// End to end: a page written with only `tags` (no explicit
+ /// `entities`) — the shape of essentially every page on a mature
+ /// store — has no entity index until the one-shot startup backfill
+ /// runs, after which both the entity retrieval stream and `as_of`
+ /// find it. This is the fix that made `as_of` do anything on a real
+ /// deployment (docs/temporal.md).
+ #[tokio::test]
+ async fn startup_backfill_lets_as_of_find_tag_only_pages() {
+ let tmp = TempDir::new().unwrap();
+ let created;
+ let (ws, proj, page_id) = {
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "temporal", None)
+ .await
+ .unwrap();
+
+ let mut p = sample_page(ws, proj, "concepts/writer.md", "the writer actor");
+ p.frontmatter_json = serde_json::json!({"tags": ["SQLite"]});
+ p.entities = Vec::new();
+ let id = store.writer.upsert_page(p).await.unwrap();
+
+ // Nothing in the entity index yet — the pre-fix state.
+ let none = store
+ .reader
+ .entity_hits_for_project(ws, proj, "sqlite", 10, None)
+ .await
+ .unwrap();
+ assert!(none.is_empty(), "no entities before backfill: {none:?}");
+ created = jiff::Timestamp::now().as_microsecond();
+ (ws, proj, id)
+ // `store` drops here: the writer thread joins and the WAL flushes.
+ };
+
+ // Reopen: `Store::open` runs the one-shot entity backfill.
+ let store = Store::open(tmp.path()).unwrap();
+
+ let now = store
+ .reader
+ .entity_hits_for_project(ws, proj, "sqlite", 10, None)
+ .await
+ .unwrap();
+ assert_eq!(now.len(), 1, "backfill populated the entity index: {now:?}");
+ assert_eq!(now[0].hit.id, page_id);
+
+ // `as_of` an instant after the page was written finds it too — the
+ // window opened at the version's creation.
+ let later = store
+ .reader
+ .entity_hits_for_project_at(ws, proj, "sqlite", 10, None, Some(created + 1))
+ .await
+ .unwrap();
+ assert_eq!(later.len(), 1, "{later:?}");
+ assert_eq!(later[0].hit.id, page_id);
+ }
+
/// Post-audit regression: retirement WITHOUT a successor (a decay
/// tombstone) closes the entity-link window too — `as_of` instants
/// after retirement must not resurrect the page.
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index 65a6c07c..36c19413 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -708,6 +708,98 @@ fn stamped_frontmatter(mut conformed: serde_json::Value, now_us: i64) -> StoreRe
Ok(serde_json::to_string(&conformed)?)
}
+/// What a [`backfill_entity_index`] pass did.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+pub struct EntityBackfillSummary {
+ /// Latest pages that gained at least one entity link.
+ pub pages_backfilled: u64,
+ /// Entity→page links inserted.
+ pub links_added: u64,
+}
+
+/// One-shot, idempotent backfill of the entity index from existing
+/// frontmatter (docs/temporal.md).
+///
+/// The entity retrieval stream and every `as_of` timeline read from
+/// `entities` / `entity_page_links`, which are populated only when a page
+/// is written with entities. Historically those came from an LLM
+/// consolidator's `entities:` field, absent on the bulk of a real store —
+/// bootstrapped pages predate it and stable pages are never
+/// re-consolidated — so both features sat empty on mature deployments.
+///
+/// This derives each latest page's entities from its `tags`/`entities`
+/// frontmatter (the exact rule the write path now uses,
+/// [`ai_memory_core::frontmatter_entity_names`]) and attaches the links
+/// the page version never got, with `valid_from` = the version's
+/// `created_at` — the same window semantics the V56 migration used, and
+/// honest because those tags have lived in the frontmatter since that
+/// version was written.
+///
+/// Only latest pages that currently have NO entity links are considered,
+/// so a store whose pages were written through the fixed path is a cheap
+/// no-op and a re-run inserts nothing (the link insert is also
+/// `ON CONFLICT DO NOTHING`). Superseded versions are left untouched.
+pub fn backfill_entity_index(conn: &mut Connection) -> StoreResult<EntityBackfillSummary> {
+ type CandidateRow = (Vec<u8>, Vec<u8>, Vec<u8>, String, i64);
+ let tx = conn.transaction()?;
+ let mut summary = EntityBackfillSummary::default();
+ {
+ let rows: Vec<CandidateRow> = {
+ let mut stmt = tx.prepare(
+ "SELECT p.id, p.workspace_id, p.project_id, p.frontmatter_json, p.created_at \
+ FROM pages p \
+ WHERE p.is_latest = 1 \
+ AND NOT EXISTS ( \
+ SELECT 1 FROM entity_page_links l WHERE l.page_id = p.id \
+ )",
+ )?;
+ stmt.query_map([], |row| {
+ Ok((
+ row.get(0)?,
+ row.get(1)?,
+ row.get(2)?,
+ row.get(3)?,
+ row.get(4)?,
+ ))
+ })?
+ .collect::<Result<_, _>>()?
+ };
+ for (page_id, ws, proj, fm_str, created_at) in rows {
+ let fm: serde_json::Value = serde_json::from_str(&fm_str).unwrap_or_default();
+ let names = ai_memory_core::frontmatter_entity_names(&fm);
+ if names.is_empty() {
+ continue;
+ }
+ let mut page_gained_a_link = false;
+ for name in names {
+ let entity_id: Vec<u8> = tx.query_row(
+ "INSERT INTO entities (id, workspace_id, project_id, name, created_at) \
+ VALUES (?1, ?2, ?3, ?4, ?5) \
+ ON CONFLICT (workspace_id, project_id, name) DO UPDATE SET name = excluded.name \
+ RETURNING id",
+ params![EntityId::new().as_bytes(), &ws, &proj, name, created_at],
+ |row| row.get(0),
+ )?;
+ let inserted = tx.execute(
+ "INSERT INTO entity_page_links (entity_id, page_id, valid_from) \
+ VALUES (?1, ?2, ?3) \
+ ON CONFLICT (entity_id, page_id) DO NOTHING",
+ params![entity_id, &page_id, created_at],
+ )?;
+ if inserted > 0 {
+ summary.links_added += 1;
+ page_gained_a_link = true;
+ }
+ }
+ if page_gained_a_link {
+ summary.pages_backfilled += 1;
+ }
+ }
+ }
+ tx.commit()?;
+ Ok(summary)
+}
+
pub(crate) fn upsert_page_in_tx(
tx: &rusqlite::Transaction<'_>,
page: &NewPage,
@@ -5847,6 +5939,82 @@ pub(crate) mod tests {
}
}
+ /// A page written before entity extraction — tags in frontmatter but
+ /// no entity links — is backfilled from those tags, with the link's
+ /// window opening at the page version's creation. A page written with
+ /// entities is left alone, and a re-run is a no-op.
+ #[test]
+ fn backfill_entity_index_populates_from_tags_idempotently() {
+ let (_tmp, mut conn, ws, proj) = fresh_db();
+
+ // Pre-fix page: tags present, but the write set no entities, so it
+ // has no entity links — exactly what a mature store looks like.
+ let mut stale = page(ws, proj, "concepts/writer.md", "the writer actor");
+ stale.frontmatter_json = serde_json::json!({"tags": ["SQLite", "Concurrency"]});
+ stale.entities = Vec::new();
+ let stale_id = upsert_page(&mut conn, &stale).unwrap();
+ let stale_created: i64 = conn
+ .query_row(
+ "SELECT created_at FROM pages WHERE id = ?1",
+ params![stale_id.as_bytes()],
+ |r| r.get(0),
+ )
+ .unwrap();
+
+ // Already-indexed page: written with entities through the normal
+ // path. The backfill must not touch it.
+ let mut indexed = page(ws, proj, "concepts/search.md", "fts and vectors");
+ indexed.entities = vec!["fts5".into()];
+ upsert_page(&mut conn, &indexed).unwrap();
+ let links_before: i64 = conn
+ .query_row("SELECT COUNT(*) FROM entity_page_links", [], |r| r.get(0))
+ .unwrap();
+
+ let summary = backfill_entity_index(&mut conn).unwrap();
+ assert_eq!(summary.pages_backfilled, 1, "only the tag-only page");
+ assert_eq!(summary.links_added, 2, "sqlite + concurrency");
+
+ // Entities exist, normalized, and the links carry the page
+ // version's own creation instant as the window open (docs/temporal).
+ let names: Vec<String> = {
+ let mut stmt = conn
+ .prepare(
+ "SELECT e.name FROM entities e \
+ JOIN entity_page_links l ON l.entity_id = e.id \
+ WHERE l.page_id = ?1 ORDER BY e.name",
+ )
+ .unwrap();
+ let rows = stmt
+ .query_map(params![stale_id.as_bytes()], |r| r.get(0))
+ .unwrap();
+ rows.collect::<Result<_, _>>().unwrap()
+ };
+ assert_eq!(names, vec!["concurrency".to_string(), "sqlite".to_string()]);
+ let valid_from: i64 = conn
+ .query_row(
+ "SELECT DISTINCT valid_from FROM entity_page_links WHERE page_id = ?1",
+ params![stale_id.as_bytes()],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(
+ valid_from, stale_created,
+ "window opens at the version's created_at"
+ );
+
+ // Idempotent: a second pass finds no candidates and inserts nothing.
+ let again = backfill_entity_index(&mut conn).unwrap();
+ assert_eq!(again, EntityBackfillSummary::default(), "re-run is a no-op");
+ let links_after: i64 = conn
+ .query_row("SELECT COUNT(*) FROM entity_page_links", [], |r| r.get(0))
+ .unwrap();
+ assert_eq!(
+ links_after,
+ links_before + 2,
+ "the pre-existing entity page kept its single link; only the stale page gained two"
+ );
+ }
+
#[test]
fn page_feedback_is_scoped_rebuildable_and_audited() {
let (_tmp, mut conn, ws, proj) = fresh_db();
diff --git a/crates/ai-memory-wiki/src/wiki.rs b/crates/ai-memory-wiki/src/wiki.rs
index 7d8096c6..b9841c76 100644
--- a/crates/ai-memory-wiki/src/wiki.rs
+++ b/crates/ai-memory-wiki/src/wiki.rs
@@ -1980,30 +1980,48 @@ struct IndexMetadata {
entities: Vec<String>,
}
-/// Parse the optional frontmatter `entities:` list — salient nouns the
-/// consolidator extracted, indexed by the store as a retrieval stream.
+/// Derive a page's indexed entities — the salient nouns it is *about* —
+/// from its frontmatter, for the entity retrieval stream and the `as_of`
+/// entity-timeline queries (docs/temporal.md).
///
-/// Unlike `expires_at`, malformed entries are *dropped* rather than
-/// rejected: entities are a soft ranking signal, and refusing a whole
-/// page write because one hand-edited entity is 80 characters long
-/// would trade a real page for a marginal signal. A non-array value is
-/// still an error — that's a structural mistake, not a bad item.
+/// Two sources, merged and normalised:
+///
+/// 1. The explicit `entities:` list an LLM consolidator emits when it
+/// runs. Highest precision, but absent on the bulk of a real store —
+/// bootstrapped pages predate it and stable pages are never
+/// re-consolidated, so relying on it alone leaves the entity index
+/// (and therefore `as_of`) empty on any mature deployment.
+/// 2. The frontmatter `tags:` list, which nearly every page carries and
+/// which is, by construction, "what this page is about" — the same
+/// thing entities are meant to capture. Deriving from tags populates
+/// the index deterministically, with no LLM and no re-consolidation,
+/// for the whole corpus. Broad tags do not distort ranking: the
+/// entity stream weights each match by inverse page-frequency, so a
+/// tag shared by many pages contributes proportionally little.
+///
+/// Explicit entities come first so they win the per-page cap
+/// ([`normalize_entities`] dedupes and bounds the result). A non-array
+/// `entities` value is still a structural error; `tags` is treated
+/// leniently (a malformed or missing list simply contributes nothing),
+/// because a bad tag must never fail a page write.
pub(crate) fn parse_entities(
path: &PagePath,
frontmatter: &serde_json::Value,
) -> WikiResult<Vec<String>> {
- let raw = match frontmatter.get("entities") {
- None | Some(serde_json::Value::Null) => return Ok(Vec::new()),
- Some(serde_json::Value::Array(items)) => items,
+ // Strict structural check on the write path only: a non-array
+ // `entities` value is a mistake worth surfacing. `tags` stays lenient
+ // (see below); the derivation itself is shared with the store's
+ // one-shot backfill so the two never drift.
+ match frontmatter.get("entities") {
+ None | Some(serde_json::Value::Null | serde_json::Value::Array(_)) => {}
Some(_) => {
return Err(ai_memory_wiki_error(&format!(
"invalid non-array entities in frontmatter for {}",
path.as_str()
)));
}
- };
- let strings = raw.iter().filter_map(|v| v.as_str());
- Ok(ai_memory_core::normalize_entities(strings))
+ }
+ Ok(ai_memory_core::frontmatter_entity_names(frontmatter))
}
/// Parse the optional frontmatter `expires_at:` key. Accepts RFC3339
@@ -4452,6 +4470,44 @@ mod tests {
assert!(err.to_string().contains("non-array entities"), "{err}");
}
+ /// Entities are derived from `tags` too, so the index (and `as_of`)
+ /// populate on real stores whose pages predate LLM entity extraction.
+ #[test]
+ fn parse_entities_derives_from_tags() {
+ let path = PagePath::new("concepts/x.md").unwrap();
+
+ // Tags alone populate entities — the common case on a mature store.
+ assert_eq!(
+ parse_entities(&path, &serde_json::json!({"tags": ["Storage", "SQLite"]})).unwrap(),
+ vec!["storage".to_string(), "sqlite".to_string()],
+ "tags become entities, normalised",
+ );
+
+ // Explicit entities come first (they win the cap), tags fill in,
+ // and a tag duplicating an entity is de-duplicated.
+ assert_eq!(
+ parse_entities(
+ &path,
+ &serde_json::json!({"entities": ["FTS5"], "tags": ["fts5", "search"]}),
+ )
+ .unwrap(),
+ vec!["fts5".to_string(), "search".to_string()],
+ "explicit first, tags merged, duplicates dropped",
+ );
+
+ // A malformed tags list is lenient — it must never fail a write —
+ // while a malformed entities list still errors.
+ assert_eq!(
+ parse_entities(
+ &path,
+ &serde_json::json!({"entities": ["ok"], "tags": "not-a-list"}),
+ )
+ .unwrap(),
+ vec!["ok".to_string()],
+ "non-array tags contribute nothing rather than erroring",
+ );
+ }
+
/// Two projects in one workspace, one ended session in the first with a
/// consolidated `sessions/<id>.md` page written through the wiki.
async fn session_with_page(
diff --git a/docs/temporal.md b/docs/temporal.md
index e2c12053..51c5e386 100644
--- a/docs/temporal.md
+++ b/docs/temporal.md
@@ -73,3 +73,28 @@ time. FTS / vector / graph streams are deliberately skipped — mixing
"current text relevance" with "historical entity validity" in one
ranked list would answer neither question honestly. Omit `as_of` and
nothing changes.
+
+## Where entities come from
+
+`as_of` and the entity retrieval stream both read the entity index, so
+they only answer once pages actually carry entities. Entities are the
+salient nouns a page is *about*, and they are populated from each page's
+frontmatter at write time:
+
+- an explicit `entities:` list, when an LLM consolidator emits one; and
+- the page's `tags:` list, which nearly every page carries — tags *are*
+ "what this page is about", so they seed the entity index deterministically
+ with no LLM pass and no re-consolidation.
+
+Broad tags do not distort ranking: the entity stream weights each match
+by inverse page-frequency, so a tag shared by many pages contributes
+proportionally little.
+
+Existing stores self-heal. On the first start after upgrading, a
+one-shot, idempotent backfill scans latest pages that have no entity
+links yet, derives their entities from the same frontmatter, and opens
+each link's validity window at the page version's own `created_at` — so
+`as_of` works historically, not just from the upgrade forward. A store
+whose pages were all written through the current path finds no
+candidates and the pass is a no-op. No command is required; a manual
+`reindex` is only needed if you deliberately cleared the index.