Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(entities): populate the entity index from tags so as_of works
AkitaOnRails38ded9a58413
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1d12d6fc..fa882713 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Fixed+- `as_of` time-travel queries and the entity retrieval stream now work+ on real stores. Both read the entity index, which was populated only+ from an LLM consolidator's `entities:` frontmatter — absent on the+ bulk of a mature store (bootstrapped pages predate it; stable pages+ are never re-consolidated), so the index sat empty and `as_of`+ returned nothing. Entities are now also derived from each page's+ `tags:` (the nouns a page is already labelled with) at write time, and+ a one-shot idempotent backfill runs on the first start after upgrading+ to populate the index for existing pages from their frontmatter —+ opening each entity-link window at the page version's own `created_at`+ so historical `as_of` is correct. Broad tags don't distort ranking+ (the entity stream is inverse-frequency weighted), and a store already+ written through the current path is a no-op. See `docs/temporal.md`.- The Windows PowerShell wrapper (`ai-memory.ps1`) no longer crashes with`git.exe : fatal: not a git repository … NativeCommandError` when runoutside a git repository — which broke `ai-memory status` and everydiff --git a/crates/ai-memory-core/src/lib.rs b/crates/ai-memory-core/src/lib.rsindex 40fde265..f61af500 100644--- a/crates/ai-memory-core/src/lib.rs+++ b/crates/ai-memory-core/src/lib.rs@@ -62,7 +62,7 @@ pub use ids::{pub use observation::{NewObservation, NewSession, Observation, ObservationKind};pub use page::{FeedbackKind, LinkTarget, MAX_ENTITIES_PER_PAGE, MAX_ENTITY_LEN, NewPage, Page, Relation, Tier,- normalize_entities, normalize_entity,+ frontmatter_entity_names, normalize_entities, normalize_entity,};pub use routing_snippet::{MARKER_END, MARKER_START, SNIPPET_BODY, find_marker_line, full_block};pub use sanitize::{diff --git a/crates/ai-memory-core/src/page.rs b/crates/ai-memory-core/src/page.rsindex ded6fe04..d31c652b 100644--- a/crates/ai-memory-core/src/page.rs+++ b/crates/ai-memory-core/src/page.rs@@ -151,6 +151,33 @@ whereout}+/// Derive a page's indexed entity names from its frontmatter: the+/// explicit `entities` list an LLM consolidator may emit, plus the+/// `tags` list nearly every page carries. Both are arrays of strings+/// meaning "what this page is about"; merging them populates the entity+/// retrieval stream and `as_of` timelines deterministically, without+/// depending on an LLM pass that a mature store's pages never received.+///+/// Explicit entities come first so they win the per-page cap. Lenient by+/// design — a missing or non-array field simply contributes nothing, so+/// this never fails on hand-edited frontmatter; the strict structural+/// check for the write path lives in the wiki layer.+#[must_use]+pub fn frontmatter_entity_names(frontmatter: &serde_json::Value) -> Vec<String> {+ fn strings(frontmatter: &serde_json::Value, key: &str) -> Vec<String> {+ frontmatter+ .get(key)+ .and_then(serde_json::Value::as_array)+ .into_iter()+ .flatten()+ .filter_map(|v| v.as_str().map(str::to_owned))+ .collect()+ }+ let mut merged = strings(frontmatter, "entities");+ merged.extend(strings(frontmatter, "tags"));+ normalize_entities(merged)+}+/// A link target discovered in a page body.////// A bare `[[path]]` / `[label](path)` resolves within the source page's@@ -439,4 +466,27 @@ mod tests {.collect();assert_eq!(normalize_entities(&many).len(), MAX_ENTITIES_PER_PAGE);}++ #[test]+ fn frontmatter_entity_names_merges_entities_and_tags_leniently() {+ // Explicit entities first (they win the cap), then tags; dupes drop.+ assert_eq!(+ frontmatter_entity_names(&serde_json::json!({+ "entities": ["FTS5"],+ "tags": ["fts5", "Search"],+ })),+ vec!["fts5".to_string(), "search".to_string()],+ );+ // Tags alone are enough — the common case on a mature store.+ assert_eq!(+ frontmatter_entity_names(&serde_json::json!({"tags": ["Storage"]})),+ vec!["storage".to_string()],+ );+ // Missing or malformed fields contribute nothing, never panic.+ assert!(frontmatter_entity_names(&serde_json::json!({})).is_empty());+ assert!(+ frontmatter_entity_names(&serde_json::json!({"tags": "not-a-list", "entities": 7}))+ .is_empty(),+ );+ }}diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex 31a90c0f..cfdb0b5c 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -50,9 +50,10 @@ pub use error::{StoreError, StoreResult};pub use maintenance::MaintenanceJob;pub use ops::{AdmittedSession, CompactSummary, Compaction, DeleteWorkspaceSummary, EmbedOutcome,- EmbeddingWrite, HookSessionAdmission, IngestObservationOutcome, LifecycleOnlyEndOutcome,- MoveSessionSummary, MoveSummary, ObservationPruneOutcome, OkfMigratedPage, PagesMode,- PurgeSessionSummary, PurgeSummary, ReorgSummary, purge_session, record_embed_failure,+ EmbeddingWrite, EntityBackfillSummary, HookSessionAdmission, IngestObservationOutcome,+ LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary, ObservationPruneOutcome,+ OkfMigratedPage, PagesMode, PurgeSessionSummary, PurgeSummary, ReorgSummary,+ backfill_entity_index, purge_session, record_embed_failure,};pub use reader::{ActivityWindow, AgentSessionCount, AuditEvent, AuditLogFilter, AutoImproveCandidateSession,@@ -132,6 +133,23 @@ impl Store {migrations::run(&mut conn)?;conn.pragma_update(None, "foreign_keys", "ON")?;+ // One-shot, idempotent entity-index backfill (docs/temporal.md).+ // Populates entities/entity_page_links from existing frontmatter+ // for pages that predate entity extraction, so the entity+ // retrieval stream and `as_of` timelines work on an upgraded+ // store without a manual reindex. A store already populated+ // through the write path finds no candidate pages and returns+ // immediately. Runs single-threaded here, before the writer actor+ // spawns, so it completes before the server accepts traffic.+ let entity_backfill = ops::backfill_entity_index(&mut conn)?;+ if entity_backfill.links_added > 0 {+ tracing::info!(+ pages = entity_backfill.pages_backfilled,+ links = entity_backfill.links_added,+ "entity index backfilled from existing frontmatter"+ );+ }+let writer = WriterHandle::spawn(conn);let reader = ReaderPool::new(&db_path, READER_POOL_SOFT_CAP)?;Ok(Self {@@ -6404,6 +6422,68 @@ mod tests {assert!(before.is_empty(), "{before:?}");}+ /// End to end: a page written with only `tags` (no explicit+ /// `entities`) — the shape of essentially every page on a mature+ /// store — has no entity index until the one-shot startup backfill+ /// runs, after which both the entity retrieval stream and `as_of`+ /// find it. This is the fix that made `as_of` do anything on a real+ /// deployment (docs/temporal.md).+ #[tokio::test]+ async fn startup_backfill_lets_as_of_find_tag_only_pages() {+ let tmp = TempDir::new().unwrap();+ let created;+ let (ws, proj, page_id) = {+ let store = Store::open(tmp.path()).unwrap();+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "temporal", None)+ .await+ .unwrap();++ let mut p = sample_page(ws, proj, "concepts/writer.md", "the writer actor");+ p.frontmatter_json = serde_json::json!({"tags": ["SQLite"]});+ p.entities = Vec::new();+ let id = store.writer.upsert_page(p).await.unwrap();++ // Nothing in the entity index yet — the pre-fix state.+ let none = store+ .reader+ .entity_hits_for_project(ws, proj, "sqlite", 10, None)+ .await+ .unwrap();+ assert!(none.is_empty(), "no entities before backfill: {none:?}");+ created = jiff::Timestamp::now().as_microsecond();+ (ws, proj, id)+ // `store` drops here: the writer thread joins and the WAL flushes.+ };++ // Reopen: `Store::open` runs the one-shot entity backfill.+ let store = Store::open(tmp.path()).unwrap();++ let now = store+ .reader+ .entity_hits_for_project(ws, proj, "sqlite", 10, None)+ .await+ .unwrap();+ assert_eq!(now.len(), 1, "backfill populated the entity index: {now:?}");+ assert_eq!(now[0].hit.id, page_id);++ // `as_of` an instant after the page was written finds it too — the+ // window opened at the version's creation.+ let later = store+ .reader+ .entity_hits_for_project_at(ws, proj, "sqlite", 10, None, Some(created + 1))+ .await+ .unwrap();+ assert_eq!(later.len(), 1, "{later:?}");+ assert_eq!(later[0].hit.id, page_id);+ }+/// Post-audit regression: retirement WITHOUT a successor (a decay/// tombstone) closes the entity-link window too — `as_of` instants/// after retirement must not resurrect the page.diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex 65a6c07c..36c19413 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -708,6 +708,98 @@ fn stamped_frontmatter(mut conformed: serde_json::Value, now_us: i64) -> StoreReOk(serde_json::to_string(&conformed)?)}+/// What a [`backfill_entity_index`] pass did.+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]+pub struct EntityBackfillSummary {+ /// Latest pages that gained at least one entity link.+ pub pages_backfilled: u64,+ /// Entity→page links inserted.+ pub links_added: u64,+}++/// One-shot, idempotent backfill of the entity index from existing+/// frontmatter (docs/temporal.md).+///+/// The entity retrieval stream and every `as_of` timeline read from+/// `entities` / `entity_page_links`, which are populated only when a page+/// is written with entities. Historically those came from an LLM+/// consolidator's `entities:` field, absent on the bulk of a real store —+/// bootstrapped pages predate it and stable pages are never+/// re-consolidated — so both features sat empty on mature deployments.+///+/// This derives each latest page's entities from its `tags`/`entities`+/// frontmatter (the exact rule the write path now uses,+/// [`ai_memory_core::frontmatter_entity_names`]) and attaches the links+/// the page version never got, with `valid_from` = the version's+/// `created_at` — the same window semantics the V56 migration used, and+/// honest because those tags have lived in the frontmatter since that+/// version was written.+///+/// Only latest pages that currently have NO entity links are considered,+/// so a store whose pages were written through the fixed path is a cheap+/// no-op and a re-run inserts nothing (the link insert is also+/// `ON CONFLICT DO NOTHING`). Superseded versions are left untouched.+pub fn backfill_entity_index(conn: &mut Connection) -> StoreResult<EntityBackfillSummary> {+ type CandidateRow = (Vec<u8>, Vec<u8>, Vec<u8>, String, i64);+ let tx = conn.transaction()?;+ let mut summary = EntityBackfillSummary::default();+ {+ let rows: Vec<CandidateRow> = {+ let mut stmt = tx.prepare(+ "SELECT p.id, p.workspace_id, p.project_id, p.frontmatter_json, p.created_at \+ FROM pages p \+ WHERE p.is_latest = 1 \+ AND NOT EXISTS ( \+ SELECT 1 FROM entity_page_links l WHERE l.page_id = p.id \+ )",+ )?;+ stmt.query_map([], |row| {+ Ok((+ row.get(0)?,+ row.get(1)?,+ row.get(2)?,+ row.get(3)?,+ row.get(4)?,+ ))+ })?+ .collect::<Result<_, _>>()?+ };+ for (page_id, ws, proj, fm_str, created_at) in rows {+ let fm: serde_json::Value = serde_json::from_str(&fm_str).unwrap_or_default();+ let names = ai_memory_core::frontmatter_entity_names(&fm);+ if names.is_empty() {+ continue;+ }+ let mut page_gained_a_link = false;+ for name in names {+ let entity_id: Vec<u8> = tx.query_row(+ "INSERT INTO entities (id, workspace_id, project_id, name, created_at) \+ VALUES (?1, ?2, ?3, ?4, ?5) \+ ON CONFLICT (workspace_id, project_id, name) DO UPDATE SET name = excluded.name \+ RETURNING id",+ params![EntityId::new().as_bytes(), &ws, &proj, name, created_at],+ |row| row.get(0),+ )?;+ let inserted = tx.execute(+ "INSERT INTO entity_page_links (entity_id, page_id, valid_from) \+ VALUES (?1, ?2, ?3) \+ ON CONFLICT (entity_id, page_id) DO NOTHING",+ params![entity_id, &page_id, created_at],+ )?;+ if inserted > 0 {+ summary.links_added += 1;+ page_gained_a_link = true;+ }+ }+ if page_gained_a_link {+ summary.pages_backfilled += 1;+ }+ }+ }+ tx.commit()?;+ Ok(summary)+}+pub(crate) fn upsert_page_in_tx(tx: &rusqlite::Transaction<'_>,page: &NewPage,@@ -5847,6 +5939,82 @@ pub(crate) mod tests {}}+ /// A page written before entity extraction — tags in frontmatter but+ /// no entity links — is backfilled from those tags, with the link's+ /// window opening at the page version's creation. A page written with+ /// entities is left alone, and a re-run is a no-op.+ #[test]+ fn backfill_entity_index_populates_from_tags_idempotently() {+ let (_tmp, mut conn, ws, proj) = fresh_db();++ // Pre-fix page: tags present, but the write set no entities, so it+ // has no entity links — exactly what a mature store looks like.+ let mut stale = page(ws, proj, "concepts/writer.md", "the writer actor");+ stale.frontmatter_json = serde_json::json!({"tags": ["SQLite", "Concurrency"]});+ stale.entities = Vec::new();+ let stale_id = upsert_page(&mut conn, &stale).unwrap();+ let stale_created: i64 = conn+ .query_row(+ "SELECT created_at FROM pages WHERE id = ?1",+ params![stale_id.as_bytes()],+ |r| r.get(0),+ )+ .unwrap();++ // Already-indexed page: written with entities through the normal+ // path. The backfill must not touch it.+ let mut indexed = page(ws, proj, "concepts/search.md", "fts and vectors");+ indexed.entities = vec!["fts5".into()];+ upsert_page(&mut conn, &indexed).unwrap();+ let links_before: i64 = conn+ .query_row("SELECT COUNT(*) FROM entity_page_links", [], |r| r.get(0))+ .unwrap();++ let summary = backfill_entity_index(&mut conn).unwrap();+ assert_eq!(summary.pages_backfilled, 1, "only the tag-only page");+ assert_eq!(summary.links_added, 2, "sqlite + concurrency");++ // Entities exist, normalized, and the links carry the page+ // version's own creation instant as the window open (docs/temporal).+ let names: Vec<String> = {+ let mut stmt = conn+ .prepare(+ "SELECT e.name FROM entities e \+ JOIN entity_page_links l ON l.entity_id = e.id \+ WHERE l.page_id = ?1 ORDER BY e.name",+ )+ .unwrap();+ let rows = stmt+ .query_map(params![stale_id.as_bytes()], |r| r.get(0))+ .unwrap();+ rows.collect::<Result<_, _>>().unwrap()+ };+ assert_eq!(names, vec!["concurrency".to_string(), "sqlite".to_string()]);+ let valid_from: i64 = conn+ .query_row(+ "SELECT DISTINCT valid_from FROM entity_page_links WHERE page_id = ?1",+ params![stale_id.as_bytes()],+ |r| r.get(0),+ )+ .unwrap();+ assert_eq!(+ valid_from, stale_created,+ "window opens at the version's created_at"+ );++ // Idempotent: a second pass finds no candidates and inserts nothing.+ let again = backfill_entity_index(&mut conn).unwrap();+ assert_eq!(again, EntityBackfillSummary::default(), "re-run is a no-op");+ let links_after: i64 = conn+ .query_row("SELECT COUNT(*) FROM entity_page_links", [], |r| r.get(0))+ .unwrap();+ assert_eq!(+ links_after,+ links_before + 2,+ "the pre-existing entity page kept its single link; only the stale page gained two"+ );+ }+#[test]fn page_feedback_is_scoped_rebuildable_and_audited() {let (_tmp, mut conn, ws, proj) = fresh_db();diff --git a/crates/ai-memory-wiki/src/wiki.rs b/crates/ai-memory-wiki/src/wiki.rsindex 7d8096c6..b9841c76 100644--- a/crates/ai-memory-wiki/src/wiki.rs+++ b/crates/ai-memory-wiki/src/wiki.rs@@ -1980,30 +1980,48 @@ struct IndexMetadata {entities: Vec<String>,}-/// Parse the optional frontmatter `entities:` list — salient nouns the-/// consolidator extracted, indexed by the store as a retrieval stream.+/// Derive a page's indexed entities — the salient nouns it is *about* —+/// from its frontmatter, for the entity retrieval stream and the `as_of`+/// entity-timeline queries (docs/temporal.md).///-/// Unlike `expires_at`, malformed entries are *dropped* rather than-/// rejected: entities are a soft ranking signal, and refusing a whole-/// page write because one hand-edited entity is 80 characters long-/// would trade a real page for a marginal signal. A non-array value is-/// still an error — that's a structural mistake, not a bad item.+/// Two sources, merged and normalised:+///+/// 1. The explicit `entities:` list an LLM consolidator emits when it+/// runs. Highest precision, but absent on the bulk of a real store —+/// bootstrapped pages predate it and stable pages are never+/// re-consolidated, so relying on it alone leaves the entity index+/// (and therefore `as_of`) empty on any mature deployment.+/// 2. The frontmatter `tags:` list, which nearly every page carries and+/// which is, by construction, "what this page is about" — the same+/// thing entities are meant to capture. Deriving from tags populates+/// the index deterministically, with no LLM and no re-consolidation,+/// for the whole corpus. Broad tags do not distort ranking: the+/// entity stream weights each match by inverse page-frequency, so a+/// tag shared by many pages contributes proportionally little.+///+/// Explicit entities come first so they win the per-page cap+/// ([`normalize_entities`] dedupes and bounds the result). A non-array+/// `entities` value is still a structural error; `tags` is treated+/// leniently (a malformed or missing list simply contributes nothing),+/// because a bad tag must never fail a page write.pub(crate) fn parse_entities(path: &PagePath,frontmatter: &serde_json::Value,) -> WikiResult<Vec<String>> {- let raw = match frontmatter.get("entities") {- None | Some(serde_json::Value::Null) => return Ok(Vec::new()),- Some(serde_json::Value::Array(items)) => items,+ // Strict structural check on the write path only: a non-array+ // `entities` value is a mistake worth surfacing. `tags` stays lenient+ // (see below); the derivation itself is shared with the store's+ // one-shot backfill so the two never drift.+ match frontmatter.get("entities") {+ None | Some(serde_json::Value::Null | serde_json::Value::Array(_)) => {}Some(_) => {return Err(ai_memory_wiki_error(&format!("invalid non-array entities in frontmatter for {}",path.as_str())));}- };- let strings = raw.iter().filter_map(|v| v.as_str());- Ok(ai_memory_core::normalize_entities(strings))+ }+ Ok(ai_memory_core::frontmatter_entity_names(frontmatter))}/// Parse the optional frontmatter `expires_at:` key. Accepts RFC3339@@ -4452,6 +4470,44 @@ mod tests {assert!(err.to_string().contains("non-array entities"), "{err}");}+ /// Entities are derived from `tags` too, so the index (and `as_of`)+ /// populate on real stores whose pages predate LLM entity extraction.+ #[test]+ fn parse_entities_derives_from_tags() {+ let path = PagePath::new("concepts/x.md").unwrap();++ // Tags alone populate entities — the common case on a mature store.+ assert_eq!(+ parse_entities(&path, &serde_json::json!({"tags": ["Storage", "SQLite"]})).unwrap(),+ vec!["storage".to_string(), "sqlite".to_string()],+ "tags become entities, normalised",+ );++ // Explicit entities come first (they win the cap), tags fill in,+ // and a tag duplicating an entity is de-duplicated.+ assert_eq!(+ parse_entities(+ &path,+ &serde_json::json!({"entities": ["FTS5"], "tags": ["fts5", "search"]}),+ )+ .unwrap(),+ vec!["fts5".to_string(), "search".to_string()],+ "explicit first, tags merged, duplicates dropped",+ );++ // A malformed tags list is lenient — it must never fail a write —+ // while a malformed entities list still errors.+ assert_eq!(+ parse_entities(+ &path,+ &serde_json::json!({"entities": ["ok"], "tags": "not-a-list"}),+ )+ .unwrap(),+ vec!["ok".to_string()],+ "non-array tags contribute nothing rather than erroring",+ );+ }+/// Two projects in one workspace, one ended session in the first with a/// consolidated `sessions/<id>.md` page written through the wiki.async fn session_with_page(diff --git a/docs/temporal.md b/docs/temporal.mdindex e2c12053..51c5e386 100644--- a/docs/temporal.md+++ b/docs/temporal.md@@ -73,3 +73,28 @@ time. FTS / vector / graph streams are deliberately skipped — mixing"current text relevance" with "historical entity validity" in oneranked list would answer neither question honestly. Omit `as_of` andnothing changes.++## Where entities come from++`as_of` and the entity retrieval stream both read the entity index, so+they only answer once pages actually carry entities. Entities are the+salient nouns a page is *about*, and they are populated from each page's+frontmatter at write time:++- an explicit `entities:` list, when an LLM consolidator emits one; and+- the page's `tags:` list, which nearly every page carries — tags *are*+ "what this page is about", so they seed the entity index deterministically+ with no LLM pass and no re-consolidation.++Broad tags do not distort ranking: the entity stream weights each match+by inverse page-frequency, so a tag shared by many pages contributes+proportionally little.++Existing stores self-heal. On the first start after upgrading, a+one-shot, idempotent backfill scans latest pages that have no entity+links yet, derives their entities from the same frontmatter, and opens+each link's validity window at the page version's own `created_at` — so+`as_of` works historically, not just from the upgrade forward. A store+whose pages were all written through the current path finds no+candidates and the pass is a no-op. No command is required; a manual+`reindex` is only needed if you deliberately cleared the index.