Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(wiki): reconcile can tombstone pages whose file disappeared (opt-in)
Maxsuel Einstein5a9ce4ac53df
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 54083f34..cd43917c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Added+- The watcher's reconcile pass can now optionally tombstone a wiki page whose+ file has disappeared from disk, closing part of the gap left by #929 (the+ watcher only reconciled create/modify events, so a deleted file stayed+ indexed until `ai-memory delete-page` ran explicitly). Opt-in and default+ `false` (`[maintenance] reconcile_tombstones_deleted_pages`), so existing+ installs see no behavior change unless they turn it on. When enabled, a+ page's file must be observed missing on two consecutive 30s reconcile+ passes, re-verified against the live filesystem immediately before acting;+ a circuit breaker refuses to act on a whole scope when more than `max(3,+ 50%)` of its candidate pages look missing in one pass, or when a+ non-partial walk finds nothing at all (either shape is far more likely a+ walk/mount problem than genuine mass deletion); and reserved/indexed-but-+ unwalked paths (`bootstrap.md`, `_meta.md`, `_pending/` sidecars, and+ `sessions/*.md` — excluded because a same-workspace `move-session` re-home+ can leave one with a correct row and no file, a separate pre-existing bug)+ are never candidates. The action is a soft tombstone (`is_latest = 0` ++ `superseded_at`, the same shape decay eviction already uses), picked up by+ the SAME aged-tombstone hard-delete sweep decay eviction uses — it is not+ exempt from it. What actually protects it: it is never destroyed while its+ chain has no successor; if the file returns, the new version re-links onto+ the tombstoned chain instead of starting a fresh, disconnected one, so+ nothing is left orphaned for that sweep to destroy. It never runs the+ blocking admission gate (nothing can refuse it), but does fire-and-forget+ any non-blocking observer/mirror webhook on success. `restore-page` and+ version history remain intact. See `docs/okf.md` for the full design.+ (#964)- `[consolidation] input_token_safety_margin` (float, default `0.8`, validatedto `(0.0, 1.0]`) scales the approximate char-count input budget. The`max_input_tokens` budget uses a flat chars-per-token heuristic thatdiff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex f219c45f..cad09650 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -1049,7 +1049,13 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {// Reader attached unconditionally: admission name-resolution uses it// when a chain is configured, and the startup scope-manifest backfill// (below) always needs it to enumerate scopes.- .with_store_reader(store.reader.clone());+ .with_store_reader(store.reader.clone())+ // `[maintenance] reconcile_tombstones_deleted_pages` (#929), read once+ // by `Config::load` above and threaded through here — the one place+ // this flag is consulted outside `Config`, per invariant #1.+ .with_reconcile_tombstones_deleted_pages(+ config.maintenance.reconcile_tombstones_deleted_pages,+ );// Attach the admission webhook chain (operator-configured via// `[[admission_webhooks]]` in config.toml or `AI_MEMORY_ADMISSION_WEBHOOKS__N__*`// env vars). Empty config = no chain attached, zero overhead. The store@@ -3611,6 +3617,7 @@ mod tests {forget_sweep_interval_secs: 1,lint_interval_secs: 1,embedding_backfill_interval_secs: 1,+ reconcile_tombstones_deleted_pages: false,},AutoImproveSettings::default(),store.reader.clone(),@@ -3649,12 +3656,14 @@ mod tests {forget_sweep_interval_secs: 0,lint_interval_secs: 60,embedding_backfill_interval_secs: 0,+ reconcile_tombstones_deleted_pages: false,},MaintenanceSettings {enabled: true,forget_sweep_interval_secs: 60,lint_interval_secs: 0,embedding_backfill_interval_secs: 0,+ reconcile_tombstones_deleted_pages: false,},] {let (_tmp, store, wiki, _ws, _first, _second) = two_project_wiki().await;diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex fe70c356..e0bb216e 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -1243,6 +1243,32 @@ pub struct MaintenanceSettings {/// Interval for embedding backfill. `0` disables this job./// Defaults to off because it may call a paid provider.pub embedding_backfill_interval_secs: u64,+ /// Opt-in reconcile-delete safety net (#929). When `true`, the watcher's+ /// 30s reconcile pass tombstones (`is_latest = 0` + `superseded_at`, never+ /// a filesystem touch or a BLOCKING admission dispatch) an OKF-imported+ /// content page (session summary pages are excluded — a same-workspace+ /// `move-session` re-home can leave one with a correct row and no file, a+ /// separate pre-existing bug) whose file has been missing on two+ /// consecutive passes, after a circuit breaker that refuses to act on a+ /// scope where more than `max(3, 50%)` of its candidate pages look+ /// missing at once, or where a non-partial walk finds nothing at all+ /// (see `ai_memory_wiki::watcher::reconcile_delete_breaker_threshold`) —+ /// either shape is far more likely a walk/mount problem (an unmounted+ /// volume, a git checkout mid-walk) than genuine deletions.+ ///+ /// The tombstone is NOT exempt from the aged-tombstone hard-delete sweep+ /// (`hard_delete_after_days`) — the actual guarantee is narrower: a+ /// reconcile tombstone is never itself destroyed while its chain has no+ /// successor. If the file returns, the new version re-links to the+ /// tombstoned chain (`ops::upsert_page_in_tx`'s resurrection path)+ /// instead of starting fresh, so nothing is orphaned for that sweep to+ /// destroy.+ ///+ /// Defaults to `false`: with this off, reconcile's behavior is+ /// byte-identical to before this feature existed — deletions still+ /// require `ai-memory delete-page`, and nothing new is logged. Doc:+ /// `docs/okf.md`, `docs/install.md`.+ pub reconcile_tombstones_deleted_pages: bool,}impl Default for MaintenanceSettings {@@ -1252,6 +1278,7 @@ impl Default for MaintenanceSettings {forget_sweep_interval_secs: 86_400,lint_interval_secs: 86_400,embedding_backfill_interval_secs: 0,+ reconcile_tombstones_deleted_pages: false,}}}diff --git a/crates/ai-memory-consolidate/src/sweep.rs b/crates/ai-memory-consolidate/src/sweep.rsindex acb4bdf3..52c4f149 100644--- a/crates/ai-memory-consolidate/src/sweep.rs+++ b/crates/ai-memory-consolidate/src/sweep.rs@@ -15,9 +15,16 @@//! contradiction is visible before the delete lands.//!//! Hard-delete pass cleans up tombstones older than-//! `hard_delete_after_days` and their supersession ancestry. A page recreated-//! at the same path is a separate live chain and is preserved. Ordinary-//! supersession rows are safe because only decay writes `superseded_at`.+//! `hard_delete_after_days` and their supersession ancestry. Ordinary+//! supersession rows are safe because only a tombstone (decay eviction, or+//! the watcher's opt-in reconcile-delete safety net, #929 — the two are+//! deliberately indistinguishable here) writes `superseded_at`. A page+//! rewritten at a tombstoned path re-links onto that chain via `supersedes`+//! and clears the old row's `superseded_at` (`ops::upsert_page_in_tx`'s+//! resurrection path), so the old version becomes an ordinary, protected+//! chain member instead of an orphan this pass would otherwise hard-delete —+//! this is what keeps a reconcile tombstone's false positive from being+//! permanently destroyed once the file reappears.//!//! Observation prune pass (opt-in, disabled unless//! `ObservationRetention::days > 0`) deletes raw observations older than thediff --git a/crates/ai-memory-consolidate/tests/suite/lifecycle.rs b/crates/ai-memory-consolidate/tests/suite/lifecycle.rsindex a46f7fed..30b59a6d 100644--- a/crates/ai-memory-consolidate/tests/suite/lifecycle.rs+++ b/crates/ai-memory-consolidate/tests/suite/lifecycle.rs@@ -1047,7 +1047,18 @@ async fn hard_delete_preserves_a_page_recreated_at_the_same_path() {.await.unwrap();assert!(cleanup.evicted.is_empty());- assert_eq!(cleanup.hard_deleted, 2);+ // #929 (B1 review fix): `reindex_page`'s upsert now resurrects a tombstoned+ // chain instead of starting a disconnected one — see+ // `ops::upsert_page_in_tx`'s resurrection path and+ // `ops::reconcile_tombstone_resurrects_instead_of_orphaning_on_recreate`.+ // The recreated file re-links onto `second` via `supersedes` and clears+ // `second`'s `superseded_at`, which is what actually protects the old+ // chain from `hard_delete_decayed_page_chain`: that function's root query+ // requires `superseded_at IS NOT NULL`, and a resurrected row no longer+ // has it set — exactly the same protection ordinary supersession-chain+ // members already had. So this cleanup pass no longer has any tombstone+ // root left to walk from, and hard-deletes nothing.+ assert_eq!(cleanup.hard_deleted, 0);assert!(wiki.abs_path(ws, proj, &path).exists());let recreated = store.reader@@ -1072,7 +1083,35 @@ async fn hard_delete_preserves_a_page_recreated_at_the_same_path() {|row| row.get(0),).unwrap();- assert_eq!(old_rows, 0);+ assert_eq!(+ old_rows, 2,+ "the old chain must survive, resurrected rather than deleted"+ );+ let supersedes: Option<Vec<u8>> = conn+ .query_row(+ "SELECT supersedes FROM pages WHERE id = ?1",+ params![recreated.as_bytes()],+ |row| row.get(0),+ )+ .unwrap();+ assert_eq!(+ supersedes.as_deref(),+ Some(&second.as_bytes()[..]),+ "the recreated version must supersede the tombstoned chain"+ );+ let second_superseded_at: Option<i64> = conn+ .query_row(+ "SELECT superseded_at FROM pages WHERE id = ?1",+ params![second.as_bytes()],+ |row| row.get(0),+ )+ .unwrap();+ assert_eq!(+ second_superseded_at, None,+ "resurrection must clear the old tombstone's superseded_at"+ );+ drop(conn);+assert!(store.reader@@ -1083,6 +1122,25 @@ async fn hard_delete_preserves_a_page_recreated_at_the_same_path() {.any(|hit| hit.id == recreated),"the recreated page and its FTS entry must survive old-chain cleanup");++ // A further cleanup pass must not find anything to hard-delete either:+ // the resurrected chain no longer has a tombstone root at all.+ let second_pass = run_sweep(+ &store.reader,+ &store.writer,+ Some(&wiki),+ ws,+ proj,+ &DecayParams {+ cold_threshold: 0.0,+ hard_delete_after_days: 0,+ ..DecayParams::default()+ },+ false,+ )+ .await+ .unwrap();+ assert_eq!(second_pass.hard_deleted, 0);}fn ws_dir_for(tmp: &TempDir, ws: WorkspaceId, proj: ProjectId) -> std::path::PathBuf {diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex 4f1d3195..d223eee9 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -1463,13 +1463,48 @@ pub(crate) fn upsert_page_in_tx(existing,});}+ // Tombstone resurrection (#929). A tombstone is any `is_latest = 0` row+ // with `superseded_at` set — decay eviction and the reconcile-delete+ // safety net both mark one this way, and they are deliberately+ // indistinguishable here (docs/okf.md): both are "the store believes+ // this path has no file", and both deserve the same resurrection+ // behavior if a fresh write proves that belief wrong.+ //+ // Without this, a fresh write at a tombstoned path started a brand-new,+ // disconnected chain (`supersedes = NULL`), leaving the OLD chain an+ // unreachable orphan. For a reconcile tombstone that "no chain has a+ // successor" is exactly what made it eligible for+ // `hard_delete_decayed_page_chain`'s recursive walk — so a false+ // positive that self-healed (the file came back) still got permanently+ // hard-deleted ~`hard_delete_after_days` later, contradicting the+ // safety net's own "never destroys anything" premise.+ //+ // The fix re-links the new version onto the tombstoned chain via+ // `supersedes`, exactly like an ordinary edit supersedes the page it+ // replaces, and clears the old row's `superseded_at`. That second step+ // is what actually protects it: `decay_tombstones_before` and+ // `hard_delete_decayed_page_chain` both key their eligibility on+ // `superseded_at IS NOT NULL` — an ordinary supersession-chain member+ // never sets it, which is the existing mechanism that keeps normal+ // history off the aged-tombstone sweep. Clearing it here puts the+ // resurrected row under that exact same protection, rather than+ // inventing a new one. `valid_to` is left as the tombstone wrote it+ // (the instant the page actually went missing), not bumped to `now`.+ let tombstone = most_recent_tombstone_at_path(tx, page)?;+ if let Some(tombstone_id) = &tombstone {+ tx.execute(+ "UPDATE pages SET superseded_at = NULL WHERE id = ?1",+ params![tombstone_id],+ )?;+ }let frontmatter_str = stamped_frontmatter(conformed, now)?;let new_id = PageId::new();tx.execute("INSERT INTO pages \(id, workspace_id, project_id, path, path_search, title, tier, body, body_sha256, \- frontmatter_json, is_latest, pinned, author_id, created_at, updated_at, expires_at, valid_from, compacted_at) \- VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, 1, ?11, ?12, ?13, ?13, ?14, ?13, ?15)",+ frontmatter_json, is_latest, supersedes, pinned, author_id, created_at, updated_at, \+ expires_at, valid_from, compacted_at) \+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, 1, ?11, ?12, ?13, ?14, ?14, ?15, ?14, ?16)",params![new_id.as_bytes(),page.workspace_id.as_bytes(),@@ -1481,6 +1516,7 @@ pub(crate) fn upsert_page_in_tx(page.body,body_sha256.as_slice(),frontmatter_str,+ tombstone,i64::from(page.pinned),page.author_id.map(|id| id.as_bytes().to_vec()),now,@@ -1494,7 +1530,11 @@ pub(crate) fn upsert_page_in_tx(insert_evidence_in_tx(tx, &new_id, &page.evidence, now)?;audit(tx,- "create_page",+ if tombstone.is_some() {+ "resurrect_tombstoned_page"+ } else {+ "create_page"+ },Some(page.workspace_id.as_bytes()),Some(page.project_id.as_bytes()),Some(new_id.as_bytes()),@@ -1506,6 +1546,32 @@ pub(crate) fn upsert_page_in_tx(Ok(new_id)}+/// The most recently tombstoned row at `page`'s path, if any: the newest+/// `is_latest = 0` version with `superseded_at` set (decay eviction or the+/// reconcile-delete safety net — see the resurrection comment in+/// [`upsert_page_in_tx`]). At most one row can match at a time in practice+/// (resurrecting one clears its `superseded_at`), but the ordering makes the+/// choice well-defined even if that invariant is ever violated.+fn most_recent_tombstone_at_path(+ tx: &rusqlite::Transaction<'_>,+ page: &NewPage,+) -> StoreResult<Option<Vec<u8>>> {+ Ok(tx+ .query_row(+ "SELECT id FROM pages \+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 \+ AND is_latest = 0 AND superseded_at IS NOT NULL \+ ORDER BY superseded_at DESC LIMIT 1",+ params![+ page.workspace_id.as_bytes(),+ page.project_id.as_bytes(),+ page.path.as_str(),+ ],+ |row| row.get::<_, Vec<u8>>(0),+ )+ .optional()?)+}+/// The live page this create would share a file with on a case-folding or/// normalizing filesystem, if any.///@@ -2841,6 +2907,82 @@ pub fn soft_delete_for_decay_if_latest(Ok(affected != 0)}+/// Tombstone the expected latest page whose on-disk file the watcher's+/// reconcile pass found missing on two consecutive passes (see #929 — an+/// opt-in safety net, `[maintenance] reconcile_tombstones_deleted_pages`).+///+/// Same shape as [`soft_delete_for_decay_if_latest`]: the full identity ++/// latest-id check runs in the same transaction as the write, so a page+/// rewritten after the caller last observed it is left untouched. Sets+/// `is_latest = 0` + `superseded_at`/`valid_to`, exactly like decay eviction.+/// The caller (`Wiki::tombstone_missing_page_if_latest`) does not dispatch+/// the admission/mirror webhook chain (this is a background safety net+/// reacting to an already-vanished file, not a user-initiated delete) and+/// never touches the filesystem (there is nothing to touch — the file is+/// already gone).+///+/// The precise durability guarantee (this tombstone is NOT exempt from the+/// aged-tombstone hard-delete sweep — it uses the exact same+/// `hard_delete_after_days`, tier/pin-agnostic path decay eviction does,+/// [`decay_tombstones_before`], [`hard_delete_decayed_page_chain`]): a+/// reconcile tombstone is never itself destroyed while its chain has no+/// successor. If the file comes back, `upsert_page_in_tx`'s resurrection+/// path re-links the new version onto this tombstoned chain via+/// `supersedes` and clears `superseded_at`, so the false positive's history+/// becomes an ordinary, protected supersession-chain member — nothing is+/// left orphaned for the sweep to destroy. Only a chain that is genuinely+/// never rewritten again is eventually hard-deleted, same as any other aged+/// decay tombstone.+pub fn soft_delete_for_reconcile_if_latest(+ conn: &mut Connection,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: &PagePath,+ expected_latest_id: PageId,+) -> StoreResult<bool> {+ let now = Timestamp::now().as_microsecond();+ let tx = conn.transaction()?;+ let affected = tx.execute(+ "UPDATE pages \+ SET is_latest = 0, superseded_at = ?1, valid_to = ?1 \+ WHERE id = ?2 \+ AND workspace_id = ?3 \+ AND project_id = ?4 \+ AND path = ?5 \+ AND is_latest = 1",+ params![+ now,+ expected_latest_id.as_bytes(),+ workspace_id.as_bytes(),+ project_id.as_bytes(),+ path.as_str(),+ ],+ )?;+ if affected != 0 {+ // Same entity-timeline closure as decay eviction (issue #656): an+ // open window on a tombstoned page must not resurrect retired+ // knowledge under `as_of`.+ tx.execute(+ "UPDATE entity_page_links SET superseded_at = ?1 \+ WHERE page_id = ?2 AND superseded_at IS NULL",+ params![now, expected_latest_id.as_bytes()],+ )?;+ audit(+ &tx,+ "soft_delete_for_reconcile",+ Some(workspace_id.as_bytes()),+ Some(project_id.as_bytes()),+ Some(expected_latest_id.as_bytes()),+ // Background reconcile safety net, not a user-attributable edit —+ // same convention as decay's `None` author.+ None,+ now,+ )?;+ }+ tx.commit()?;+ Ok(affected != 0)+}+/// Delete every version of a page (by path) from the index. Used when the/// wiki file is removed (`Wiki::delete_page`): the watcher does not handle/// file deletions, so the derived rows must be dropped explicitly or the@@ -9977,6 +10119,139 @@ pub(crate) mod tests {assert!(supersedes.is_some(), "new row must link to its predecessor");}+ /// B1 (#929 review): a page whose file disappears, gets reconcile-+ /// tombstoned, and then reappears must NOT start a brand-new,+ /// disconnected chain. Before this fix `upsert_page_in_tx`'s "no+ /// existing `is_latest = 1` row" branch always inserted with+ /// `supersedes = NULL`, leaving the tombstoned chain an orphan that+ /// `hard_delete_decayed_page_chain` would permanently destroy once+ /// `hard_delete_after_days` passed — even though the "deletion" was a+ /// false positive that had already self-healed. Exercises the exact+ /// mechanism end to end: tombstone -> file returns (a fresh+ /// `upsert_page`) -> hard-delete sweep with a cutoff equivalent to+ /// `hard_delete_after_days = 0` -> the prior (tombstoned) version must+ /// still exist and be reachable via the chain.+ #[test]+ fn reconcile_tombstone_resurrects_instead_of_orphaning_on_recreate() {+ let (_tmp, mut conn, ws, proj) = fresh_db();+ let path = PagePath::new("concepts/flaky.md").unwrap();+ let original_id =+ upsert_page(&mut conn, &page(ws, proj, path.as_str(), "v1 body")).unwrap();++ // The file vanishes and the watcher's reconcile-delete safety net+ // tombstones it (mirrors `Wiki::tombstone_missing_page_if_latest`).+ assert!(+ soft_delete_for_reconcile_if_latest(&mut conn, ws, proj, &path, original_id).unwrap(),+ "precondition: the tombstone must take"+ );+ let tombstoned_superseded_at: Option<i64> = conn+ .query_row(+ "SELECT superseded_at FROM pages WHERE id = ?1",+ params![original_id.as_bytes()],+ |r| r.get(0),+ )+ .unwrap();+ assert!(+ tombstoned_superseded_at.is_some(),+ "precondition: the tombstone sets superseded_at"+ );++ // The false positive self-heals: the file comes back and gets+ // reindexed, which is a fresh `upsert_page` call at the same path.+ let resurrected_id = upsert_page(+ &mut conn,+ &page(ws, proj, path.as_str(), "v1 body, but back"),+ )+ .unwrap();+ assert_ne!(original_id, resurrected_id);++ // The new version must re-link onto the tombstoned chain...+ let supersedes: Option<Vec<u8>> = conn+ .query_row(+ "SELECT supersedes FROM pages WHERE id = ?1",+ params![resurrected_id.as_bytes()],+ |r| r.get(0),+ )+ .unwrap();+ assert_eq!(+ supersedes.as_deref(),+ Some(&original_id.as_bytes()[..]),+ "the resurrected version must supersede the tombstoned chain, not start fresh"+ );+ // ...and the old row must no longer look like an eligible tombstone:+ // clearing `superseded_at` is what actually protects it, since+ // `hard_delete_decayed_page_chain`'s root query does not care who+ // points at a row via `supersedes`, only whether that row itself+ // still has `superseded_at IS NOT NULL`.+ let cleared_superseded_at: Option<i64> = conn+ .query_row(+ "SELECT superseded_at FROM pages WHERE id = ?1",+ params![original_id.as_bytes()],+ |r| r.get(0),+ )+ .unwrap();+ assert_eq!(+ cleared_superseded_at, None,+ "resurrection must clear the old row's superseded_at"+ );++ // Simulate the forget sweep's hard-delete pass with+ // `hard_delete_after_days = 0` (an immediate cutoff: `now`).+ let cutoff_now = jiff::Timestamp::now().as_microsecond();+ let candidates = {+ let mut stmt = conn+ .prepare(+ "SELECT id FROM pages WHERE workspace_id = ?1 AND project_id = ?2 \+ AND path = ?3 AND is_latest = 0 AND superseded_at IS NOT NULL \+ AND superseded_at <= ?4",+ )+ .unwrap();+ let rows = stmt+ .query_map(+ params![ws.as_bytes(), proj.as_bytes(), path.as_str(), cutoff_now],+ |r| r.get::<_, Vec<u8>>(0),+ )+ .unwrap();+ rows.collect::<Result<Vec<_>, _>>().unwrap()+ };+ assert!(+ candidates.is_empty(),+ "a resurrected chain's old version must not even be a hard-delete candidate: {candidates:?}"+ );+ let deleted = hard_delete_decayed_page_chain(+ &mut conn,+ ws,+ proj,+ &path,+ original_id,+ Some(resurrected_id),+ cutoff_now,+ )+ .unwrap();+ assert_eq!(+ deleted, 0,+ "the sweep must refuse to delete a chain member that is no longer a tombstone root"+ );++ // The prior version is still there and reachable via the chain.+ let still_exists: i64 = conn+ .query_row(+ "SELECT COUNT(*) FROM pages WHERE id = ?1",+ params![original_id.as_bytes()],+ |r| r.get(0),+ )+ .unwrap();+ assert_eq!(still_exists, 1, "the tombstoned version must survive");+ let reachable_via_chain: Vec<u8> = conn+ .query_row(+ "SELECT supersedes FROM pages WHERE id = ?1",+ params![resurrected_id.as_bytes()],+ |r| r.get(0),+ )+ .unwrap();+ assert_eq!(reachable_via_chain, original_id.as_bytes());+ }+/// Idempotency: re-upserting the same body should NOT create a/// second row. The watcher's reconciliation calls upsert_page on/// every file on every tick — without this, a quiet repo woulddiff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rsindex eb33734a..966066e4 100644--- a/crates/ai-memory-store/src/reader.rs+++ b/crates/ai-memory-store/src/reader.rs@@ -4379,12 +4379,53 @@ impl ReaderPool {.await}+ /// Return `(PageId, PagePath)` for every `is_latest = 1` page under a+ /// scope, cheaper than [`decay_candidates`](Self::decay_candidates) when+ /// only identity is needed. Used by the watcher's reconcile-delete+ /// safety net (#929) to snapshot the store's view of "pages that should+ /// have a file on disk" before walking the tree, so a page written via+ /// the API mid-walk is never mistaken for one the walk simply hasn't+ /// reached yet.+ ///+ /// # Errors+ /// Propagates any SQL or pool error.+ pub async fn latest_page_ids(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ ) -> StoreResult<Vec<(PageId, PagePath)>> {+ self.with_conn(move |conn| {+ let mut stmt = conn.prepare(+ "SELECT id, path FROM pages \+ WHERE workspace_id = ?1 AND project_id = ?2 AND is_latest = 1",+ )?;+ let rows = stmt.query_map(+ params![workspace_id.as_bytes(), project_id.as_bytes()],+ |row| Ok((row.get::<_, Vec<u8>>(0)?, row.get::<_, String>(1)?)),+ )?;+ let mut out = Vec::new();+ for row in rows {+ let (id, path) = row?;+ out.push((PageId::from_slice(&id)?, PagePath::new(path)?));+ }+ Ok(out)+ })+ .await+ }+/// Return decay tombstones old enough for permanent cleanup.///- /// `superseded_at` is written only by the forget-sweep eviction path, so- /// it is the tombstone discriminator regardless of whether a rewritten- /// page head also has a `supersedes` ancestor. The caller still routes- /// each result through the wiki layer before deleting its version chain.+ /// `superseded_at` is written by the forget-sweep eviction path AND by+ /// the watcher's opt-in reconcile-delete safety net (#929) — the two are+ /// deliberately indistinguishable here (docs/okf.md), so it remains the+ /// tombstone discriminator regardless of whether a rewritten page head+ /// also has a `supersedes` ancestor. A tombstone whose path was rewritten+ /// after it was marked (the false-positive self-healing, or an ordinary+ /// decay-then-recreate) has its `superseded_at` cleared by+ /// `ops::upsert_page_in_tx`'s resurrection path and so drops out of this+ /// query — only a chain with no successor ever reaches here. The caller+ /// still routes each result through the wiki layer before deleting its+ /// version chain.////// # Errors/// Propagates any SQL or pool error.diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rsindex ab9df7f0..56568c68 100644--- a/crates/ai-memory-store/src/writer.rs+++ b/crates/ai-memory-store/src/writer.rs@@ -340,6 +340,13 @@ pub(crate) enum WriteCmd {expected_latest_id: PageId,reply: oneshot::Sender<StoreResult<bool>>,},+ SoftDeleteForReconcileIfLatest {+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: PagePath,+ expected_latest_id: PageId,+ reply: oneshot::Sender<StoreResult<bool>>,+ },HardDeleteDecayedPageChain {workspace_id: WorkspaceId,project_id: ProjectId,@@ -1836,6 +1843,30 @@ impl WriterHandle {rx.await.map_err(|_| StoreError::WriterClosed)?}+ /// Tombstone the expected latest page whose file the watcher's reconcile+ /// pass found missing on two consecutive passes (opt-in, see #929).+ ///+ /// # Errors+ /// Returns [`StoreError::WriterClosed`] or propagates SQL errors.+ pub async fn soft_delete_for_reconcile_if_latest(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: PagePath,+ expected_latest_id: PageId,+ ) -> StoreResult<bool> {+ let (tx, rx) = oneshot::channel();+ self.send(WriteCmd::SoftDeleteForReconcileIfLatest {+ workspace_id,+ project_id,+ path,+ expected_latest_id,+ reply: tx,+ })+ .await?;+ rx.await.map_err(|_| StoreError::WriterClosed)?+ }+/// Permanently delete one eligible decay tombstone and its ancestry chain./// The expected latest-page state is checked in the same transaction so a/// page recreated at the same path cannot be removed accidentally.@@ -3570,6 +3601,22 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {);send_or_warn(reply, result, "soft_delete_for_decay_if_latest");}+ WriteCmd::SoftDeleteForReconcileIfLatest {+ workspace_id,+ project_id,+ path,+ expected_latest_id,+ reply,+ } => {+ let result = ops::soft_delete_for_reconcile_if_latest(+ &mut conn,+ workspace_id,+ project_id,+ &path,+ expected_latest_id,+ );+ send_or_warn(reply, result, "soft_delete_for_reconcile_if_latest");+ }WriteCmd::HardDeleteDecayedPageChain {workspace_id,project_id,diff --git a/crates/ai-memory-wiki/src/watcher.rs b/crates/ai-memory-wiki/src/watcher.rsindex 7b56f5f7..3a0693de 100644--- a/crates/ai-memory-wiki/src/watcher.rs+++ b/crates/ai-memory-wiki/src/watcher.rs@@ -28,11 +28,12 @@//! External writes drive store updates; internal writes drive disk +//! store updates via [`Wiki::write_page`].+use std::collections::{HashMap, HashSet};use std::path::Path;use std::str::FromStr;use std::time::Duration;-use ai_memory_core::{PagePath, ProjectId, WorkspaceId};+use ai_memory_core::{PageId, PagePath, ProjectId, WorkspaceId};use notify::{EventKind, RecursiveMode};use notify_debouncer_full::{DebounceEventResult, Debouncer, RecommendedCache, new_debouncer_opt};use tokio::sync::mpsc;@@ -159,6 +160,13 @@ async fn run_loop(let mut consecutive_failures: u32 = 0;const DEGRADED_AFTER: u32 = 5;+ // Reconcile-delete safety net (#929, opt-in — see `MissingStreaks`):+ // per-`(workspace, project, path)` count of consecutive passes a+ // candidate page's file has looked missing. Lives here, not on `Wiki`,+ // for the same reason `consecutive_failures` does: it is this task's own+ // pass-to-pass memory, not shared server state.+ let mut missing_streaks: MissingStreaks = HashMap::new();+loop {tokio::select! {biased;@@ -170,7 +178,7 @@ async fn run_loop(handle_event(&wiki, event).await;}_ = tick.tick() => {- match reconcile(&wiki).await {+ match reconcile(&wiki, &mut missing_streaks).await {Ok(_) => {if consecutive_failures > 0 {tracing::info!(@@ -368,15 +376,258 @@ pub(crate) struct ReconcileStats {/// Session pages left on disk by a purge whose file cleanup failed, and/// deliberately not re-indexed (#701).pub skipped_purged_sessions: usize,+ /// Pages tombstoned by the reconcile-delete safety net (#929) after their+ /// file was missing on two consecutive passes. Always `0` unless+ /// `[maintenance] reconcile_tombstones_deleted_pages` is enabled.+ pub tombstoned_missing: usize,+ /// Scopes where the reconcile-delete circuit breaker tripped this pass+ /// (design item #4) — too many candidates looked missing at once, so+ /// nothing in that scope was tombstoned or counted toward a streak.+ pub circuit_broken_scopes: usize,+}++/// Per-`(workspace, project, path)` streak of consecutive reconcile passes a+/// candidate page's file has looked missing, plus the [`PageId`] last+/// observed missing (so a path superseded by a new version between passes+/// restarts its streak rather than crediting the old version's absence to+/// the new one). Design item #5 requires two consecutive passes before a+/// tombstone is even attempted; this is that memory, owned by the watcher's+/// `run_loop` across ticks the same way `consecutive_failures` is.+type MissingStreaks = HashMap<(WorkspaceId, ProjectId, PagePath), (PageId, u32)>;++/// Reconcile-delete circuit breaker (design item #4): absolute floor. Even a+/// scope where every one of a handful of candidates looks "missing" must+/// clear this many before the breaker can trip — a two- or three-page+/// project isn't permanently blocked from ever tombstoning a real deletion+/// just because 100% of a tiny candidate set matches.+const RECONCILE_DELETE_BREAKER_MIN_CANDIDATES: usize = 3;++/// Reconcile-delete circuit breaker (design item #4): fraction floor, applied+/// alongside the absolute floor above. Half (or more) of a scope's candidate+/// pages vanishing inside one 30s reconcile window is far more likely a+/// walk/mount problem — an unmounted volume, a git checkout mid-walk, a+/// project directory being renamed — than that many genuine deletions+/// landing in the same pass, so the pass is treated as suspect and nothing+/// in that scope is tombstoned this round.+const RECONCILE_DELETE_BREAKER_FRACTION: f64 = 0.5;++/// The missing-candidate count strictly above which the circuit breaker+/// trips, for a scope with `total_candidates` walk-eligible pages tracked+/// before the walk ran this pass.+fn reconcile_delete_breaker_threshold(total_candidates: usize) -> usize {+ #[allow(clippy::cast_precision_loss)]+ let total = total_candidates as f64;+ let fraction_floor = (total * RECONCILE_DELETE_BREAKER_FRACTION).ceil();+ #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)]+ let fraction_floor = fraction_floor as usize;+ fraction_floor.max(RECONCILE_DELETE_BREAKER_MIN_CANDIDATES)+}++/// Whether `path` is a shape [`walk_markdown`] could ever have returned,+/// applied to a DB row instead of a filesystem entry (design item #1).+///+/// `bootstrap.md`, `_meta.md`, and `_pending/` sidecars are indexed rows the+/// walk structurally never returns, so their absence from a walk's page list+/// proves nothing about whether their file still exists — without this+/// filter every one of them would look "missing" on every single pass.+///+/// More conservative than the walk's own ledger check+/// ([`is_reserved_page_file`]), which reads a candidate ledger file's first+/// line to tell a real hook ledger from an ordinary page that merely happens+/// to be named `log.md`. Here the file is gone by definition — there is+/// nothing left to read — so any `log.md` / `log-YYYY-MM.md`-shaped path is+/// excluded from candidacy unconditionally, erring toward never tombstoning+/// rather than guessing.+///+/// `sessions/<id>.md` pages are excluded too, for an unrelated reason (#929+/// review, B2): a same-workspace `move-session` re-home updates the store+/// row's `(workspace_id, project_id)` but deliberately does not move the+/// file (the wiki layer skips that on purpose — see `Wiki::move_session_page`+/// and its callers), so a session summary page can have a perfectly correct,+/// live DB row with no file at its *new* scope. That is a separate,+/// pre-existing bug in `move-session`'s file relocation, not fixed here —+/// but this mechanism would otherwise read that gap as "the file was+/// deleted" and tombstone a real, live page. This safety net is meant for+/// OKF-imported content pages (concepts, decisions, gotchas — the paths+/// #929's own repro used), never session-summary pages, so the whole shape+/// is excluded until `move-session` is fixed to relocate the file too+/// (tracked as a follow-up).+fn could_have_been_walked(path: &PagePath) -> bool {+ if is_pending_path(path) {+ return false;+ }+ if is_manifest_filename(path) || path.as_str() == "bootstrap.md" {+ return false;+ }+ if crate::wiki::session_id_for_page(path).is_some() {+ return false;+ }+ if crate::ledger::is_log_ledger_filename(path) {+ return false;+ }+ true+}++/// The reconcile-delete safety net (#929, design items #1–#6) for one+/// project scope, run after that scope's walk has completed. No-op unless+/// `[maintenance] reconcile_tombstones_deleted_pages` is enabled.+///+/// `pre_walk_snapshot` must have been taken from the store *before*+/// `walked`'s walk started (design item #2) — that ordering is what makes a+/// page written via the API mid-walk safe: it is either absent from the+/// snapshot (never a candidate) or present in `walked` (found on disk), and+/// either way it can never look like a deletion.+#[allow(clippy::too_many_arguments)]+async fn reconcile_missing_pages(+ wiki: &Wiki,+ ws: WorkspaceId,+ proj: ProjectId,+ pre_walk_snapshot: &[(PageId, PagePath)],+ walked: &HashSet<PagePath>,+ walk_partial: bool,+ streaks: &mut MissingStreaks,+ stats: &mut ReconcileStats,+) {+ if pre_walk_snapshot.is_empty() {+ return;+ }++ // A page found on disk this pass can never contribute to a future+ // tombstone from an earlier streak — clear it regardless of what happens+ // to the rest of this scope below. This runs even on a partial walk+ // (S1, #929 review): only ABSENCE evidence from a partial walk is+ // suspect (design item #3, below) — presence evidence a partial walk DID+ // manage to observe is exactly as trustworthy as any other pass's, and+ // losing it would needlessly re-arm a streak the page had already+ // cleared.+ for (_, path) in pre_walk_snapshot.iter().filter(|(_, p)| walked.contains(p)) {+ streaks.remove(&(ws, proj, path.clone()));+ }++ if walk_partial {+ // Design item #3: a `NotFound` anywhere in this scope's walk (a+ // vanished subdirectory, or the whole project directory) means some+ // branch of the tree was unreadable rather than every page under it+ // having actually been deleted one by one. Skip NEW missing-page+ // evidence for this pass — streaks for pages already missing before+ // this pass are left exactly as they were (neither incremented nor+ // cleared), so a later, complete pass still needs its own two+ // consecutive observations before anything is tombstoned.+ debug!(+ workspace = %ws,+ project = %proj,+ "reconcile-delete: partial walk this pass (a directory was unreadable); not \+ treating this scope's absent pages as new deletion evidence",+ );+ return;+ }++ let missing: Vec<&(PageId, PagePath)> = pre_walk_snapshot+ .iter()+ .filter(|(_, p)| !walked.contains(p))+ .collect();+ if missing.is_empty() {+ return;+ }++ // Design item #4: circuit breaker. A suspect pass makes zero progress+ // toward the two-pass threshold for any of its candidates — not just+ // "don't tombstone yet", but "don't count this pass at all" — so a+ // walk/mount blip can never contribute half of the two observations a+ // real deletion needs.+ //+ // S3 (#929 review): an empty `walked` set with a non-empty snapshot is+ // ALWAYS suspect, regardless of the `max(3, 50%)` math below — that+ // fraction still has a floor of `RECONCILE_DELETE_BREAKER_MIN_CANDIDATES`,+ // so a scope with only one or two tracked candidates could have every+ // single one look missing without ever tripping the percentage breaker.+ // "the walk found NOTHING at all" is exactly the "directory exists but+ // came back empty" mount-failure signature (an unmounted volume, a+ // bind-mount that briefly resolves to an empty stub) that the threshold+ // alone would miss for a small scope.+ let threshold = reconcile_delete_breaker_threshold(pre_walk_snapshot.len());+ let empty_walk_is_suspect = walked.is_empty();+ if missing.len() > threshold || empty_walk_is_suspect {+ tracing::warn!(+ workspace = %ws,+ project = %proj,+ missing = missing.len(),+ total_candidates = pre_walk_snapshot.len(),+ threshold,+ empty_walk = empty_walk_is_suspect,+ "reconcile-delete: circuit breaker tripped — treating this as a walk/mount \+ problem rather than that many genuine deletions in one pass; nothing in this \+ scope is tombstoned or counted toward a streak this round",+ );+ stats.circuit_broken_scopes += 1;+ return;+ }++ for (id, path) in missing {+ let key = (ws, proj, path.clone());+ let streak = streaks.entry(key.clone()).or_insert((*id, 0));+ if streak.0 != *id {+ // A different page version now occupies this path than the one+ // whose absence started this streak (e.g. recreated then+ // rewritten between passes) — restart rather than credit the old+ // version's absence to the new one.+ *streak = (*id, 0);+ }+ streak.1 += 1;+ if streak.1 < 2 {+ continue;+ }+ // Missing on two consecutive passes: attempt the tombstone. Whatever+ // the outcome, drop the streak — a page still missing after this+ // starts a fresh streak next pass rather than retrying every tick.+ streaks.remove(&key);+ match wiki+ .tombstone_missing_page_if_latest(ws, proj, path, *id)+ .await+ {+ Ok(true) => {+ stats.tombstoned_missing += 1;+ tracing::info!(+ workspace = %ws,+ project = %proj,+ path = %path,+ "reconcile-delete: tombstoned a page whose file was missing on two \+ consecutive reconcile passes",+ );+ }+ Ok(false) => {+ debug!(+ workspace = %ws,+ project = %proj,+ path = %path,+ "reconcile-delete: tombstone candidate skipped (page changed, or its file \+ reappeared before the final recheck)",+ );+ }+ Err(e) => {+ warn!(+ workspace = %ws,+ project = %proj,+ path = %path,+ error = %e,+ "reconcile-delete: tombstone attempt failed",+ );+ }+ }+ }}-async fn reconcile(wiki: &Wiki) -> WikiResult<ReconcileStats> {+async fn reconcile(+ wiki: &Wiki,+ missing_streaks: &mut MissingStreaks,+) -> WikiResult<ReconcileStats> {let root = wiki.root().to_path_buf();// Walk all per-project subdirectories: <ws_uuid>/<proj_uuid>/let project_dirs = tokio::task::spawn_blocking(move || walk_project_dirs(&root)).await.map_err(|e| WikiError::Io(std::io::Error::other(e.to_string())))??;+ let tombstones_enabled = wiki.reconcile_tombstones_deleted_pages();let mut stats = ReconcileStats::default();for (ws, proj, proj_root) in project_dirs {// The directory name parses as a valid UUID pair, but that does not@@ -401,14 +652,45 @@ async fn reconcile(wiki: &Wiki) -> WikiResult<ReconcileStats> {stats.skipped_orphans += 1;continue;}- let pages = tokio::task::spawn_blocking(move || walk_markdown(&proj_root))- .await- .map_err(|e| WikiError::Io(std::io::Error::other(e.to_string())))??;++ // Design item #2: snapshot the store's `is_latest = 1` pages for this+ // scope BEFORE the walk runs, not after — see `reconcile_missing_pages`.+ // Filtered to paths the walk could ever return, so reserved/pending+ // rows are never candidates (design item #1). A snapshot failure+ // disables the safety net for this scope this pass only; it never+ // blocks the ordinary reindex below.+ let pre_walk_snapshot = if tombstones_enabled {+ match wiki.latest_page_ids(ws, proj).await {+ Ok(rows) => rows+ .into_iter()+ .filter(|(_, p)| could_have_been_walked(p))+ .collect::<Vec<_>>(),+ Err(e) => {+ warn!(+ workspace = %ws,+ project = %proj,+ error = %e,+ "reconcile-delete: candidate snapshot failed; skipping this scope's \+ missing-page check this pass",+ );+ Vec::new()+ }+ }+ } else {+ Vec::new()+ };++ let (pages, walk_partial) =+ tokio::task::spawn_blocking(move || walk_markdown_partial(&proj_root))+ .await+ .map_err(|e| WikiError::Io(std::io::Error::other(e.to_string())))??;// Once per directory, not once per page: a project accumulates one// tombstone per purge, and the set is only consulted for the// `sessions/<id>.md` paths that a session purge could have left behind.let purged = wiki.purged_sessions(ws, proj).await?;+ let mut walked: HashSet<PagePath> = HashSet::with_capacity(pages.len());for path in pages {+ walked.insert(path.clone());if crate::wiki::is_purged_session_page(&path, &purged) {debug!(path = %path,@@ -423,6 +705,20 @@ async fn reconcile(wiki: &Wiki) -> WikiResult<ReconcileStats> {stats.indexed += 1;}}++ if tombstones_enabled {+ reconcile_missing_pages(+ wiki,+ ws,+ proj,+ &pre_walk_snapshot,+ &walked,+ walk_partial,+ missing_streaks,+ &mut stats,+ )+ .await;+ }}// debug!, not info!: this fires every RECONCILE_INTERVAL regardless of// activity, so at info it is ~half the default server log (#894). Its@@ -432,6 +728,8 @@ async fn reconcile(wiki: &Wiki) -> WikiResult<ReconcileStats> {indexed = stats.indexed,skipped_orphans = stats.skipped_orphans,skipped_purged_sessions = stats.skipped_purged_sessions,+ tombstoned_missing = stats.tombstoned_missing,+ circuit_broken_scopes = stats.circuit_broken_scopes,"reconciliation pass complete",);Ok(stats)@@ -532,12 +830,31 @@ fn extract_project_dir_ids(}pub(crate) fn walk_markdown(root: &Path) -> WikiResult<Vec<PagePath>> {+ walk_markdown_partial(root).map(|(pages, _partial)| pages)+}++/// Like [`walk_markdown`], but also reports whether any directory in the+/// tree — including `root` itself — could not be read because it no longer+/// exists.+///+/// A `true` second element means the walk is **partial**: some branch+/// vanished mid-walk (a git checkout, a bind-mount hiccup, a concurrent+/// project move) rather than every page under it having actually been+/// deleted one at a time. The reconcile-delete safety net (#929 design item+/// #3) relies on this to tell those two situations apart — treating a+/// partial walk's absent pages as deletion evidence would tombstone pages+/// that are still there, just temporarily unreadable.+pub(crate) fn walk_markdown_partial(root: &Path) -> WikiResult<(Vec<PagePath>, bool)> {let mut out = Vec::new();+ let mut partial = false;let mut stack = vec![root.to_path_buf()];while let Some(dir) = stack.pop() {let read = match std::fs::read_dir(&dir) {Ok(r) => r,- Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue,+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => {+ partial = true;+ continue;+ }Err(e) => return Err(WikiError::Io(e)),};for entry in read {@@ -575,7 +892,7 @@ pub(crate) fn walk_markdown(root: &Path) -> WikiResult<Vec<PagePath>> {}}}- Ok(out)+ Ok((out, partial))}pub(crate) fn is_pending_path(page_path: &PagePath) -> bool {@@ -770,7 +1087,7 @@ mod tests {"precondition: the purge made the page unsearchable",);- let stats = reconcile(&wiki).await.unwrap();+ let stats = reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();assert_eq!(stats.skipped_orphans, 0,"a session purge leaves the project row, so the orphan guard passes",@@ -1004,7 +1321,7 @@ mod tests {let handle = WatcherHandle::start(wiki.clone()).unwrap();// Hit reconcile manually instead of waiting 30s.- reconcile(&wiki).await.unwrap();+ reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();let hits = store.reader@@ -1040,7 +1357,7 @@ mod tests {std::fs::write(orphan_dir.join("index.md"), "seeded shell\n").unwrap();std::fs::write(orphan_dir.join("stale.md"), "orphantoken content\n").unwrap();- let stats = reconcile(&wiki).await.unwrap();+ let stats = reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();assert_eq!(stats.skipped_orphans, 1,@@ -1193,7 +1510,7 @@ mod tests {let handle = WatcherHandle::start(wiki.clone()).unwrap();// Trigger the reconciliation pass directly.- reconcile(&wiki).await.unwrap();+ reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();// Only `real.md` should land in the index.let hits = store@@ -1306,7 +1623,7 @@ mod tests {.unwrap();let handle = WatcherHandle::start(wiki.clone()).unwrap();- reconcile(&wiki).await.unwrap();+ reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();let page_hits = store.reader@@ -1506,4 +1823,707 @@ mod tests {"unchanged content must short-circuit to the same id");}++ // --- #929: reconcile-delete safety net (opt-in tombstone-on-missing-file) ---++ fn proj_dir(tmp: &TempDir, ws: WorkspaceId, proj: ProjectId) -> std::path::PathBuf {+ tmp.path()+ .join("wiki")+ .join(ws.to_string())+ .join(proj.to_string())+ }++ /// `setup()` with the reconcile-delete safety net opted in. Kept separate+ /// from `setup()` so every other watcher test keeps exercising the+ /// default-off path unchanged.+ async fn setup_reconcile_delete() -> (TempDir, Store, Wiki, WorkspaceId, ProjectId) {+ let (tmp, store, wiki, ws, proj) = setup().await;+ (+ tmp,+ store,+ wiki.with_reconcile_tombstones_deleted_pages(true),+ ws,+ proj,+ )+ }++ /// Write `body` at `path` under the project directory and index it,+ /// returning the resulting `PageId`.+ async fn write_and_index(+ wiki: &Wiki,+ tmp: &TempDir,+ ws: WorkspaceId,+ proj: ProjectId,+ path: &str,+ body: &str,+ ) -> PageId {+ let dir = proj_dir(tmp, ws, proj);+ let full = dir.join(path);+ std::fs::create_dir_all(full.parent().unwrap()).unwrap();+ std::fs::write(&full, body).unwrap();+ wiki.reindex_page(ws, proj, PagePath::new(path).unwrap())+ .await+ .unwrap()+ }++ /// With the feature at its default (off), a page whose file disappears is+ /// never tombstoned, no matter how many reconcile passes run — reconcile's+ /// behavior is unchanged from before this feature existed (#929).+ #[tokio::test]+ async fn reconcile_delete_disabled_by_default_never_tombstones_a_missing_page() {+ let (tmp, store, wiki, ws, proj) = setup().await;+ let path = PagePath::new("gone.md").unwrap();+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "here for now").await;+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();++ let mut streaks = MissingStreaks::new();+ for _ in 0..5 {+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(stats.tombstoned_missing, 0);+ }+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(id),+ "deletions still require `ai-memory delete-page` when the feature is off",+ );+ }++ /// The core happy path, and the two-pass bite-check: the first pass a+ /// page's file is found missing must NOT tombstone it (streak = 1); only+ /// the second CONSECUTIVE pass does (design item #5).+ #[tokio::test]+ async fn reconcile_delete_tombstones_only_after_two_consecutive_missing_passes() {+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;+ let path = PagePath::new("gone.md").unwrap();+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "here for now").await;+ // A control page that stays present the whole test, so the scope's+ // walk is never literally empty (S3, #929 review: an empty walk with+ // a non-empty snapshot is its own, unconditional breaker trip).+ write_and_index(&wiki, &tmp, ws, proj, "control.md", "always here").await;++ let mut streaks = MissingStreaks::new();+ let baseline = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(baseline.tombstoned_missing, 0);++ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();++ let first_miss = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(+ first_miss.tombstoned_missing, 0,+ "one missing pass must not be enough"+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(id),+ "still latest after only one missing pass",+ );++ let second_miss = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(+ second_miss.tombstoned_missing, 1,+ "two consecutive missing passes must tombstone"+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ None,+ );+ }++ /// `could_have_been_walked` (design item #1, plus the B2 sessions+ /// exclusion) is the path-only filter applied to the DB-side candidate+ /// snapshot, mirroring the walk's own skip rules. `_pending/` sidecars+ /// can never even reach it as a candidate — `reindex_page` itself+ /// refuses to index one (see `refusing to index pending proposal+ /// sidecar` in `wiki.rs`) — so this unit test is the only way to+ /// exercise that branch directly.+ #[test]+ fn could_have_been_walked_excludes_every_reserved_shape() {+ let session_path = format!("sessions/{}.md", ai_memory_core::SessionId::new());+ for excluded in [+ "_pending",+ "_pending/draft.md",+ "_pending/review-bot/notes.md",+ "bootstrap.md",+ "_meta.md",+ "notes/_meta.md",+ "log.md",+ "log-2024-01.md",+ session_path.as_str(),+ ] {+ assert!(+ !could_have_been_walked(&PagePath::new(excluded).unwrap()),+ "{excluded} must never be a reconcile-delete candidate",+ );+ }+ // "sessions/abc.md" does NOT parse as a `SessionId` (not a UUID), so+ // it is an ordinary page shape, not a session summary — proving the+ // exclusion is keyed on `session_id_for_page`, not merely the+ // `sessions/` prefix.+ for included in ["notes/plan.md", "sessions/abc.md", "log-review.md"] {+ assert!(+ could_have_been_walked(&PagePath::new(included).unwrap()),+ "{included} is an ordinary page and must remain a candidate",+ );+ }+ }++ /// Reserved/indexed-but-unwalked paths (design item #1) are never+ /// candidates, even across many consecutive passes with their files gone:+ /// `bootstrap.md` and a legacy ledger row (`log.md`-shaped). Unlike the+ /// walk, `reindex_page` does not itself refuse these two paths, so they+ /// can be indexed rows exactly the way an older store or a direct API+ /// write could leave them.+ #[tokio::test]+ async fn reconcile_delete_reserved_paths_are_never_candidates() {+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;+ let paths = ["bootstrap.md", "log.md"];+ let mut ids = Vec::new();+ for p in paths {+ let id = write_and_index(&wiki, &tmp, ws, proj, p, "reserved-shaped content").await;+ ids.push((PagePath::new(p).unwrap(), id));+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(p)).unwrap();+ }++ let mut streaks = MissingStreaks::new();+ for _ in 0..5 {+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(stats.tombstoned_missing, 0);+ }+ for (path, id) in ids {+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(id),+ "{path} must never be treated as deletion evidence",+ );+ }+ }++ /// An atomic-save pattern (delete-then-recreate) must survive: if the+ /// file is back by the very next pass, the streak clears and nothing is+ /// ever tombstoned — the two-pass requirement plus this reappearance+ /// check is what makes that safe beyond the existing debounce window.+ #[tokio::test]+ async fn reconcile_delete_atomic_save_pattern_survives() {+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;+ let path = PagePath::new("flaky.md").unwrap();+ let id1 = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "v1").await;+ // Control page (S3, #929 review): without it, the scope's walk goes+ // fully empty the moment `flaky.md` is missing, and the survival this+ // test checks for would be explained by the empty-walk breaker+ // tripping rather than by the streak/reappearance mechanism it's+ // actually meant to exercise.+ write_and_index(&wiki, &tmp, ws, proj, "control.md", "always here").await;++ let mut streaks = MissingStreaks::new();+ reconcile(&wiki, &mut streaks).await.unwrap();++ // Pass N: file momentarily missing (streak = 1).+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();+ let miss = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(miss.tombstoned_missing, 0);++ // Recreated before the next pass — the "atomic save" completing.+ let id2 = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "v2").await;+ assert_ne!(id1, id2, "a content change mints a new version");+ let recovered = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(recovered.tombstoned_missing, 0);++ // One more pass with the file still present must not retroactively+ // tombstone anything either — the streak was cleared, not merely paused.+ let stable = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(stable.tombstoned_missing, 0);+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(id2),+ );+ }++ /// S4 (#929 review): the test above recreates the file with DIFFERENT+ /// content, so it never actually proves the streak was CLEARED by the+ /// SAME page reappearing — a new `PageId` alone (via the `streak.0 !=+ /// *id` restart in `reconcile_missing_pages`) would make that test pass+ /// even if the "found in `walked`" clearing loop were deleted entirely.+ /// This test forces the identical-content path: missing (streak = 1) ->+ /// present again with the SAME body (so `upsert_page`'s content+ /// short-circuit returns the SAME id, exercising the presence-clears-+ /// streak branch specifically) -> missing again (must restart at+ /// streak = 1, not continue to 2) -> not tombstoned; a further pass+ /// still missing then reaches streak = 2 and IS tombstoned — proving two+ /// FRESH consecutive misses were required, not two misses ever observed.+ #[tokio::test]+ async fn reconcile_delete_streak_clears_on_identical_content_reappearance() {+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;+ let path = PagePath::new("flaky-identical.md").unwrap();+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "same body always").await;+ // Control page (S3, #929 review — see the other test's comment).+ write_and_index(&wiki, &tmp, ws, proj, "control.md", "always here").await;++ let mut streaks = MissingStreaks::new();+ reconcile(&wiki, &mut streaks).await.unwrap();++ // Miss #1.+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();+ let first_miss = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(first_miss.tombstoned_missing, 0, "streak = 1, not enough");++ // Reappears with IDENTICAL content: `upsert_page`'s sha256+ // short-circuit returns the SAME id, so this exercises "found in+ // `walked`" clearing a streak, not the `streak.0 != *id` restart.+ let same_id =+ write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "same body always").await;+ assert_eq!(+ same_id, id,+ "precondition: identical content must not mint a new id"+ );+ let recovered = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(recovered.tombstoned_missing, 0);++ // Missing again: this must be a FRESH streak (= 1), not a+ // continuation (which would already be 2 and tombstone here).+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();+ let second_streak_first_miss = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(+ second_streak_first_miss.tombstoned_missing, 0,+ "the streak must have restarted at 1, not continued to 2"+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(id),+ "still latest after only one fresh missing pass"+ );++ // Still missing, second FRESH consecutive pass: now it tombstones.+ let second_streak_second_miss = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(second_streak_second_miss.tombstoned_missing, 1);+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ None,+ );+ }++ /// Circuit breaker (design item #4): when most of a scope's candidates+ /// look missing in one pass, nothing is tombstoned and a `warn` fires —+ /// that shape is far more likely a walk/mount problem than genuine mass+ /// deletion. Bite-check: with the breaker removed, this test fails+ /// (verified manually while writing it, then restored).+ #[tokio::test]+ async fn reconcile_delete_circuit_breaker_blocks_when_most_candidates_vanish() {+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;+ let mut ids = Vec::new();+ for i in 0..10 {+ let p = format!("bulk/{i}.md");+ let id = write_and_index(&wiki, &tmp, ws, proj, &p, "bulk content").await;+ ids.push((PagePath::new(p).unwrap(), id));+ }++ let mut streaks = MissingStreaks::new();+ reconcile(&wiki, &mut streaks).await.unwrap();++ // 60% vanish in one pass.+ for (path, _) in ids.iter().take(6) {+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();+ }+ for pass in 0..3 {+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(+ stats.tombstoned_missing, 0,+ "pass {pass}: the breaker must block every one of the 6 missing pages"+ );+ assert_eq!(stats.circuit_broken_scopes, 1, "pass {pass}");+ }+ for (path, id) in &ids {+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(*id),+ "{path} must survive a tripped breaker",+ );+ }+ }++ /// Control for the breaker test: a small fraction missing (well under+ /// `max(3, 50%)`) proceeds normally and tombstones after two consecutive+ /// passes, exactly like the non-bulk case.+ #[tokio::test]+ async fn reconcile_delete_circuit_breaker_does_not_block_a_small_fraction() {+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;+ let mut ids = Vec::new();+ for i in 0..10 {+ let p = format!("bulk/{i}.md");+ let id = write_and_index(&wiki, &tmp, ws, proj, &p, "bulk content").await;+ ids.push((PagePath::new(p).unwrap(), id));+ }++ let mut streaks = MissingStreaks::new();+ reconcile(&wiki, &mut streaks).await.unwrap();++ // 10% vanish: well under the breaker threshold (max(3, 50%) = 5 of 10).+ let (missing_path, _missing_id) = ids[0].clone();+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(missing_path.as_str())).unwrap();++ let first = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(first.circuit_broken_scopes, 0);+ assert_eq!(first.tombstoned_missing, 0, "first missing pass");++ let second = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(second.circuit_broken_scopes, 0);+ assert_eq!(+ second.tombstoned_missing, 1,+ "second consecutive missing pass"+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, missing_path.as_str().to_string())+ .await+ .unwrap(),+ None,+ );+ for (path, id) in ids.iter().skip(1) {+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(*id),+ "{path} was never missing",+ );+ }+ }++ /// S3 (#929 review): an empty (but NOT partial/`NotFound`) walk is always+ /// suspect, even for a scope too small to ever trip the `max(3, 50%)`+ /// math — 2 candidates, both missing, is only 100% but the absolute+ /// floor of 3 never trips. This is the "directory exists but came back+ /// empty" mount-failure signature (an unmounted volume, a bind-mount+ /// briefly resolving to an empty stub): the walk succeeds (no+ /// `NotFound`, so `walk_partial` is false) but finds nothing at all.+ #[tokio::test]+ async fn reconcile_delete_circuit_breaker_trips_on_empty_walk_even_for_a_small_scope() {+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;+ let mut ids = Vec::new();+ for i in 0..2 {+ let p = format!("small/{i}.md");+ let id = write_and_index(&wiki, &tmp, ws, proj, &p, "small scope content").await;+ ids.push((PagePath::new(p).unwrap(), id));+ }++ let mut streaks = MissingStreaks::new();+ reconcile(&wiki, &mut streaks).await.unwrap();++ // Both files vanish, but the directory itself stays put — an+ // ordinary (non-partial) walk that simply finds nothing.+ for (path, _) in &ids {+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();+ }+ for pass in 0..3 {+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(+ stats.tombstoned_missing, 0,+ "pass {pass}: an empty walk must never be treated as evidence, however small \+ the scope"+ );+ assert_eq!(stats.circuit_broken_scopes, 1, "pass {pass}");+ }+ for (path, id) in &ids {+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(*id),+ "{path} must survive an empty-walk breaker trip",+ );+ }+ }++ /// Partial walk (design item #3): the whole project directory vanishing+ /// mid-pass (a git checkout, a bind-mount hiccup) must not be treated as+ /// every page under it having been deleted. Restoring the directory lets+ /// the very next pass proceed normally — the reappear-after-restore case.+ #[tokio::test]+ async fn reconcile_delete_partial_walk_skips_a_scope_whose_project_directory_vanished() {+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;+ let path = PagePath::new("survivor.md").unwrap();+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "still around").await;++ let dir = proj_dir(&tmp, ws, proj);+ let mut streaks = MissingStreaks::new();+ reconcile(&wiki, &mut streaks).await.unwrap();++ // Simulate the whole project directory vanishing mid-walk (walk_partial+ // becomes true because `walk_markdown_partial`'s own root read fails).+ std::fs::remove_dir_all(&dir).unwrap();+ for pass in 0..3 {+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(+ stats.tombstoned_missing, 0,+ "pass {pass}: a partial walk must never produce deletion evidence"+ );+ }+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(id),+ "the page must survive many passes of a vanished project directory",+ );++ // Restore: the directory and file come back, and reconcile resumes+ // ordinary behavior instead of carrying over any partial-pass state.+ std::fs::create_dir_all(&dir).unwrap();+ std::fs::write(dir.join(path.as_str()), "still around").unwrap();+ let restored = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(restored.tombstoned_missing, 0);+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(id),+ );+ }++ /// S1 (#929 review): a partial walk's ABSENCE evidence is suspect, but+ /// its PRESENCE evidence is not — a page the walk actually found, even+ /// during a partial pass, must still clear an earlier streak. Otherwise+ /// a page that legitimately reappeared could be tombstoned later purely+ /// because an unrelated directory's `NotFound` earlier suppressed the+ /// clearing that presence should have triggered.+ ///+ /// Calls `reconcile_missing_pages` directly rather than orchestrating a+ /// real partial walk through the filesystem: a subdirectory deleted+ /// *before* `reconcile()` runs is never even listed by `read_dir`, so it+ /// produces no `NotFound` at all (that's only reachable via a genuine+ /// race — the directory vanishing *during* the walk, mid-traversal,+ /// which a single-threaded test cannot deterministically force). Driving+ /// the function directly exercises the exact contract instead.+ #[tokio::test]+ async fn reconcile_delete_partial_walk_still_clears_streaks_for_pages_it_did_find() {+ let (tmp, _store, wiki, ws, proj) = setup_reconcile_delete().await;+ let path = PagePath::new("flaky.md").unwrap();+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "here").await;++ // Seed a pre-existing streak of 1, as if an earlier complete pass had+ // already observed this exact page missing once.+ let mut streaks = MissingStreaks::new();+ streaks.insert((ws, proj, path.clone()), (id, 1));++ let snapshot = vec![(id, path.clone())];+ let mut walked: HashSet<PagePath> = HashSet::new();+ walked.insert(path.clone());+ let mut stats = ReconcileStats::default();++ // This pass is partial (some unrelated directory vanished elsewhere+ // in the scope), but it DID find this exact page present.+ reconcile_missing_pages(+ &wiki,+ ws,+ proj,+ &snapshot,+ &walked,+ true,+ &mut streaks,+ &mut stats,+ )+ .await;++ assert!(+ !streaks.contains_key(&(ws, proj, path.clone())),+ "presence evidence from a partial walk must still clear the streak"+ );+ assert_eq!(stats.tombstoned_missing, 0);++ // Bite-check performed manually while writing this test: hoisting+ // the clearing loop back inside the `if walk_partial { return; }`+ // branch (the pre-S1 shape) makes this assertion fail, since the+ // seeded streak survives untouched.+ }++ /// Cross-project isolation (the same adversarial shape #929's rejected+ /// prior attempt was tested against): two projects with a page at the+ /// same relative path, only one of which loses its file, must not let+ /// one project's missing-page streak or tombstone affect the other's.+ #[tokio::test]+ async fn reconcile_delete_cross_project_isolation() {+ let (tmp, store, wiki, ws, proj_a) = setup_reconcile_delete().await;+ let proj_b = store+ .writer+ .get_or_create_project(ws, "scratch-b", None)+ .await+ .unwrap();+ let path = PagePath::new("shared/name.md").unwrap();+ let _id_a = write_and_index(&wiki, &tmp, ws, proj_a, path.as_str(), "project a").await;+ let id_b = write_and_index(&wiki, &tmp, ws, proj_b, path.as_str(), "project b").await;+ // Control page in proj_a (S3, #929 review): keeps that scope's walk+ // from being literally empty once `shared/name.md` goes missing,+ // which would otherwise trip the unconditional empty-walk breaker.+ write_and_index(&wiki, &tmp, ws, proj_a, "control.md", "always here").await;++ let mut streaks = MissingStreaks::new();+ reconcile(&wiki, &mut streaks).await.unwrap();+ std::fs::remove_file(proj_dir(&tmp, ws, proj_a).join(path.as_str())).unwrap();++ reconcile(&wiki, &mut streaks).await.unwrap();+ let final_stats = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(final_stats.tombstoned_missing, 1);++ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj_a, path.as_str().to_string())+ .await+ .unwrap(),+ None,+ "project A's page was tombstoned",+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj_b, path.as_str().to_string())+ .await+ .unwrap(),+ Some(id_b),+ "project B's identically-pathed page must survive untouched",+ );+ }++ /// B2 (#929 review): a same-workspace `move-session` re-home can leave a+ /// session page's DB row live in the destination project with no file+ /// there. Reproduced end to end through the real bug, not a synthetic+ /// row edit: the session's own record already says it lives in+ /// `proj_b` (so `Wiki::move_session_page(from=(ws,proj_b),+ /// to=(ws,proj_b), ...)` sees `from == to` and skips the file step+ /// entirely — see its doc comment), while its `sessions/<id>.md` page+ /// is a straggler still sitting in `proj_a`. `ops::move_session`'s+ /// page re-stamp is keyed purely on path + "not already in the target+ /// scope", not on `from`, so it sweeps that straggler page into+ /// `proj_b` regardless — leaving a live, correct row at `proj_b` with+ /// no file there, ever. This is a separate, pre-existing bug in+ /// `move-session`'s file relocation (not fixed here); this test only+ /// proves the reconcile-delete safety net's `sessions/*.md` exclusion+ /// keeps it from destroying that page.+ #[tokio::test]+ async fn reconcile_delete_never_tombstones_a_session_page_orphaned_by_move_session() {+ let (tmp, store, wiki, ws, proj_a) = setup_reconcile_delete().await;+ let proj_b = store+ .writer+ .get_or_create_project(ws, "scratch-b", None)+ .await+ .unwrap();+ let sid = ai_memory_core::SessionId::new();+ // The session's own record already lives in proj_b.+ store+ .writer+ .begin_session(ai_memory_core::NewSession {+ occurred_at: None,+ id: sid,+ workspace_id: ws,+ project_id: proj_b,+ agent_kind: ai_memory_core::AgentKind::ClaudeCode,+ cwd: None,+ actor_user: None,+ })+ .await+ .unwrap();+ // Its page is a straggler, still in proj_a.+ let page_path = format!("sessions/{sid}.md");+ let page_id = write_and_index(&wiki, &tmp, ws, proj_a, &page_path, "session summary").await;+ // A control page that actually lives in proj_b, so proj_b's on-disk+ // directory exists and its walk is complete (not partial) — without+ // this, an absent proj_b directory would make the walk itself hit+ // `NotFound` and get skipped by the PARTIAL-walk guard (design item+ // #3) instead of by the sessions exclusion this test means to+ // isolate. Confirmed by disabling the sessions exclusion locally+ // while writing this test: without a proj_b control page the test+ // still passed for the wrong reason (partial walk), and only+ // failed as expected once this control page made the walk complete.+ write_and_index(&wiki, &tmp, ws, proj_b, "control.md", "always here").await;++ // The from==to re-home: the wiki sees no scope change and skips the+ // file step, but the store still sweeps the straggler page into+ // proj_b (see `ops::move_session`'s `rehome` branch).+ let outcome = wiki+ .move_session_page(+ sid,+ (ws, proj_b),+ (ws, proj_b),+ ai_memory_store::PagesMode::Move,+ None,+ None,+ )+ .await+ .unwrap();+ assert_eq!(+ outcome.summary.page_versions_moved, 1,+ "precondition: the page re-stamp must have actually run"+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj_b, page_path.as_str().to_string())+ .await+ .unwrap(),+ Some(page_id),+ "precondition: the page row now lives in proj_b",+ );+ assert!(+ !proj_dir(&tmp, ws, proj_b).join(&page_path).exists(),+ "precondition: proj_b has no file for it — the pre-existing move-session bug",+ );++ let mut streaks = MissingStreaks::new();+ for pass in 0..3 {+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();+ assert_eq!(+ stats.tombstoned_missing, 0,+ "pass {pass}: a session page must never be reconcile-delete candidate"+ );+ }+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj_b, page_path.as_str().to_string())+ .await+ .unwrap(),+ Some(page_id),+ "the orphaned-by-move-session page must survive untouched",+ );+ }}diff --git a/crates/ai-memory-wiki/src/wiki.rs b/crates/ai-memory-wiki/src/wiki.rsindex b529cd8f..f332b11d 100644--- a/crates/ai-memory-wiki/src/wiki.rs+++ b/crates/ai-memory-wiki/src/wiki.rs@@ -184,6 +184,12 @@ pub struct Wiki {/// write after the scope's first — the store query and the two manifest/// reads happen once per scope per process, never per page.manifested_scopes: Arc<Mutex<HashSet<(WorkspaceId, ProjectId)>>>,+ /// `[maintenance] reconcile_tombstones_deleted_pages` (#929), read once by+ /// `Config::load` and threaded here via [`Wiki::with_reconcile_tombstones_deleted_pages`].+ /// OFF by default: with this `false`, the watcher's reconcile pass never+ /// tombstones a page whose file vanished — deletions still require+ /// `ai-memory delete-page`, unchanged from before this feature existed.+ reconcile_tombstones_deleted_pages: bool,}/// Key uniquely identifying a page for per-path write serialization.@@ -211,6 +217,7 @@ impl Wiki {mutation_lock: Arc::new(RwLock::new(())),page_locks: Arc::new(std::sync::Mutex::new(HashMap::new())),manifested_scopes: Arc::new(Mutex::new(HashSet::new())),+ reconcile_tombstones_deleted_pages: false,})}@@ -245,6 +252,24 @@ impl Wiki {self}+ /// Opt into the watcher reconcile pass's tombstone-on-missing-file safety+ /// net (`[maintenance] reconcile_tombstones_deleted_pages`, #929). `false`+ /// (the default from [`Wiki::new`]) leaves reconcile's behavior exactly+ /// as it was before this feature: deletions require `ai-memory+ /// delete-page`.+ #[must_use]+ pub fn with_reconcile_tombstones_deleted_pages(mut self, enabled: bool) -> Self {+ self.reconcile_tombstones_deleted_pages = enabled;+ self+ }++ /// Whether the watcher's reconcile pass may tombstone a page whose file+ /// disappeared. See [`Wiki::with_reconcile_tombstones_deleted_pages`].+ #[must_use]+ pub(crate) fn reconcile_tombstones_deleted_pages(&self) -> bool {+ self.reconcile_tombstones_deleted_pages+ }+/// Replace the default built-in-only sanitizer with one carrying/// the operator's `[sanitize].extra_patterns` + `allowlist`.#[must_use]@@ -910,6 +935,108 @@ impl Wiki {.await}+ /// `(PageId, PagePath)` of every `is_latest = 1` page in a scope, for the+ /// watcher's reconcile-delete safety net (#929) to snapshot before it+ /// walks the wiki tree.+ ///+ /// # Errors+ /// Returns [`WikiError`] when the store reader is unavailable, or on a+ /// store error.+ pub(crate) async fn latest_page_ids(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ ) -> WikiResult<Vec<(PageId, PagePath)>> {+ let reader = self.store_reader.as_ref().ok_or_else(|| {+ ai_memory_wiki_error("reconcile-delete candidate snapshot requires a store reader")+ })?;+ reader+ .latest_page_ids(workspace_id, project_id)+ .await+ .map_err(Into::into)+ }++ /// Tombstone the expected latest page after the watcher's reconcile pass+ /// found its file missing on two consecutive passes (opt-in, #929;+ /// `[maintenance] reconcile_tombstones_deleted_pages`).+ ///+ /// Deliberately does NOT go through [`Wiki::remove_page_locked`]: unlike+ /// `evict_page_if_latest` (decay) or `delete_page_if_latest` (explicit+ /// delete), this is a background safety net reacting to a file that is+ /// already gone, not a user-initiated action, so it never runs the+ /// BLOCKING admission gate (`AdmissionChain::notify`/`authorize`) —+ /// nothing gets to refuse it — and it never touches the filesystem —+ /// there is nothing on disk to quarantine or remove. It DOES+ /// fire-and-forget the chain's non-blocking observer/mirror webhooks on+ /// success (#929 review, S6), same as `remove_page_locked`'s post-write+ /// `dispatch_async`, so a mirror learns about the tombstone instead of+ /// silently diverging from the source of truth.+ ///+ /// Re-checks both the store's latest-id and the file's actual absence+ /// while holding the exclusive mutation guard, immediately before+ /// writing — the final gate design item #6 calls for, closing the window+ /// between the reconcile pass's second "missing" observation and this+ /// call (e.g. a concurrent `write_page` recreating the path).+ ///+ /// # Errors+ /// Returns [`WikiError`] when the store reader is unavailable, or on a+ /// filesystem or store error.+ pub(crate) async fn tombstone_missing_page_if_latest(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: &PagePath,+ expected_latest_id: PageId,+ ) -> WikiResult<bool> {+ let _guard = self.mutation_lock.write().await;+ self.ensure_project_workspace(workspace_id, project_id)+ .await?;+ let reader = self+ .store_reader+ .as_ref()+ .ok_or_else(|| ai_memory_wiki_error("reconcile tombstone requires a store reader"))?;+ let current = reader+ .latest_page_id_by_ids(workspace_id, project_id, path.as_str().to_string())+ .await?;+ if current != Some(expected_latest_id) {+ return Ok(false);+ }+ let abs = self.abs_path(workspace_id, project_id, path);+ if abs.try_exists()? {+ // The file came back (e.g. a slow atomic-save pattern beyond the+ // two-pass window, or a deliberate rewrite mid-check). Leave the+ // page alone; the next reconcile tick reindexes it normally.+ return Ok(false);+ }+ let tombstoned = self+ .writer+ .soft_delete_for_reconcile_if_latest(+ workspace_id,+ project_id,+ path.clone(),+ expected_latest_id,+ )+ .await?;+ if tombstoned && let Some(chain) = &self.admission_chain {+ // Non-blocking observers/mirrors only (#929 review, S6) — never+ // the blocking `notify`/`authorize` gate: this is a background+ // safety net, not a user-initiated delete, so nothing gets to+ // refuse it. But a mirror that never hears about the tombstone+ // silently diverges from the source of truth, which is a real+ // correctness issue too; fire-and-forget informs it the same way+ // `remove_page_locked` does for `evict_page_if_latest` and+ // `delete_page_if_latest`, just skipping their blocking half.+ let mut ctx = AdmissionContext {+ op: AdmissionOp::Delete,+ ..Default::default()+ };+ self.resolve_admission_names(workspace_id, project_id, &mut ctx)+ .await;+ chain.dispatch_async(Some(path.as_str()), &serde_json::Value::Null, "", &ctx);+ }+ Ok(tombstoned)+ }+async fn remove_page_locked(&self,workspace_id: WorkspaceId,@@ -4325,6 +4452,360 @@ mod tests {);}+ // --- #929: reconcile-delete tombstone op (`tombstone_missing_page_if_latest`) ---++ #[tokio::test]+ async fn reconcile_tombstone_refuses_a_stale_latest_id() {+ let tmp = TempDir::new().unwrap();+ let (store, wiki, ws, proj) = scoped(&tmp).await;+ let path = PagePath::new("sessions/stale-reconcile.md").unwrap();+ let stale = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "old candidate",+ serde_json::json!({}),+ ))+ .await+ .unwrap();+ let current = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "new current body",+ serde_json::json!({}),+ ))+ .await+ .unwrap();+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();++ assert!(+ !wiki+ .tombstone_missing_page_if_latest(ws, proj, &path, stale)+ .await+ .unwrap(),+ "a stale expected id must leave the current version untouched"+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(current),+ );+ }++ /// Design item #6: the final recheck, immediately before acting, must+ /// refuse when the file has actually come back — even though the caller+ /// (the watcher's reconcile pass) only calls this after two consecutive+ /// "missing" observations, a page can still be recreated in the narrow+ /// window between the second observation and this call.+ #[tokio::test]+ async fn reconcile_tombstone_refuses_when_the_file_still_exists() {+ let tmp = TempDir::new().unwrap();+ let (store, wiki, ws, proj) = scoped(&tmp).await;+ let path = PagePath::new("sessions/present-reconcile.md").unwrap();+ let page_id = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "still here",+ serde_json::json!({}),+ ))+ .await+ .unwrap();++ assert!(+ !wiki+ .tombstone_missing_page_if_latest(ws, proj, &path, page_id)+ .await+ .unwrap(),+ "the file exists on disk; the final recheck must refuse to tombstone it"+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(page_id),+ );+ assert!(wiki.abs_path(ws, proj, &path).exists());+ }++ /// The tombstone op is a background safety net reacting to a file that is+ /// already gone, not a user-initiated delete: it must never run the+ /// BLOCKING admission gate, which could refuse it. A webhook configured+ /// to reject every `Delete` event proves this by NOT blocking the+ /// tombstone — if `tombstone_missing_page_if_latest` went through the+ /// same path as `evict_page_if_latest`/`delete_page_if_latest`, this+ /// webhook would turn the call into an error+ /// (`rejecting_delete_admission_leaves_decay_candidate_live` shows+ /// exactly that for the decay path). It DOES still fire-and-forget any+ /// non-blocking observer/mirror webhook on success — see+ /// `reconcile_tombstone_dispatches_non_blocking_observer_webhook` below+ /// (#929 review, S6).+ #[tokio::test]+ async fn reconcile_tombstone_does_not_dispatch_admission_webhook() {+ use axum::Router;+ use axum::http::StatusCode;+ use axum::routing::post;+ use tokio::net::TcpListener;++ let app = Router::new().route(+ "/reject",+ post(|| async { (StatusCode::FORBIDDEN, "must never be called") }),+ );+ let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();+ let addr = listener.local_addr().unwrap();+ tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });++ let tmp = TempDir::new().unwrap();+ let (store, wiki, ws, proj) = scoped(&tmp).await;+ let path = PagePath::new("sessions/webhook-reconcile.md").unwrap();+ let page_id = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "vanishing",+ serde_json::json!({}),+ ))+ .await+ .unwrap();+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();+ let wiki = wiki.with_admission_chain(+ AdmissionChain::new(vec![WebhookConfig {+ name: "reconcile-guard".into(),+ url: format!("http://{addr}/reject"),+ timeout_ms: 1_000,+ failure_policy: FailurePolicy::Reject,+ events: vec![AdmissionOp::Delete],+ blocking: true,+ }])+ .unwrap(),+ );++ assert!(+ wiki.tombstone_missing_page_if_latest(ws, proj, &path, page_id)+ .await+ .unwrap(),+ "no admission dispatch means the rejecting webhook is never consulted"+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ None,+ "the page is no longer latest",+ );+ }++ /// S6 (#929 review): the tombstone op skips the BLOCKING admission gate+ /// (proven above), but a mirror that never hears about the tombstone at+ /// all would silently diverge from the source of truth — a real+ /// correctness issue on its own. A non-blocking observer webhook must+ /// still be fired, fire-and-forget, on a successful tombstone.+ #[tokio::test]+ async fn reconcile_tombstone_dispatches_non_blocking_observer_webhook() {+ use axum::Router;+ use axum::extract::State;+ use axum::http::StatusCode;+ use axum::routing::post;+ use std::sync::Arc;+ use std::sync::atomic::{AtomicBool, Ordering};+ use tokio::net::TcpListener;++ let observed = Arc::new(AtomicBool::new(false));+ let app = Router::new()+ .route(+ "/observe",+ post(|State(flag): State<Arc<AtomicBool>>| async move {+ flag.store(true, Ordering::SeqCst);+ StatusCode::OK+ }),+ )+ .with_state(observed.clone());+ let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();+ let addr = listener.local_addr().unwrap();+ tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });++ let tmp = TempDir::new().unwrap();+ let (_store, wiki, ws, proj) = scoped(&tmp).await;+ let path = PagePath::new("notes/mirror-reconcile.md").unwrap();+ let page_id = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "vanishing",+ serde_json::json!({}),+ ))+ .await+ .unwrap();+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();+ let wiki = wiki.with_admission_chain(+ AdmissionChain::new(vec![WebhookConfig {+ name: "mirror-observer".into(),+ url: format!("http://{addr}/observe"),+ timeout_ms: 1_000,+ failure_policy: FailurePolicy::Ignore,+ events: vec![AdmissionOp::Delete],+ blocking: false,+ }])+ .unwrap(),+ );++ assert!(+ wiki.tombstone_missing_page_if_latest(ws, proj, &path, page_id)+ .await+ .unwrap()+ );++ // `dispatch_async` fires the request off the caller's path; give it a+ // bounded window to land instead of asserting immediately.+ let mut delivered = false;+ for _ in 0..50 {+ if observed.load(Ordering::SeqCst) {+ delivered = true;+ break;+ }+ tokio::time::sleep(std::time::Duration::from_millis(20)).await;+ }+ assert!(+ delivered,+ "the non-blocking observer webhook must be notified of the tombstone"+ );+ }++ /// The op never touches the filesystem (design item #7: there is nothing+ /// to touch — the file is already gone). Proven by the project+ /// directory's file listing being byte-for-byte identical before and+ /// after the call: no quarantine artifact, no recreated file, nothing.+ #[tokio::test]+ async fn reconcile_tombstone_never_touches_the_filesystem() {+ let tmp = TempDir::new().unwrap();+ let (_store, wiki, ws, proj) = scoped(&tmp).await;+ let path = PagePath::new("notes/vanishing-fs.md").unwrap();+ let page_id = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "gone soon",+ serde_json::json!({}),+ ))+ .await+ .unwrap();+ let proj_dir = wiki.project_root(ws, proj);+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();++ let listing_before = list_dir_recursive(&proj_dir);+ assert!(+ wiki.tombstone_missing_page_if_latest(ws, proj, &path, page_id)+ .await+ .unwrap()+ );+ let listing_after = list_dir_recursive(&proj_dir);+ assert_eq!(+ listing_before, listing_after,+ "the tombstone op must not create, remove, or rename any file"+ );+ }++ /// `restore-page`/wiki history depend on this: a reconcile tombstone is+ /// `is_latest = 0` + `superseded_at`, mirroring decay eviction exactly —+ /// and is picked up by the SAME aged-tombstone sweep decay eviction uses+ /// (`hard_delete_after_days`, tier/pin-agnostic), not exempted from it.+ /// Proven by fetching the row back through the same query that sweep+ /// uses ([`ai_memory_store::ReaderPool::decay_tombstones_before`]): if+ /// the row were gone, or if only `is_latest` had flipped without+ /// `superseded_at`, it would not show up there. What actually protects a+ /// reconcile tombstone from that sweep permanently destroying a+ /// self-healed false positive is `upsert_page_in_tx`'s resurrection path+ /// (`ops.rs`): if the file comes back, the new version re-links onto+ /// this chain and clears `superseded_at`, which is covered separately by+ /// `reconcile_tombstone_resurrects_instead_of_orphaning_on_recreate`.+ #[tokio::test]+ async fn reconcile_tombstone_is_a_soft_delete_visible_to_normal_retention() {+ let tmp = TempDir::new().unwrap();+ let (store, wiki, ws, proj) = scoped(&tmp).await;+ let path = PagePath::new("notes/tombstone-shape.md").unwrap();+ let page_id = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "about to vanish",+ serde_json::json!({}),+ ))+ .await+ .unwrap();+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();++ assert!(+ wiki.tombstone_missing_page_if_latest(ws, proj, &path, page_id)+ .await+ .unwrap()+ );++ // is_latest = 0: no longer the scope's current version.+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ None,+ );+ // superseded_at set (not just is_latest): the row surfaces through the+ // exact query the decay retention sweep uses to find eligible aged+ // tombstones — proving it is a real tombstone, not merely hidden, and+ // that hard cleanup is left to that same normal retention path.+ let far_future = jiff::Timestamp::now().as_microsecond() + 365 * 24 * 3_600 * 1_000_000;+ let tombstones = store+ .reader+ .decay_tombstones_before(ws, proj, far_future)+ .await+ .unwrap();+ assert!(+ tombstones.iter().any(|t| t.id == page_id && t.path == path),+ "the reconcile tombstone must be indistinguishable from a decay tombstone to the \+ retention sweep: {tombstones:?}",+ );+ }++ fn list_dir_recursive(root: &Path) -> Vec<String> {+ fn walk(dir: &Path, root: &Path, out: &mut Vec<String>) {+ let Ok(entries) = std::fs::read_dir(dir) else {+ return;+ };+ for entry in entries.flatten() {+ let path = entry.path();+ if path.is_dir() {+ walk(&path, root, out);+ } else {+ out.push(+ path.strip_prefix(root)+ .unwrap_or(&path)+ .to_string_lossy()+ .into_owned(),+ );+ }+ }+ }+ let mut out = Vec::new();+ walk(root, root, &mut out);+ out.sort();+ out+ }+#[tokio::test]async fn decay_cleanup_refuses_to_reindex_a_symlinked_recreation() {let tmp = TempDir::new().unwrap();diff --git a/docs/install.md b/docs/install.mdindex 1ab0fbab..aa01ba8a 100644--- a/docs/install.md+++ b/docs/install.md@@ -2217,6 +2217,26 @@ or overdue job runs once after a bounded startup delay. Failed runs are notrecorded as successful and retry after that bounded delay. Embedding backfillremains opt-in and keeps its interval-only behavior (no startup catch-up).+`reconcile_tombstones_deleted_pages` (default `false`, also under+`[maintenance]`) is a separate, experimental opt-in: it lets the watcher's+own 30s reconcile pass (not the scheduled jobs above) tombstone an+OKF-imported content page (session summary pages are excluded — see+`docs/okf.md`) whose file has disappeared from disk, after it has been+missing on two consecutive passes and survived a circuit breaker that+refuses to act when more than `max(3, 50%)` of a scope's candidate pages+look missing at once, or when a walk finds nothing at all. The tombstone is+soft (`is_latest = 0` + `superseded_at`, the same shape decay eviction uses)+and is picked up by the same aged-tombstone hard-delete sweep — it is not+exempt from it. What actually protects it: a reconcile tombstone is never+itself destroyed while its chain has no successor; if the file returns, the+new version re-links to the tombstoned chain instead of starting fresh, so+nothing is orphaned for that sweep to destroy. It never runs the blocking+admission gate (nothing can refuse it), but does fire-and-forget any+non-blocking observer/mirror webhook. With it off (the default), reconcile's+behavior is unchanged: a deleted file still requires `ai-memory delete-page`.+See+`docs/okf.md` for the full design.+---## Bootstrap mid-projectdiff --git a/docs/okf.md b/docs/okf.mdindex 25611239..ce48242c 100644--- a/docs/okf.md+++ b/docs/okf.md@@ -143,6 +143,57 @@ so any later rewrite of an affected page (a restore, a hand edit, a`stale_after` that differs from `expires_at` was not derived byai-memory and is left alone.+## Reconcile-delete safety net (opt-in, #929)++The watcher's 30s reconcile pass only ever reindexes create/modify events by+default — a page whose file disappears from disk stays indexed until+`ai-memory delete-page` removes it explicitly. `[maintenance]+reconcile_tombstones_deleted_pages` (default `false`) opts into a background+safety net that closes that gap, designed to be safe even though it adds an+automatic mutation to a background loop:++- A page's file must be observed missing on **two consecutive** reconcile+ passes (not one) before anything happens, and the check is re-verified+ against the live filesystem immediately before acting.+- Only pages the walk could ever have returned are eligible — `bootstrap.md`,+ `_meta.md`, and `_pending/` sidecars are never candidates, and a project+ whose walk hit `NotFound` (a vanished subdirectory, a whole project+ directory gone mid-pass) contributes no "missing" evidence for that pass.+ `sessions/*.md` pages are excluded too, for an unrelated reason: a+ same-workspace `move-session` re-home can leave a correct DB row with no+ file at its new scope (a separate, pre-existing bug in `move-session`'s+ file relocation — tracked as a follow-up, not fixed here). This mechanism+ is meant for OKF-imported content pages only.+- A circuit breaker refuses to act on a whole scope when more than `max(3,+ 50%)` of its candidate pages look missing in one pass, OR when a+ non-partial walk finds nothing at all in a scope that has candidates — the+ latter catches the "directory exists but came back empty" mount-failure+ signature for scopes too small to ever trip the percentage math. Either+ shape is far more likely a walk/mount problem (an unmounted volume, a git+ checkout mid-walk) than genuine mass deletion, and is logged at `warn`.+- The action itself is a soft tombstone (`is_latest = 0` + `superseded_at`),+ the same row shape decay eviction already uses, and is picked up by the+ exact same aged-tombstone hard-delete sweep decay eviction uses+ (`hard_delete_after_days`, tier/pin-agnostic) — it is NOT exempt from that+ sweep. The precise guarantee: **a reconcile tombstone is never itself+ destroyed while its chain has no successor; if the file returns, the new+ version re-links to the tombstoned chain instead of starting fresh, so+ nothing is orphaned for the 180-day sweep to destroy.** Concretely, a fresh+ write at a tombstoned path (`upsert_page_in_tx`) supersedes the tombstoned+ row via `supersedes` and clears its `superseded_at`, turning it into an+ ordinary, protected supersession-chain member — the same mechanism that+ already keeps normal edit history off that sweep. It also never dispatches+ the BLOCKING admission gate (nothing gets to refuse it — this is a+ background safety net reacting to an already-vanished file, not a+ user-initiated delete) and never touches the filesystem (there is nothing+ to write; the file is already gone); it DOES fire-and-forget any+ non-blocking observer/mirror webhook on success, so a mirror learns about+ the tombstone instead of silently diverging. `restore-page` and the page's+ version history remain intact.+- With the flag off (the default), behavior is byte-identical to before this+ feature existed: nothing new is logged, and reconcile never mutates the+ store.+## Tests (each with a control that must fail on a broken build)- Round-trip: page → OKF file on disk → parsed back identical.@@ -161,8 +212,8 @@ so any later rewrite of an affected page (a restore, a hand edit, adirectory and letting the watcher (or `reindex`) ingest them IS theimport path. Overwriting an already-imported concept file gets its newversion embedded the same way a brand-new file does — no manual- `ai-memory embed` needed. Deleting one does not yet remove it from the- index: the watcher only reconciles create/modify events, so a deleted- concept file still needs an explicit `ai-memory delete-page` (tracked in- #929).+ `ai-memory embed` needed. Deleting one does not remove it from the index by+ default: the watcher only reconciles create/modify events, so a deleted+ concept file needs an explicit `ai-memory delete-page` unless the opt-in+ safety net below is enabled (#929).- Retrieval regression: LongMemEval baseline re-run; no material drop.diff --git a/docs/security-boundaries.md b/docs/security-boundaries.mdindex 935d8d42..2f02b04a 100644--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@ -46,6 +46,7 @@ boundary not yet built.| 10b | Session purge is scope+owner-bound (no cross-session/project over-delete); `purge-session`'s `dry_run` always wins over `confirm`, same as `purge-project` | `ai-memory-store/src/ops.rs` `purge_session` — selection scoped to `(workspace_id, project_id)` and keyed on this session's own `summary_page_id` **or** `path='sessions/<sid>.md'` + `json_extract(frontmatter_json,'$.session_id')=<sid>` (frontmatter owner, not the recursive latest-chain); `in_scope==0 → NotFound` fail-closed; whole op in one transaction; `admin.rs` `handle_purge_session` checks `req.dry_run` unconditionally, before `req.confirm` | `ops.rs` `purge_session_leaves_a_sibling_session_in_the_same_project_intact`, `…refuses_a_session_from_another_project_and_deletes_nothing`, `…refuses_a_session_from_another_workspace`, `…does_not_delete_an_identically_pathed_page_in_another_project`, `…removes_older_summary_versions_without_deleting_prior_manual_page` (#862); `admin_purge.rs` `purge_session_confirm_true_and_dry_run_true_still_only_previews` — `{"confirm": true, "dry_run": true}` deletes nothing; `ops.rs` `purge_session_dry_run_changes_nothing` — every table a session purge touches, the `purged_sessions` tombstone, and the `audit_log` row are identical before/after a `PurgeMode::Preview` run; `ops.rs` `purge_session_counts_collateral_damage_in_another_project` and `admin_purge.rs` `purge_session_dry_run_and_confirmed_purge_both_report_collateral_damage_in_another_project` — purging session S collaterally deletes an observation, and orphans a handoff's session reference, in a *different* project (via `S`'s own id cascading), and both the preview and the confirmed report count it; `admin_purge.rs` `purge_session_dry_run_refuses_a_session_outside_its_named_scope` — the same session previewed under a different real project (same workspace) or a different real workspace whose project shares the name both 404 with no counts in the body, with the session's own scope as the 200 control | STRONG || 10c | Ledger reclaim is content-gated, not filename-gated (invariant #16) | `ai-memory-store/src/ops.rs` `reclaim_ledger_versions` — a candidate row must pass `ai_memory_core::log_ledger::body_opens_with_log_ledger` on its own `body`, and only `is_latest=0 AND superseded_at IS NULL` rows are eligible, so decay-owned rows stay with `forget-sweep`; confirm gate in `admin.rs` `handle_reclaim_ledger_versions` | `reclaim_ledger_versions.rs` `a_prose_page_wearing_the_ledger_name_keeps_its_whole_version_chain`, `an_ordinary_page_chain_in_another_project_is_untouched`, `a_decay_tombstone_stays_with_the_sweep_that_owns_it`; `admin_reclaim_ledger_versions.rs` `deleting_without_confirm_is_refused_and_changes_nothing` (with dry-run and confirmed controls) | STRONG || 10d | Workspace delete's own row counts are scope-bound to `workspace_id` (a workspace's *own* rows are never over- or under-counted by a sibling workspace's), its cross-workspace observation/handoff cascade is reported by both preview and confirm rather than prevented (same as `purge-project` one level down), the non-empty guard fails closed without `force`, and `delete-workspace`'s `dry_run` always wins | `ai-memory-store/src/ops.rs` `delete_workspace` — every direct count and the `DELETE FROM workspaces` itself are scoped by `workspace_id = ?1`; the two `collateral_*` counts are scoped by `workspace_id != ?1` so a row is counted as one or the other but never both; `projects_deleted > 0 && !force → WorkspaceNotEmpty` fail-closed, and a missing workspace is `NotFound`, both checked identically under `PurgeMode::Preview` and `PurgeMode::Commit`; whole op in one transaction; `admin.rs` `handle_delete_workspace` checks `req.dry_run` unconditionally, before building the confirmed path, mirroring `reclaim-ledger-versions`/`purge-project`; root-only gate is the shared `/admin/*` multi-user auth ladder, not a separate check | `admin_move.rs` `delete_workspace_dry_run_never_counts_a_different_workspaces_rows` — previewing workspace A counts only its own project/page/session/observation/handoff rows, not workspace B's (which holds two like-named projects and its own full row set), and a session living in B that stamps an observation directly into A is counted in A's own `observations_deleted`, never in `collateral_observations_deleted` (the two counts are mutually exclusive by construction); `admin_move.rs` `delete_workspace_force_true_and_dry_run_true_still_only_previews` — `{"force": true, "dry_run": true}` deletes nothing; `ops.rs` `delete_workspace_dry_run_changes_nothing` — every table's row count, including `workspaces` itself and the `purged_scopes` tombstone (the two tables a confirmed delete's own transaction writes to), is identical before/after a `PurgeMode::Preview` run; `ops.rs` `delete_workspace_dry_run_and_confirmed_delete_both_report_collateral_damage_in_another_workspace` and `admin_move.rs` of the same name — deleting workspace W collaterally deletes an observation, and orphans a handoff's session reference, in a *different* workspace (via `sessions` cascading out of W), and both the preview and the confirmed response count it identically — the cascade itself is reported, not blocked, by either; `admin_move.rs` `delete_workspace_dry_run_refuses_non_empty_without_force` / `delete_workspace_dry_run_nonexistent_workspace_returns_404` — same `409`/`404` a confirmed delete would give; `admin.rs` `multiuser_delete_workspace_dry_run_rejects_db_user_and_anonymous` — root reaches the preview handler (404 control), DB user gets 403, anonymous gets 401 | STRONG |+| 10e | Reconcile-tombstone of a disk-deleted page is opt-in, soft, scope-bound, and false-positive-guarded (invariant #16, #929 delete-half) | `ai-memory-wiki/src/watcher.rs` — off by default (`[maintenance] reconcile_tombstones_deleted_pages`); a page is tombstoned only after its file is observed missing on two consecutive passes (streak resets if the path's `PageId` changed), re-verified against the live filesystem immediately before acting, with reserved/indexed-but-unwalked paths (`bootstrap.md`, `_meta.md`, `_pending/`, `sessions/*.md`) excluded and a circuit breaker that skips a whole scope when `> max(3, 50%)` of its pages look missing or a non-partial walk finds nothing; `ai-memory-store/src/ops.rs` `soft_delete_for_reconcile_if_latest` is a soft tombstone (`is_latest=0` + `superseded_at`, decay's shape) through the single writer actor in one transaction — never a filesystem write — scoped to the page's own `(workspace_id, project_id)` | `watcher.rs` `reconcile_delete_disabled_by_default_never_tombstones_a_missing_page`, `reconcile_delete_cross_project_isolation` (project A's deletion never tombstones project B's same-path page), `reconcile_tombstone_refuses_a_stale_latest_id`, `reconcile_tombstone_refuses_when_the_file_still_exists`, `reconcile_tombstone_never_touches_the_filesystem`; `ops.rs` `reconcile_tombstone_resurrects_instead_of_orphaning_on_recreate` | STRONG || 11a | Hook backpressure (202/429) + bounded fan-out (invariant #5) | `ai-memory-hooks/src/router.rs` semaphore→429, 202 immediately, `MAX_HOOK_BATCH_ITEMS`, bounded LRU limiter | `router.rs` `handle_hook_returns_429_when_ingest_saturated`, `ingest_rate_limiter_is_bounded` | STRONG || 11b | Capture exclusions drop before storage | `ai-memory-hooks` `capture_policy.rs` `inspect`→`Drop` (before semaphore/spawn), including shell commands whose arguments name an ignored path (`match_command`); generated OpenCode/OMP/Pi/OpenClaw integrations mirror it in `render_shared.rs` `ts_capture_policy_v1` (`captureMatchCommand`) | `capture_policy.rs` per-agent `…honors_exclusions` tests, `shell_fixture_vectors` (shared `capture-policy.json` `shell` drop/keep vectors: tool aliases incl. OpenClaw/Devin `exec`, `workdir`, glob directories, Windows paths), `shell_tool_shapes_of_every_adapter_honor_exclusions`, `fixture_vectors` (incl. TS-adapter `bash`/`exec` vectors); `router.rs` `capture_protocol_shell_decisions_survive_server_reinspection`; `hook.rs` `shell_command_reading_an_ignored_path_is_dropped_before_spool`; `render_shared.rs` `generated_capture_policy_v1_node_runtime_evidence` (runs the same fixture sections against the emitted TypeScript; `#[ignore]` locally, run with `--ignored` under Node 24 by the Linux CI test job) | STRONG || 11c | Capture hook ≤200ms budget (invariant #5) | `hooks/_lib.sh` capture path `curl --max-time 0.2` (context-fetch 1.0s and background drain 2.0s are separate, larger-budget paths) | none (shell-script timeout; hard to unit-test) — watch on any capture-path change | WATCH |