Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(wiki): reconcile can tombstone pages whose file disappeared (opt-in)

Maxsuel Einstein5a9ce4ac53df
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 54083f34..cd43917c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- The watcher's reconcile pass can now optionally tombstone a wiki page whose
+ file has disappeared from disk, closing part of the gap left by #929 (the
+ watcher only reconciled create/modify events, so a deleted file stayed
+ indexed until `ai-memory delete-page` ran explicitly). Opt-in and default
+ `false` (`[maintenance] reconcile_tombstones_deleted_pages`), so existing
+ installs see no behavior change unless they turn it on. When enabled, a
+ page's file must be observed missing on two consecutive 30s reconcile
+ passes, re-verified against the live filesystem immediately before acting;
+ a circuit breaker refuses to act on a whole scope when more than `max(3,
+ 50%)` of its candidate pages look missing in one pass, or when a
+ non-partial walk finds nothing at all (either shape is far more likely a
+ walk/mount problem than genuine mass deletion); and reserved/indexed-but-
+ unwalked paths (`bootstrap.md`, `_meta.md`, `_pending/` sidecars, and
+ `sessions/*.md` — excluded because a same-workspace `move-session` re-home
+ can leave one with a correct row and no file, a separate pre-existing bug)
+ are never candidates. The action is a soft tombstone (`is_latest = 0` +
+ `superseded_at`, the same shape decay eviction already uses), picked up by
+ the SAME aged-tombstone hard-delete sweep decay eviction uses — it is not
+ exempt from it. What actually protects it: it is never destroyed while its
+ chain has no successor; if the file returns, the new version re-links onto
+ the tombstoned chain instead of starting a fresh, disconnected one, so
+ nothing is left orphaned for that sweep to destroy. It never runs the
+ blocking admission gate (nothing can refuse it), but does fire-and-forget
+ any non-blocking observer/mirror webhook on success. `restore-page` and
+ version history remain intact. See `docs/okf.md` for the full design.
+ (#964)
- `[consolidation] input_token_safety_margin` (float, default `0.8`, validated
to `(0.0, 1.0]`) scales the approximate char-count input budget. The
`max_input_tokens` budget uses a flat chars-per-token heuristic that
diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rs
index f219c45f..cad09650 100644
--- a/crates/ai-memory-cli/src/commands/serve.rs
+++ b/crates/ai-memory-cli/src/commands/serve.rs
@@ -1049,7 +1049,13 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {
// Reader attached unconditionally: admission name-resolution uses it
// when a chain is configured, and the startup scope-manifest backfill
// (below) always needs it to enumerate scopes.
- .with_store_reader(store.reader.clone());
+ .with_store_reader(store.reader.clone())
+ // `[maintenance] reconcile_tombstones_deleted_pages` (#929), read once
+ // by `Config::load` above and threaded through here — the one place
+ // this flag is consulted outside `Config`, per invariant #1.
+ .with_reconcile_tombstones_deleted_pages(
+ config.maintenance.reconcile_tombstones_deleted_pages,
+ );
// Attach the admission webhook chain (operator-configured via
// `[[admission_webhooks]]` in config.toml or `AI_MEMORY_ADMISSION_WEBHOOKS__N__*`
// env vars). Empty config = no chain attached, zero overhead. The store
@@ -3611,6 +3617,7 @@ mod tests {
forget_sweep_interval_secs: 1,
lint_interval_secs: 1,
embedding_backfill_interval_secs: 1,
+ reconcile_tombstones_deleted_pages: false,
},
AutoImproveSettings::default(),
store.reader.clone(),
@@ -3649,12 +3656,14 @@ mod tests {
forget_sweep_interval_secs: 0,
lint_interval_secs: 60,
embedding_backfill_interval_secs: 0,
+ reconcile_tombstones_deleted_pages: false,
},
MaintenanceSettings {
enabled: true,
forget_sweep_interval_secs: 60,
lint_interval_secs: 0,
embedding_backfill_interval_secs: 0,
+ reconcile_tombstones_deleted_pages: false,
},
] {
let (_tmp, store, wiki, _ws, _first, _second) = two_project_wiki().await;
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index fe70c356..e0bb216e 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -1243,6 +1243,32 @@ pub struct MaintenanceSettings {
/// Interval for embedding backfill. `0` disables this job.
/// Defaults to off because it may call a paid provider.
pub embedding_backfill_interval_secs: u64,
+ /// Opt-in reconcile-delete safety net (#929). When `true`, the watcher's
+ /// 30s reconcile pass tombstones (`is_latest = 0` + `superseded_at`, never
+ /// a filesystem touch or a BLOCKING admission dispatch) an OKF-imported
+ /// content page (session summary pages are excluded — a same-workspace
+ /// `move-session` re-home can leave one with a correct row and no file, a
+ /// separate pre-existing bug) whose file has been missing on two
+ /// consecutive passes, after a circuit breaker that refuses to act on a
+ /// scope where more than `max(3, 50%)` of its candidate pages look
+ /// missing at once, or where a non-partial walk finds nothing at all
+ /// (see `ai_memory_wiki::watcher::reconcile_delete_breaker_threshold`) —
+ /// either shape is far more likely a walk/mount problem (an unmounted
+ /// volume, a git checkout mid-walk) than genuine deletions.
+ ///
+ /// The tombstone is NOT exempt from the aged-tombstone hard-delete sweep
+ /// (`hard_delete_after_days`) — the actual guarantee is narrower: a
+ /// reconcile tombstone is never itself destroyed while its chain has no
+ /// successor. If the file returns, the new version re-links to the
+ /// tombstoned chain (`ops::upsert_page_in_tx`'s resurrection path)
+ /// instead of starting fresh, so nothing is orphaned for that sweep to
+ /// destroy.
+ ///
+ /// Defaults to `false`: with this off, reconcile's behavior is
+ /// byte-identical to before this feature existed — deletions still
+ /// require `ai-memory delete-page`, and nothing new is logged. Doc:
+ /// `docs/okf.md`, `docs/install.md`.
+ pub reconcile_tombstones_deleted_pages: bool,
}
impl Default for MaintenanceSettings {
@@ -1252,6 +1278,7 @@ impl Default for MaintenanceSettings {
forget_sweep_interval_secs: 86_400,
lint_interval_secs: 86_400,
embedding_backfill_interval_secs: 0,
+ reconcile_tombstones_deleted_pages: false,
}
}
}
diff --git a/crates/ai-memory-consolidate/src/sweep.rs b/crates/ai-memory-consolidate/src/sweep.rs
index acb4bdf3..52c4f149 100644
--- a/crates/ai-memory-consolidate/src/sweep.rs
+++ b/crates/ai-memory-consolidate/src/sweep.rs
@@ -15,9 +15,16 @@
//! contradiction is visible before the delete lands.
//!
//! Hard-delete pass cleans up tombstones older than
-//! `hard_delete_after_days` and their supersession ancestry. A page recreated
-//! at the same path is a separate live chain and is preserved. Ordinary
-//! supersession rows are safe because only decay writes `superseded_at`.
+//! `hard_delete_after_days` and their supersession ancestry. Ordinary
+//! supersession rows are safe because only a tombstone (decay eviction, or
+//! the watcher's opt-in reconcile-delete safety net, #929 — the two are
+//! deliberately indistinguishable here) writes `superseded_at`. A page
+//! rewritten at a tombstoned path re-links onto that chain via `supersedes`
+//! and clears the old row's `superseded_at` (`ops::upsert_page_in_tx`'s
+//! resurrection path), so the old version becomes an ordinary, protected
+//! chain member instead of an orphan this pass would otherwise hard-delete —
+//! this is what keeps a reconcile tombstone's false positive from being
+//! permanently destroyed once the file reappears.
//!
//! Observation prune pass (opt-in, disabled unless
//! `ObservationRetention::days > 0`) deletes raw observations older than the
diff --git a/crates/ai-memory-consolidate/tests/suite/lifecycle.rs b/crates/ai-memory-consolidate/tests/suite/lifecycle.rs
index a46f7fed..30b59a6d 100644
--- a/crates/ai-memory-consolidate/tests/suite/lifecycle.rs
+++ b/crates/ai-memory-consolidate/tests/suite/lifecycle.rs
@@ -1047,7 +1047,18 @@ async fn hard_delete_preserves_a_page_recreated_at_the_same_path() {
.await
.unwrap();
assert!(cleanup.evicted.is_empty());
- assert_eq!(cleanup.hard_deleted, 2);
+ // #929 (B1 review fix): `reindex_page`'s upsert now resurrects a tombstoned
+ // chain instead of starting a disconnected one — see
+ // `ops::upsert_page_in_tx`'s resurrection path and
+ // `ops::reconcile_tombstone_resurrects_instead_of_orphaning_on_recreate`.
+ // The recreated file re-links onto `second` via `supersedes` and clears
+ // `second`'s `superseded_at`, which is what actually protects the old
+ // chain from `hard_delete_decayed_page_chain`: that function's root query
+ // requires `superseded_at IS NOT NULL`, and a resurrected row no longer
+ // has it set — exactly the same protection ordinary supersession-chain
+ // members already had. So this cleanup pass no longer has any tombstone
+ // root left to walk from, and hard-deletes nothing.
+ assert_eq!(cleanup.hard_deleted, 0);
assert!(wiki.abs_path(ws, proj, &path).exists());
let recreated = store
.reader
@@ -1072,7 +1083,35 @@ async fn hard_delete_preserves_a_page_recreated_at_the_same_path() {
|row| row.get(0),
)
.unwrap();
- assert_eq!(old_rows, 0);
+ assert_eq!(
+ old_rows, 2,
+ "the old chain must survive, resurrected rather than deleted"
+ );
+ let supersedes: Option<Vec<u8>> = conn
+ .query_row(
+ "SELECT supersedes FROM pages WHERE id = ?1",
+ params![recreated.as_bytes()],
+ |row| row.get(0),
+ )
+ .unwrap();
+ assert_eq!(
+ supersedes.as_deref(),
+ Some(&second.as_bytes()[..]),
+ "the recreated version must supersede the tombstoned chain"
+ );
+ let second_superseded_at: Option<i64> = conn
+ .query_row(
+ "SELECT superseded_at FROM pages WHERE id = ?1",
+ params![second.as_bytes()],
+ |row| row.get(0),
+ )
+ .unwrap();
+ assert_eq!(
+ second_superseded_at, None,
+ "resurrection must clear the old tombstone's superseded_at"
+ );
+ drop(conn);
+
assert!(
store
.reader
@@ -1083,6 +1122,25 @@ async fn hard_delete_preserves_a_page_recreated_at_the_same_path() {
.any(|hit| hit.id == recreated),
"the recreated page and its FTS entry must survive old-chain cleanup"
);
+
+ // A further cleanup pass must not find anything to hard-delete either:
+ // the resurrected chain no longer has a tombstone root at all.
+ let second_pass = run_sweep(
+ &store.reader,
+ &store.writer,
+ Some(&wiki),
+ ws,
+ proj,
+ &DecayParams {
+ cold_threshold: 0.0,
+ hard_delete_after_days: 0,
+ ..DecayParams::default()
+ },
+ false,
+ )
+ .await
+ .unwrap();
+ assert_eq!(second_pass.hard_deleted, 0);
}
fn ws_dir_for(tmp: &TempDir, ws: WorkspaceId, proj: ProjectId) -> std::path::PathBuf {
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index 4f1d3195..d223eee9 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -1463,13 +1463,48 @@ pub(crate) fn upsert_page_in_tx(
existing,
});
}
+ // Tombstone resurrection (#929). A tombstone is any `is_latest = 0` row
+ // with `superseded_at` set — decay eviction and the reconcile-delete
+ // safety net both mark one this way, and they are deliberately
+ // indistinguishable here (docs/okf.md): both are "the store believes
+ // this path has no file", and both deserve the same resurrection
+ // behavior if a fresh write proves that belief wrong.
+ //
+ // Without this, a fresh write at a tombstoned path started a brand-new,
+ // disconnected chain (`supersedes = NULL`), leaving the OLD chain an
+ // unreachable orphan. For a reconcile tombstone that "no chain has a
+ // successor" is exactly what made it eligible for
+ // `hard_delete_decayed_page_chain`'s recursive walk — so a false
+ // positive that self-healed (the file came back) still got permanently
+ // hard-deleted ~`hard_delete_after_days` later, contradicting the
+ // safety net's own "never destroys anything" premise.
+ //
+ // The fix re-links the new version onto the tombstoned chain via
+ // `supersedes`, exactly like an ordinary edit supersedes the page it
+ // replaces, and clears the old row's `superseded_at`. That second step
+ // is what actually protects it: `decay_tombstones_before` and
+ // `hard_delete_decayed_page_chain` both key their eligibility on
+ // `superseded_at IS NOT NULL` — an ordinary supersession-chain member
+ // never sets it, which is the existing mechanism that keeps normal
+ // history off the aged-tombstone sweep. Clearing it here puts the
+ // resurrected row under that exact same protection, rather than
+ // inventing a new one. `valid_to` is left as the tombstone wrote it
+ // (the instant the page actually went missing), not bumped to `now`.
+ let tombstone = most_recent_tombstone_at_path(tx, page)?;
+ if let Some(tombstone_id) = &tombstone {
+ tx.execute(
+ "UPDATE pages SET superseded_at = NULL WHERE id = ?1",
+ params![tombstone_id],
+ )?;
+ }
let frontmatter_str = stamped_frontmatter(conformed, now)?;
let new_id = PageId::new();
tx.execute(
"INSERT INTO pages \
(id, workspace_id, project_id, path, path_search, title, tier, body, body_sha256, \
- frontmatter_json, is_latest, pinned, author_id, created_at, updated_at, expires_at, valid_from, compacted_at) \
- VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, 1, ?11, ?12, ?13, ?13, ?14, ?13, ?15)",
+ frontmatter_json, is_latest, supersedes, pinned, author_id, created_at, updated_at, \
+ expires_at, valid_from, compacted_at) \
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, 1, ?11, ?12, ?13, ?14, ?14, ?15, ?14, ?16)",
params![
new_id.as_bytes(),
page.workspace_id.as_bytes(),
@@ -1481,6 +1516,7 @@ pub(crate) fn upsert_page_in_tx(
page.body,
body_sha256.as_slice(),
frontmatter_str,
+ tombstone,
i64::from(page.pinned),
page.author_id.map(|id| id.as_bytes().to_vec()),
now,
@@ -1494,7 +1530,11 @@ pub(crate) fn upsert_page_in_tx(
insert_evidence_in_tx(tx, &new_id, &page.evidence, now)?;
audit(
tx,
- "create_page",
+ if tombstone.is_some() {
+ "resurrect_tombstoned_page"
+ } else {
+ "create_page"
+ },
Some(page.workspace_id.as_bytes()),
Some(page.project_id.as_bytes()),
Some(new_id.as_bytes()),
@@ -1506,6 +1546,32 @@ pub(crate) fn upsert_page_in_tx(
Ok(new_id)
}
+/// The most recently tombstoned row at `page`'s path, if any: the newest
+/// `is_latest = 0` version with `superseded_at` set (decay eviction or the
+/// reconcile-delete safety net — see the resurrection comment in
+/// [`upsert_page_in_tx`]). At most one row can match at a time in practice
+/// (resurrecting one clears its `superseded_at`), but the ordering makes the
+/// choice well-defined even if that invariant is ever violated.
+fn most_recent_tombstone_at_path(
+ tx: &rusqlite::Transaction<'_>,
+ page: &NewPage,
+) -> StoreResult<Option<Vec<u8>>> {
+ Ok(tx
+ .query_row(
+ "SELECT id FROM pages \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 \
+ AND is_latest = 0 AND superseded_at IS NOT NULL \
+ ORDER BY superseded_at DESC LIMIT 1",
+ params![
+ page.workspace_id.as_bytes(),
+ page.project_id.as_bytes(),
+ page.path.as_str(),
+ ],
+ |row| row.get::<_, Vec<u8>>(0),
+ )
+ .optional()?)
+}
+
/// The live page this create would share a file with on a case-folding or
/// normalizing filesystem, if any.
///
@@ -2841,6 +2907,82 @@ pub fn soft_delete_for_decay_if_latest(
Ok(affected != 0)
}
+/// Tombstone the expected latest page whose on-disk file the watcher's
+/// reconcile pass found missing on two consecutive passes (see #929 — an
+/// opt-in safety net, `[maintenance] reconcile_tombstones_deleted_pages`).
+///
+/// Same shape as [`soft_delete_for_decay_if_latest`]: the full identity +
+/// latest-id check runs in the same transaction as the write, so a page
+/// rewritten after the caller last observed it is left untouched. Sets
+/// `is_latest = 0` + `superseded_at`/`valid_to`, exactly like decay eviction.
+/// The caller (`Wiki::tombstone_missing_page_if_latest`) does not dispatch
+/// the admission/mirror webhook chain (this is a background safety net
+/// reacting to an already-vanished file, not a user-initiated delete) and
+/// never touches the filesystem (there is nothing to touch — the file is
+/// already gone).
+///
+/// The precise durability guarantee (this tombstone is NOT exempt from the
+/// aged-tombstone hard-delete sweep — it uses the exact same
+/// `hard_delete_after_days`, tier/pin-agnostic path decay eviction does,
+/// [`decay_tombstones_before`], [`hard_delete_decayed_page_chain`]): a
+/// reconcile tombstone is never itself destroyed while its chain has no
+/// successor. If the file comes back, `upsert_page_in_tx`'s resurrection
+/// path re-links the new version onto this tombstoned chain via
+/// `supersedes` and clears `superseded_at`, so the false positive's history
+/// becomes an ordinary, protected supersession-chain member — nothing is
+/// left orphaned for the sweep to destroy. Only a chain that is genuinely
+/// never rewritten again is eventually hard-deleted, same as any other aged
+/// decay tombstone.
+pub fn soft_delete_for_reconcile_if_latest(
+ conn: &mut Connection,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ path: &PagePath,
+ expected_latest_id: PageId,
+) -> StoreResult<bool> {
+ let now = Timestamp::now().as_microsecond();
+ let tx = conn.transaction()?;
+ let affected = tx.execute(
+ "UPDATE pages \
+ SET is_latest = 0, superseded_at = ?1, valid_to = ?1 \
+ WHERE id = ?2 \
+ AND workspace_id = ?3 \
+ AND project_id = ?4 \
+ AND path = ?5 \
+ AND is_latest = 1",
+ params![
+ now,
+ expected_latest_id.as_bytes(),
+ workspace_id.as_bytes(),
+ project_id.as_bytes(),
+ path.as_str(),
+ ],
+ )?;
+ if affected != 0 {
+ // Same entity-timeline closure as decay eviction (issue #656): an
+ // open window on a tombstoned page must not resurrect retired
+ // knowledge under `as_of`.
+ tx.execute(
+ "UPDATE entity_page_links SET superseded_at = ?1 \
+ WHERE page_id = ?2 AND superseded_at IS NULL",
+ params![now, expected_latest_id.as_bytes()],
+ )?;
+ audit(
+ &tx,
+ "soft_delete_for_reconcile",
+ Some(workspace_id.as_bytes()),
+ Some(project_id.as_bytes()),
+ Some(expected_latest_id.as_bytes()),
+ // Background reconcile safety net, not a user-attributable edit —
+ // same convention as decay's `None` author.
+ None,
+ now,
+ )?;
+ }
+ tx.commit()?;
+ Ok(affected != 0)
+}
+
/// Delete every version of a page (by path) from the index. Used when the
/// wiki file is removed (`Wiki::delete_page`): the watcher does not handle
/// file deletions, so the derived rows must be dropped explicitly or the
@@ -9977,6 +10119,139 @@ pub(crate) mod tests {
assert!(supersedes.is_some(), "new row must link to its predecessor");
}
+ /// B1 (#929 review): a page whose file disappears, gets reconcile-
+ /// tombstoned, and then reappears must NOT start a brand-new,
+ /// disconnected chain. Before this fix `upsert_page_in_tx`'s "no
+ /// existing `is_latest = 1` row" branch always inserted with
+ /// `supersedes = NULL`, leaving the tombstoned chain an orphan that
+ /// `hard_delete_decayed_page_chain` would permanently destroy once
+ /// `hard_delete_after_days` passed — even though the "deletion" was a
+ /// false positive that had already self-healed. Exercises the exact
+ /// mechanism end to end: tombstone -> file returns (a fresh
+ /// `upsert_page`) -> hard-delete sweep with a cutoff equivalent to
+ /// `hard_delete_after_days = 0` -> the prior (tombstoned) version must
+ /// still exist and be reachable via the chain.
+ #[test]
+ fn reconcile_tombstone_resurrects_instead_of_orphaning_on_recreate() {
+ let (_tmp, mut conn, ws, proj) = fresh_db();
+ let path = PagePath::new("concepts/flaky.md").unwrap();
+ let original_id =
+ upsert_page(&mut conn, &page(ws, proj, path.as_str(), "v1 body")).unwrap();
+
+ // The file vanishes and the watcher's reconcile-delete safety net
+ // tombstones it (mirrors `Wiki::tombstone_missing_page_if_latest`).
+ assert!(
+ soft_delete_for_reconcile_if_latest(&mut conn, ws, proj, &path, original_id).unwrap(),
+ "precondition: the tombstone must take"
+ );
+ let tombstoned_superseded_at: Option<i64> = conn
+ .query_row(
+ "SELECT superseded_at FROM pages WHERE id = ?1",
+ params![original_id.as_bytes()],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert!(
+ tombstoned_superseded_at.is_some(),
+ "precondition: the tombstone sets superseded_at"
+ );
+
+ // The false positive self-heals: the file comes back and gets
+ // reindexed, which is a fresh `upsert_page` call at the same path.
+ let resurrected_id = upsert_page(
+ &mut conn,
+ &page(ws, proj, path.as_str(), "v1 body, but back"),
+ )
+ .unwrap();
+ assert_ne!(original_id, resurrected_id);
+
+ // The new version must re-link onto the tombstoned chain...
+ let supersedes: Option<Vec<u8>> = conn
+ .query_row(
+ "SELECT supersedes FROM pages WHERE id = ?1",
+ params![resurrected_id.as_bytes()],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(
+ supersedes.as_deref(),
+ Some(&original_id.as_bytes()[..]),
+ "the resurrected version must supersede the tombstoned chain, not start fresh"
+ );
+ // ...and the old row must no longer look like an eligible tombstone:
+ // clearing `superseded_at` is what actually protects it, since
+ // `hard_delete_decayed_page_chain`'s root query does not care who
+ // points at a row via `supersedes`, only whether that row itself
+ // still has `superseded_at IS NOT NULL`.
+ let cleared_superseded_at: Option<i64> = conn
+ .query_row(
+ "SELECT superseded_at FROM pages WHERE id = ?1",
+ params![original_id.as_bytes()],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(
+ cleared_superseded_at, None,
+ "resurrection must clear the old row's superseded_at"
+ );
+
+ // Simulate the forget sweep's hard-delete pass with
+ // `hard_delete_after_days = 0` (an immediate cutoff: `now`).
+ let cutoff_now = jiff::Timestamp::now().as_microsecond();
+ let candidates = {
+ let mut stmt = conn
+ .prepare(
+ "SELECT id FROM pages WHERE workspace_id = ?1 AND project_id = ?2 \
+ AND path = ?3 AND is_latest = 0 AND superseded_at IS NOT NULL \
+ AND superseded_at <= ?4",
+ )
+ .unwrap();
+ let rows = stmt
+ .query_map(
+ params![ws.as_bytes(), proj.as_bytes(), path.as_str(), cutoff_now],
+ |r| r.get::<_, Vec<u8>>(0),
+ )
+ .unwrap();
+ rows.collect::<Result<Vec<_>, _>>().unwrap()
+ };
+ assert!(
+ candidates.is_empty(),
+ "a resurrected chain's old version must not even be a hard-delete candidate: {candidates:?}"
+ );
+ let deleted = hard_delete_decayed_page_chain(
+ &mut conn,
+ ws,
+ proj,
+ &path,
+ original_id,
+ Some(resurrected_id),
+ cutoff_now,
+ )
+ .unwrap();
+ assert_eq!(
+ deleted, 0,
+ "the sweep must refuse to delete a chain member that is no longer a tombstone root"
+ );
+
+ // The prior version is still there and reachable via the chain.
+ let still_exists: i64 = conn
+ .query_row(
+ "SELECT COUNT(*) FROM pages WHERE id = ?1",
+ params![original_id.as_bytes()],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(still_exists, 1, "the tombstoned version must survive");
+ let reachable_via_chain: Vec<u8> = conn
+ .query_row(
+ "SELECT supersedes FROM pages WHERE id = ?1",
+ params![resurrected_id.as_bytes()],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(reachable_via_chain, original_id.as_bytes());
+ }
+
/// Idempotency: re-upserting the same body should NOT create a
/// second row. The watcher's reconciliation calls upsert_page on
/// every file on every tick — without this, a quiet repo would
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index eb33734a..966066e4 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -4379,12 +4379,53 @@ impl ReaderPool {
.await
}
+ /// Return `(PageId, PagePath)` for every `is_latest = 1` page under a
+ /// scope, cheaper than [`decay_candidates`](Self::decay_candidates) when
+ /// only identity is needed. Used by the watcher's reconcile-delete
+ /// safety net (#929) to snapshot the store's view of "pages that should
+ /// have a file on disk" before walking the tree, so a page written via
+ /// the API mid-walk is never mistaken for one the walk simply hasn't
+ /// reached yet.
+ ///
+ /// # Errors
+ /// Propagates any SQL or pool error.
+ pub async fn latest_page_ids(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ ) -> StoreResult<Vec<(PageId, PagePath)>> {
+ self.with_conn(move |conn| {
+ let mut stmt = conn.prepare(
+ "SELECT id, path FROM pages \
+ WHERE workspace_id = ?1 AND project_id = ?2 AND is_latest = 1",
+ )?;
+ let rows = stmt.query_map(
+ params![workspace_id.as_bytes(), project_id.as_bytes()],
+ |row| Ok((row.get::<_, Vec<u8>>(0)?, row.get::<_, String>(1)?)),
+ )?;
+ let mut out = Vec::new();
+ for row in rows {
+ let (id, path) = row?;
+ out.push((PageId::from_slice(&id)?, PagePath::new(path)?));
+ }
+ Ok(out)
+ })
+ .await
+ }
+
/// Return decay tombstones old enough for permanent cleanup.
///
- /// `superseded_at` is written only by the forget-sweep eviction path, so
- /// it is the tombstone discriminator regardless of whether a rewritten
- /// page head also has a `supersedes` ancestor. The caller still routes
- /// each result through the wiki layer before deleting its version chain.
+ /// `superseded_at` is written by the forget-sweep eviction path AND by
+ /// the watcher's opt-in reconcile-delete safety net (#929) — the two are
+ /// deliberately indistinguishable here (docs/okf.md), so it remains the
+ /// tombstone discriminator regardless of whether a rewritten page head
+ /// also has a `supersedes` ancestor. A tombstone whose path was rewritten
+ /// after it was marked (the false-positive self-healing, or an ordinary
+ /// decay-then-recreate) has its `superseded_at` cleared by
+ /// `ops::upsert_page_in_tx`'s resurrection path and so drops out of this
+ /// query — only a chain with no successor ever reaches here. The caller
+ /// still routes each result through the wiki layer before deleting its
+ /// version chain.
///
/// # Errors
/// Propagates any SQL or pool error.
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index ab9df7f0..56568c68 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -340,6 +340,13 @@ pub(crate) enum WriteCmd {
expected_latest_id: PageId,
reply: oneshot::Sender<StoreResult<bool>>,
},
+ SoftDeleteForReconcileIfLatest {
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ path: PagePath,
+ expected_latest_id: PageId,
+ reply: oneshot::Sender<StoreResult<bool>>,
+ },
HardDeleteDecayedPageChain {
workspace_id: WorkspaceId,
project_id: ProjectId,
@@ -1836,6 +1843,30 @@ impl WriterHandle {
rx.await.map_err(|_| StoreError::WriterClosed)?
}
+ /// Tombstone the expected latest page whose file the watcher's reconcile
+ /// pass found missing on two consecutive passes (opt-in, see #929).
+ ///
+ /// # Errors
+ /// Returns [`StoreError::WriterClosed`] or propagates SQL errors.
+ pub async fn soft_delete_for_reconcile_if_latest(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ path: PagePath,
+ expected_latest_id: PageId,
+ ) -> StoreResult<bool> {
+ let (tx, rx) = oneshot::channel();
+ self.send(WriteCmd::SoftDeleteForReconcileIfLatest {
+ workspace_id,
+ project_id,
+ path,
+ expected_latest_id,
+ reply: tx,
+ })
+ .await?;
+ rx.await.map_err(|_| StoreError::WriterClosed)?
+ }
+
/// Permanently delete one eligible decay tombstone and its ancestry chain.
/// The expected latest-page state is checked in the same transaction so a
/// page recreated at the same path cannot be removed accidentally.
@@ -3570,6 +3601,22 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
);
send_or_warn(reply, result, "soft_delete_for_decay_if_latest");
}
+ WriteCmd::SoftDeleteForReconcileIfLatest {
+ workspace_id,
+ project_id,
+ path,
+ expected_latest_id,
+ reply,
+ } => {
+ let result = ops::soft_delete_for_reconcile_if_latest(
+ &mut conn,
+ workspace_id,
+ project_id,
+ &path,
+ expected_latest_id,
+ );
+ send_or_warn(reply, result, "soft_delete_for_reconcile_if_latest");
+ }
WriteCmd::HardDeleteDecayedPageChain {
workspace_id,
project_id,
diff --git a/crates/ai-memory-wiki/src/watcher.rs b/crates/ai-memory-wiki/src/watcher.rs
index 7b56f5f7..3a0693de 100644
--- a/crates/ai-memory-wiki/src/watcher.rs
+++ b/crates/ai-memory-wiki/src/watcher.rs
@@ -28,11 +28,12 @@
//! External writes drive store updates; internal writes drive disk +
//! store updates via [`Wiki::write_page`].
+use std::collections::{HashMap, HashSet};
use std::path::Path;
use std::str::FromStr;
use std::time::Duration;
-use ai_memory_core::{PagePath, ProjectId, WorkspaceId};
+use ai_memory_core::{PageId, PagePath, ProjectId, WorkspaceId};
use notify::{EventKind, RecursiveMode};
use notify_debouncer_full::{DebounceEventResult, Debouncer, RecommendedCache, new_debouncer_opt};
use tokio::sync::mpsc;
@@ -159,6 +160,13 @@ async fn run_loop(
let mut consecutive_failures: u32 = 0;
const DEGRADED_AFTER: u32 = 5;
+ // Reconcile-delete safety net (#929, opt-in — see `MissingStreaks`):
+ // per-`(workspace, project, path)` count of consecutive passes a
+ // candidate page's file has looked missing. Lives here, not on `Wiki`,
+ // for the same reason `consecutive_failures` does: it is this task's own
+ // pass-to-pass memory, not shared server state.
+ let mut missing_streaks: MissingStreaks = HashMap::new();
+
loop {
tokio::select! {
biased;
@@ -170,7 +178,7 @@ async fn run_loop(
handle_event(&wiki, event).await;
}
_ = tick.tick() => {
- match reconcile(&wiki).await {
+ match reconcile(&wiki, &mut missing_streaks).await {
Ok(_) => {
if consecutive_failures > 0 {
tracing::info!(
@@ -368,15 +376,258 @@ pub(crate) struct ReconcileStats {
/// Session pages left on disk by a purge whose file cleanup failed, and
/// deliberately not re-indexed (#701).
pub skipped_purged_sessions: usize,
+ /// Pages tombstoned by the reconcile-delete safety net (#929) after their
+ /// file was missing on two consecutive passes. Always `0` unless
+ /// `[maintenance] reconcile_tombstones_deleted_pages` is enabled.
+ pub tombstoned_missing: usize,
+ /// Scopes where the reconcile-delete circuit breaker tripped this pass
+ /// (design item #4) — too many candidates looked missing at once, so
+ /// nothing in that scope was tombstoned or counted toward a streak.
+ pub circuit_broken_scopes: usize,
+}
+
+/// Per-`(workspace, project, path)` streak of consecutive reconcile passes a
+/// candidate page's file has looked missing, plus the [`PageId`] last
+/// observed missing (so a path superseded by a new version between passes
+/// restarts its streak rather than crediting the old version's absence to
+/// the new one). Design item #5 requires two consecutive passes before a
+/// tombstone is even attempted; this is that memory, owned by the watcher's
+/// `run_loop` across ticks the same way `consecutive_failures` is.
+type MissingStreaks = HashMap<(WorkspaceId, ProjectId, PagePath), (PageId, u32)>;
+
+/// Reconcile-delete circuit breaker (design item #4): absolute floor. Even a
+/// scope where every one of a handful of candidates looks "missing" must
+/// clear this many before the breaker can trip — a two- or three-page
+/// project isn't permanently blocked from ever tombstoning a real deletion
+/// just because 100% of a tiny candidate set matches.
+const RECONCILE_DELETE_BREAKER_MIN_CANDIDATES: usize = 3;
+
+/// Reconcile-delete circuit breaker (design item #4): fraction floor, applied
+/// alongside the absolute floor above. Half (or more) of a scope's candidate
+/// pages vanishing inside one 30s reconcile window is far more likely a
+/// walk/mount problem — an unmounted volume, a git checkout mid-walk, a
+/// project directory being renamed — than that many genuine deletions
+/// landing in the same pass, so the pass is treated as suspect and nothing
+/// in that scope is tombstoned this round.
+const RECONCILE_DELETE_BREAKER_FRACTION: f64 = 0.5;
+
+/// The missing-candidate count strictly above which the circuit breaker
+/// trips, for a scope with `total_candidates` walk-eligible pages tracked
+/// before the walk ran this pass.
+fn reconcile_delete_breaker_threshold(total_candidates: usize) -> usize {
+ #[allow(clippy::cast_precision_loss)]
+ let total = total_candidates as f64;
+ let fraction_floor = (total * RECONCILE_DELETE_BREAKER_FRACTION).ceil();
+ #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)]
+ let fraction_floor = fraction_floor as usize;
+ fraction_floor.max(RECONCILE_DELETE_BREAKER_MIN_CANDIDATES)
+}
+
+/// Whether `path` is a shape [`walk_markdown`] could ever have returned,
+/// applied to a DB row instead of a filesystem entry (design item #1).
+///
+/// `bootstrap.md`, `_meta.md`, and `_pending/` sidecars are indexed rows the
+/// walk structurally never returns, so their absence from a walk's page list
+/// proves nothing about whether their file still exists — without this
+/// filter every one of them would look "missing" on every single pass.
+///
+/// More conservative than the walk's own ledger check
+/// ([`is_reserved_page_file`]), which reads a candidate ledger file's first
+/// line to tell a real hook ledger from an ordinary page that merely happens
+/// to be named `log.md`. Here the file is gone by definition — there is
+/// nothing left to read — so any `log.md` / `log-YYYY-MM.md`-shaped path is
+/// excluded from candidacy unconditionally, erring toward never tombstoning
+/// rather than guessing.
+///
+/// `sessions/<id>.md` pages are excluded too, for an unrelated reason (#929
+/// review, B2): a same-workspace `move-session` re-home updates the store
+/// row's `(workspace_id, project_id)` but deliberately does not move the
+/// file (the wiki layer skips that on purpose — see `Wiki::move_session_page`
+/// and its callers), so a session summary page can have a perfectly correct,
+/// live DB row with no file at its *new* scope. That is a separate,
+/// pre-existing bug in `move-session`'s file relocation, not fixed here —
+/// but this mechanism would otherwise read that gap as "the file was
+/// deleted" and tombstone a real, live page. This safety net is meant for
+/// OKF-imported content pages (concepts, decisions, gotchas — the paths
+/// #929's own repro used), never session-summary pages, so the whole shape
+/// is excluded until `move-session` is fixed to relocate the file too
+/// (tracked as a follow-up).
+fn could_have_been_walked(path: &PagePath) -> bool {
+ if is_pending_path(path) {
+ return false;
+ }
+ if is_manifest_filename(path) || path.as_str() == "bootstrap.md" {
+ return false;
+ }
+ if crate::wiki::session_id_for_page(path).is_some() {
+ return false;
+ }
+ if crate::ledger::is_log_ledger_filename(path) {
+ return false;
+ }
+ true
+}
+
+/// The reconcile-delete safety net (#929, design items #1–#6) for one
+/// project scope, run after that scope's walk has completed. No-op unless
+/// `[maintenance] reconcile_tombstones_deleted_pages` is enabled.
+///
+/// `pre_walk_snapshot` must have been taken from the store *before*
+/// `walked`'s walk started (design item #2) — that ordering is what makes a
+/// page written via the API mid-walk safe: it is either absent from the
+/// snapshot (never a candidate) or present in `walked` (found on disk), and
+/// either way it can never look like a deletion.
+#[allow(clippy::too_many_arguments)]
+async fn reconcile_missing_pages(
+ wiki: &Wiki,
+ ws: WorkspaceId,
+ proj: ProjectId,
+ pre_walk_snapshot: &[(PageId, PagePath)],
+ walked: &HashSet<PagePath>,
+ walk_partial: bool,
+ streaks: &mut MissingStreaks,
+ stats: &mut ReconcileStats,
+) {
+ if pre_walk_snapshot.is_empty() {
+ return;
+ }
+
+ // A page found on disk this pass can never contribute to a future
+ // tombstone from an earlier streak — clear it regardless of what happens
+ // to the rest of this scope below. This runs even on a partial walk
+ // (S1, #929 review): only ABSENCE evidence from a partial walk is
+ // suspect (design item #3, below) — presence evidence a partial walk DID
+ // manage to observe is exactly as trustworthy as any other pass's, and
+ // losing it would needlessly re-arm a streak the page had already
+ // cleared.
+ for (_, path) in pre_walk_snapshot.iter().filter(|(_, p)| walked.contains(p)) {
+ streaks.remove(&(ws, proj, path.clone()));
+ }
+
+ if walk_partial {
+ // Design item #3: a `NotFound` anywhere in this scope's walk (a
+ // vanished subdirectory, or the whole project directory) means some
+ // branch of the tree was unreadable rather than every page under it
+ // having actually been deleted one by one. Skip NEW missing-page
+ // evidence for this pass — streaks for pages already missing before
+ // this pass are left exactly as they were (neither incremented nor
+ // cleared), so a later, complete pass still needs its own two
+ // consecutive observations before anything is tombstoned.
+ debug!(
+ workspace = %ws,
+ project = %proj,
+ "reconcile-delete: partial walk this pass (a directory was unreadable); not \
+ treating this scope's absent pages as new deletion evidence",
+ );
+ return;
+ }
+
+ let missing: Vec<&(PageId, PagePath)> = pre_walk_snapshot
+ .iter()
+ .filter(|(_, p)| !walked.contains(p))
+ .collect();
+ if missing.is_empty() {
+ return;
+ }
+
+ // Design item #4: circuit breaker. A suspect pass makes zero progress
+ // toward the two-pass threshold for any of its candidates — not just
+ // "don't tombstone yet", but "don't count this pass at all" — so a
+ // walk/mount blip can never contribute half of the two observations a
+ // real deletion needs.
+ //
+ // S3 (#929 review): an empty `walked` set with a non-empty snapshot is
+ // ALWAYS suspect, regardless of the `max(3, 50%)` math below — that
+ // fraction still has a floor of `RECONCILE_DELETE_BREAKER_MIN_CANDIDATES`,
+ // so a scope with only one or two tracked candidates could have every
+ // single one look missing without ever tripping the percentage breaker.
+ // "the walk found NOTHING at all" is exactly the "directory exists but
+ // came back empty" mount-failure signature (an unmounted volume, a
+ // bind-mount that briefly resolves to an empty stub) that the threshold
+ // alone would miss for a small scope.
+ let threshold = reconcile_delete_breaker_threshold(pre_walk_snapshot.len());
+ let empty_walk_is_suspect = walked.is_empty();
+ if missing.len() > threshold || empty_walk_is_suspect {
+ tracing::warn!(
+ workspace = %ws,
+ project = %proj,
+ missing = missing.len(),
+ total_candidates = pre_walk_snapshot.len(),
+ threshold,
+ empty_walk = empty_walk_is_suspect,
+ "reconcile-delete: circuit breaker tripped — treating this as a walk/mount \
+ problem rather than that many genuine deletions in one pass; nothing in this \
+ scope is tombstoned or counted toward a streak this round",
+ );
+ stats.circuit_broken_scopes += 1;
+ return;
+ }
+
+ for (id, path) in missing {
+ let key = (ws, proj, path.clone());
+ let streak = streaks.entry(key.clone()).or_insert((*id, 0));
+ if streak.0 != *id {
+ // A different page version now occupies this path than the one
+ // whose absence started this streak (e.g. recreated then
+ // rewritten between passes) — restart rather than credit the old
+ // version's absence to the new one.
+ *streak = (*id, 0);
+ }
+ streak.1 += 1;
+ if streak.1 < 2 {
+ continue;
+ }
+ // Missing on two consecutive passes: attempt the tombstone. Whatever
+ // the outcome, drop the streak — a page still missing after this
+ // starts a fresh streak next pass rather than retrying every tick.
+ streaks.remove(&key);
+ match wiki
+ .tombstone_missing_page_if_latest(ws, proj, path, *id)
+ .await
+ {
+ Ok(true) => {
+ stats.tombstoned_missing += 1;
+ tracing::info!(
+ workspace = %ws,
+ project = %proj,
+ path = %path,
+ "reconcile-delete: tombstoned a page whose file was missing on two \
+ consecutive reconcile passes",
+ );
+ }
+ Ok(false) => {
+ debug!(
+ workspace = %ws,
+ project = %proj,
+ path = %path,
+ "reconcile-delete: tombstone candidate skipped (page changed, or its file \
+ reappeared before the final recheck)",
+ );
+ }
+ Err(e) => {
+ warn!(
+ workspace = %ws,
+ project = %proj,
+ path = %path,
+ error = %e,
+ "reconcile-delete: tombstone attempt failed",
+ );
+ }
+ }
+ }
}
-async fn reconcile(wiki: &Wiki) -> WikiResult<ReconcileStats> {
+async fn reconcile(
+ wiki: &Wiki,
+ missing_streaks: &mut MissingStreaks,
+) -> WikiResult<ReconcileStats> {
let root = wiki.root().to_path_buf();
// Walk all per-project subdirectories: <ws_uuid>/<proj_uuid>/
let project_dirs = tokio::task::spawn_blocking(move || walk_project_dirs(&root))
.await
.map_err(|e| WikiError::Io(std::io::Error::other(e.to_string())))??;
+ let tombstones_enabled = wiki.reconcile_tombstones_deleted_pages();
let mut stats = ReconcileStats::default();
for (ws, proj, proj_root) in project_dirs {
// The directory name parses as a valid UUID pair, but that does not
@@ -401,14 +652,45 @@ async fn reconcile(wiki: &Wiki) -> WikiResult<ReconcileStats> {
stats.skipped_orphans += 1;
continue;
}
- let pages = tokio::task::spawn_blocking(move || walk_markdown(&proj_root))
- .await
- .map_err(|e| WikiError::Io(std::io::Error::other(e.to_string())))??;
+
+ // Design item #2: snapshot the store's `is_latest = 1` pages for this
+ // scope BEFORE the walk runs, not after — see `reconcile_missing_pages`.
+ // Filtered to paths the walk could ever return, so reserved/pending
+ // rows are never candidates (design item #1). A snapshot failure
+ // disables the safety net for this scope this pass only; it never
+ // blocks the ordinary reindex below.
+ let pre_walk_snapshot = if tombstones_enabled {
+ match wiki.latest_page_ids(ws, proj).await {
+ Ok(rows) => rows
+ .into_iter()
+ .filter(|(_, p)| could_have_been_walked(p))
+ .collect::<Vec<_>>(),
+ Err(e) => {
+ warn!(
+ workspace = %ws,
+ project = %proj,
+ error = %e,
+ "reconcile-delete: candidate snapshot failed; skipping this scope's \
+ missing-page check this pass",
+ );
+ Vec::new()
+ }
+ }
+ } else {
+ Vec::new()
+ };
+
+ let (pages, walk_partial) =
+ tokio::task::spawn_blocking(move || walk_markdown_partial(&proj_root))
+ .await
+ .map_err(|e| WikiError::Io(std::io::Error::other(e.to_string())))??;
// Once per directory, not once per page: a project accumulates one
// tombstone per purge, and the set is only consulted for the
// `sessions/<id>.md` paths that a session purge could have left behind.
let purged = wiki.purged_sessions(ws, proj).await?;
+ let mut walked: HashSet<PagePath> = HashSet::with_capacity(pages.len());
for path in pages {
+ walked.insert(path.clone());
if crate::wiki::is_purged_session_page(&path, &purged) {
debug!(
path = %path,
@@ -423,6 +705,20 @@ async fn reconcile(wiki: &Wiki) -> WikiResult<ReconcileStats> {
stats.indexed += 1;
}
}
+
+ if tombstones_enabled {
+ reconcile_missing_pages(
+ wiki,
+ ws,
+ proj,
+ &pre_walk_snapshot,
+ &walked,
+ walk_partial,
+ missing_streaks,
+ &mut stats,
+ )
+ .await;
+ }
}
// debug!, not info!: this fires every RECONCILE_INTERVAL regardless of
// activity, so at info it is ~half the default server log (#894). Its
@@ -432,6 +728,8 @@ async fn reconcile(wiki: &Wiki) -> WikiResult<ReconcileStats> {
indexed = stats.indexed,
skipped_orphans = stats.skipped_orphans,
skipped_purged_sessions = stats.skipped_purged_sessions,
+ tombstoned_missing = stats.tombstoned_missing,
+ circuit_broken_scopes = stats.circuit_broken_scopes,
"reconciliation pass complete",
);
Ok(stats)
@@ -532,12 +830,31 @@ fn extract_project_dir_ids(
}
pub(crate) fn walk_markdown(root: &Path) -> WikiResult<Vec<PagePath>> {
+ walk_markdown_partial(root).map(|(pages, _partial)| pages)
+}
+
+/// Like [`walk_markdown`], but also reports whether any directory in the
+/// tree — including `root` itself — could not be read because it no longer
+/// exists.
+///
+/// A `true` second element means the walk is **partial**: some branch
+/// vanished mid-walk (a git checkout, a bind-mount hiccup, a concurrent
+/// project move) rather than every page under it having actually been
+/// deleted one at a time. The reconcile-delete safety net (#929 design item
+/// #3) relies on this to tell those two situations apart — treating a
+/// partial walk's absent pages as deletion evidence would tombstone pages
+/// that are still there, just temporarily unreadable.
+pub(crate) fn walk_markdown_partial(root: &Path) -> WikiResult<(Vec<PagePath>, bool)> {
let mut out = Vec::new();
+ let mut partial = false;
let mut stack = vec![root.to_path_buf()];
while let Some(dir) = stack.pop() {
let read = match std::fs::read_dir(&dir) {
Ok(r) => r,
- Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue,
+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
+ partial = true;
+ continue;
+ }
Err(e) => return Err(WikiError::Io(e)),
};
for entry in read {
@@ -575,7 +892,7 @@ pub(crate) fn walk_markdown(root: &Path) -> WikiResult<Vec<PagePath>> {
}
}
}
- Ok(out)
+ Ok((out, partial))
}
pub(crate) fn is_pending_path(page_path: &PagePath) -> bool {
@@ -770,7 +1087,7 @@ mod tests {
"precondition: the purge made the page unsearchable",
);
- let stats = reconcile(&wiki).await.unwrap();
+ let stats = reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();
assert_eq!(
stats.skipped_orphans, 0,
"a session purge leaves the project row, so the orphan guard passes",
@@ -1004,7 +1321,7 @@ mod tests {
let handle = WatcherHandle::start(wiki.clone()).unwrap();
// Hit reconcile manually instead of waiting 30s.
- reconcile(&wiki).await.unwrap();
+ reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();
let hits = store
.reader
@@ -1040,7 +1357,7 @@ mod tests {
std::fs::write(orphan_dir.join("index.md"), "seeded shell\n").unwrap();
std::fs::write(orphan_dir.join("stale.md"), "orphantoken content\n").unwrap();
- let stats = reconcile(&wiki).await.unwrap();
+ let stats = reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();
assert_eq!(
stats.skipped_orphans, 1,
@@ -1193,7 +1510,7 @@ mod tests {
let handle = WatcherHandle::start(wiki.clone()).unwrap();
// Trigger the reconciliation pass directly.
- reconcile(&wiki).await.unwrap();
+ reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();
// Only `real.md` should land in the index.
let hits = store
@@ -1306,7 +1623,7 @@ mod tests {
.unwrap();
let handle = WatcherHandle::start(wiki.clone()).unwrap();
- reconcile(&wiki).await.unwrap();
+ reconcile(&wiki, &mut MissingStreaks::new()).await.unwrap();
let page_hits = store
.reader
@@ -1506,4 +1823,707 @@ mod tests {
"unchanged content must short-circuit to the same id"
);
}
+
+ // --- #929: reconcile-delete safety net (opt-in tombstone-on-missing-file) ---
+
+ fn proj_dir(tmp: &TempDir, ws: WorkspaceId, proj: ProjectId) -> std::path::PathBuf {
+ tmp.path()
+ .join("wiki")
+ .join(ws.to_string())
+ .join(proj.to_string())
+ }
+
+ /// `setup()` with the reconcile-delete safety net opted in. Kept separate
+ /// from `setup()` so every other watcher test keeps exercising the
+ /// default-off path unchanged.
+ async fn setup_reconcile_delete() -> (TempDir, Store, Wiki, WorkspaceId, ProjectId) {
+ let (tmp, store, wiki, ws, proj) = setup().await;
+ (
+ tmp,
+ store,
+ wiki.with_reconcile_tombstones_deleted_pages(true),
+ ws,
+ proj,
+ )
+ }
+
+ /// Write `body` at `path` under the project directory and index it,
+ /// returning the resulting `PageId`.
+ async fn write_and_index(
+ wiki: &Wiki,
+ tmp: &TempDir,
+ ws: WorkspaceId,
+ proj: ProjectId,
+ path: &str,
+ body: &str,
+ ) -> PageId {
+ let dir = proj_dir(tmp, ws, proj);
+ let full = dir.join(path);
+ std::fs::create_dir_all(full.parent().unwrap()).unwrap();
+ std::fs::write(&full, body).unwrap();
+ wiki.reindex_page(ws, proj, PagePath::new(path).unwrap())
+ .await
+ .unwrap()
+ }
+
+ /// With the feature at its default (off), a page whose file disappears is
+ /// never tombstoned, no matter how many reconcile passes run — reconcile's
+ /// behavior is unchanged from before this feature existed (#929).
+ #[tokio::test]
+ async fn reconcile_delete_disabled_by_default_never_tombstones_a_missing_page() {
+ let (tmp, store, wiki, ws, proj) = setup().await;
+ let path = PagePath::new("gone.md").unwrap();
+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "here for now").await;
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();
+
+ let mut streaks = MissingStreaks::new();
+ for _ in 0..5 {
+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(stats.tombstoned_missing, 0);
+ }
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(id),
+ "deletions still require `ai-memory delete-page` when the feature is off",
+ );
+ }
+
+ /// The core happy path, and the two-pass bite-check: the first pass a
+ /// page's file is found missing must NOT tombstone it (streak = 1); only
+ /// the second CONSECUTIVE pass does (design item #5).
+ #[tokio::test]
+ async fn reconcile_delete_tombstones_only_after_two_consecutive_missing_passes() {
+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let path = PagePath::new("gone.md").unwrap();
+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "here for now").await;
+ // A control page that stays present the whole test, so the scope's
+ // walk is never literally empty (S3, #929 review: an empty walk with
+ // a non-empty snapshot is its own, unconditional breaker trip).
+ write_and_index(&wiki, &tmp, ws, proj, "control.md", "always here").await;
+
+ let mut streaks = MissingStreaks::new();
+ let baseline = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(baseline.tombstoned_missing, 0);
+
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();
+
+ let first_miss = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(
+ first_miss.tombstoned_missing, 0,
+ "one missing pass must not be enough"
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(id),
+ "still latest after only one missing pass",
+ );
+
+ let second_miss = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(
+ second_miss.tombstoned_missing, 1,
+ "two consecutive missing passes must tombstone"
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ None,
+ );
+ }
+
+ /// `could_have_been_walked` (design item #1, plus the B2 sessions
+ /// exclusion) is the path-only filter applied to the DB-side candidate
+ /// snapshot, mirroring the walk's own skip rules. `_pending/` sidecars
+ /// can never even reach it as a candidate — `reindex_page` itself
+ /// refuses to index one (see `refusing to index pending proposal
+ /// sidecar` in `wiki.rs`) — so this unit test is the only way to
+ /// exercise that branch directly.
+ #[test]
+ fn could_have_been_walked_excludes_every_reserved_shape() {
+ let session_path = format!("sessions/{}.md", ai_memory_core::SessionId::new());
+ for excluded in [
+ "_pending",
+ "_pending/draft.md",
+ "_pending/review-bot/notes.md",
+ "bootstrap.md",
+ "_meta.md",
+ "notes/_meta.md",
+ "log.md",
+ "log-2024-01.md",
+ session_path.as_str(),
+ ] {
+ assert!(
+ !could_have_been_walked(&PagePath::new(excluded).unwrap()),
+ "{excluded} must never be a reconcile-delete candidate",
+ );
+ }
+ // "sessions/abc.md" does NOT parse as a `SessionId` (not a UUID), so
+ // it is an ordinary page shape, not a session summary — proving the
+ // exclusion is keyed on `session_id_for_page`, not merely the
+ // `sessions/` prefix.
+ for included in ["notes/plan.md", "sessions/abc.md", "log-review.md"] {
+ assert!(
+ could_have_been_walked(&PagePath::new(included).unwrap()),
+ "{included} is an ordinary page and must remain a candidate",
+ );
+ }
+ }
+
+ /// Reserved/indexed-but-unwalked paths (design item #1) are never
+ /// candidates, even across many consecutive passes with their files gone:
+ /// `bootstrap.md` and a legacy ledger row (`log.md`-shaped). Unlike the
+ /// walk, `reindex_page` does not itself refuse these two paths, so they
+ /// can be indexed rows exactly the way an older store or a direct API
+ /// write could leave them.
+ #[tokio::test]
+ async fn reconcile_delete_reserved_paths_are_never_candidates() {
+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let paths = ["bootstrap.md", "log.md"];
+ let mut ids = Vec::new();
+ for p in paths {
+ let id = write_and_index(&wiki, &tmp, ws, proj, p, "reserved-shaped content").await;
+ ids.push((PagePath::new(p).unwrap(), id));
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(p)).unwrap();
+ }
+
+ let mut streaks = MissingStreaks::new();
+ for _ in 0..5 {
+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(stats.tombstoned_missing, 0);
+ }
+ for (path, id) in ids {
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(id),
+ "{path} must never be treated as deletion evidence",
+ );
+ }
+ }
+
+ /// An atomic-save pattern (delete-then-recreate) must survive: if the
+ /// file is back by the very next pass, the streak clears and nothing is
+ /// ever tombstoned — the two-pass requirement plus this reappearance
+ /// check is what makes that safe beyond the existing debounce window.
+ #[tokio::test]
+ async fn reconcile_delete_atomic_save_pattern_survives() {
+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let path = PagePath::new("flaky.md").unwrap();
+ let id1 = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "v1").await;
+ // Control page (S3, #929 review): without it, the scope's walk goes
+ // fully empty the moment `flaky.md` is missing, and the survival this
+ // test checks for would be explained by the empty-walk breaker
+ // tripping rather than by the streak/reappearance mechanism it's
+ // actually meant to exercise.
+ write_and_index(&wiki, &tmp, ws, proj, "control.md", "always here").await;
+
+ let mut streaks = MissingStreaks::new();
+ reconcile(&wiki, &mut streaks).await.unwrap();
+
+ // Pass N: file momentarily missing (streak = 1).
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();
+ let miss = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(miss.tombstoned_missing, 0);
+
+ // Recreated before the next pass — the "atomic save" completing.
+ let id2 = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "v2").await;
+ assert_ne!(id1, id2, "a content change mints a new version");
+ let recovered = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(recovered.tombstoned_missing, 0);
+
+ // One more pass with the file still present must not retroactively
+ // tombstone anything either — the streak was cleared, not merely paused.
+ let stable = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(stable.tombstoned_missing, 0);
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(id2),
+ );
+ }
+
+ /// S4 (#929 review): the test above recreates the file with DIFFERENT
+ /// content, so it never actually proves the streak was CLEARED by the
+ /// SAME page reappearing — a new `PageId` alone (via the `streak.0 !=
+ /// *id` restart in `reconcile_missing_pages`) would make that test pass
+ /// even if the "found in `walked`" clearing loop were deleted entirely.
+ /// This test forces the identical-content path: missing (streak = 1) ->
+ /// present again with the SAME body (so `upsert_page`'s content
+ /// short-circuit returns the SAME id, exercising the presence-clears-
+ /// streak branch specifically) -> missing again (must restart at
+ /// streak = 1, not continue to 2) -> not tombstoned; a further pass
+ /// still missing then reaches streak = 2 and IS tombstoned — proving two
+ /// FRESH consecutive misses were required, not two misses ever observed.
+ #[tokio::test]
+ async fn reconcile_delete_streak_clears_on_identical_content_reappearance() {
+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let path = PagePath::new("flaky-identical.md").unwrap();
+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "same body always").await;
+ // Control page (S3, #929 review — see the other test's comment).
+ write_and_index(&wiki, &tmp, ws, proj, "control.md", "always here").await;
+
+ let mut streaks = MissingStreaks::new();
+ reconcile(&wiki, &mut streaks).await.unwrap();
+
+ // Miss #1.
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();
+ let first_miss = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(first_miss.tombstoned_missing, 0, "streak = 1, not enough");
+
+ // Reappears with IDENTICAL content: `upsert_page`'s sha256
+ // short-circuit returns the SAME id, so this exercises "found in
+ // `walked`" clearing a streak, not the `streak.0 != *id` restart.
+ let same_id =
+ write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "same body always").await;
+ assert_eq!(
+ same_id, id,
+ "precondition: identical content must not mint a new id"
+ );
+ let recovered = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(recovered.tombstoned_missing, 0);
+
+ // Missing again: this must be a FRESH streak (= 1), not a
+ // continuation (which would already be 2 and tombstone here).
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();
+ let second_streak_first_miss = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(
+ second_streak_first_miss.tombstoned_missing, 0,
+ "the streak must have restarted at 1, not continued to 2"
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(id),
+ "still latest after only one fresh missing pass"
+ );
+
+ // Still missing, second FRESH consecutive pass: now it tombstones.
+ let second_streak_second_miss = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(second_streak_second_miss.tombstoned_missing, 1);
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ None,
+ );
+ }
+
+ /// Circuit breaker (design item #4): when most of a scope's candidates
+ /// look missing in one pass, nothing is tombstoned and a `warn` fires —
+ /// that shape is far more likely a walk/mount problem than genuine mass
+ /// deletion. Bite-check: with the breaker removed, this test fails
+ /// (verified manually while writing it, then restored).
+ #[tokio::test]
+ async fn reconcile_delete_circuit_breaker_blocks_when_most_candidates_vanish() {
+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let mut ids = Vec::new();
+ for i in 0..10 {
+ let p = format!("bulk/{i}.md");
+ let id = write_and_index(&wiki, &tmp, ws, proj, &p, "bulk content").await;
+ ids.push((PagePath::new(p).unwrap(), id));
+ }
+
+ let mut streaks = MissingStreaks::new();
+ reconcile(&wiki, &mut streaks).await.unwrap();
+
+ // 60% vanish in one pass.
+ for (path, _) in ids.iter().take(6) {
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();
+ }
+ for pass in 0..3 {
+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(
+ stats.tombstoned_missing, 0,
+ "pass {pass}: the breaker must block every one of the 6 missing pages"
+ );
+ assert_eq!(stats.circuit_broken_scopes, 1, "pass {pass}");
+ }
+ for (path, id) in &ids {
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(*id),
+ "{path} must survive a tripped breaker",
+ );
+ }
+ }
+
+ /// Control for the breaker test: a small fraction missing (well under
+ /// `max(3, 50%)`) proceeds normally and tombstones after two consecutive
+ /// passes, exactly like the non-bulk case.
+ #[tokio::test]
+ async fn reconcile_delete_circuit_breaker_does_not_block_a_small_fraction() {
+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let mut ids = Vec::new();
+ for i in 0..10 {
+ let p = format!("bulk/{i}.md");
+ let id = write_and_index(&wiki, &tmp, ws, proj, &p, "bulk content").await;
+ ids.push((PagePath::new(p).unwrap(), id));
+ }
+
+ let mut streaks = MissingStreaks::new();
+ reconcile(&wiki, &mut streaks).await.unwrap();
+
+ // 10% vanish: well under the breaker threshold (max(3, 50%) = 5 of 10).
+ let (missing_path, _missing_id) = ids[0].clone();
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(missing_path.as_str())).unwrap();
+
+ let first = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(first.circuit_broken_scopes, 0);
+ assert_eq!(first.tombstoned_missing, 0, "first missing pass");
+
+ let second = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(second.circuit_broken_scopes, 0);
+ assert_eq!(
+ second.tombstoned_missing, 1,
+ "second consecutive missing pass"
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, missing_path.as_str().to_string())
+ .await
+ .unwrap(),
+ None,
+ );
+ for (path, id) in ids.iter().skip(1) {
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(*id),
+ "{path} was never missing",
+ );
+ }
+ }
+
+ /// S3 (#929 review): an empty (but NOT partial/`NotFound`) walk is always
+ /// suspect, even for a scope too small to ever trip the `max(3, 50%)`
+ /// math — 2 candidates, both missing, is only 100% but the absolute
+ /// floor of 3 never trips. This is the "directory exists but came back
+ /// empty" mount-failure signature (an unmounted volume, a bind-mount
+ /// briefly resolving to an empty stub): the walk succeeds (no
+ /// `NotFound`, so `walk_partial` is false) but finds nothing at all.
+ #[tokio::test]
+ async fn reconcile_delete_circuit_breaker_trips_on_empty_walk_even_for_a_small_scope() {
+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let mut ids = Vec::new();
+ for i in 0..2 {
+ let p = format!("small/{i}.md");
+ let id = write_and_index(&wiki, &tmp, ws, proj, &p, "small scope content").await;
+ ids.push((PagePath::new(p).unwrap(), id));
+ }
+
+ let mut streaks = MissingStreaks::new();
+ reconcile(&wiki, &mut streaks).await.unwrap();
+
+ // Both files vanish, but the directory itself stays put — an
+ // ordinary (non-partial) walk that simply finds nothing.
+ for (path, _) in &ids {
+ std::fs::remove_file(proj_dir(&tmp, ws, proj).join(path.as_str())).unwrap();
+ }
+ for pass in 0..3 {
+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(
+ stats.tombstoned_missing, 0,
+ "pass {pass}: an empty walk must never be treated as evidence, however small \
+ the scope"
+ );
+ assert_eq!(stats.circuit_broken_scopes, 1, "pass {pass}");
+ }
+ for (path, id) in &ids {
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(*id),
+ "{path} must survive an empty-walk breaker trip",
+ );
+ }
+ }
+
+ /// Partial walk (design item #3): the whole project directory vanishing
+ /// mid-pass (a git checkout, a bind-mount hiccup) must not be treated as
+ /// every page under it having been deleted. Restoring the directory lets
+ /// the very next pass proceed normally — the reappear-after-restore case.
+ #[tokio::test]
+ async fn reconcile_delete_partial_walk_skips_a_scope_whose_project_directory_vanished() {
+ let (tmp, store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let path = PagePath::new("survivor.md").unwrap();
+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "still around").await;
+
+ let dir = proj_dir(&tmp, ws, proj);
+ let mut streaks = MissingStreaks::new();
+ reconcile(&wiki, &mut streaks).await.unwrap();
+
+ // Simulate the whole project directory vanishing mid-walk (walk_partial
+ // becomes true because `walk_markdown_partial`'s own root read fails).
+ std::fs::remove_dir_all(&dir).unwrap();
+ for pass in 0..3 {
+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(
+ stats.tombstoned_missing, 0,
+ "pass {pass}: a partial walk must never produce deletion evidence"
+ );
+ }
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(id),
+ "the page must survive many passes of a vanished project directory",
+ );
+
+ // Restore: the directory and file come back, and reconcile resumes
+ // ordinary behavior instead of carrying over any partial-pass state.
+ std::fs::create_dir_all(&dir).unwrap();
+ std::fs::write(dir.join(path.as_str()), "still around").unwrap();
+ let restored = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(restored.tombstoned_missing, 0);
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(id),
+ );
+ }
+
+ /// S1 (#929 review): a partial walk's ABSENCE evidence is suspect, but
+ /// its PRESENCE evidence is not — a page the walk actually found, even
+ /// during a partial pass, must still clear an earlier streak. Otherwise
+ /// a page that legitimately reappeared could be tombstoned later purely
+ /// because an unrelated directory's `NotFound` earlier suppressed the
+ /// clearing that presence should have triggered.
+ ///
+ /// Calls `reconcile_missing_pages` directly rather than orchestrating a
+ /// real partial walk through the filesystem: a subdirectory deleted
+ /// *before* `reconcile()` runs is never even listed by `read_dir`, so it
+ /// produces no `NotFound` at all (that's only reachable via a genuine
+ /// race — the directory vanishing *during* the walk, mid-traversal,
+ /// which a single-threaded test cannot deterministically force). Driving
+ /// the function directly exercises the exact contract instead.
+ #[tokio::test]
+ async fn reconcile_delete_partial_walk_still_clears_streaks_for_pages_it_did_find() {
+ let (tmp, _store, wiki, ws, proj) = setup_reconcile_delete().await;
+ let path = PagePath::new("flaky.md").unwrap();
+ let id = write_and_index(&wiki, &tmp, ws, proj, path.as_str(), "here").await;
+
+ // Seed a pre-existing streak of 1, as if an earlier complete pass had
+ // already observed this exact page missing once.
+ let mut streaks = MissingStreaks::new();
+ streaks.insert((ws, proj, path.clone()), (id, 1));
+
+ let snapshot = vec![(id, path.clone())];
+ let mut walked: HashSet<PagePath> = HashSet::new();
+ walked.insert(path.clone());
+ let mut stats = ReconcileStats::default();
+
+ // This pass is partial (some unrelated directory vanished elsewhere
+ // in the scope), but it DID find this exact page present.
+ reconcile_missing_pages(
+ &wiki,
+ ws,
+ proj,
+ &snapshot,
+ &walked,
+ true,
+ &mut streaks,
+ &mut stats,
+ )
+ .await;
+
+ assert!(
+ !streaks.contains_key(&(ws, proj, path.clone())),
+ "presence evidence from a partial walk must still clear the streak"
+ );
+ assert_eq!(stats.tombstoned_missing, 0);
+
+ // Bite-check performed manually while writing this test: hoisting
+ // the clearing loop back inside the `if walk_partial { return; }`
+ // branch (the pre-S1 shape) makes this assertion fail, since the
+ // seeded streak survives untouched.
+ }
+
+ /// Cross-project isolation (the same adversarial shape #929's rejected
+ /// prior attempt was tested against): two projects with a page at the
+ /// same relative path, only one of which loses its file, must not let
+ /// one project's missing-page streak or tombstone affect the other's.
+ #[tokio::test]
+ async fn reconcile_delete_cross_project_isolation() {
+ let (tmp, store, wiki, ws, proj_a) = setup_reconcile_delete().await;
+ let proj_b = store
+ .writer
+ .get_or_create_project(ws, "scratch-b", None)
+ .await
+ .unwrap();
+ let path = PagePath::new("shared/name.md").unwrap();
+ let _id_a = write_and_index(&wiki, &tmp, ws, proj_a, path.as_str(), "project a").await;
+ let id_b = write_and_index(&wiki, &tmp, ws, proj_b, path.as_str(), "project b").await;
+ // Control page in proj_a (S3, #929 review): keeps that scope's walk
+ // from being literally empty once `shared/name.md` goes missing,
+ // which would otherwise trip the unconditional empty-walk breaker.
+ write_and_index(&wiki, &tmp, ws, proj_a, "control.md", "always here").await;
+
+ let mut streaks = MissingStreaks::new();
+ reconcile(&wiki, &mut streaks).await.unwrap();
+ std::fs::remove_file(proj_dir(&tmp, ws, proj_a).join(path.as_str())).unwrap();
+
+ reconcile(&wiki, &mut streaks).await.unwrap();
+ let final_stats = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(final_stats.tombstoned_missing, 1);
+
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj_a, path.as_str().to_string())
+ .await
+ .unwrap(),
+ None,
+ "project A's page was tombstoned",
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj_b, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(id_b),
+ "project B's identically-pathed page must survive untouched",
+ );
+ }
+
+ /// B2 (#929 review): a same-workspace `move-session` re-home can leave a
+ /// session page's DB row live in the destination project with no file
+ /// there. Reproduced end to end through the real bug, not a synthetic
+ /// row edit: the session's own record already says it lives in
+ /// `proj_b` (so `Wiki::move_session_page(from=(ws,proj_b),
+ /// to=(ws,proj_b), ...)` sees `from == to` and skips the file step
+ /// entirely — see its doc comment), while its `sessions/<id>.md` page
+ /// is a straggler still sitting in `proj_a`. `ops::move_session`'s
+ /// page re-stamp is keyed purely on path + "not already in the target
+ /// scope", not on `from`, so it sweeps that straggler page into
+ /// `proj_b` regardless — leaving a live, correct row at `proj_b` with
+ /// no file there, ever. This is a separate, pre-existing bug in
+ /// `move-session`'s file relocation (not fixed here); this test only
+ /// proves the reconcile-delete safety net's `sessions/*.md` exclusion
+ /// keeps it from destroying that page.
+ #[tokio::test]
+ async fn reconcile_delete_never_tombstones_a_session_page_orphaned_by_move_session() {
+ let (tmp, store, wiki, ws, proj_a) = setup_reconcile_delete().await;
+ let proj_b = store
+ .writer
+ .get_or_create_project(ws, "scratch-b", None)
+ .await
+ .unwrap();
+ let sid = ai_memory_core::SessionId::new();
+ // The session's own record already lives in proj_b.
+ store
+ .writer
+ .begin_session(ai_memory_core::NewSession {
+ occurred_at: None,
+ id: sid,
+ workspace_id: ws,
+ project_id: proj_b,
+ agent_kind: ai_memory_core::AgentKind::ClaudeCode,
+ cwd: None,
+ actor_user: None,
+ })
+ .await
+ .unwrap();
+ // Its page is a straggler, still in proj_a.
+ let page_path = format!("sessions/{sid}.md");
+ let page_id = write_and_index(&wiki, &tmp, ws, proj_a, &page_path, "session summary").await;
+ // A control page that actually lives in proj_b, so proj_b's on-disk
+ // directory exists and its walk is complete (not partial) — without
+ // this, an absent proj_b directory would make the walk itself hit
+ // `NotFound` and get skipped by the PARTIAL-walk guard (design item
+ // #3) instead of by the sessions exclusion this test means to
+ // isolate. Confirmed by disabling the sessions exclusion locally
+ // while writing this test: without a proj_b control page the test
+ // still passed for the wrong reason (partial walk), and only
+ // failed as expected once this control page made the walk complete.
+ write_and_index(&wiki, &tmp, ws, proj_b, "control.md", "always here").await;
+
+ // The from==to re-home: the wiki sees no scope change and skips the
+ // file step, but the store still sweeps the straggler page into
+ // proj_b (see `ops::move_session`'s `rehome` branch).
+ let outcome = wiki
+ .move_session_page(
+ sid,
+ (ws, proj_b),
+ (ws, proj_b),
+ ai_memory_store::PagesMode::Move,
+ None,
+ None,
+ )
+ .await
+ .unwrap();
+ assert_eq!(
+ outcome.summary.page_versions_moved, 1,
+ "precondition: the page re-stamp must have actually run"
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj_b, page_path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(page_id),
+ "precondition: the page row now lives in proj_b",
+ );
+ assert!(
+ !proj_dir(&tmp, ws, proj_b).join(&page_path).exists(),
+ "precondition: proj_b has no file for it — the pre-existing move-session bug",
+ );
+
+ let mut streaks = MissingStreaks::new();
+ for pass in 0..3 {
+ let stats = reconcile(&wiki, &mut streaks).await.unwrap();
+ assert_eq!(
+ stats.tombstoned_missing, 0,
+ "pass {pass}: a session page must never be reconcile-delete candidate"
+ );
+ }
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj_b, page_path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(page_id),
+ "the orphaned-by-move-session page must survive untouched",
+ );
+ }
}
diff --git a/crates/ai-memory-wiki/src/wiki.rs b/crates/ai-memory-wiki/src/wiki.rs
index b529cd8f..f332b11d 100644
--- a/crates/ai-memory-wiki/src/wiki.rs
+++ b/crates/ai-memory-wiki/src/wiki.rs
@@ -184,6 +184,12 @@ pub struct Wiki {
/// write after the scope's first — the store query and the two manifest
/// reads happen once per scope per process, never per page.
manifested_scopes: Arc<Mutex<HashSet<(WorkspaceId, ProjectId)>>>,
+ /// `[maintenance] reconcile_tombstones_deleted_pages` (#929), read once by
+ /// `Config::load` and threaded here via [`Wiki::with_reconcile_tombstones_deleted_pages`].
+ /// OFF by default: with this `false`, the watcher's reconcile pass never
+ /// tombstones a page whose file vanished — deletions still require
+ /// `ai-memory delete-page`, unchanged from before this feature existed.
+ reconcile_tombstones_deleted_pages: bool,
}
/// Key uniquely identifying a page for per-path write serialization.
@@ -211,6 +217,7 @@ impl Wiki {
mutation_lock: Arc::new(RwLock::new(())),
page_locks: Arc::new(std::sync::Mutex::new(HashMap::new())),
manifested_scopes: Arc::new(Mutex::new(HashSet::new())),
+ reconcile_tombstones_deleted_pages: false,
})
}
@@ -245,6 +252,24 @@ impl Wiki {
self
}
+ /// Opt into the watcher reconcile pass's tombstone-on-missing-file safety
+ /// net (`[maintenance] reconcile_tombstones_deleted_pages`, #929). `false`
+ /// (the default from [`Wiki::new`]) leaves reconcile's behavior exactly
+ /// as it was before this feature: deletions require `ai-memory
+ /// delete-page`.
+ #[must_use]
+ pub fn with_reconcile_tombstones_deleted_pages(mut self, enabled: bool) -> Self {
+ self.reconcile_tombstones_deleted_pages = enabled;
+ self
+ }
+
+ /// Whether the watcher's reconcile pass may tombstone a page whose file
+ /// disappeared. See [`Wiki::with_reconcile_tombstones_deleted_pages`].
+ #[must_use]
+ pub(crate) fn reconcile_tombstones_deleted_pages(&self) -> bool {
+ self.reconcile_tombstones_deleted_pages
+ }
+
/// Replace the default built-in-only sanitizer with one carrying
/// the operator's `[sanitize].extra_patterns` + `allowlist`.
#[must_use]
@@ -910,6 +935,108 @@ impl Wiki {
.await
}
+ /// `(PageId, PagePath)` of every `is_latest = 1` page in a scope, for the
+ /// watcher's reconcile-delete safety net (#929) to snapshot before it
+ /// walks the wiki tree.
+ ///
+ /// # Errors
+ /// Returns [`WikiError`] when the store reader is unavailable, or on a
+ /// store error.
+ pub(crate) async fn latest_page_ids(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ ) -> WikiResult<Vec<(PageId, PagePath)>> {
+ let reader = self.store_reader.as_ref().ok_or_else(|| {
+ ai_memory_wiki_error("reconcile-delete candidate snapshot requires a store reader")
+ })?;
+ reader
+ .latest_page_ids(workspace_id, project_id)
+ .await
+ .map_err(Into::into)
+ }
+
+ /// Tombstone the expected latest page after the watcher's reconcile pass
+ /// found its file missing on two consecutive passes (opt-in, #929;
+ /// `[maintenance] reconcile_tombstones_deleted_pages`).
+ ///
+ /// Deliberately does NOT go through [`Wiki::remove_page_locked`]: unlike
+ /// `evict_page_if_latest` (decay) or `delete_page_if_latest` (explicit
+ /// delete), this is a background safety net reacting to a file that is
+ /// already gone, not a user-initiated action, so it never runs the
+ /// BLOCKING admission gate (`AdmissionChain::notify`/`authorize`) —
+ /// nothing gets to refuse it — and it never touches the filesystem —
+ /// there is nothing on disk to quarantine or remove. It DOES
+ /// fire-and-forget the chain's non-blocking observer/mirror webhooks on
+ /// success (#929 review, S6), same as `remove_page_locked`'s post-write
+ /// `dispatch_async`, so a mirror learns about the tombstone instead of
+ /// silently diverging from the source of truth.
+ ///
+ /// Re-checks both the store's latest-id and the file's actual absence
+ /// while holding the exclusive mutation guard, immediately before
+ /// writing — the final gate design item #6 calls for, closing the window
+ /// between the reconcile pass's second "missing" observation and this
+ /// call (e.g. a concurrent `write_page` recreating the path).
+ ///
+ /// # Errors
+ /// Returns [`WikiError`] when the store reader is unavailable, or on a
+ /// filesystem or store error.
+ pub(crate) async fn tombstone_missing_page_if_latest(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ path: &PagePath,
+ expected_latest_id: PageId,
+ ) -> WikiResult<bool> {
+ let _guard = self.mutation_lock.write().await;
+ self.ensure_project_workspace(workspace_id, project_id)
+ .await?;
+ let reader = self
+ .store_reader
+ .as_ref()
+ .ok_or_else(|| ai_memory_wiki_error("reconcile tombstone requires a store reader"))?;
+ let current = reader
+ .latest_page_id_by_ids(workspace_id, project_id, path.as_str().to_string())
+ .await?;
+ if current != Some(expected_latest_id) {
+ return Ok(false);
+ }
+ let abs = self.abs_path(workspace_id, project_id, path);
+ if abs.try_exists()? {
+ // The file came back (e.g. a slow atomic-save pattern beyond the
+ // two-pass window, or a deliberate rewrite mid-check). Leave the
+ // page alone; the next reconcile tick reindexes it normally.
+ return Ok(false);
+ }
+ let tombstoned = self
+ .writer
+ .soft_delete_for_reconcile_if_latest(
+ workspace_id,
+ project_id,
+ path.clone(),
+ expected_latest_id,
+ )
+ .await?;
+ if tombstoned && let Some(chain) = &self.admission_chain {
+ // Non-blocking observers/mirrors only (#929 review, S6) — never
+ // the blocking `notify`/`authorize` gate: this is a background
+ // safety net, not a user-initiated delete, so nothing gets to
+ // refuse it. But a mirror that never hears about the tombstone
+ // silently diverges from the source of truth, which is a real
+ // correctness issue too; fire-and-forget informs it the same way
+ // `remove_page_locked` does for `evict_page_if_latest` and
+ // `delete_page_if_latest`, just skipping their blocking half.
+ let mut ctx = AdmissionContext {
+ op: AdmissionOp::Delete,
+ ..Default::default()
+ };
+ self.resolve_admission_names(workspace_id, project_id, &mut ctx)
+ .await;
+ chain.dispatch_async(Some(path.as_str()), &serde_json::Value::Null, "", &ctx);
+ }
+ Ok(tombstoned)
+ }
+
async fn remove_page_locked(
&self,
workspace_id: WorkspaceId,
@@ -4325,6 +4452,360 @@ mod tests {
);
}
+ // --- #929: reconcile-delete tombstone op (`tombstone_missing_page_if_latest`) ---
+
+ #[tokio::test]
+ async fn reconcile_tombstone_refuses_a_stale_latest_id() {
+ let tmp = TempDir::new().unwrap();
+ let (store, wiki, ws, proj) = scoped(&tmp).await;
+ let path = PagePath::new("sessions/stale-reconcile.md").unwrap();
+ let stale = wiki
+ .write_page(req(
+ ws,
+ proj,
+ path.as_str(),
+ "old candidate",
+ serde_json::json!({}),
+ ))
+ .await
+ .unwrap();
+ let current = wiki
+ .write_page(req(
+ ws,
+ proj,
+ path.as_str(),
+ "new current body",
+ serde_json::json!({}),
+ ))
+ .await
+ .unwrap();
+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();
+
+ assert!(
+ !wiki
+ .tombstone_missing_page_if_latest(ws, proj, &path, stale)
+ .await
+ .unwrap(),
+ "a stale expected id must leave the current version untouched"
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(current),
+ );
+ }
+
+ /// Design item #6: the final recheck, immediately before acting, must
+ /// refuse when the file has actually come back — even though the caller
+ /// (the watcher's reconcile pass) only calls this after two consecutive
+ /// "missing" observations, a page can still be recreated in the narrow
+ /// window between the second observation and this call.
+ #[tokio::test]
+ async fn reconcile_tombstone_refuses_when_the_file_still_exists() {
+ let tmp = TempDir::new().unwrap();
+ let (store, wiki, ws, proj) = scoped(&tmp).await;
+ let path = PagePath::new("sessions/present-reconcile.md").unwrap();
+ let page_id = wiki
+ .write_page(req(
+ ws,
+ proj,
+ path.as_str(),
+ "still here",
+ serde_json::json!({}),
+ ))
+ .await
+ .unwrap();
+
+ assert!(
+ !wiki
+ .tombstone_missing_page_if_latest(ws, proj, &path, page_id)
+ .await
+ .unwrap(),
+ "the file exists on disk; the final recheck must refuse to tombstone it"
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ Some(page_id),
+ );
+ assert!(wiki.abs_path(ws, proj, &path).exists());
+ }
+
+ /// The tombstone op is a background safety net reacting to a file that is
+ /// already gone, not a user-initiated delete: it must never run the
+ /// BLOCKING admission gate, which could refuse it. A webhook configured
+ /// to reject every `Delete` event proves this by NOT blocking the
+ /// tombstone — if `tombstone_missing_page_if_latest` went through the
+ /// same path as `evict_page_if_latest`/`delete_page_if_latest`, this
+ /// webhook would turn the call into an error
+ /// (`rejecting_delete_admission_leaves_decay_candidate_live` shows
+ /// exactly that for the decay path). It DOES still fire-and-forget any
+ /// non-blocking observer/mirror webhook on success — see
+ /// `reconcile_tombstone_dispatches_non_blocking_observer_webhook` below
+ /// (#929 review, S6).
+ #[tokio::test]
+ async fn reconcile_tombstone_does_not_dispatch_admission_webhook() {
+ use axum::Router;
+ use axum::http::StatusCode;
+ use axum::routing::post;
+ use tokio::net::TcpListener;
+
+ let app = Router::new().route(
+ "/reject",
+ post(|| async { (StatusCode::FORBIDDEN, "must never be called") }),
+ );
+ let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let addr = listener.local_addr().unwrap();
+ tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
+
+ let tmp = TempDir::new().unwrap();
+ let (store, wiki, ws, proj) = scoped(&tmp).await;
+ let path = PagePath::new("sessions/webhook-reconcile.md").unwrap();
+ let page_id = wiki
+ .write_page(req(
+ ws,
+ proj,
+ path.as_str(),
+ "vanishing",
+ serde_json::json!({}),
+ ))
+ .await
+ .unwrap();
+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();
+ let wiki = wiki.with_admission_chain(
+ AdmissionChain::new(vec![WebhookConfig {
+ name: "reconcile-guard".into(),
+ url: format!("http://{addr}/reject"),
+ timeout_ms: 1_000,
+ failure_policy: FailurePolicy::Reject,
+ events: vec![AdmissionOp::Delete],
+ blocking: true,
+ }])
+ .unwrap(),
+ );
+
+ assert!(
+ wiki.tombstone_missing_page_if_latest(ws, proj, &path, page_id)
+ .await
+ .unwrap(),
+ "no admission dispatch means the rejecting webhook is never consulted"
+ );
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ None,
+ "the page is no longer latest",
+ );
+ }
+
+ /// S6 (#929 review): the tombstone op skips the BLOCKING admission gate
+ /// (proven above), but a mirror that never hears about the tombstone at
+ /// all would silently diverge from the source of truth — a real
+ /// correctness issue on its own. A non-blocking observer webhook must
+ /// still be fired, fire-and-forget, on a successful tombstone.
+ #[tokio::test]
+ async fn reconcile_tombstone_dispatches_non_blocking_observer_webhook() {
+ use axum::Router;
+ use axum::extract::State;
+ use axum::http::StatusCode;
+ use axum::routing::post;
+ use std::sync::Arc;
+ use std::sync::atomic::{AtomicBool, Ordering};
+ use tokio::net::TcpListener;
+
+ let observed = Arc::new(AtomicBool::new(false));
+ let app = Router::new()
+ .route(
+ "/observe",
+ post(|State(flag): State<Arc<AtomicBool>>| async move {
+ flag.store(true, Ordering::SeqCst);
+ StatusCode::OK
+ }),
+ )
+ .with_state(observed.clone());
+ let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let addr = listener.local_addr().unwrap();
+ tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
+
+ let tmp = TempDir::new().unwrap();
+ let (_store, wiki, ws, proj) = scoped(&tmp).await;
+ let path = PagePath::new("notes/mirror-reconcile.md").unwrap();
+ let page_id = wiki
+ .write_page(req(
+ ws,
+ proj,
+ path.as_str(),
+ "vanishing",
+ serde_json::json!({}),
+ ))
+ .await
+ .unwrap();
+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();
+ let wiki = wiki.with_admission_chain(
+ AdmissionChain::new(vec![WebhookConfig {
+ name: "mirror-observer".into(),
+ url: format!("http://{addr}/observe"),
+ timeout_ms: 1_000,
+ failure_policy: FailurePolicy::Ignore,
+ events: vec![AdmissionOp::Delete],
+ blocking: false,
+ }])
+ .unwrap(),
+ );
+
+ assert!(
+ wiki.tombstone_missing_page_if_latest(ws, proj, &path, page_id)
+ .await
+ .unwrap()
+ );
+
+ // `dispatch_async` fires the request off the caller's path; give it a
+ // bounded window to land instead of asserting immediately.
+ let mut delivered = false;
+ for _ in 0..50 {
+ if observed.load(Ordering::SeqCst) {
+ delivered = true;
+ break;
+ }
+ tokio::time::sleep(std::time::Duration::from_millis(20)).await;
+ }
+ assert!(
+ delivered,
+ "the non-blocking observer webhook must be notified of the tombstone"
+ );
+ }
+
+ /// The op never touches the filesystem (design item #7: there is nothing
+ /// to touch — the file is already gone). Proven by the project
+ /// directory's file listing being byte-for-byte identical before and
+ /// after the call: no quarantine artifact, no recreated file, nothing.
+ #[tokio::test]
+ async fn reconcile_tombstone_never_touches_the_filesystem() {
+ let tmp = TempDir::new().unwrap();
+ let (_store, wiki, ws, proj) = scoped(&tmp).await;
+ let path = PagePath::new("notes/vanishing-fs.md").unwrap();
+ let page_id = wiki
+ .write_page(req(
+ ws,
+ proj,
+ path.as_str(),
+ "gone soon",
+ serde_json::json!({}),
+ ))
+ .await
+ .unwrap();
+ let proj_dir = wiki.project_root(ws, proj);
+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();
+
+ let listing_before = list_dir_recursive(&proj_dir);
+ assert!(
+ wiki.tombstone_missing_page_if_latest(ws, proj, &path, page_id)
+ .await
+ .unwrap()
+ );
+ let listing_after = list_dir_recursive(&proj_dir);
+ assert_eq!(
+ listing_before, listing_after,
+ "the tombstone op must not create, remove, or rename any file"
+ );
+ }
+
+ /// `restore-page`/wiki history depend on this: a reconcile tombstone is
+ /// `is_latest = 0` + `superseded_at`, mirroring decay eviction exactly —
+ /// and is picked up by the SAME aged-tombstone sweep decay eviction uses
+ /// (`hard_delete_after_days`, tier/pin-agnostic), not exempted from it.
+ /// Proven by fetching the row back through the same query that sweep
+ /// uses ([`ai_memory_store::ReaderPool::decay_tombstones_before`]): if
+ /// the row were gone, or if only `is_latest` had flipped without
+ /// `superseded_at`, it would not show up there. What actually protects a
+ /// reconcile tombstone from that sweep permanently destroying a
+ /// self-healed false positive is `upsert_page_in_tx`'s resurrection path
+ /// (`ops.rs`): if the file comes back, the new version re-links onto
+ /// this chain and clears `superseded_at`, which is covered separately by
+ /// `reconcile_tombstone_resurrects_instead_of_orphaning_on_recreate`.
+ #[tokio::test]
+ async fn reconcile_tombstone_is_a_soft_delete_visible_to_normal_retention() {
+ let tmp = TempDir::new().unwrap();
+ let (store, wiki, ws, proj) = scoped(&tmp).await;
+ let path = PagePath::new("notes/tombstone-shape.md").unwrap();
+ let page_id = wiki
+ .write_page(req(
+ ws,
+ proj,
+ path.as_str(),
+ "about to vanish",
+ serde_json::json!({}),
+ ))
+ .await
+ .unwrap();
+ std::fs::remove_file(wiki.abs_path(ws, proj, &path)).unwrap();
+
+ assert!(
+ wiki.tombstone_missing_page_if_latest(ws, proj, &path, page_id)
+ .await
+ .unwrap()
+ );
+
+ // is_latest = 0: no longer the scope's current version.
+ assert_eq!(
+ store
+ .reader
+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())
+ .await
+ .unwrap(),
+ None,
+ );
+ // superseded_at set (not just is_latest): the row surfaces through the
+ // exact query the decay retention sweep uses to find eligible aged
+ // tombstones — proving it is a real tombstone, not merely hidden, and
+ // that hard cleanup is left to that same normal retention path.
+ let far_future = jiff::Timestamp::now().as_microsecond() + 365 * 24 * 3_600 * 1_000_000;
+ let tombstones = store
+ .reader
+ .decay_tombstones_before(ws, proj, far_future)
+ .await
+ .unwrap();
+ assert!(
+ tombstones.iter().any(|t| t.id == page_id && t.path == path),
+ "the reconcile tombstone must be indistinguishable from a decay tombstone to the \
+ retention sweep: {tombstones:?}",
+ );
+ }
+
+ fn list_dir_recursive(root: &Path) -> Vec<String> {
+ fn walk(dir: &Path, root: &Path, out: &mut Vec<String>) {
+ let Ok(entries) = std::fs::read_dir(dir) else {
+ return;
+ };
+ for entry in entries.flatten() {
+ let path = entry.path();
+ if path.is_dir() {
+ walk(&path, root, out);
+ } else {
+ out.push(
+ path.strip_prefix(root)
+ .unwrap_or(&path)
+ .to_string_lossy()
+ .into_owned(),
+ );
+ }
+ }
+ }
+ let mut out = Vec::new();
+ walk(root, root, &mut out);
+ out.sort();
+ out
+ }
+
#[tokio::test]
async fn decay_cleanup_refuses_to_reindex_a_symlinked_recreation() {
let tmp = TempDir::new().unwrap();
diff --git a/docs/install.md b/docs/install.md
index 1ab0fbab..aa01ba8a 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -2217,6 +2217,26 @@ or overdue job runs once after a bounded startup delay. Failed runs are not
recorded as successful and retry after that bounded delay. Embedding backfill
remains opt-in and keeps its interval-only behavior (no startup catch-up).
+`reconcile_tombstones_deleted_pages` (default `false`, also under
+`[maintenance]`) is a separate, experimental opt-in: it lets the watcher's
+own 30s reconcile pass (not the scheduled jobs above) tombstone an
+OKF-imported content page (session summary pages are excluded — see
+`docs/okf.md`) whose file has disappeared from disk, after it has been
+missing on two consecutive passes and survived a circuit breaker that
+refuses to act when more than `max(3, 50%)` of a scope's candidate pages
+look missing at once, or when a walk finds nothing at all. The tombstone is
+soft (`is_latest = 0` + `superseded_at`, the same shape decay eviction uses)
+and is picked up by the same aged-tombstone hard-delete sweep — it is not
+exempt from it. What actually protects it: a reconcile tombstone is never
+itself destroyed while its chain has no successor; if the file returns, the
+new version re-links to the tombstoned chain instead of starting fresh, so
+nothing is orphaned for that sweep to destroy. It never runs the blocking
+admission gate (nothing can refuse it), but does fire-and-forget any
+non-blocking observer/mirror webhook. With it off (the default), reconcile's
+behavior is unchanged: a deleted file still requires `ai-memory delete-page`.
+See
+`docs/okf.md` for the full design.
+
---
## Bootstrap mid-project
diff --git a/docs/okf.md b/docs/okf.md
index 25611239..ce48242c 100644
--- a/docs/okf.md
+++ b/docs/okf.md
@@ -143,6 +143,57 @@ so any later rewrite of an affected page (a restore, a hand edit, a
`stale_after` that differs from `expires_at` was not derived by
ai-memory and is left alone.
+## Reconcile-delete safety net (opt-in, #929)
+
+The watcher's 30s reconcile pass only ever reindexes create/modify events by
+default — a page whose file disappears from disk stays indexed until
+`ai-memory delete-page` removes it explicitly. `[maintenance]
+reconcile_tombstones_deleted_pages` (default `false`) opts into a background
+safety net that closes that gap, designed to be safe even though it adds an
+automatic mutation to a background loop:
+
+- A page's file must be observed missing on **two consecutive** reconcile
+ passes (not one) before anything happens, and the check is re-verified
+ against the live filesystem immediately before acting.
+- Only pages the walk could ever have returned are eligible — `bootstrap.md`,
+ `_meta.md`, and `_pending/` sidecars are never candidates, and a project
+ whose walk hit `NotFound` (a vanished subdirectory, a whole project
+ directory gone mid-pass) contributes no "missing" evidence for that pass.
+ `sessions/*.md` pages are excluded too, for an unrelated reason: a
+ same-workspace `move-session` re-home can leave a correct DB row with no
+ file at its new scope (a separate, pre-existing bug in `move-session`'s
+ file relocation — tracked as a follow-up, not fixed here). This mechanism
+ is meant for OKF-imported content pages only.
+- A circuit breaker refuses to act on a whole scope when more than `max(3,
+ 50%)` of its candidate pages look missing in one pass, OR when a
+ non-partial walk finds nothing at all in a scope that has candidates — the
+ latter catches the "directory exists but came back empty" mount-failure
+ signature for scopes too small to ever trip the percentage math. Either
+ shape is far more likely a walk/mount problem (an unmounted volume, a git
+ checkout mid-walk) than genuine mass deletion, and is logged at `warn`.
+- The action itself is a soft tombstone (`is_latest = 0` + `superseded_at`),
+ the same row shape decay eviction already uses, and is picked up by the
+ exact same aged-tombstone hard-delete sweep decay eviction uses
+ (`hard_delete_after_days`, tier/pin-agnostic) — it is NOT exempt from that
+ sweep. The precise guarantee: **a reconcile tombstone is never itself
+ destroyed while its chain has no successor; if the file returns, the new
+ version re-links to the tombstoned chain instead of starting fresh, so
+ nothing is orphaned for the 180-day sweep to destroy.** Concretely, a fresh
+ write at a tombstoned path (`upsert_page_in_tx`) supersedes the tombstoned
+ row via `supersedes` and clears its `superseded_at`, turning it into an
+ ordinary, protected supersession-chain member — the same mechanism that
+ already keeps normal edit history off that sweep. It also never dispatches
+ the BLOCKING admission gate (nothing gets to refuse it — this is a
+ background safety net reacting to an already-vanished file, not a
+ user-initiated delete) and never touches the filesystem (there is nothing
+ to write; the file is already gone); it DOES fire-and-forget any
+ non-blocking observer/mirror webhook on success, so a mirror learns about
+ the tombstone instead of silently diverging. `restore-page` and the page's
+ version history remain intact.
+- With the flag off (the default), behavior is byte-identical to before this
+ feature existed: nothing new is logged, and reconcile never mutates the
+ store.
+
## Tests (each with a control that must fail on a broken build)
- Round-trip: page → OKF file on disk → parsed back identical.
@@ -161,8 +212,8 @@ so any later rewrite of an affected page (a restore, a hand edit, a
directory and letting the watcher (or `reindex`) ingest them IS the
import path. Overwriting an already-imported concept file gets its new
version embedded the same way a brand-new file does — no manual
- `ai-memory embed` needed. Deleting one does not yet remove it from the
- index: the watcher only reconciles create/modify events, so a deleted
- concept file still needs an explicit `ai-memory delete-page` (tracked in
- #929).
+ `ai-memory embed` needed. Deleting one does not remove it from the index by
+ default: the watcher only reconciles create/modify events, so a deleted
+ concept file needs an explicit `ai-memory delete-page` unless the opt-in
+ safety net below is enabled (#929).
- Retrieval regression: LongMemEval baseline re-run; no material drop.
diff --git a/docs/security-boundaries.md b/docs/security-boundaries.md
index 935d8d42..2f02b04a 100644
--- a/docs/security-boundaries.md
+++ b/docs/security-boundaries.md
@@ -46,6 +46,7 @@ boundary not yet built.
| 10b | Session purge is scope+owner-bound (no cross-session/project over-delete); `purge-session`'s `dry_run` always wins over `confirm`, same as `purge-project` | `ai-memory-store/src/ops.rs` `purge_session` — selection scoped to `(workspace_id, project_id)` and keyed on this session's own `summary_page_id` **or** `path='sessions/<sid>.md'` + `json_extract(frontmatter_json,'$.session_id')=<sid>` (frontmatter owner, not the recursive latest-chain); `in_scope==0 → NotFound` fail-closed; whole op in one transaction; `admin.rs` `handle_purge_session` checks `req.dry_run` unconditionally, before `req.confirm` | `ops.rs` `purge_session_leaves_a_sibling_session_in_the_same_project_intact`, `…refuses_a_session_from_another_project_and_deletes_nothing`, `…refuses_a_session_from_another_workspace`, `…does_not_delete_an_identically_pathed_page_in_another_project`, `…removes_older_summary_versions_without_deleting_prior_manual_page` (#862); `admin_purge.rs` `purge_session_confirm_true_and_dry_run_true_still_only_previews` — `{"confirm": true, "dry_run": true}` deletes nothing; `ops.rs` `purge_session_dry_run_changes_nothing` — every table a session purge touches, the `purged_sessions` tombstone, and the `audit_log` row are identical before/after a `PurgeMode::Preview` run; `ops.rs` `purge_session_counts_collateral_damage_in_another_project` and `admin_purge.rs` `purge_session_dry_run_and_confirmed_purge_both_report_collateral_damage_in_another_project` — purging session S collaterally deletes an observation, and orphans a handoff's session reference, in a *different* project (via `S`'s own id cascading), and both the preview and the confirmed report count it; `admin_purge.rs` `purge_session_dry_run_refuses_a_session_outside_its_named_scope` — the same session previewed under a different real project (same workspace) or a different real workspace whose project shares the name both 404 with no counts in the body, with the session's own scope as the 200 control | STRONG |
| 10c | Ledger reclaim is content-gated, not filename-gated (invariant #16) | `ai-memory-store/src/ops.rs` `reclaim_ledger_versions` — a candidate row must pass `ai_memory_core::log_ledger::body_opens_with_log_ledger` on its own `body`, and only `is_latest=0 AND superseded_at IS NULL` rows are eligible, so decay-owned rows stay with `forget-sweep`; confirm gate in `admin.rs` `handle_reclaim_ledger_versions` | `reclaim_ledger_versions.rs` `a_prose_page_wearing_the_ledger_name_keeps_its_whole_version_chain`, `an_ordinary_page_chain_in_another_project_is_untouched`, `a_decay_tombstone_stays_with_the_sweep_that_owns_it`; `admin_reclaim_ledger_versions.rs` `deleting_without_confirm_is_refused_and_changes_nothing` (with dry-run and confirmed controls) | STRONG |
| 10d | Workspace delete's own row counts are scope-bound to `workspace_id` (a workspace's *own* rows are never over- or under-counted by a sibling workspace's), its cross-workspace observation/handoff cascade is reported by both preview and confirm rather than prevented (same as `purge-project` one level down), the non-empty guard fails closed without `force`, and `delete-workspace`'s `dry_run` always wins | `ai-memory-store/src/ops.rs` `delete_workspace` — every direct count and the `DELETE FROM workspaces` itself are scoped by `workspace_id = ?1`; the two `collateral_*` counts are scoped by `workspace_id != ?1` so a row is counted as one or the other but never both; `projects_deleted > 0 && !force → WorkspaceNotEmpty` fail-closed, and a missing workspace is `NotFound`, both checked identically under `PurgeMode::Preview` and `PurgeMode::Commit`; whole op in one transaction; `admin.rs` `handle_delete_workspace` checks `req.dry_run` unconditionally, before building the confirmed path, mirroring `reclaim-ledger-versions`/`purge-project`; root-only gate is the shared `/admin/*` multi-user auth ladder, not a separate check | `admin_move.rs` `delete_workspace_dry_run_never_counts_a_different_workspaces_rows` — previewing workspace A counts only its own project/page/session/observation/handoff rows, not workspace B's (which holds two like-named projects and its own full row set), and a session living in B that stamps an observation directly into A is counted in A's own `observations_deleted`, never in `collateral_observations_deleted` (the two counts are mutually exclusive by construction); `admin_move.rs` `delete_workspace_force_true_and_dry_run_true_still_only_previews` — `{"force": true, "dry_run": true}` deletes nothing; `ops.rs` `delete_workspace_dry_run_changes_nothing` — every table's row count, including `workspaces` itself and the `purged_scopes` tombstone (the two tables a confirmed delete's own transaction writes to), is identical before/after a `PurgeMode::Preview` run; `ops.rs` `delete_workspace_dry_run_and_confirmed_delete_both_report_collateral_damage_in_another_workspace` and `admin_move.rs` of the same name — deleting workspace W collaterally deletes an observation, and orphans a handoff's session reference, in a *different* workspace (via `sessions` cascading out of W), and both the preview and the confirmed response count it identically — the cascade itself is reported, not blocked, by either; `admin_move.rs` `delete_workspace_dry_run_refuses_non_empty_without_force` / `delete_workspace_dry_run_nonexistent_workspace_returns_404` — same `409`/`404` a confirmed delete would give; `admin.rs` `multiuser_delete_workspace_dry_run_rejects_db_user_and_anonymous` — root reaches the preview handler (404 control), DB user gets 403, anonymous gets 401 | STRONG |
+| 10e | Reconcile-tombstone of a disk-deleted page is opt-in, soft, scope-bound, and false-positive-guarded (invariant #16, #929 delete-half) | `ai-memory-wiki/src/watcher.rs` — off by default (`[maintenance] reconcile_tombstones_deleted_pages`); a page is tombstoned only after its file is observed missing on two consecutive passes (streak resets if the path's `PageId` changed), re-verified against the live filesystem immediately before acting, with reserved/indexed-but-unwalked paths (`bootstrap.md`, `_meta.md`, `_pending/`, `sessions/*.md`) excluded and a circuit breaker that skips a whole scope when `> max(3, 50%)` of its pages look missing or a non-partial walk finds nothing; `ai-memory-store/src/ops.rs` `soft_delete_for_reconcile_if_latest` is a soft tombstone (`is_latest=0` + `superseded_at`, decay's shape) through the single writer actor in one transaction — never a filesystem write — scoped to the page's own `(workspace_id, project_id)` | `watcher.rs` `reconcile_delete_disabled_by_default_never_tombstones_a_missing_page`, `reconcile_delete_cross_project_isolation` (project A's deletion never tombstones project B's same-path page), `reconcile_tombstone_refuses_a_stale_latest_id`, `reconcile_tombstone_refuses_when_the_file_still_exists`, `reconcile_tombstone_never_touches_the_filesystem`; `ops.rs` `reconcile_tombstone_resurrects_instead_of_orphaning_on_recreate` | STRONG |
| 11a | Hook backpressure (202/429) + bounded fan-out (invariant #5) | `ai-memory-hooks/src/router.rs` semaphore→429, 202 immediately, `MAX_HOOK_BATCH_ITEMS`, bounded LRU limiter | `router.rs` `handle_hook_returns_429_when_ingest_saturated`, `ingest_rate_limiter_is_bounded` | STRONG |
| 11b | Capture exclusions drop before storage | `ai-memory-hooks` `capture_policy.rs` `inspect`→`Drop` (before semaphore/spawn), including shell commands whose arguments name an ignored path (`match_command`); generated OpenCode/OMP/Pi/OpenClaw integrations mirror it in `render_shared.rs` `ts_capture_policy_v1` (`captureMatchCommand`) | `capture_policy.rs` per-agent `…honors_exclusions` tests, `shell_fixture_vectors` (shared `capture-policy.json` `shell` drop/keep vectors: tool aliases incl. OpenClaw/Devin `exec`, `workdir`, glob directories, Windows paths), `shell_tool_shapes_of_every_adapter_honor_exclusions`, `fixture_vectors` (incl. TS-adapter `bash`/`exec` vectors); `router.rs` `capture_protocol_shell_decisions_survive_server_reinspection`; `hook.rs` `shell_command_reading_an_ignored_path_is_dropped_before_spool`; `render_shared.rs` `generated_capture_policy_v1_node_runtime_evidence` (runs the same fixture sections against the emitted TypeScript; `#[ignore]` locally, run with `--ignored` under Node 24 by the Linux CI test job) | STRONG |
| 11c | Capture hook ≤200ms budget (invariant #5) | `hooks/_lib.sh` capture path `curl --max-time 0.2` (context-fetch 1.0s and background drain 2.0s are separate, larger-budget paths) | none (shell-script timeout; hard to unit-test) — watch on any capture-path change | WATCH |