Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(retention): make decay eviction durable (#391)
AkitaOnRailscd318733b343
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c2818794..0758b87d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Fixed+- Forget-sweep decay now removes the authoritative Markdown file while+ conditionally tombstoning the selected page, preventing reconciliation from+ resurrecting evicted content. Aged cleanup recognizes rewritten heads,+ deletes their complete `supersedes` ancestry, ignores lifetime access counts,+ and preserves a newer page recreated at the same path. The corrected scoped+ tombstone index is added by a new migration rather than rewriting history+ (#391).- The `pages_fts_rows` and `observations_fts_rows` status counters now countindexed documents instead of content-table rows. Both FTS tables use externalcontent, so `SELECT COUNT(*)` against them was answered from `pages` /diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex 1d20e6a2..6e45eee7 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -1134,7 +1134,7 @@ async fn run_scheduled_sweep_tick({Ok(report) => {outcome.candidates_evaluated += report.candidates_evaluated;- outcome.evicted += report.evicted.len();+ outcome.evicted += report.evicted.iter().filter(|page| page.deleted).count();outcome.expired += report.expired.len();outcome.hard_deleted += report.hard_deleted;}@@ -2291,7 +2291,9 @@ mod tests {async fn two_project_wiki() -> (TempDir, Store, Wiki, WorkspaceId, ProjectId, ProjectId) {let tmp = TempDir::new().unwrap();let store = Store::open(tmp.path()).unwrap();- let wiki = Wiki::new(tmp.path(), store.writer.clone()).unwrap();+ let wiki = Wiki::new(tmp.path(), store.writer.clone())+ .unwrap()+ .with_store_reader(store.reader.clone());let ws = store.writer.get_or_create_workspace("default")diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex ac002de8..4424e635 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -57,7 +57,7 @@ pub struct DecaySettings {pub salience_default: f64,/// Soft-delete threshold.pub cold_threshold: f64,- /// Delay before hard-deleting an untouched soft-deleted page.+ /// Grace period before permanently deleting an evicted version chain.pub hard_delete_after_days: i64,/// Optional weight for the number of distinct authenticated readers.pub breadth_weight: f64,@@ -175,8 +175,8 @@ pub struct Config {pub embedding_base_url: Option<String>,/// M8 retention-sweep parameters. The defaults give an ~80-day/// "survival floor" for unused episodic content (above the cold- /// threshold), followed by ~180 days of soft-delete buffer before- /// hard-deletion. Tune `decay.lambda` down to slow decay or+ /// threshold), followed by ~180 days of tombstone grace before permanent+ /// version-chain deletion. Tune `decay.lambda` down to slow decay or/// `decay.cold_threshold` to evict more / less aggressively.pub decay: DecaySettings,/// Server-side scheduled maintenance. Jobs run outside hook latency.diff --git a/crates/ai-memory-consolidate/src/sweep.rs b/crates/ai-memory-consolidate/src/sweep.rsindex d319c020..e48224ab 100644--- a/crates/ai-memory-consolidate/src/sweep.rs+++ b/crates/ai-memory-consolidate/src/sweep.rs@@ -2,10 +2,11 @@//!//! Walks the `is_latest = 1` pages for a project, computes the retention score//! for each via [`ai_memory_store::retention_score_with_breadth`],-//! and soft-deletes those below threshold. Semantic / procedural /-//! working tiers are skipped (M8 policy: semantic compounds, only-//! episodic decays). Pinned pages (schema flag OR `pinned: true` in-//! frontmatter) are exempt regardless of tier.+//! and evicts those below threshold through the wiki layer: the authoritative+//! Markdown file is removed and the exact latest row becomes a decay+//! tombstone. Semantic / procedural / working tiers are skipped (M8 policy:+//! semantic compounds, only episodic decays). Pinned pages (schema flag OR+//! `pinned: true` in frontmatter) are exempt regardless of tier.//!//! TTL pass: pages whose frontmatter `expires_at:` is in the past are//! hard-deleted through the wiki layer (file + rows), regardless of@@ -13,10 +14,10 @@//! than a pin. `memory_lint` warns about pinned+expiring combos so the//! contradiction is visible before the delete lands.//!-//! Hard-delete pass cleans up rows soft-deleted more than-//! `hard_delete_after_days` ago that received zero subsequent access.-//! M7 supersession rows are safe: they have `supersedes IS NOT NULL`-//! and therefore never match the hard-delete predicate.+//! Hard-delete pass cleans up tombstones older than+//! `hard_delete_after_days` and their supersession ancestry. A page recreated+//! at the same path is a separate live chain and is preserved. Ordinary+//! supersession rows are safe because only decay writes `superseded_at`.use std::collections::HashMap;@@ -32,7 +33,7 @@ use thiserror::Error;/// One evicted page surfaced in the [`SweepReport`].#[derive(Debug, Clone, Serialize)]pub struct EvictedPage {- /// Identifier of the soft-deleted page.+ /// Identifier of the page selected for eviction.pub id: PageId,/// Relative wiki path.pub path: String,@@ -42,6 +43,9 @@ pub struct EvictedPage {pub age_days: f64,/// Total access count.pub access_count: u32,+ /// `true` when the Markdown removal and tombstone write landed. Always+ /// `false` on `dry_run`; failures are retried by the next sweep.+ pub deleted: bool,}/// One TTL-expired page surfaced in the [`SweepReport`].@@ -67,13 +71,13 @@ pub struct SweepReport {pub dry_run: bool,/// Total candidates evaluated (all tiers, before filtering).pub candidates_evaluated: usize,- /// Pages that fell below the cold threshold (soft-deleted unless- /// `dry_run`).+ /// Pages that fell below the cold threshold (evicted through the wiki+ /// layer unless `dry_run`).pub evicted: Vec<EvictedPage>,/// Pages past their frontmatter `expires_at:` TTL (hard-deleted/// through the wiki layer unless `dry_run`).pub expired: Vec<ExpiredPage>,- /// Number of older soft-deleted rows hard-deleted on this pass.+ /// Number of page-version rows permanently deleted on this pass.pub hard_deleted: usize,}@@ -90,21 +94,21 @@ pub enum SweepError {}const US_PER_DAY: f64 = 86_400_000_000.0;+const US_PER_DAY_I64: i64 = 86_400_000_000;/// Run a sweep against the given workspace/project.///-/// `wiki` routes TTL deletions through the wiki layer so the markdown-/// file is removed together with the rows (deleting only store rows-/// would let the watcher re-index the file). Callers without a wiki-/// handle (bare-store tests) pass `None`; expired pages are then reported-/// but left intact. A store-only delete would leave the authoritative file-/// behind for the watcher to re-index.+/// `wiki` routes decay and TTL deletions through the wiki layer so the+/// Markdown source of truth is removed with the index mutation. Callers+/// without a wiki handle (bare-store tests) pass `None`; affected pages are+/// reported but left intact. A store-only mutation would let reconciliation+/// re-index the authoritative file.////// # Errors/// Propagates any store error encountered while reading candidates or-/// writing soft-deletions. Per-page TTL delete failures (rejecting-/// admission webhook, IO error) are reported in the-/// [`SweepReport::expired`] entries instead of aborting the sweep.+/// writing tombstones. Per-page Wiki failures (rejecting admission+/// webhook, IO error) are reported in the corresponding entry instead of+/// aborting the sweep.pub async fn run_sweep(reader: &ReaderPool,writer: &WriterHandle,@@ -135,7 +139,7 @@ pub async fn run_sweep(#[allow(clippy::too_many_arguments)]pub async fn run_sweep_with_breadth(reader: &ReaderPool,- writer: &WriterHandle,+ _writer: &WriterHandle,wiki: Option<&Wiki>,workspace_id: WorkspaceId,project_id: ProjectId,@@ -152,7 +156,6 @@ pub async fn run_sweep_with_breadth(let now_us = Timestamp::now().as_microsecond();let mut evicted = Vec::new();- let mut to_evict_ids: Vec<PageId> = Vec::new();let mut expired: Vec<ExpiredPage> = Vec::new();for c in &candidates {@@ -190,8 +193,8 @@ pub async fn run_sweep_with_breadth(retention: score,age_days,access_count: c.access_count,+ deleted: false,});- to_evict_ids.push(c.id);}}@@ -227,12 +230,60 @@ pub async fn run_sweep_with_breadth(}}}- if !to_evict_ids.is_empty() {- writer.soft_delete_for_decay(to_evict_ids).await?;+ for page in &mut evicted {+ let path = match ai_memory_core::PagePath::new(page.path.clone()) {+ Ok(path) => path,+ Err(_) => continue,+ };+ let result = match wiki {+ Some(wiki) => wiki+ .evict_page_if_latest(workspace_id, project_id, &path, page.id, None)+ .await+ .map_err(|error| error.to_string()),+ None => Err("wiki unavailable; refusing store-only decay eviction".to_string()),+ };+ match result {+ Ok(true) => page.deleted = true,+ Ok(false) => tracing::debug!(+ path = %page.path,+ "forget sweep: page changed after decay selection; skipping stale eviction"+ ),+ Err(error) => tracing::warn!(+ path = %page.path,+ error,+ "forget sweep: decay eviction failed; retrying on next sweep"+ ),+ }+ }++ if let Some(wiki) = wiki {+ let grace_days = params.hard_delete_after_days.max(0);+ let cutoff_us = Timestamp::now()+ .as_microsecond()+ .saturating_sub(grace_days.saturating_mul(US_PER_DAY_I64));+ let tombstones = reader+ .decay_tombstones_before(workspace_id, project_id, cutoff_us)+ .await?;+ for tombstone in tombstones {+ match wiki+ .hard_delete_decay_tombstone(+ workspace_id,+ project_id,+ &tombstone.path,+ tombstone.id,+ cutoff_us,+ )+ .await+ {+ Ok(deleted) => hard_deleted += deleted,+ Err(error) => tracing::warn!(+ path = %tombstone.path,+ %error,+ "forget sweep: decay tombstone cleanup failed; retrying on next sweep"+ ),+ }+ }}- hard_deleted = writer- .hard_delete_decayed(workspace_id, project_id, params.hard_delete_after_days)- .await?;}Ok(SweepReport {diff --git a/crates/ai-memory-consolidate/tests/lifecycle.rs b/crates/ai-memory-consolidate/tests/lifecycle.rsindex 2f12b9df..65e16a22 100644--- a/crates/ai-memory-consolidate/tests/lifecycle.rs+++ b/crates/ai-memory-consolidate/tests/lifecycle.rs@@ -262,7 +262,10 @@ async fn m8_retention_lifecycle_end_to_end() {assert_eq!(counts_before.pages_latest as usize, FIXTURES.len());assert_eq!(counts_before.pages_all as usize, FIXTURES.len());- let real = run_sweep(+ // Destructive decay without the authoritative Wiki handle must fail+ // closed. Mutating only SQLite would leave the files for reconciliation+ // to recreate as live pages.+ let no_wiki = run_sweep(&store.reader,&store.writer,None,@@ -272,10 +275,38 @@ async fn m8_retention_lifecycle_end_to_end() {/* dry_run */ false,).await+ .expect("store-only sweep");+ assert!(no_wiki.evicted.iter().all(|page| !page.deleted));+ let counts_after_refusal = store+ .reader+ .status_counts()+ .await+ .expect("counts after refusal");+ assert_eq!(+ counts_after_refusal.pages_latest,+ counts_before.pages_latest+ );+ assert_eq!(counts_after_refusal.pages_all, counts_before.pages_all);++ let real = run_sweep(+ &store.reader,+ &store.writer,+ Some(&wiki),+ ws,+ proj,+ ¶ms,+ /* dry_run */ false,+ )+ .await.expect("real sweep");assert!(!real.dry_run);let expected_evicted_count = FIXTURES.iter().filter(|f| f.expected_evicted).count();assert_eq!(real.evicted.len(), expected_evicted_count);+ assert!(+ real.evicted.iter().all(|page| page.deleted),+ "every selected page should complete its wiki-backed eviction: {:?}",+ real.evicted,+ );let counts_after = store.reader.status_counts().await.expect("counts after");assert_eq!(@@ -288,6 +319,15 @@ async fn m8_retention_lifecycle_end_to_end() {FIXTURES.len(),"soft-delete preserves the row (kept for 180d hard-delete window)",);+ let project_root = wiki.project_root(ws, proj);+ for fixture in FIXTURES {+ assert_eq!(+ project_root.join(fixture.path).exists(),+ !fixture.expected_evicted,+ "the Markdown source must be removed exactly for evicted pages: {}",+ fixture.path,+ );+ }// ── Phase 6 — FTS5 invariants ───────────────────────────────// Keywords unique to evicted pages should disappear from search.@@ -690,7 +730,9 @@ async fn aged_decay_cleanup_stays_within_the_requested_scope() {.get_or_create_project(other_ws, "target", None).await.expect("other workspace project");- let wiki = Wiki::new(tmp.path(), store.writer.clone()).expect("wiki");+ let wiki = Wiki::new(tmp.path(), store.writer.clone())+ .expect("wiki")+ .with_store_reader(store.reader.clone());for (workspace_id, project_id, path, entity) in [(ws, target, "sessions/target.md", "target-entity"),@@ -724,11 +766,13 @@ async fn aged_decay_cleanup_stays_within_the_requested_scope() {let conn = rusqlite::Connection::open(store.db_path()).expect("open aux conn");conn.pragma_update(None, "busy_timeout", 5_000).unwrap();conn.execute(- "UPDATE pages SET is_latest = 0, superseded_at = 1, access_count = 0",+ "UPDATE pages SET is_latest = 0, superseded_at = 1, access_count = 7",[],).expect("age tombstone fixtures");drop(conn);+ std::fs::remove_file(wiki.abs_path(ws, target, &PagePath::new("sessions/target.md").unwrap()))+ .expect("simulate wiki-backed target eviction");let params = DecayParams {hard_delete_after_days: 0,@@ -737,7 +781,7 @@ async fn aged_decay_cleanup_stays_within_the_requested_scope() {let report = run_sweep(&store.reader,&store.writer,- None,+ Some(&wiki),ws,target,¶ms,@@ -791,6 +835,245 @@ async fn aged_decay_cleanup_stays_within_the_requested_scope() {);}+#[tokio::test]+async fn rewritten_decay_eviction_removes_the_file_and_entire_version_chain() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("open store");+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .expect("workspace");+ let proj = store+ .writer+ .get_or_create_project(ws, "rewritten", None)+ .await+ .expect("project");+ let wiki = Wiki::new(tmp.path(), store.writer.clone())+ .expect("wiki")+ .with_store_reader(store.reader.clone());+ let path = PagePath::new("sessions/rewritten.md").unwrap();++ let write = |body: &str, entity: &str| WritePageRequest {+ workspace_id: ws,+ project_id: proj,+ path: path.clone(),+ frontmatter: serde_json::json!({"entities": [entity]}),+ body: body.to_string(),+ tier: Tier::Episodic,+ pinned: false,+ title: Some("Rewritten".into()),+ admission_ctx: None,+ author_id: None,+ actor: ai_memory_core::ActorContext::anonymous(),+ };+ let first = wiki+ .write_page(write("# Rewritten\nfirst-version-marker", "first-entity"))+ .await+ .unwrap();+ let second = wiki+ .write_page(write("# Rewritten\nsecond-version-marker", "second-entity"))+ .await+ .unwrap();++ let conn = rusqlite::Connection::open(store.db_path()).unwrap();+ conn.execute(+ "UPDATE pages SET updated_at = 1, access_count = 7 WHERE id = ?1",+ params![second.as_bytes()],+ )+ .unwrap();+ drop(conn);++ let report = run_sweep(+ &store.reader,+ &store.writer,+ Some(&wiki),+ ws,+ proj,+ &DecayParams {+ hard_delete_after_days: 0,+ ..DecayParams::default()+ },+ false,+ )+ .await+ .unwrap();+ assert_eq!(report.evicted.len(), 1);+ assert!(report.evicted[0].deleted);+ assert_eq!(+ report.hard_deleted, 2,+ "the evicted head and its superseded ancestor are both purged"+ );+ assert!(!wiki.abs_path(ws, proj, &path).exists());++ let conn = rusqlite::Connection::open(store.db_path()).unwrap();+ let remaining: i64 = conn+ .query_row(+ "SELECT count(*) FROM pages WHERE id IN (?1, ?2)",+ params![first.as_bytes(), second.as_bytes()],+ |row| row.get(0),+ )+ .unwrap();+ let entities: i64 = conn+ .query_row(+ "SELECT count(*) FROM entities WHERE workspace_id = ?1 AND project_id = ?2",+ params![ws.as_bytes(), proj.as_bytes()],+ |row| row.get(0),+ )+ .unwrap();+ assert_eq!(remaining, 0);+ assert_eq!(+ entities, 0,+ "orphaned entity rows are cleaned with the chain"+ );+ assert!(+ store+ .reader+ .search_pages_for_project(ws, proj, "second-version-marker".into(), 5, None)+ .await+ .unwrap()+ .is_empty(),+ "the deleted chain must leave no searchable FTS entry"+ );+}++#[tokio::test]+async fn hard_delete_preserves_a_page_recreated_at_the_same_path() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("open store");+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .expect("workspace");+ let proj = store+ .writer+ .get_or_create_project(ws, "recreated", None)+ .await+ .expect("project");+ let wiki = Wiki::new(tmp.path(), store.writer.clone())+ .expect("wiki")+ .with_store_reader(store.reader.clone());+ let path = PagePath::new("sessions/recreated.md").unwrap();++ let write = |body: &str| WritePageRequest {+ workspace_id: ws,+ project_id: proj,+ path: path.clone(),+ frontmatter: serde_json::json!({}),+ body: body.to_string(),+ tier: Tier::Episodic,+ pinned: false,+ title: Some("Recreated".into()),+ admission_ctx: None,+ author_id: None,+ actor: ai_memory_core::ActorContext::anonymous(),+ };+ let first = wiki+ .write_page(write("# Recreated\nold one"))+ .await+ .unwrap();+ let second = wiki+ .write_page(write("# Recreated\nold two"))+ .await+ .unwrap();+ let conn = rusqlite::Connection::open(store.db_path()).unwrap();+ conn.execute(+ "UPDATE pages SET updated_at = 1 WHERE id = ?1",+ params![second.as_bytes()],+ )+ .unwrap();+ drop(conn);++ let grace = run_sweep(+ &store.reader,+ &store.writer,+ Some(&wiki),+ ws,+ proj,+ &DecayParams {+ hard_delete_after_days: 30,+ ..DecayParams::default()+ },+ false,+ )+ .await+ .unwrap();+ assert!(grace.evicted[0].deleted);+ assert_eq!(grace.hard_deleted, 0);+ assert!(!wiki.abs_path(ws, proj, &path).exists());++ // Simulate an external editor recreating the authoritative file before+ // the watcher gets a chance to index it. Cleanup must notice and reindex+ // this file rather than treating the absent latest row as permission to+ // delete it.+ std::fs::write(+ wiki.abs_path(ws, proj, &path),+ "# Recreated\nnew-live-marker",+ )+ .unwrap();+ let conn = rusqlite::Connection::open(store.db_path()).unwrap();+ conn.execute(+ "UPDATE pages SET superseded_at = 1 WHERE id = ?1",+ params![second.as_bytes()],+ )+ .unwrap();+ drop(conn);++ let cleanup = run_sweep(+ &store.reader,+ &store.writer,+ Some(&wiki),+ ws,+ proj,+ &DecayParams {+ cold_threshold: 0.0,+ hard_delete_after_days: 0,+ ..DecayParams::default()+ },+ false,+ )+ .await+ .unwrap();+ assert!(cleanup.evicted.is_empty());+ assert_eq!(cleanup.hard_deleted, 2);+ assert!(wiki.abs_path(ws, proj, &path).exists());+ let recreated = store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap()+ .expect("the external recreation is indexed before cleanup");+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(recreated),+ );++ let conn = rusqlite::Connection::open(store.db_path()).unwrap();+ let old_rows: i64 = conn+ .query_row(+ "SELECT count(*) FROM pages WHERE id IN (?1, ?2)",+ params![first.as_bytes(), second.as_bytes()],+ |row| row.get(0),+ )+ .unwrap();+ assert_eq!(old_rows, 0);+ assert!(+ store+ .reader+ .search_pages_for_project(ws, proj, "new-live-marker".into(), 5, None)+ .await+ .unwrap()+ .iter()+ .any(|hit| hit.id == recreated),+ "the recreated page and its FTS entry must survive old-chain cleanup"+ );+}+fn ws_dir_for(tmp: &TempDir, ws: WorkspaceId, proj: ProjectId) -> std::path::PathBuf {tmp.path().join("wiki")diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rsindex 87eb8b13..c82afee4 100644--- a/crates/ai-memory-mcp/src/server.rs+++ b/crates/ai-memory-mcp/src/server.rs@@ -2280,8 +2280,8 @@ impl AiMemoryServer {#[tool(description = "Run the retention sweep: walk is_latest=1 \episodic pages, score them with the agentmemory-style retention \formula (salience * exp(-lambda * age) + sigma * log(1 + accesses) \- * exp(-mu * days_since_access)), and soft-delete those below the \- cold threshold. Semantic / procedural / pinned pages are exempt. \+ * exp(-mu * days_since_access)), and evict those below the cold \+ threshold through the wiki layer. Semantic / procedural / pinned pages are exempt. \Pass dry_run=true to preview.")]async fn memory_forget_sweep(&self,diff --git a/crates/ai-memory-store/migrations/V49__decay_tombstone_index.sql b/crates/ai-memory-store/migrations/V49__decay_tombstone_index.sqlnew file mode 100644index 00000000..e0bb30b1--- /dev/null+++ b/crates/ai-memory-store/migrations/V49__decay_tombstone_index.sql@@ -0,0 +1,10 @@+-- A decay tombstone is identified by superseded_at, the only column written+-- by the forget-sweep eviction path. Rewritten page heads legitimately carry+-- a non-NULL supersedes pointer, so filtering those rows out made their+-- tombstones permanent and left this partial index unusable by the corrected+-- cleanup query.+DROP INDEX idx_pages_evicted;++CREATE INDEX idx_pages_evicted+ ON pages(workspace_id, project_id, superseded_at)+ WHERE is_latest = 0 AND superseded_at IS NOT NULL;diff --git a/crates/ai-memory-store/src/decay.rs b/crates/ai-memory-store/src/decay.rsindex 842cf3a8..40d41fa1 100644--- a/crates/ai-memory-store/src/decay.rs+++ b/crates/ai-memory-store/src/decay.rs@@ -27,8 +27,8 @@ pub struct DecayParams {pub salience_default: f64,/// Below this score, an episodic page is a soft-delete candidate.pub cold_threshold: f64,- /// Days a soft-deleted (sweep-evicted) page must survive before- /// hard-delete, *with* zero subsequent access.+ /// Days an evicted page's tombstone and version ancestry survive before+ /// permanent deletion.pub hard_delete_after_days: i64,}diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex 606d4875..d8280da7 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -49,7 +49,7 @@ pub use ops::{pub use reader::{ActivityWindow, AgentSessionCount, AutoImproveCandidateSession, BriefPageBody, BriefingPage,BriefingSnapshot, ClientActivity, ContaminationFinding, ContaminationReport,- ContaminationSummary, DecayCandidate, DerivedIndexStatus, EmbeddingTripleCount,+ ContaminationSummary, DecayCandidate, DecayTombstone, DerivedIndexStatus, EmbeddingTripleCount,FeedbackFinding, GraphVia, HealthDetail, HealthPage, ObservationHit, OpenSession, PageAuthor,PageHit, PageHitWithMeta, PageLinks, PageMeta, PageSummary, ProjectSummary, ReaderPool,ReindexTargetStatus, RelatedPage, RrfContributions, ScopeRow, SearchExplain,diff --git a/crates/ai-memory-store/src/migrations.rs b/crates/ai-memory-store/src/migrations.rsindex be2a9ac0..1c1e424d 100644--- a/crates/ai-memory-store/src/migrations.rs+++ b/crates/ai-memory-store/src/migrations.rs@@ -732,4 +732,34 @@ mod tests {assert_eq!(schema_object_count(&conn, kind, name), 1, "missing {name}");}}++ #[test]+ fn v48_to_v49_replaces_the_decay_tombstone_index_predicate() {+ let mut conn = Connection::open_in_memory().unwrap();+ run_to(&mut conn, 48).unwrap();++ let before: String = conn+ .query_row(+ "SELECT sql FROM sqlite_master WHERE type = 'index' AND name = 'idx_pages_evicted'",+ [],+ |row| row.get(0),+ )+ .unwrap();+ assert!(before.contains("supersedes IS NULL"), "{before}");++ run(&mut conn).unwrap();+ let after: String = conn+ .query_row(+ "SELECT sql FROM sqlite_master WHERE type = 'index' AND name = 'idx_pages_evicted'",+ [],+ |row| row.get(0),+ )+ .unwrap();+ assert!(+ after.contains("workspace_id, project_id, superseded_at"),+ "{after}"+ );+ assert!(after.contains("superseded_at IS NOT NULL"), "{after}");+ assert!(!after.contains("supersedes IS NULL"), "{after}");+ }}diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex 6eb1f676..ec6c9593 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -1442,38 +1442,52 @@ pub fn record_page_feedback(Ok(Some((page_id, next_salience)))}-/// Mark a set of `is_latest=1` pages as soft-deleted by the forget-/// sweep. Distinguished from M7 supersession by `supersedes IS NULL`.-pub fn soft_delete_for_decay(conn: &mut Connection, page_ids: &[PageId]) -> StoreResult<usize> {- if page_ids.is_empty() {- return Ok(0);- }+/// Mark the expected latest page as evicted by the forget sweep.+///+/// The full identity and latest-id check share one transaction so a stale+/// sweep candidate cannot evict a page that was rewritten after selection.+/// `superseded_at` is the decay-tombstone marker; unlike `supersedes`, it is+/// never populated by ordinary page versioning.+pub fn soft_delete_for_decay_if_latest(+ conn: &mut Connection,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: &PagePath,+ expected_latest_id: PageId,+) -> StoreResult<bool> {let now = Timestamp::now().as_microsecond();- let mut affected = 0usize;let tx = conn.transaction()?;- {- let mut stmt = tx.prepare(- "UPDATE pages \- SET is_latest = 0, superseded_at = ?1 \- WHERE id = ?2 AND is_latest = 1",+ let affected = tx.execute(+ "UPDATE pages \+ SET is_latest = 0, superseded_at = ?1 \+ WHERE id = ?2 \+ AND workspace_id = ?3 \+ AND project_id = ?4 \+ AND path = ?5 \+ AND is_latest = 1",+ params![+ now,+ expected_latest_id.as_bytes(),+ workspace_id.as_bytes(),+ project_id.as_bytes(),+ path.as_str(),+ ],+ )?;+ if affected != 0 {+ audit(+ &tx,+ "soft_delete_for_decay",+ Some(workspace_id.as_bytes()),+ Some(project_id.as_bytes()),+ Some(expected_latest_id.as_bytes()),+ // Decay is a scheduled/admin system operation rather than a+ // user-attributable page edit.+ None,+ now,)?;- for id in page_ids {- affected += stmt.execute(params![now, id.as_bytes()])?;- }}- audit(- &tx,- "soft_delete_for_decay",- None,- None,- None,- // Decay sweep is a system op (scheduled / admin-triggered) — no- // user-attributable actor at the row level.- None,- Timestamp::now().as_microsecond(),- )?;tx.commit()?;- Ok(affected)+ Ok(affected != 0)}/// Delete every version of a page (by path) from the index. Used when the@@ -1578,29 +1592,67 @@ fn delete_page_inner(Ok(rows > 0)}-/// Hard-delete rows in one workspace/project that were soft-deleted by an-/// earlier sweep at least `hard_delete_after_days` ago AND received zero-/// subsequent accesses. Safe: M7 supersedes-chain pages have a non-null-/// `supersedes` so they never match. Orphaned entity-index rows from those-/// page deletions are removed in the same transaction.-pub fn hard_delete_decayed_pages(+/// Permanently delete one eligible decay tombstone and its ancestry chain.+///+/// The tombstone id, full path identity, cutoff, and current latest-page state+/// are all rechecked in the transaction. This lets the wiki layer remove the+/// authoritative file only when the path has no newer live page, while still+/// allowing an old chain to be cleaned after that path was deliberately+/// recreated. Ordinary supersession rows have `superseded_at IS NULL` and+/// cannot become roots of this deletion.+pub fn hard_delete_decayed_page_chain(conn: &mut Connection,workspace_id: WorkspaceId,project_id: ProjectId,- hard_delete_after_days: i64,+ path: &PagePath,+ tombstone_id: PageId,+ expected_latest_id: Option<PageId>,+ cutoff_us: i64,) -> StoreResult<usize> {- let cutoff = Timestamp::now().as_microsecond() - hard_delete_after_days * 86_400_000_000;let tx = conn.transaction()?;+ let current_latest: Option<Vec<u8>> = tx+ .query_row(+ "SELECT id FROM pages \+ WHERE workspace_id = ?1 AND project_id = ?2 AND path = ?3 AND is_latest = 1",+ params![+ workspace_id.as_bytes(),+ project_id.as_bytes(),+ path.as_str()+ ],+ |row| row.get(0),+ )+ .optional()?;+ let expected_latest = expected_latest_id.map(|id| id.as_bytes().to_vec());+ if current_latest != expected_latest {+ return Ok(0);+ }let n = tx.execute(- "DELETE FROM pages \- WHERE workspace_id = ?1 \- AND project_id = ?2 \- AND is_latest = 0 \- AND supersedes IS NULL \- AND superseded_at IS NOT NULL \- AND superseded_at < ?3 \- AND access_count = 0",- params![workspace_id.as_bytes(), project_id.as_bytes(), cutoff],+ "WITH RECURSIVE decay_chain(id) AS ( \+ SELECT id FROM pages \+ WHERE id = ?1 \+ AND workspace_id = ?2 \+ AND project_id = ?3 \+ AND path = ?4 \+ AND is_latest = 0 \+ AND superseded_at IS NOT NULL \+ AND superseded_at <= ?5 \+ UNION \+ SELECT parent.id \+ FROM decay_chain c \+ JOIN pages child ON child.id = c.id \+ JOIN pages parent ON parent.id = child.supersedes \+ WHERE parent.workspace_id = ?2 \+ AND parent.project_id = ?3 \+ AND parent.path = ?4 \+ ) \+ DELETE FROM pages WHERE id IN (SELECT id FROM decay_chain)",+ params![+ tombstone_id.as_bytes(),+ workspace_id.as_bytes(),+ project_id.as_bytes(),+ path.as_str(),+ cutoff_us,+ ],)?;if n > 0 {tx.execute(@@ -1611,6 +1663,15 @@ pub fn hard_delete_decayed_pages()",params![workspace_id.as_bytes(), project_id.as_bytes()],)?;+ audit(+ &tx,+ "hard_delete_decayed",+ Some(workspace_id.as_bytes()),+ Some(project_id.as_bytes()),+ Some(tombstone_id.as_bytes()),+ None,+ Timestamp::now().as_microsecond(),+ )?;}tx.commit()?;Ok(n)diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rsindex c3fa98c5..5ccad44c 100644--- a/crates/ai-memory-store/src/reader.rs+++ b/crates/ai-memory-store/src/reader.rs@@ -2509,6 +2509,47 @@ impl ReaderPool {.await}+ /// Return decay tombstones old enough for permanent cleanup.+ ///+ /// `superseded_at` is written only by the forget-sweep eviction path, so+ /// it is the tombstone discriminator regardless of whether a rewritten+ /// page head also has a `supersedes` ancestor. The caller still routes+ /// each result through the wiki layer before deleting its version chain.+ ///+ /// # Errors+ /// Propagates any SQL or pool error.+ pub async fn decay_tombstones_before(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ cutoff_us: i64,+ ) -> StoreResult<Vec<DecayTombstone>> {+ self.with_conn(move |conn| {+ let mut stmt = conn.prepare(+ "SELECT id, path FROM pages \+ WHERE workspace_id = ?1 AND project_id = ?2 \+ AND is_latest = 0 \+ AND superseded_at IS NOT NULL \+ AND superseded_at <= ?3 \+ ORDER BY superseded_at, id",+ )?;+ let rows = stmt.query_map(+ params![workspace_id.as_bytes(), project_id.as_bytes(), cutoff_us],+ |row| Ok((row.get::<_, Vec<u8>>(0)?, row.get::<_, String>(1)?)),+ )?;+ let mut out = Vec::new();+ for row in rows {+ let (id, path) = row?;+ out.push(DecayTombstone {+ id: PageId::from_slice(&id)?,+ path: PagePath::new(path)?,+ });+ }+ Ok(out)+ })+ .await+ }+/// Return the number of DISTINCT operators that reinforced each/// `is_latest = 1` page of a project, for the sweep's breadth term.///@@ -6339,6 +6380,15 @@ pub struct DecayCandidate {pub salience: Option<f64>,}+/// One forget-sweep tombstone eligible for permanent cleanup.+#[derive(Debug, Clone, Serialize)]+pub struct DecayTombstone {+ /// Evicted head whose ancestry chain is awaiting deletion.+ pub id: PageId,+ /// Relative wiki path associated with the evicted chain.+ pub path: PagePath,+}+fn row_to_decay_candidate(row: &rusqlite::Row<'_>,) -> rusqlite::Result<StoreResult<DecayCandidate>> {diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rsindex 40535672..894a1f8b 100644--- a/crates/ai-memory-store/src/writer.rs+++ b/crates/ai-memory-store/src/writer.rs@@ -207,14 +207,20 @@ pub(crate) enum WriteCmd {params: crate::decay::DecayParams,reply: oneshot::Sender<StoreResult<Option<(PageId, f64)>>>,},- SoftDeleteForDecay {- page_ids: Vec<PageId>,- reply: oneshot::Sender<StoreResult<usize>>,+ SoftDeleteForDecayIfLatest {+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: PagePath,+ expected_latest_id: PageId,+ reply: oneshot::Sender<StoreResult<bool>>,},- HardDeleteDecayed {+ HardDeleteDecayedPageChain {workspace_id: WorkspaceId,project_id: ProjectId,- hard_delete_after_days: i64,+ path: PagePath,+ tombstone_id: PageId,+ expected_latest_id: Option<PageId>,+ cutoff_us: i64,reply: oneshot::Sender<StoreResult<usize>>,},HealCatchAllRepoPaths {@@ -973,36 +979,53 @@ impl WriterHandle {rx.await.map_err(|_| StoreError::WriterClosed)?}- /// Soft-delete pages identified by the M8 forget sweep.+ /// Soft-delete the expected latest page identified by the forget sweep.////// # Errors/// Returns [`StoreError::WriterClosed`] or propagates SQL errors.- pub async fn soft_delete_for_decay(&self, page_ids: Vec<PageId>) -> StoreResult<usize> {+ pub async fn soft_delete_for_decay_if_latest(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: PagePath,+ expected_latest_id: PageId,+ ) -> StoreResult<bool> {let (tx, rx) = oneshot::channel();- self.send(WriteCmd::SoftDeleteForDecay {- page_ids,+ self.send(WriteCmd::SoftDeleteForDecayIfLatest {+ workspace_id,+ project_id,+ path,+ expected_latest_id,reply: tx,}).await?;rx.await.map_err(|_| StoreError::WriterClosed)?}- /// Hard-delete pages in one workspace/project that were soft-deleted by- /// the sweep more than `hard_delete_after_days` ago.+ /// Permanently delete one eligible decay tombstone and its ancestry chain.+ /// The expected latest-page state is checked in the same transaction so a+ /// page recreated at the same path cannot be removed accidentally.////// # Errors/// Returns [`StoreError::WriterClosed`] or propagates SQL errors.- pub async fn hard_delete_decayed(+ #[allow(clippy::too_many_arguments)]+ pub async fn hard_delete_decayed_page_chain(&self,workspace_id: WorkspaceId,project_id: ProjectId,- hard_delete_after_days: i64,+ path: PagePath,+ tombstone_id: PageId,+ expected_latest_id: Option<PageId>,+ cutoff_us: i64,) -> StoreResult<usize> {let (tx, rx) = oneshot::channel();- self.send(WriteCmd::HardDeleteDecayed {+ self.send(WriteCmd::HardDeleteDecayedPageChain {workspace_id,project_id,- hard_delete_after_days,+ path,+ tombstone_id,+ expected_latest_id,+ cutoff_us,reply: tx,}).await?;@@ -1851,23 +1874,41 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {let result = ops::bump_client_activity(&mut conn, &entries);send_or_warn(reply, result, "bump_client_activity");}- WriteCmd::SoftDeleteForDecay { page_ids, reply } => {- let result = ops::soft_delete_for_decay(&mut conn, &page_ids);- send_or_warn(reply, result, "soft_delete_for_decay");+ WriteCmd::SoftDeleteForDecayIfLatest {+ workspace_id,+ project_id,+ path,+ expected_latest_id,+ reply,+ } => {+ let result = ops::soft_delete_for_decay_if_latest(+ &mut conn,+ workspace_id,+ project_id,+ &path,+ expected_latest_id,+ );+ send_or_warn(reply, result, "soft_delete_for_decay_if_latest");}- WriteCmd::HardDeleteDecayed {+ WriteCmd::HardDeleteDecayedPageChain {workspace_id,project_id,- hard_delete_after_days,+ path,+ tombstone_id,+ expected_latest_id,+ cutoff_us,reply,} => {- let result = ops::hard_delete_decayed_pages(+ let result = ops::hard_delete_decayed_page_chain(&mut conn,workspace_id,project_id,- hard_delete_after_days,+ &path,+ tombstone_id,+ expected_latest_id,+ cutoff_us,);- send_or_warn(reply, result, "hard_delete_decayed_pages");+ send_or_warn(reply, result, "hard_delete_decayed_page_chain");}WriteCmd::HealCatchAllRepoPaths { home, reply } => {let result = ops::heal_catch_all_repo_paths(&mut conn, home.as_deref());diff --git a/crates/ai-memory-wiki/src/wiki.rs b/crates/ai-memory-wiki/src/wiki.rsindex eff52db0..2fadd040 100644--- a/crates/ai-memory-wiki/src/wiki.rs+++ b/crates/ai-memory-wiki/src/wiki.rs@@ -33,6 +33,16 @@ pub struct ReindexSummary {pub pages: usize,}+enum PageStoreRemoval {+ Delete {+ author_id: Option<UserId>,+ expected_latest_id: Option<PageId>,+ },+ Decay {+ expected_latest_id: PageId,+ },+}+/// Wiki filesystem handle.////// Owns the path of the wiki root (`<data_dir>/wiki/`) and a cloneable@@ -475,13 +485,15 @@ impl Wiki {let _guard = self.mutation_lock.read().await;self.ensure_project_workspace(workspace_id, project_id).await?;- self.delete_page_locked(+ self.remove_page_locked(workspace_id,project_id,path,admission_ctx,- author_id,- None,+ PageStoreRemoval::Delete {+ author_id,+ expected_latest_id: None,+ },).await.map(|_| ())@@ -518,25 +530,63 @@ impl Wiki {if current != Some(expected_latest_id) {return Ok(false);}- self.delete_page_locked(+ self.remove_page_locked(workspace_id,project_id,path,admission_ctx,- None,- Some(expected_latest_id),+ PageStoreRemoval::Delete {+ author_id: None,+ expected_latest_id: Some(expected_latest_id),+ },).await}- async fn delete_page_locked(+ /// Remove the authoritative file and soft-delete the expected latest page+ /// as a decay tombstone. A stale candidate leaves both disk and store+ /// untouched.+ ///+ /// # Errors+ /// Returns [`WikiError`] when the store reader is unavailable, or on a+ /// filesystem, store, or rejecting-webhook error.+ pub async fn evict_page_if_latest(&self,workspace_id: WorkspaceId,project_id: ProjectId,path: &PagePath,+ expected_latest_id: PageId,admission_ctx: Option<AdmissionContext>,- author_id: Option<ai_memory_core::UserId>,- expected_latest_id: Option<PageId>,+ ) -> WikiResult<bool> {+ let _guard = self.mutation_lock.write().await;+ self.ensure_project_workspace(workspace_id, project_id)+ .await?;+ let reader = self.store_reader.as_ref().ok_or_else(|| {+ ai_memory_wiki_error("conditional page eviction requires a store reader")+ })?;+ let current = reader+ .latest_page_id_by_ids(workspace_id, project_id, path.as_str().to_string())+ .await?;+ if current != Some(expected_latest_id) {+ return Ok(false);+ }+ self.remove_page_locked(+ workspace_id,+ project_id,+ path,+ admission_ctx,+ PageStoreRemoval::Decay { expected_latest_id },+ )+ .await+ }++ async fn remove_page_locked(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: &PagePath,+ admission_ctx: Option<AdmissionContext>,+ removal: PageStoreRemoval,) -> WikiResult<bool> {let mut resolved_ctx = None;if let Some(chain) = &self.admission_chain {@@ -554,17 +604,33 @@ impl Wiki {Err(e) => return Err(crate::WikiError::Io(e)),};- let delete_result = match expected_latest_id {- Some(expected) => {+ let delete_result = match removal {+ PageStoreRemoval::Delete {+ expected_latest_id: Some(expected),+ ..+ } => {self.writer.delete_page_if_latest(workspace_id, project_id, path.clone(), expected).await}- None => self+ PageStoreRemoval::Delete {+ author_id,+ expected_latest_id: None,+ } => self.writer.delete_page(workspace_id, project_id, path.clone(), author_id).await.map(|()| true),+ PageStoreRemoval::Decay { expected_latest_id } => {+ self.writer+ .soft_delete_for_decay_if_latest(+ workspace_id,+ project_id,+ path.clone(),+ expected_latest_id,+ )+ .await+ }};let deleted = match delete_result {Ok(deleted) => deleted,@@ -588,6 +654,57 @@ impl Wiki {Ok(true)}+ /// Permanently delete one aged decay tombstone and its ancestry chain.+ ///+ /// An existing Markdown file is always preserved. If the watcher has not+ /// indexed it yet (including a legacy pre-fix eviction or an external+ /// recreation), this method reindexes it under the exclusive mutation+ /// guard before deleting only the old chain rooted at `tombstone_id`.+ /// The writer repeats the observed latest-id check so a concurrent store+ /// mutation fails closed.+ ///+ /// # Errors+ /// Returns [`WikiError`] when the store reader is unavailable, or on a+ /// filesystem, store, or rejecting-webhook error.+ pub async fn hard_delete_decay_tombstone(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: &PagePath,+ tombstone_id: PageId,+ cutoff_us: i64,+ ) -> WikiResult<usize> {+ let _guard = self.mutation_lock.write().await;+ self.ensure_project_workspace(workspace_id, project_id)+ .await?;+ let reader = self+ .store_reader+ .as_ref()+ .ok_or_else(|| ai_memory_wiki_error("decay cleanup requires a store reader"))?;+ let mut current_latest = reader+ .latest_page_id_by_ids(workspace_id, project_id, path.as_str().to_string())+ .await?;+ let abs = self.abs_path(workspace_id, project_id, path);+ if current_latest.is_none() && abs.try_exists()? {+ current_latest = Some(+ self.reindex_page_locked(workspace_id, project_id, path.clone())+ .await?,+ );+ }++ self.writer+ .hard_delete_decayed_page_chain(+ workspace_id,+ project_id,+ path.clone(),+ tombstone_id,+ current_latest,+ cutoff_us,+ )+ .await+ .map_err(Into::into)+ }+/// Purge a whole project's wiki directory. When an admission chain is/// attached, it is notified (`op=purge_project`, no page path) BEFORE the/// directory is removed, so a mirror can drop the project. A `Reject`@@ -1002,6 +1119,16 @@ impl Wiki {self.ensure_project_workspace(workspace_id, project_id).await?;+ self.reindex_page_locked(workspace_id, project_id, path)+ .await+ }++ async fn reindex_page_locked(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ path: PagePath,+ ) -> WikiResult<PageId> {let md = self.read_page(workspace_id, project_id, &path)?;let title = derive_title(&md.frontmatter, &md.body, &path);let links = extract_links(&md.body, &path);@@ -1849,7 +1976,7 @@ fn restore_quarantined_file(quarantined: &Option<PathBuf>, path: &Path, page_patpath = %page_path.as_str(),quarantine = %quarantine.display(),%error,- "delete_page: conditional/DB delete failed and restoring quarantined file also failed"+ "page removal: conditional store mutation failed and restoring quarantined file also failed");}}@@ -2678,6 +2805,109 @@ mod tests {assert!(hits[0].snippet.contains("compile"));}+ #[tokio::test]+ async fn decay_eviction_refuses_a_stale_latest_id_without_touching_disk() {+ let tmp = TempDir::new().unwrap();+ let (store, wiki, ws, proj) = scoped(&tmp).await;+ let path = PagePath::new("sessions/stale-decay.md").unwrap();+ let stale = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "old candidate",+ serde_json::json!({"tier": "episodic"}),+ ))+ .await+ .unwrap();+ let current = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "new current body",+ serde_json::json!({"tier": "episodic"}),+ ))+ .await+ .unwrap();++ assert!(+ !wiki+ .evict_page_if_latest(ws, proj, &path, stale, None)+ .await+ .unwrap()+ );+ assert!(+ std::fs::read_to_string(wiki.abs_path(ws, proj, &path))+ .unwrap()+ .contains("new current body")+ );+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(current),+ );+ }++ #[tokio::test]+ async fn rejecting_delete_admission_leaves_decay_candidate_live() {+ use axum::Router;+ use axum::http::StatusCode;+ use axum::routing::post;+ use tokio::net::TcpListener;++ let app = Router::new().route(+ "/reject",+ post(|| async { (StatusCode::FORBIDDEN, "retention denied") }),+ );+ let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();+ let addr = listener.local_addr().unwrap();+ tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });++ let tmp = TempDir::new().unwrap();+ let (store, wiki, ws, proj) = scoped(&tmp).await;+ let path = PagePath::new("sessions/admission-decay.md").unwrap();+ let page_id = wiki+ .write_page(req(+ ws,+ proj,+ path.as_str(),+ "must remain live",+ serde_json::json!({"tier": "episodic"}),+ ))+ .await+ .unwrap();+ let wiki = wiki.with_admission_chain(+ AdmissionChain::new(vec![WebhookConfig {+ name: "retention-guard".into(),+ url: format!("http://{addr}/reject"),+ timeout_ms: 1_000,+ failure_policy: FailurePolicy::Reject,+ events: vec![AdmissionOp::Delete],+ blocking: true,+ }])+ .unwrap(),+ );++ assert!(+ wiki.evict_page_if_latest(ws, proj, &path, page_id, None)+ .await+ .is_err()+ );+ assert!(wiki.abs_path(ws, proj, &path).exists());+ assert_eq!(+ store+ .reader+ .latest_page_id_by_ids(ws, proj, path.as_str().to_string())+ .await+ .unwrap(),+ Some(page_id),+ );+ }+#[tokio::test]async fn write_page_rolls_back_file_when_store_upsert_fails() {let tmp = TempDir::new().unwrap();diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex bf5c3699..739b811f 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -137,11 +137,13 @@ from hook paths.7. The forget sweep runs on demand and on the server's `[maintenance]`schedule: pages past their frontmatter `expires_at:` TTL arehard-deleted through the wiki layer (file + rows, pin or not);- pages with `retention < cold_threshold` are soft-deleted;- soft-deletions older than `hard_delete_after_days` with no subsequent- access get purged only within that sweep's resolved workspace/project,- together with entity-index rows orphaned by the purge. Semantic / pinned /- freshly-touched pages survive.+ pages with `retention < cold_threshold` are evicted through the wiki layer,+ which removes the authoritative file and leaves a decay tombstone;+ tombstones older than `hard_delete_after_days` are purged with their full+ version ancestry only within that sweep's resolved workspace/project,+ together with entity-index rows orphaned by the purge. A newer page recreated+ at the same path is preserved. Semantic / pinned / freshly-touched pages+ survive.Scheduled sweep, rule-based lint, and opt-in embedding backfill ticksenumerate every existing workspace/project scope before doing per-projectwork, matching the auto-improvement scheduler's store-wide scope model. A@@ -255,7 +257,7 @@ separately gated Claude Code assistant/Stop excerpt remains capped at 2 KB.| Table | What ||---|---|| `workspaces`, `projects` | Top of the 3-tuple identity coordinate. |-| `pages` | Versioned wiki pages with `is_latest` + `supersedes` chain. M8 columns: `last_accessed_at`, `access_count`, `superseded_at`. M9 cols: `embedding_provider`, `embedding_model`, `embedding_dim`. V36: `expires_at` (frontmatter TTL). V37: `salience` (NULL = `salience_default`; derived from `page_feedback`). |+| `pages` | Versioned wiki pages with `is_latest` + `supersedes` chain. M8 columns: `last_accessed_at`, `access_count`, and decay-only tombstone marker `superseded_at`. M9 cols: `embedding_provider`, `embedding_model`, `embedding_dim`. V36: `expires_at` (frontmatter TTL). V37: `salience` (NULL = `salience_default`; derived from `page_feedback`). || `pages_fts` | FTS5 virtual table over `(title, body)`, auto-synced by triggers. || `sessions`, `observations` | Sanitized, bounded lifecycle-hook projections. `sessions.ended_observation_count` is the stable generation watermark for resumed-session re-end eligibility; wall clocks are not used for that decision. They are an operational audit trail, not a complete native transcript. || `session_consolidation_jobs` | Durable, observation-generation-idempotent queue for opt-in SessionEnd LLM consolidation. One bounded server worker leases jobs, retries provider failures with backoff, and recovers expired leases after restart. |@@ -362,7 +364,7 @@ invariants below.| `memory_auto_improve` | write | Manually review a completed session and apply or stage validated wiki edits through the auto-improvement approval path. Without a session ID, selects the newest completed session with no persisted auto-improvement run so repeated calls advance through preflight skips; an explicit ID remains rerunnable. The server also schedules review for new sessions; `[auto_improve] require_approval = true` leaves proposals pending for manual review. || `memory_write_page` | destructive | Write durable wiki knowledge when the user explicitly asks to remember/annotate it. `scope: "global"` writes into the reserved `_global` preferences scope; optional `expires_at` sets an RFC3339 or date-only TTL. || `memory_delete_page` | destructive | Delete a single page by exact `path`. Fires the admission chain (op=delete); idempotent. |-| `memory_forget_sweep` | destructive | Retention pass: soft-delete cold pages, purge aged tombstones, and hard-delete TTL-expired pages through the wiki layer. `dry_run=true` for preview. |+| `memory_forget_sweep` | destructive | Retention pass: evict cold pages through the wiki layer, purge aged tombstone ancestry, and hard-delete TTL-expired pages. `dry_run=true` for preview. || `memory_lint` | destructive | Rule-based + LLM contradiction findings → `wiki/_lint/`. || `memory_install_self_routing` | read-only | Return the canonical slim routing snippet plus managed Agent Skill payloads and target hints for CLAUDE.md / AGENTS.md installs. |@@ -492,7 +494,7 @@ log_level = "info"lambda = 0.02 # ↓ to forget less aggressivelysigma = 0.6 # ↑ to reward query-hits moremu = 0.04 # ↑ if recent hits should count more-cold_threshold = 0.20 # below this → soft-delete+cold_threshold = 0.20 # below this → remove file + retain tombstonehard_delete_after_days = 180breadth_weight = 0.0 # opt-in reward for distinct operatorsdiff --git a/docs/admission-webhooks.md b/docs/admission-webhooks.mdindex ef663e3d..82ab95d5 100644--- a/docs/admission-webhooks.md+++ b/docs/admission-webhooks.md@@ -129,6 +129,12 @@ and `/admin/delete-workspace` paths, and before source teardown inEach webhook opts into the ops it cares about via `events`; the chain checks theop against `WebhookConfig::events` before dispatching.+Forget-sweep decay eviction uses the same conditional `delete` notification+before removing the Markdown file and writing its tombstone. Aged cleanup+never removes a file: if Markdown has reappeared at the path, it is preserved+and reindexed before only the old SQLite version chain is removed. There is no+file-mirror event for that DB-only history purge.+A copy-purge `/admin/move-project` fires **two** webhook events fromone request: one or more `write_page` notifications as the pages copyinto the destination, then one terminal `purge_project` notification@@ -158,13 +164,10 @@ terminal `purge_project` notification is unchanged.the scope and the acting operator. A handoff is SQLite rows, so a file mirrorhas no file change to reconcile from one; what it gets is the lifecycleevent, not a page.-- **Forget-sweep decay soft-delete and aged-tombstone cleanup** —- project-scoped and DB-only (`is_latest=0` / row delete); the markdown file- stays on disk, so there is nothing for a file mirror to do. This exemption- does not include frontmatter TTL expiry: that path uses the wiki's normal- conditional page delete, removes the file and rows, and runs the- `delete_page` admission chain. (`purge_project` remains the bulk-removal- path.)+- **Aged decay-history cleanup** — only the old SQLite version chain changes;+ any Markdown file at the path remains authoritative and is reindexed first+ when necessary. Initial decay eviction and frontmatter TTL expiry use the+ conditional `delete` admission path.- **`rename-project`** — a `projects.name` column update; the on-disk pathis the stable UUID, so no file moves and nothing to propagate.- **`rename-workspace`** — a `workspaces.name` column update plus refresheddiff --git a/docs/design-decisions.md b/docs/design-decisions.mdindex 3074e889..38383b49 100644--- a/docs/design-decisions.md+++ b/docs/design-decisions.md@@ -159,7 +159,7 @@ Three scheduled MCP operations:- **`memory_query`** (called by agent on demand): project-scoped FTS + lexical entity + graph retrieval, with optional vectors, RRF-fused before bounded authority and optional LLM reranking. Agentmemory's earlier triple-stream result motivated the fusion shape.- **`memory_lint`** (scheduled hourly + on session-end): scans for contradictions, orphan pages, broken links, stale claims, low-confidence + zero-reinforcement entries. Pure LLM with strict JSON output.-Decay/forget runs as a separate `memory_forget_sweep` job: applies the retention formula; soft-deletes via `is_latest=false` + `superseded_at`; hard-deletes only after 180 days *and* zero accesses. Never silently destroys anything user-pinned.+Decay/forget runs as a separate `memory_forget_sweep` job: applies the retention formula; removes the Markdown source while tombstoning via `is_latest=false` + `superseded_at`; then hard-deletes the tombstone's full version ancestry after the configured grace period. Lifetime access counters influence the retention score before eviction but do not block cleanup afterward. Never silently destroys anything user-pinned, and never path-deletes a newer recreation.Auto-improvement work stays separate from normal session consolidation. Thereviewer notes and staged design are in@@ -221,7 +221,7 @@ basic-memory has ~25 tools, agentmemory has 53. Both have user confusion as a re| `memory_read_page` | Read a full page body by exact path or top search hit | read-only || `memory_delete_page` | Delete a single exact-path page with admission hooks | destructive || `memory_feedback` | Record bounded page-quality feedback; adjust episodic retention and flag stale/wrong current versions for lint review | write |-| `memory_forget_sweep` | Retention sweep (M8); soft-delete below cold threshold; `dry_run=true` previews | destructive |+| `memory_forget_sweep` | Retention sweep (M8); wiki-backed eviction below cold threshold; `dry_run=true` previews | destructive || `memory_lint` | Rule-based + optional LLM contradiction findings → `wiki/_lint/<date>.md` | destructive || `memory_install_self_routing` | Returns the canonical slim CLAUDE.md / AGENTS.md routing block, managed Agent Skill payloads, target hints, and overwrite guidance | read-only |