Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(sweep): opt-in bounded retention for raw observations (#508)
Djalma Júnior4756e153b927
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7070993f..275c57d0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Added+- Bounded raw-observation retention as an opt-in fourth pass of the M8 forget+ sweep, disabled by default. Nothing in the tree could delete an `observations`+ row for age: the sweep acts on pages only, so raw capture grew without limit.+ Measured on a three-month-old install, `observations` plus its FTS shadow and+ four indexes were ~5.4 GB of a 5.53 GB store (5,236,232 rows), against 0.03 GB+ of `pages` — the 2,365 compiled pages that hold the durable value. The nightly+ backup tarball was 2.7 GB, almost all of it capture already distilled into+ 30 MB of Markdown. `decay.observation_retention_days` (default `0` = disabled)+ now lets an operator bound that, and `decay.observation_prune_batch` (default+ `5000`) bounds each transaction. (#508)++ The pass deletes an observation only when its session was already consolidated+ into a summary page that is still live, and the row is older than the+ configured age. That is three gates on columns the schema already maintains:+ `sessions.summary_page_id` is written only by the session-end path, so it+ already implies `ended_at IS NOT NULL`; `pages.superseded_at IS NULL` excludes+ a session whose page decay has evicted, because that session's raw rows are+ now the only surviving copy; and a hard-deleted page has already NULLed the+ pointer through `ON DELETE SET NULL`, so the join finds nothing. The prune+ therefore runs LAST, after this same run's evictions and hard-deletes, rather+ than letting raw capture outlive its distillation by one sweep. A session with+ no `sessions` row, or one still running, matches nothing and can never lose a+ row.++ Each batch is one transaction sent as its own message to the single-writer+ actor, so every other pending write interleaves between batches and a+ multi-million row prune never holds the write lock across the run. Measured on+ a 300,000-row store built from the shipped migrations: ~23-27 ms per 5,000-row+ batch including the FTS trigger work, plus ~2-4 ms to commit. The session-end+ observation watermark (`sessions.ended_observation_count`) is repaired in the+ same transaction and only downward, so a resumed session's genuinely new work+ is still read as new work instead of `AlreadyEnded`. One `prune_observations`+ audit row is written per batch that actually deleted, inside that batch's+ transaction. Zero migrations: every access is an index seek on+ `idx_observations_project_created` and `idx_observations_session`, both of+ which predate this change.++ The cost is stated rather than hidden, and it is not just disk: observations+ are the input to consolidation, so pruning is irreversible — a pruned session+ can never be re-consolidated (not with a better model, a better prompt, or a+ fixed consolidator bug) and its summary page becomes the only surviving+ account of that session. Concretely: the raw transcript+ view returns empty, `raw_hits` can no longer match the exact original wording,+ a manual auto-improve rerun rejects with `too_few_observations`, and a+ `move_session` with `PagesMode::Regenerate` has nothing left to rebuild the+ page from. All of it is reachable only by setting a positive age. Note that+ SQLite does not return freed pages to the OS without `VACUUM`, so the `.db`+ file will not shrink after the first prune — the `ai-memory backup` tarball+ will, because it is a fresh online copy.+### Fixed- The CHANGELOG frozen-section check no longer fires on a branch that merged`main` after a release. It compared line ranges since the merge base, so adiff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex e4fce548..bf5a32e1 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -7,8 +7,9 @@ use std::sync::Arc;use std::time::Duration;use ai_memory_consolidate::{- AutoImproveReviewConfig, Consolidator, EmbedBackfillOptions, ScheduledAutoImproveSettings,- run_auto_improve_scheduler_tick, run_embedding_backfill, run_lint, run_sweep_with_breadth,+ AutoImproveReviewConfig, Consolidator, EmbedBackfillOptions, ObservationRetention,+ ScheduledAutoImproveSettings, run_auto_improve_scheduler_tick, run_embedding_backfill,+ run_lint, run_sweep_with_options,};use ai_memory_core::{ActiveProject, ProjectId, Sanitizer, WorkspaceId};use ai_memory_hooks::{@@ -17,7 +18,7 @@ use ai_memory_hooks::{};use ai_memory_llm::{Embedder, LlmProvider, ProviderHealth, build_embedder, build_provider};use ai_memory_mcp::{- AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_decay_breadth,+ AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_sweep_tuning,};use ai_memory_store::{ReaderPool, Store, WriterHandle};use ai_memory_web::{WebMountSpec, mount_web_router, normalize_prefix, web_base_href};@@ -525,6 +526,7 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {.with_wiki(wiki.clone()).with_decay_params(decay_params).with_decay_breadth_weight(config.decay.breadth_weight)+ .with_observation_retention(config.decay.observation_retention()).with_auto_improve_require_approval(config.auto_improve.require_approval).with_auto_improve_review_config(auto_improve_review_config_from_settings(&config.auto_improve,@@ -687,7 +689,7 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {sanitizer: sanitizer.clone(),data_dir: config.data_dir.clone(),});- let admin = admin_router_with_decay_breadth(+ let admin = admin_router_with_sweep_tuning(AdminState {ingest_metrics: ingest_metrics.clone(),writer: store.writer.clone(),@@ -717,6 +719,7 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {trusted_proxy_identity: trusted_proxy_identity_enabled(&config.auth),},config.decay.breadth_weight,+ config.decay.observation_retention(),);// Multi-rung auth assembly:// - rung 0 (no bearer_token configured) → AuthState::new@@ -966,6 +969,7 @@ async fn start_maintenance_scheduler(&wiki,&decay.decay_params(),decay.breadth_weight,+ decay.observation_retention(),).await?;if outcome.errors > 0 {@@ -980,6 +984,7 @@ async fn start_maintenance_scheduler(evicted = outcome.evicted,expired = outcome.expired,hard_deleted = outcome.hard_deleted,+ observations_pruned = outcome.observations_pruned,errors = outcome.errors,elapsed_ms = started.elapsed().as_millis(),"scheduled forget sweep completed"@@ -1195,6 +1200,7 @@ struct ScheduledSweepTickOutcome {evicted: usize,expired: usize,hard_deleted: usize,+ observations_pruned: usize,errors: usize,}@@ -1204,6 +1210,7 @@ async fn run_scheduled_sweep_tick(wiki: &Wiki,decay: &ai_memory_store::DecayParams,breadth_weight: f64,+ retention: ObservationRetention,) -> Result<ScheduledSweepTickOutcome> {let scopes = reader.list_all_scopes().await?;let mut outcome = ScheduledSweepTickOutcome {@@ -1212,7 +1219,7 @@ async fn run_scheduled_sweep_tick(};for scope in scopes {- match run_sweep_with_breadth(+ match run_sweep_with_options(reader,writer,Some(wiki),@@ -1220,6 +1227,7 @@ async fn run_scheduled_sweep_tick(scope.project_id,decay,breadth_weight,+ retention,false,).await@@ -1229,6 +1237,7 @@ async fn run_scheduled_sweep_tick(outcome.evicted += report.evicted.iter().filter(|page| page.deleted).count();outcome.expired += report.expired.len();outcome.hard_deleted += report.hard_deleted;+ outcome.observations_pruned += report.observations_pruned;}Err(e) => {outcome.errors += 1;@@ -2521,9 +2530,16 @@ mod tests {cold_threshold: 2.0,..ai_memory_store::DecayParams::default()};- let outcome = run_scheduled_sweep_tick(&store.reader, &store.writer, &wiki, &decay, 0.0)- .await- .unwrap();+ let outcome = run_scheduled_sweep_tick(+ &store.reader,+ &store.writer,+ &wiki,+ &decay,+ 0.0,+ ObservationRetention::default(),+ )+ .await+ .unwrap();assert_eq!(outcome.scopes, 2);assert_eq!(outcome.errors, 0);diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex 8634d79e..47638b01 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -61,6 +61,11 @@ pub struct DecaySettings {pub hard_delete_after_days: i64,/// Optional weight for the number of distinct authenticated readers.pub breadth_weight: f64,+ /// Age past which a consolidated session's raw observations may be pruned.+ /// `0` disables the pass; nothing is deleted until an operator opts in.+ pub observation_retention_days: i64,+ /// Observation rows deleted per prune transaction.+ pub observation_prune_batch: usize,}impl Default for DecaySettings {@@ -74,6 +79,8 @@ impl Default for DecaySettings {cold_threshold: base.cold_threshold,hard_delete_after_days: base.hard_delete_after_days,breadth_weight: 0.0,+ observation_retention_days: 0,+ observation_prune_batch: ai_memory_consolidate::DEFAULT_OBSERVATION_PRUNE_BATCH,}}}@@ -91,6 +98,19 @@ impl DecaySettings {hard_delete_after_days: self.hard_delete_after_days,}}++ /// Opt-in observation prune bound consumed by the M8 sweep.+ ///+ /// Deliberately separate from [`Self::decay_params`]: keeping it out of the+ /// public `DecayParams` struct is what lets every downstream Rust caller+ /// that builds one directly keep compiling — and keep today's behaviour.+ #[must_use]+ pub fn observation_retention(self) -> ai_memory_consolidate::ObservationRetention {+ ai_memory_consolidate::ObservationRetention {+ days: self.observation_retention_days,+ batch: self.observation_prune_batch,+ }+ }}/// Top-level runtime configuration.@@ -929,6 +949,19 @@ impl Config {);}+ // Fail closed at load rather than at 3am inside a destructive pass: a+ // negative age would be a nonsensical cutoff, and a zero batch would+ // spin the prune loop forever without deleting anything.+ if config.decay.observation_retention_days < 0 {+ anyhow::bail!(+ "decay.observation_retention_days must be greater than or equal to zero \+ (0 disables observation pruning)"+ );+ }+ if config.decay.observation_prune_batch == 0 {+ anyhow::bail!("decay.observation_prune_batch must be greater than zero");+ }+// Fail at startup rather than shipping a prompt that is all scaffolding// and no observations: below this floor the fixed system prompt and page// conventions consume the entire budget, so every consolidation would@@ -1365,6 +1398,11 @@ mod tests {assert_eq!(cfg.maintenance.lint_interval_secs, 86_400);assert_eq!(cfg.maintenance.embedding_backfill_interval_secs, 0);assert_eq!(cfg.decay.breadth_weight, 0.0);+ // Observation pruning must stay OFF by default: an install that never+ // opts in keeps every raw observation it has today.+ assert_eq!(cfg.decay.observation_retention_days, 0);+ assert_eq!(cfg.decay.observation_prune_batch, 5_000);+ assert!(!cfg.decay.observation_retention().is_enabled());assert!(!cfg.slots.per_user);assert!(cfg.auto_improve.scheduler.enabled);assert_eq!(cfg.auto_improve.scheduler.interval_secs, 3_600);@@ -1433,6 +1471,27 @@ mod tests {}}+ /// A negative retention age or a zero batch is rejected at load, beside+ /// the breadth-weight guard, so a destructive pass can never be configured+ /// into a nonsensical shape.+ #[test]+ fn load_rejects_destructive_invalid_observation_retention() {+ for (key, value) in [+ ("observation_retention_days", "-1"),+ ("observation_prune_batch", "0"),+ ] {+ let tmp = TempDir::new().unwrap();+ let config_path = tmp.path().join("config.toml");+ std::fs::write(&config_path, format!("[decay]\n{key} = {value}\n")).unwrap();+ let error = Config::load(Some(&config_path), Some(tmp.path().to_path_buf()))+ .expect_err("invalid observation retention must fail closed");+ assert!(+ error.to_string().contains(key),+ "unexpected error for {key} = {value}: {error:#}"+ );+ }+ }+/// The consolidation budget must be big enough to leave room for/// observations after the fixed system prompt and page conventions./// Below the floor every consolidation would be evidence-free, so itdiff --git a/crates/ai-memory-cli/templates/config.default.toml b/crates/ai-memory-cli/templates/config.default.tomlindex 289fda9f..96196301 100644--- a/crates/ai-memory-cli/templates/config.default.toml+++ b/crates/ai-memory-cli/templates/config.default.toml@@ -80,6 +80,22 @@ log_level = "info"# - Bigger access boost (frequently-used stuff lasts longer): sigma = 1.0# - Bigger time-pressure on access: mu = 0.08# - Reward pages used by more identified operators: breadth_weight = 0.2+#+# Raw observations are the store's growth driver (a busy install can add+# gigabytes a year), while the durable value is the compiled pages. Set+# observation_retention_days to a positive number to let the sweep delete raw+# observations older than that — but ONLY for sessions already consolidated into+# a page that is still live. Unconsolidated sessions, and sessions whose page was+# evicted by decay, keep every row. 0 (the default) disables the pass entirely.+# Pruning is IRREVERSIBLE: observations are the input to consolidation, so a+# pruned session can never be re-consolidated (a better model, a better prompt,+# or a fixed consolidator bug make no difference) and its summary page becomes+# the only surviving account of that session. The 0 default is a contract: this+# pass never runs unless an operator sets a positive age here.+# SQLite does not return freed pages to the OS without VACUUM, so the .db file+# will not shrink after the first prune; the `ai-memory backup` tarball will.+# - A quarter of raw capture: observation_retention_days = 90+# - Smaller transactions on a slow disk: observation_prune_batch = 2000[decay]lambda = 0.02sigma = 0.6@@ -88,6 +104,8 @@ salience_default = 1.0cold_threshold = 0.20hard_delete_after_days = 180breadth_weight = 0.0+observation_retention_days = 0 # 0 = never prune raw observations+observation_prune_batch = 5000 # rows per prune transaction# Privacy strip.#diff --git a/crates/ai-memory-consolidate/src/lib.rs b/crates/ai-memory-consolidate/src/lib.rsindex 8345b3cd..f58e9155 100644--- a/crates/ai-memory-consolidate/src/lib.rs+++ b/crates/ai-memory-consolidate/src/lib.rs@@ -64,7 +64,10 @@ pub use embed::{EmbedBackfillCounts, EmbedBackfillError, EmbedBackfillOptions, run_embedding_backfill,};pub use lint::{LintError, LintFinding, LintOptions, LintReport, run_lint, stale_days_for};-pub use sweep::{EvictedPage, SweepError, SweepReport, run_sweep, run_sweep_with_breadth};+pub use sweep::{+ DEFAULT_OBSERVATION_PRUNE_BATCH, EvictedPage, ObservationRetention, SweepError, SweepReport,+ run_sweep, run_sweep_with_breadth, run_sweep_with_options,+};pub use types::{ConsolidatedBatch, ConsolidatedPage, ConsolidatedPageUpdate, ConsolidationOutcome, PageKind,SlotKind,diff --git a/crates/ai-memory-consolidate/src/sweep.rs b/crates/ai-memory-consolidate/src/sweep.rsindex e48224ab..b7dcfc19 100644--- a/crates/ai-memory-consolidate/src/sweep.rs+++ b/crates/ai-memory-consolidate/src/sweep.rs@@ -18,8 +18,16 @@//! `hard_delete_after_days` and their supersession ancestry. A page recreated//! at the same path is a separate live chain and is preserved. Ordinary//! supersession rows are safe because only decay writes `superseded_at`.+//!+//! Observation prune pass (opt-in, disabled unless+//! `ObservationRetention::days > 0`) deletes raw observations older than the+//! configured age, and only for sessions already distilled into a page that is+//! still live. It runs LAST so both of this run's page deletions are already+//! visible to its predicate: a session whose summary page was evicted or+//! hard-deleted by the passes above keeps its raw capture, because that capture+//! is now the only surviving copy.-use std::collections::HashMap;+use std::collections::{HashMap, HashSet};use ai_memory_core::{PageId, ProjectId, Tier, WorkspaceId};use ai_memory_store::{@@ -79,8 +87,56 @@ pub struct SweepReport {pub expired: Vec<ExpiredPage>,/// Number of page-version rows permanently deleted on this pass.pub hard_deleted: usize,+ /// Observations the prune predicate matched. Reported in both modes; `0`+ /// when observation retention is disabled, which is the default.+ pub observations_prunable: usize,+ /// Observation rows permanently deleted. Always `0` on `dry_run`, exactly+ /// like the wiki-routed passes above.+ pub observations_pruned: usize,+ /// Distinct consolidated sessions that lost rows, so the blast radius is a+ /// number in the response rather than a claim in a changelog.+ pub observation_prune_sessions: usize,+ /// Transactions the prune spent. Makes the batching bound observable.+ pub observation_prune_batches: usize,+}++/// Opt-in bound on how long raw observations outlive their consolidation.+///+/// Deliberately a type of its own rather than two more+/// [`DecayParams`] fields: the retention coefficients are a public struct that+/// downstream Rust callers construct directly, and widening it would break+/// them. Same reasoning the access-breadth coefficient already carries.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub struct ObservationRetention {+ /// Age in days past which a consolidated session's observations may be+ /// pruned. `0` disables the pass entirely — the default, so an existing+ /// install behaves exactly as it did before this pass existed.+ pub days: i64,+ /// Rows deleted per transaction.+ pub batch: usize,+}++/// Disabled: nothing is ever pruned until an operator sets a positive age.+impl Default for ObservationRetention {+ fn default() -> Self {+ Self {+ days: 0,+ batch: DEFAULT_OBSERVATION_PRUNE_BATCH,+ }+ }+}++impl ObservationRetention {+ /// `true` when the pass is enabled and can delete something.+ #[must_use]+ pub const fn is_enabled(self) -> bool {+ self.days > 0 && self.batch > 0+ }}+/// Default rows per prune transaction.+pub const DEFAULT_OBSERVATION_PRUNE_BATCH: usize = 5_000;+/// Errors raised by the sweep.#[derive(Debug, Error)]#[non_exhaustive]@@ -91,6 +147,9 @@ pub enum SweepError {/// The optional access-breadth coefficient was negative or non-finite.#[error("decay breadth_weight must be a finite number greater than or equal to zero")]InvalidBreadthWeight,+ /// The opt-in observation retention age was negative.+ #[error("decay observation_retention_days must be greater than or equal to zero")]+ InvalidObservationRetention,}const US_PER_DAY: f64 = 86_400_000_000.0;@@ -139,17 +198,54 @@ pub async fn run_sweep(#[allow(clippy::too_many_arguments)]pub async fn run_sweep_with_breadth(reader: &ReaderPool,- _writer: &WriterHandle,+ writer: &WriterHandle,+ wiki: Option<&Wiki>,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ params: &DecayParams,+ breadth_weight: f64,+ dry_run: bool,+) -> Result<SweepReport, SweepError> {+ run_sweep_with_options(+ reader,+ writer,+ wiki,+ workspace_id,+ project_id,+ params,+ breadth_weight,+ ObservationRetention::default(),+ dry_run,+ )+ .await+}++/// Run a sweep with the breadth coefficient and the opt-in observation prune.+///+/// The prune is the only pass that can delete raw capture, and it is off unless+/// `retention.days` is positive.+///+/// # Errors+/// Returns [`SweepError::InvalidObservationRetention`] for a negative age, in+/// addition to the errors documented by [`run_sweep_with_breadth`].+#[allow(clippy::too_many_arguments)]+pub async fn run_sweep_with_options(+ reader: &ReaderPool,+ writer: &WriterHandle,wiki: Option<&Wiki>,workspace_id: WorkspaceId,project_id: ProjectId,params: &DecayParams,breadth_weight: f64,+ retention: ObservationRetention,dry_run: bool,) -> Result<SweepReport, SweepError> {if !breadth_weight.is_finite() || breadth_weight < 0.0 {return Err(SweepError::InvalidBreadthWeight);}+ if retention.days < 0 {+ return Err(SweepError::InvalidObservationRetention);+ }let candidates = reader.decay_candidates(workspace_id, project_id).await?;let breadth =access_breadth_for_scoring(reader, workspace_id, project_id, breadth_weight).await?;@@ -199,6 +295,18 @@ pub async fn run_sweep_with_breadth(}let mut hard_deleted = 0usize;+ let mut observations_prunable = 0usize;+ let mut observations_pruned = 0usize;+ let mut observation_prune_sessions = 0usize;+ let mut observation_prune_batches = 0usize;+ let observation_cutoff_us = Timestamp::now()+ .as_microsecond()+ .saturating_sub(retention.days.saturating_mul(US_PER_DAY_I64));+ if retention.is_enabled() {+ observations_prunable = reader+ .prunable_observation_count(workspace_id, project_id, observation_cutoff_us)+ .await?;+ }if !dry_run {for page in &mut expired {let path = match ai_memory_core::PagePath::new(page.path.clone()) {@@ -284,6 +392,37 @@ pub async fn run_sweep_with_breadth(}}}++ // LAST on purpose. Both passes above can strip a session of its+ // distillation — decay stamps `superseded_at`, hard-delete removes the+ // row and the `ON DELETE SET NULL` foreign key clears the pointer — and+ // the prune predicate reads exactly those two columns. Running here+ // means this run's own evictions already exclude their sessions,+ // instead of a batch of raw capture outliving its page by one sweep.+ //+ // Unlike the wiki-routed passes this one runs without a `wiki` handle:+ // observations have no Markdown file, so there is no source of truth+ // for reconciliation to re-index and no store-only-mutation hazard.+ if retention.is_enabled() {+ let mut touched: HashSet<ai_memory_core::SessionId> = HashSet::new();+ loop {+ let outcome = writer+ .prune_consolidated_observations(+ workspace_id,+ project_id,+ observation_cutoff_us,+ retention.batch,+ )+ .await?;+ observation_prune_batches += 1;+ observations_pruned += outcome.deleted;+ touched.extend(outcome.sessions_touched.iter().copied());+ if outcome.deleted < retention.batch {+ break;+ }+ }+ observation_prune_sessions = touched.len();+ }}Ok(SweepReport {@@ -292,6 +431,10 @@ pub async fn run_sweep_with_breadth(evicted,expired,hard_deleted,+ observations_prunable,+ observations_pruned,+ observation_prune_sessions,+ observation_prune_batches,})}diff --git a/crates/ai-memory-consolidate/tests/observation_retention.rs b/crates/ai-memory-consolidate/tests/observation_retention.rsnew file mode 100644index 00000000..a6e9f545--- /dev/null+++ b/crates/ai-memory-consolidate/tests/observation_retention.rs@@ -0,0 +1,554 @@+//! The opt-in observation prune, end to end.+//!+//! Raw observations are ~98% of a mature store's bytes, and their durable value+//! was already distilled into pages. Deleting them is safe exactly when that+//! distillation still exists — so these tests hold the pass to the only+//! discrimination that makes it safe: a session consolidated into a live page+//! loses its old raw capture; a session with equally old capture and no page,+//! or with a page decay already evicted, keeps every row.+//!+//! They also pin the two properties an operator has to be able to trust without+//! reading the SQL: nothing is deleted at the shipped defaults, and the FTS+//! index stops matching what the prune removed.++use ai_memory_consolidate::{ObservationRetention, run_sweep, run_sweep_with_options};+use ai_memory_core::{+ AgentKind, NewObservation, NewPage, NewSession, ObservationKind, PageId, PagePath, ProjectId,+ Sanitized, Sanitizer, SessionId, Tier, WorkspaceId,+};+use ai_memory_store::{DecayParams, Store, WriterHandle};+use rusqlite::{Connection, params};+use tempfile::TempDir;++const US_PER_DAY: i64 = 86_400_000_000;+/// Well past any retention age used below, so age is never what separates the+/// fixtures — only whether the session was consolidated.+const OBS_AGE_DAYS: i64 = 400;+const RETENTION_DAYS: i64 = 90;++/// A session whose observations the prune is allowed to delete: ended, with a+/// live summary page.+const CONSOLIDATED_BODY: &str = "consolidated capture zebrafinch";+/// A session with identical, equally old capture that was never consolidated.+const RAW_BODY: &str = "unconsolidated capture zebrafinch";+/// A session that WAS consolidated, but whose page decay has since evicted —+/// the raw rows are now the only surviving copy.+const EVICTED_BODY: &str = "evicted capture zebrafinch";++struct Fixture {+ ws: WorkspaceId,+ proj: ProjectId,+ consolidated: SessionId,+ unconsolidated: SessionId,+ evicted: SessionId,+}++/// Three sessions x three observations each, all backdated to the same age.+async fn seed(store: &Store) -> Fixture {+ let ws = store+ .writer+ .get_or_create_workspace("default".to_string())+ .await+ .expect("ws");+ let proj = store+ .writer+ .get_or_create_project(ws, "retention".to_string(), None)+ .await+ .expect("proj");++ let consolidated = session(store, ws, proj, 1, CONSOLIDATED_BODY).await;+ let unconsolidated = session(store, ws, proj, 2, RAW_BODY).await;+ let evicted = session(store, ws, proj, 3, EVICTED_BODY).await;++ // Only the first two ends link a page; the third's page is evicted below.+ let page = write_page(&store.writer, ws, proj, "sessions/consolidated.md").await;+ store+ .writer+ .end_session(consolidated, Some(page))+ .await+ .expect("end consolidated");+ let evicted_page = write_page(&store.writer, ws, proj, "sessions/evicted.md").await;+ store+ .writer+ .end_session(evicted, Some(evicted_page))+ .await+ .expect("end evicted");++ let conn = aux(store);+ // Decay eviction, exactly as `soft_delete_for_decay_if_latest` writes it:+ // the row survives as a tombstone but the Markdown is gone.+ conn.execute(+ "UPDATE pages SET is_latest = 0, superseded_at = ?1 WHERE id = ?2",+ params![+ jiff::Timestamp::now().as_microsecond(),+ evicted_page.as_bytes()+ ],+ )+ .expect("evict page");+ let then_us = jiff::Timestamp::now().as_microsecond() - OBS_AGE_DAYS * US_PER_DAY;+ conn.execute("UPDATE observations SET created_at = ?1", params![then_us])+ .expect("backdate observations");++ Fixture {+ ws,+ proj,+ consolidated,+ unconsolidated,+ evicted,+ }+}++async fn session(store: &Store, ws: WorkspaceId, proj: ProjectId, n: u8, body: &str) -> SessionId {+ let mut raw = [0u8; 16];+ raw[15] = n;+ let id = SessionId::from_slice(&raw).expect("session id");+ store+ .writer+ .begin_session(NewSession {+ id,+ workspace_id: ws,+ project_id: proj,+ agent_kind: AgentKind::ClaudeCode,+ cwd: None,+ actor_user: None,+ })+ .await+ .expect("begin session");+ let sanitizer = Sanitizer::builtin();+ for i in 0..3 {+ store+ .writer+ .insert_observation(Sanitized::new(+ NewObservation {+ session_id: id,+ workspace_id: ws,+ project_id: proj,+ kind: ObservationKind::UserPrompt,+ extension: None,+ source_event: None,+ title: format!("obs {n}-{i}"),+ body: body.to_string(),+ importance: 5,+ },+ &sanitizer,+ ))+ .await+ .expect("insert observation");+ }+ id+}++async fn write_page(writer: &WriterHandle, ws: WorkspaceId, proj: ProjectId, path: &str) -> PageId {+ writer+ .upsert_page(NewPage {+ workspace_id: ws,+ project_id: proj,+ path: PagePath::new(path.to_string()).expect("path"),+ title: path.into(),+ body: "distilled".into(),+ tier: Tier::Episodic,+ frontmatter_json: serde_json::json!({}),+ pinned: false,+ links: Vec::new(),+ author_id: None,+ expires_at: None,+ entities: Vec::new(),+ })+ .await+ .expect("upsert page")+}++fn aux(store: &Store) -> Connection {+ let conn = Connection::open(store.db_path()).expect("aux conn");+ conn.pragma_update(None, "busy_timeout", 5_000).unwrap();+ conn+}++fn observations_for(conn: &Connection, session: SessionId) -> i64 {+ conn.query_row(+ "SELECT COUNT(*) FROM observations WHERE session_id = ?1",+ params![session.as_bytes()],+ |row| row.get(0),+ )+ .expect("count observations")+}++fn fts_matches(conn: &Connection, needle: &str) -> i64 {+ conn.query_row(+ "SELECT COUNT(*) FROM observations_fts WHERE observations_fts MATCH ?1",+ params![needle],+ |row| row.get(0),+ )+ .expect("fts match")+}++fn enabled() -> ObservationRetention {+ ObservationRetention {+ days: RETENTION_DAYS,+ batch: 5_000,+ }+}++/// The no-change proof. `run_sweep` is what every existing caller reaches, and+/// `ObservationRetention::default()` is what the shipped config produces. On a+/// store full of ancient consolidated capture, both must delete nothing at all.+#[tokio::test]+async fn the_default_configuration_prunes_no_observation() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("store");+ let f = seed(&store).await;+ let params = DecayParams::default();++ let report = run_sweep(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ ¶ms,+ false,+ )+ .await+ .expect("sweep");+ assert_eq!(report.observations_pruned, 0);+ assert_eq!(+ report.observations_prunable, 0,+ "disabled must not even count"+ );+ assert_eq!(report.observation_prune_batches, 0);++ let report = run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ ¶ms,+ 0.0,+ ObservationRetention::default(),+ false,+ )+ .await+ .expect("sweep");+ assert_eq!(report.observations_pruned, 0);++ let conn = aux(&store);+ for s in [f.consolidated, f.unconsolidated, f.evicted] {+ assert_eq!(+ observations_for(&conn, s),+ 3,+ "the default configuration must leave every observation in place"+ );+ }+}++/// The discrimination the whole feature rests on. Same scope, same age, same+/// body length: the ONLY difference is whether the session's work still exists+/// as a page. Consolidated loses its rows; unconsolidated and decay-evicted+/// keep theirs.+#[tokio::test]+async fn only_a_consolidated_session_with_a_live_page_loses_its_observations() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("store");+ let f = seed(&store).await;++ let report = run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ &DecayParams::default(),+ 0.0,+ enabled(),+ false,+ )+ .await+ .expect("sweep");++ assert_eq!(report.observations_pruned, 3);+ assert_eq!(report.observations_prunable, 3);+ assert_eq!(report.observation_prune_sessions, 1);++ let conn = aux(&store);+ assert_eq!(+ observations_for(&conn, f.consolidated),+ 0,+ "a consolidated session's old raw capture must be pruned"+ );+ assert_eq!(+ observations_for(&conn, f.unconsolidated),+ 3,+ "a session never consolidated into a page must keep every row"+ );+ assert_eq!(+ observations_for(&conn, f.evicted),+ 3,+ "raw capture whose summary page decay already evicted is the last copy"+ );+ assert_eq!(+ conn.query_row(+ "SELECT detail FROM audit_log WHERE op = 'prune_observations'",+ [],+ |r| r.get::<_, String>(0)+ )+ .expect("exactly one audit row for the single deleting batch"),+ "{\"deleted\":3}",+ "the audit row must record how many rows the batch deleted"+ );+}++/// `observations_fts` is `content='observations'`, so a stale index would keep+/// returning hits for rows that no longer exist. The `observations_fts_ad`+/// trigger is supposed to prevent that; assert it rather than trust it.+#[tokio::test]+async fn the_fts_index_stops_matching_pruned_bodies() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("store");+ let f = seed(&store).await;+ let conn = aux(&store);++ assert_eq!(fts_matches(&conn, "consolidated"), 3);+ assert_eq!(fts_matches(&conn, "zebrafinch"), 9);++ run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ &DecayParams::default(),+ 0.0,+ enabled(),+ false,+ )+ .await+ .expect("sweep");++ assert_eq!(+ fts_matches(&conn, "consolidated"),+ 0,+ "the FTS index must no longer match a pruned body"+ );+ assert_eq!(+ fts_matches(&conn, "zebrafinch"),+ 6,+ "only the pruned rows may leave the index"+ );+ // The drift check behind `fts_drift_status` compares these two directly.+ let rows: i64 = conn+ .query_row("SELECT COUNT(*) FROM observations", [], |r| r.get(0))+ .unwrap();+ let docsize: i64 = conn+ .query_row("SELECT COUNT(*) FROM observations_fts_docsize", [], |r| {+ r.get(0)+ })+ .unwrap();+ assert_eq!(rows, docsize, "FTS docsize must track the delete exactly");+}++/// A dry run must be able to answer "how much would this delete?" without+/// opening a write transaction. Counting is the whole point; deleting is not.+#[tokio::test]+async fn a_dry_run_counts_without_deleting() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("store");+ let f = seed(&store).await;++ let report = run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ &DecayParams::default(),+ 0.0,+ enabled(),+ true,+ )+ .await+ .expect("sweep");++ assert_eq!(report.observations_prunable, 3);+ assert_eq!(report.observations_pruned, 0);+ assert_eq!(report.observation_prune_batches, 0);+ let conn = aux(&store);+ assert_eq!(observations_for(&conn, f.consolidated), 3);+ assert_eq!(+ conn.query_row(+ "SELECT COUNT(*) FROM audit_log WHERE op = 'prune_observations'",+ [],+ |r| r.get::<_, i64>(0)+ )+ .unwrap(),+ 0,+ "a dry run must not write an audit row either"+ );+}++/// The bound that keeps a multi-million row delete off the write lock: one+/// transaction per batch, looping until a short batch. With `batch = 1` the+/// three eligible rows must cost four batches (three full, one short) and still+/// land completely.+#[tokio::test]+async fn the_prune_deletes_in_bounded_batches() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("store");+ let f = seed(&store).await;++ let report = run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ &DecayParams::default(),+ 0.0,+ ObservationRetention {+ days: RETENTION_DAYS,+ batch: 1,+ },+ false,+ )+ .await+ .expect("sweep");++ assert_eq!(report.observations_pruned, 3);+ assert_eq!(+ report.observation_prune_batches, 4,+ "each row is its own transaction, plus the short batch that ends the loop"+ );+ assert_eq!(+ report.observation_prune_sessions, 1,+ "a session split across batches must be counted once"+ );+ let conn = aux(&store);+ assert_eq!(+ conn.query_row(+ "SELECT COUNT(*) FROM audit_log WHERE op = 'prune_observations'",+ [],+ |r| r.get::<_, i64>(0)+ )+ .unwrap(),+ 3,+ "one audit row per batch that actually deleted, none for the empty one"+ );+}++/// `session_end_disposition` reads `sessions.ended_observation_count` against a+/// live `COUNT(*)`. Prune without repairing that watermark and a resumed+/// session's genuinely new work reads as `AlreadyEnded` forever.+#[tokio::test]+async fn the_session_end_watermark_follows_the_prune() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("store");+ let f = seed(&store).await;+ let conn = aux(&store);++ let watermark = |conn: &Connection, s: SessionId| -> i64 {+ conn.query_row(+ "SELECT ended_observation_count FROM sessions WHERE id = ?1",+ params![s.as_bytes()],+ |row| row.get(0),+ )+ .expect("watermark")+ };+ assert_eq!(watermark(&conn, f.consolidated), 3);++ run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ &DecayParams::default(),+ 0.0,+ enabled(),+ false,+ )+ .await+ .expect("sweep");++ assert_eq!(+ watermark(&conn, f.consolidated),+ 0,+ "the watermark must come back down to the surviving row count"+ );+ // Idempotent and one-directional: a second sweep with nothing left to+ // delete must not touch it again.+ run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ &DecayParams::default(),+ 0.0,+ enabled(),+ false,+ )+ .await+ .expect("second sweep");+ assert_eq!(watermark(&conn, f.consolidated), 0);+}++/// Observations younger than the retention age are never eligible, however+/// thoroughly their session was consolidated — the age gate is what keeps+/// consolidation, auto-improve review and the `raw_hits` fallback whole.+#[tokio::test]+async fn recent_observations_survive_a_consolidated_session() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("store");+ let f = seed(&store).await;+ let conn = aux(&store);+ let recent = jiff::Timestamp::now().as_microsecond() - US_PER_DAY;+ conn.execute(+ "UPDATE observations SET created_at = ?1 WHERE session_id = ?2",+ params![recent, f.consolidated.as_bytes()],+ )+ .expect("freshen");++ let report = run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ &DecayParams::default(),+ 0.0,+ enabled(),+ false,+ )+ .await+ .expect("sweep");++ assert_eq!(report.observations_pruned, 0);+ assert_eq!(observations_for(&conn, f.consolidated), 3);+}++/// A negative age is a nonsensical cutoff and must stop the sweep before it+/// mutates anything, exactly as an invalid breadth coefficient does.+#[tokio::test]+async fn a_negative_retention_age_stops_the_sweep() {+ let tmp = TempDir::new().expect("tempdir");+ let store = Store::open(tmp.path()).expect("store");+ let f = seed(&store).await;++ let error = run_sweep_with_options(+ &store.reader,+ &store.writer,+ None,+ f.ws,+ f.proj,+ &DecayParams::default(),+ 0.0,+ ObservationRetention {+ days: -1,+ batch: 5_000,+ },+ false,+ )+ .await+ .expect_err("a negative retention age must fail closed");+ assert!(error.to_string().contains("observation_retention_days"));+}diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rsindex 1c348de9..2f05f4a6 100644--- a/crates/ai-memory-mcp/src/admin.rs+++ b/crates/ai-memory-mcp/src/admin.rs@@ -46,10 +46,10 @@ use std::pin::Pin;use ai_memory_consolidate::{AutoImproveReviewConfig, AutoImproveTelemetryParams, AutoImproveTelemetryReport, Bootstrap,BootstrapConfig, BootstrapOutcome, BootstrapSource, CuratorParams, CuratorReport,- EmbedBackfillCounts, EmbedBackfillOptions, SourceCounts, prune_sources_to_budget,- render_auto_improve_telemetry_report_markdown, render_curator_report_markdown,- run_auto_improve_review, run_auto_improve_telemetry_report, run_curator_report_with_breadth,- run_embedding_backfill, run_lint, run_sweep_with_breadth,+ EmbedBackfillCounts, EmbedBackfillOptions, ObservationRetention, SourceCounts,+ prune_sources_to_budget, render_auto_improve_telemetry_report_markdown,+ render_curator_report_markdown, run_auto_improve_review, run_auto_improve_telemetry_report,+ run_curator_report_with_breadth, run_embedding_backfill, run_lint, run_sweep_with_options,};use ai_memory_core::{ActiveProject, AgentKind, AutoImproveProposalId, Capability, DEFAULT_PROJECT_NAME,@@ -81,8 +81,13 @@ use tracing::{info, warn};const CONTRIBUTORS_WEBHOOK_NAME: &str = "contributors";-#[derive(Clone, Copy)]-struct DecayBreadthWeight(f64);+/// Sweep knobs that live beside `DecayParams` rather than inside it, injected+/// as one Extension so adding the next one is not a third router constructor.+#[derive(Clone, Copy, Default)]+struct SweepTuning {+ breadth_weight: f64,+ retention: ObservationRetention,+}/// Shared state for the admin router.#[derive(Clone)]@@ -554,6 +559,16 @@ pub fn admin_router(state: AdminState) -> Router {/// Build the admin router with the optional distinct-reader retention weight.pub fn admin_router_with_decay_breadth(state: AdminState, breadth_weight: f64) -> Router {+ admin_router_with_sweep_tuning(state, breadth_weight, ObservationRetention::default())+}++/// Build the admin router with every sweep knob that lives outside+/// `DecayParams`, including the opt-in observation prune (disabled by default).+pub fn admin_router_with_sweep_tuning(+ state: AdminState,+ breadth_weight: f64,+ retention: ObservationRetention,+) -> Router {let state = Arc::new(state);let operational = Router::new().route("/admin/backup", post(handle_backup))@@ -627,7 +642,10 @@ pub fn admin_router_with_decay_breadth(state: AdminState, breadth_weight: f64) -require_root_for_multiuser_admin,)).with_state(state)- .layer(axum::Extension(DecayBreadthWeight(breadth_weight)))+ .layer(axum::Extension(SweepTuning {+ breadth_weight,+ retention,+ }))}async fn require_root_for_multiuser_admin(@@ -2110,7 +2128,7 @@ async fn handle_auto_improve_report(async fn handle_curator(State(state): State<Arc<AdminState>>,- axum::Extension(breadth): axum::Extension<DecayBreadthWeight>,+ axum::Extension(tuning): axum::Extension<SweepTuning>,Json(req): Json<CuratorRequest>,) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {let mode = req.mode.as_deref().map(str::trim).filter(|s| !s.is_empty());@@ -2141,7 +2159,7 @@ async fn handle_curator(&req.workspace,&req.project,params.clone(),- breadth.0,+ tuning.breadth_weight,).await.map_err(|e| internal_err(e.to_string()))?;@@ -2803,19 +2821,20 @@ struct ForgetSweepRequest {async fn handle_forget_sweep(State(state): State<Arc<AdminState>>,- axum::Extension(breadth): axum::Extension<DecayBreadthWeight>,+ axum::Extension(tuning): axum::Extension<SweepTuning>,Json(req): Json<ForgetSweepRequest>,) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {let (ws, proj) = lookup_ws_proj_no_create(&state, &req.workspace, &req.project).await?;- run_sweep_with_breadth(+ run_sweep_with_options(&state.reader,&state.writer,Some(&state.wiki),ws,proj,&state.decay_params,- breadth.0,+ tuning.breadth_weight,+ tuning.retention,req.dry_run,).awaitdiff --git a/crates/ai-memory-mcp/src/lib.rs b/crates/ai-memory-mcp/src/lib.rsindex 9176704e..24044bcd 100644--- a/crates/ai-memory-mcp/src/lib.rs+++ b/crates/ai-memory-mcp/src/lib.rs@@ -16,5 +16,6 @@ mod server;pub use actor::actor_from_headers;pub use admin::{AdminState, ScopeInvalidation, ScopeInvalidator, admin_router, admin_router_with_decay_breadth,+ admin_router_with_sweep_tuning,};pub use server::{AiMemoryServer, MEMORY_INSTRUCTIONS};diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rsindex 693b2c3f..7ddf0a89 100644--- a/crates/ai-memory-mcp/src/server.rs+++ b/crates/ai-memory-mcp/src/server.rs@@ -6,8 +6,8 @@ use std::sync::{Arc, Mutex};use std::time::{Duration, Instant};use ai_memory_consolidate::{- AutoImproveReviewConfig, Consolidator, projection::cap_text_with_marker,- run_auto_improve_review, run_lint, run_sweep_with_breadth,+ AutoImproveReviewConfig, Consolidator, ObservationRetention, projection::cap_text_with_marker,+ run_auto_improve_review, run_lint, run_sweep_with_options,};use ai_memory_core::{ActiveProject, AgentKind, FeedbackKind, HandoffId, HandoffState, NewHandoff, PageId, PagePath,@@ -381,6 +381,9 @@ pub struct AiMemoryServer {decay_params: DecayParams,/// Optional distinct-reader reinforcement coefficient.decay_breadth_weight: f64,+ /// Opt-in bound on how long raw observations outlive their consolidation.+ /// Default is disabled, so `memory_forget_sweep` deletes no raw capture.+ observation_retention: ObservationRetention,/// M9 embedder for hybrid query. When `None`, `memory_query`/// still fuses FTS5 with entity matches and graph-neighbour expansion.embedder: Option<Arc<dyn Embedder>>,@@ -1258,6 +1261,7 @@ impl AiMemoryServer {wiki: None,decay_params: DecayParams::default(),decay_breadth_weight: 0.0,+ observation_retention: ObservationRetention::default(),embedder: None,reranker: None,client_activity: Arc::new(std::sync::Mutex::new(ClientActivityBuffer::new())),@@ -1770,6 +1774,14 @@ impl AiMemoryServer {self}+ /// Set the opt-in observation prune bound used by `memory_forget_sweep`.+ /// Unset, the sweep never deletes a raw observation.+ #[must_use]+ pub fn with_observation_retention(mut self, retention: ObservationRetention) -> Self {+ self.observation_retention = retention;+ self+ }+/// Attach the wiki handle. Without this, `memory_forget_sweep`/// and `memory_lint` cannot write their report pages.#[must_use]@@ -2375,7 +2387,7 @@ impl AiMemoryServer {}/// Run the M8 forget sweep over episodic pages.- #[tool(description = "Run the retention sweep. THREE passes, and they \+ #[tool(description = "Run the retention sweep. FOUR passes, and they \differ on what they will delete. (1) TTL: pages whose frontmatter \expires_at is in the past are hard-deleted (file + rows) REGARDLESS \OF TIER OR PIN — an explicit expiry overrides a pin, so a pinned \@@ -2386,8 +2398,15 @@ impl AiMemoryServer {evicted to a tombstone; only this pass exempts semantic / procedural \/ pinned pages. (3) Hard-delete: tombstones older than \hard_delete_after_days and their supersession ancestry are removed \- permanently. The report's expired / hard_deleted counts come from \- passes 1 and 3. Pass dry_run=true to preview.")]+ permanently. (4) Observation prune: raw observations older than \+ decay.observation_retention_days are deleted permanently, in bounded \+ batches, but ONLY for sessions already consolidated into a summary \+ page that is still live — an unconsolidated session, or one whose \+ page pass 2 or 3 just removed, keeps every row. This pass is DISABLED \+ by default (observation_retention_days = 0) and deletes nothing until \+ an operator opts in; when off, observations_pruned is 0. The report's \+ expired / hard_deleted / observations_pruned counts come from passes \+ 1, 3 and 4. Pass dry_run=true to preview.")]async fn memory_forget_sweep(&self,Parameters(args): Parameters<SweepArgs>,@@ -2405,7 +2424,7 @@ impl AiMemoryServer {&aps_actor,).await?;- let report = run_sweep_with_breadth(+ let report = run_sweep_with_options(&self.reader,&self.writer,self.wiki.as_ref(),@@ -2413,6 +2432,7 @@ impl AiMemoryServer {proj,&self.decay_params,self.decay_breadth_weight,+ self.observation_retention,args.dry_run.unwrap_or(false),).await@@ -5953,7 +5973,7 @@ mod tests {/// `memory_forget_sweep` is admin-gated and destructive, and the surface/// an agent reads is the only place it can learn what the tool will- /// delete. Both surfaces must name all three passes and must not claim a+ /// delete. Both surfaces must name all four passes and must not claim a/// blanket pin exemption.////// The description previously said "Semantic / procedural / pinned pages@@ -5997,6 +6017,28 @@ mod tests {desc.contains("hard_delete_after_days"),"description must name the hard-delete pass; got: {desc}");+ // The fourth pass deletes raw capture, not pages, and can remove+ // millions of rows on a real install. An agent that cannot read it+ // here cannot know the tool touches observations at all — which is the+ // exact regression this test was written to stop.+ assert!(+ desc.contains("observation_retention_days"),+ "description must name the observation prune pass; got: {desc}"+ );+ assert!(+ desc.contains("consolidated") && desc.contains("still live"),+ "description must state that only consolidated sessions with a live \+ page are pruned; got: {desc}"+ );+ assert!(+ desc.contains("DISABLED"),+ "description must state that observation pruning is off by default; \+ got: {desc}"+ );+ assert!(+ !desc.contains("THREE passes"),+ "the three-pass wording must not come back now there are four; got: {desc}"+ );// A bare unqualified exemption sentence is exactly the regression// this guards, so reject the old wording verbatim.assert!(diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex 4792d07f..c1ee768f 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -46,8 +46,8 @@ pub use error::{StoreError, StoreResult};pub use maintenance::MaintenanceJob;pub use ops::{AdmittedSession, DeleteWorkspaceSummary, EmbeddingWrite, HookSessionAdmission,- IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary, PagesMode,- PurgeSummary, ReorgSummary,+ IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary,+ ObservationPruneOutcome, PagesMode, PurgeSummary, ReorgSummary,};pub use reader::{ActivityWindow, AgentSessionCount, AutoImproveCandidateSession, BriefPageBody, BriefingPage,diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex 41ef311c..9cb690f4 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -1957,6 +1957,130 @@ pub fn hard_delete_decayed_page_chain(Ok(n)}+/// One batch of an observation prune, as reported by+/// [`prune_consolidated_observations`].+#[derive(Debug, Default, Clone, PartialEq, Eq)]+pub struct ObservationPruneOutcome {+ /// Observation rows permanently deleted by this batch.+ pub deleted: usize,+ /// Distinct sessions that lost rows in this batch. Returned rather than+ /// counted so a caller looping over batches can deduplicate a session whose+ /// rows straddle a batch boundary.+ pub sessions_touched: Vec<SessionId>,+ /// Sessions whose `ended_observation_count` watermark was lowered back onto+ /// the surviving row count.+ pub sessions_resynced: usize,+}++/// Permanently delete one bounded batch of raw observations belonging to+/// sessions that were already distilled into a live summary page.+///+/// Raw capture is the store's growth driver, but it is also the only input+/// consolidation, auto-improvement and the `raw_hits` fallback have. Deleting+/// it is safe exactly when its durable distillation still exists, so the+/// predicate joins through `sessions.summary_page_id` — the marker+/// [`end_session_row`] writes — and requires the page it points at to still be+/// live. `superseded_at IS NOT NULL` means decay already evicted that page, so+/// its session's raw capture is the last copy and must stay; a page hard-deleted+/// by the same sweep is excluded for free, because the `ON DELETE SET NULL`+/// foreign key has already cleared the pointer. A session with no `sessions`+/// row, or one that never ended, matches nothing and can never be pruned.+///+/// `ORDER BY created_at` makes the batching deterministic: a run interrupted+/// halfway leaves a clean age boundary rather than holes, so the next run+/// resumes on a store whose invariant ("everything older than X is gone for+/// consolidated sessions") is still one comparison.+///+/// The watermark resync is not optional. `session_end_disposition` compares a+/// live `COUNT(*)` against the persisted `sessions.ended_observation_count`;+/// leaving the watermark above reality would make a resumed session's genuinely+/// new work read as `AlreadyEnded` and never be re-consolidated. The+/// `ended_observation_count > (…)` guard keeps the repair idempotent and+/// one-directional — it can only lower a watermark that is now above reality.+/// Bounding it to the ids the delete returned also repairs sessions whose row+/// lives in a different scope than the observations being pruned.+pub fn prune_consolidated_observations(+ conn: &mut Connection,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ cutoff_us: i64,+ batch: usize,+) -> StoreResult<ObservationPruneOutcome> {+ if batch == 0 {+ return Ok(ObservationPruneOutcome::default());+ }+ let tx = conn.transaction()?;+ let mut deleted = 0usize;+ let mut sessions: BTreeSet<Vec<u8>> = BTreeSet::new();+ {+ let mut stmt = tx.prepare(+ "DELETE FROM observations \+ WHERE id IN ( \+ SELECT o.id FROM observations o \+ WHERE o.workspace_id = ?1 \+ AND o.project_id = ?2 \+ AND o.created_at < ?3 \+ AND EXISTS ( \+ SELECT 1 FROM sessions s \+ JOIN pages p ON p.id = s.summary_page_id \+ WHERE s.id = o.session_id \+ AND p.superseded_at IS NULL \+ ) \+ ORDER BY o.created_at \+ LIMIT ?4 \+ ) \+ RETURNING session_id",+ )?;+ let rows = stmt.query_map(+ params![+ workspace_id.as_bytes(),+ project_id.as_bytes(),+ cutoff_us,+ u64::try_from(batch).unwrap_or(u64::MAX),+ ],+ |row| row.get::<_, Vec<u8>>(0),+ )?;+ for row in rows {+ sessions.insert(row?);+ deleted += 1;+ }+ }+ let mut sessions_touched = Vec::with_capacity(sessions.len());+ let mut sessions_resynced = 0usize;+ if deleted > 0 {+ for session_id in &sessions {+ sessions_touched.push(SessionId::from_slice(session_id)?);+ sessions_resynced += tx.execute(+ "UPDATE sessions \+ SET ended_observation_count = ( \+ SELECT COUNT(*) FROM observations o WHERE o.session_id = sessions.id \+ ) \+ WHERE id = ?1 \+ AND ended_observation_count > ( \+ SELECT COUNT(*) FROM observations o WHERE o.session_id = sessions.id \+ )",+ params![session_id],+ )?;+ }+ audit_with_detail(+ &tx,+ "prune_observations",+ Some(workspace_id.as_bytes()),+ Some(project_id.as_bytes()),+ None,+ None,+ Timestamp::now().as_microsecond(),+ &format!("{{\"deleted\":{deleted}}}"),+ )?;+ }+ tx.commit()?;+ Ok(ObservationPruneOutcome {+ deleted,+ sessions_touched,+ sessions_resynced,+ })+}+/// Insert a new handoff in state=open.pub fn insert_handoff(conn: &mut Connection, h: &NewHandoff) -> StoreResult<HandoffId> {let tx = conn.transaction()?;@@ -2458,10 +2582,36 @@ fn audit(page_id: Option<&[u8; 16]>,author_id: Option<&[u8; 16]>,at: i64,+) -> StoreResult<()> {+ audit_with_detail(+ tx,+ op,+ workspace_id,+ project_id,+ page_id,+ author_id,+ at,+ "{}",+ )+}++/// `audit`, but with a caller-supplied JSON `detail` payload. Exists so a+/// destructive batch can record what it did (e.g. the observation prune's+/// deleted-row count) instead of leaving a gap only inferable from timestamps.+#[allow(clippy::too_many_arguments)]+fn audit_with_detail(+ tx: &rusqlite::Transaction<'_>,+ op: &str,+ workspace_id: Option<&[u8; 16]>,+ project_id: Option<&[u8; 16]>,+ page_id: Option<&[u8; 16]>,+ author_id: Option<&[u8; 16]>,+ at: i64,+ detail: &str,) -> StoreResult<()> {tx.execute("INSERT INTO audit_log (at, op, workspace_id, project_id, page_id, author_id, detail) \- VALUES (?1, ?2, ?3, ?4, ?5, ?6, '{}')",+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",params![at,op,@@ -2469,6 +2619,7 @@ fn audit(project_id.map(|b| &b[..]),page_id.map(|b| &b[..]),author_id.map(|b| &b[..]),+ detail,],)?;Ok(())diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rsindex 908e781b..eeb013b6 100644--- a/crates/ai-memory-store/src/reader.rs+++ b/crates/ai-memory-store/src/reader.rs@@ -3205,6 +3205,42 @@ impl ReaderPool {.await}+ /// Count the observations the prune pass would delete for this scope.+ ///+ /// Same predicate as+ /// [`prune_consolidated_observations`](crate::ops::prune_consolidated_observations),+ /// read-only, so a dry run can report a number without any chance of a+ /// write. Kept beside the delete rather than derived from it because the+ /// dry run must never open a write transaction at all.+ ///+ /// # Errors+ /// Propagates SQL errors.+ pub async fn prunable_observation_count(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ cutoff_us: i64,+ ) -> StoreResult<usize> {+ self.with_conn(move |conn| {+ let n: i64 = conn.query_row(+ "SELECT COUNT(*) FROM observations o \+ WHERE o.workspace_id = ?1 \+ AND o.project_id = ?2 \+ AND o.created_at < ?3 \+ AND EXISTS ( \+ SELECT 1 FROM sessions s \+ JOIN pages p ON p.id = s.summary_page_id \+ WHERE s.id = o.session_id \+ AND p.superseded_at IS NULL \+ )",+ params![workspace_id.as_bytes(), project_id.as_bytes(), cutoff_us],+ |row| row.get(0),+ )?;+ Ok(usize::try_from(n).unwrap_or(0))+ })+ .await+ }+/// Return the number of DISTINCT operators that reinforced each/// `is_latest = 1` page of a project, for the sweep's breadth term.///diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rsindex a5ba035b..3d778629 100644--- a/crates/ai-memory-store/src/writer.rs+++ b/crates/ai-memory-store/src/writer.rs@@ -26,8 +26,8 @@ use crate::auto_improve::{use crate::error::{StoreError, StoreResult};use crate::ops::{self, AdmittedSession, DeleteWorkspaceSummary, EmbeddingWrite, HookSessionAdmission,- IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary, PagesMode,- PurgeSummary, ReorgSummary,+ IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary,+ ObservationPruneOutcome, PagesMode, PurgeSummary, ReorgSummary,};use crate::session_consolidation::SessionConsolidationJob;use crate::users::{self, TOKEN_HASH_LEN};@@ -246,6 +246,13 @@ pub(crate) enum WriteCmd {cutoff_us: i64,reply: oneshot::Sender<StoreResult<usize>>,},+ PruneConsolidatedObservations {+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ cutoff_us: i64,+ batch: usize,+ reply: oneshot::Sender<StoreResult<ObservationPruneOutcome>>,+ },HealCatchAllRepoPaths {home: Option<String>,reply: oneshot::Sender<StoreResult<u64>>,@@ -1136,6 +1143,35 @@ impl WriterHandle {rx.await.map_err(|_| StoreError::WriterClosed)?}+ /// Delete one bounded batch of raw observations whose session was already+ /// consolidated into a live summary page, repairing the session end+ /// watermark in the same transaction.+ ///+ /// One batch is one transaction and one mailbox message, so a multi-million+ /// row prune never holds the write lock across the whole run — every other+ /// pending write interleaves between batches.+ ///+ /// # Errors+ /// Returns [`StoreError::WriterClosed`] or propagates SQL errors.+ pub async fn prune_consolidated_observations(+ &self,+ workspace_id: WorkspaceId,+ project_id: ProjectId,+ cutoff_us: i64,+ batch: usize,+ ) -> StoreResult<ObservationPruneOutcome> {+ let (tx, rx) = oneshot::channel();+ self.send(WriteCmd::PruneConsolidatedObservations {+ workspace_id,+ project_id,+ cutoff_us,+ batch,+ reply: tx,+ })+ .await?;+ rx.await.map_err(|_| StoreError::WriterClosed)?+ }+/// NULL out catch-all project `repo_path` rows so existing installs/// self-heal on upgrade: the broad sentinels (`$HOME` and the filesystem/// root) plus any path that exists locally but is not a git work-tree@@ -2097,6 +2133,22 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {);send_or_warn(reply, result, "hard_delete_decayed_page_chain");}+ WriteCmd::PruneConsolidatedObservations {+ workspace_id,+ project_id,+ cutoff_us,+ batch,+ reply,+ } => {+ let result = ops::prune_consolidated_observations(+ &mut conn,+ workspace_id,+ project_id,+ cutoff_us,+ batch,+ );+ send_or_warn(reply, result, "prune_consolidated_observations");+ }WriteCmd::HealCatchAllRepoPaths { home, reply } => {let result = ops::heal_catch_all_repo_paths(&mut conn, home.as_deref());send_or_warn(reply, result, "heal_catch_all_repo_paths");diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex dc9164a1..cd997219 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -146,7 +146,20 @@ from hook paths.version ancestry only within that sweep's resolved workspace/project,together with entity-index rows orphaned by the purge. A newer page recreatedat the same path is preserved. Semantic / pinned / freshly-touched pages- survive.+ survive. A fourth pass, disabled unless `[decay] observation_retention_days`+ is positive, then deletes raw `observations` older than that age — but only+ for sessions already consolidated into a summary page that is still live, so+ raw capture is never removed while it is the last copy of that session's+ work. It runs last so this run's own evictions and hard-deletes already+ exclude their sessions, deletes in `observation_prune_batch` transactions so+ a multi-million row prune cannot hold the write lock, and repairs+ `sessions.ended_observation_count` downward in the same transaction. The+ prune is irreversible in a specific sense: observations are the input to+ consolidation, so a pruned session can never be re-consolidated — not with+ a better model, a better prompt, or a fixed consolidator bug — and its+ summary page becomes the only surviving account of that session. Freed+ SQLite pages are reused, not returned to the OS: the `.db` file does not+ shrink, the backup tarball does.Scheduled sweep, rule-based lint, and opt-in embedding backfill ticksenumerate every existing workspace/project scope before doing per-projectwork, matching the auto-improvement scheduler's store-wide scope model. A@@ -513,6 +526,8 @@ mu = 0.04 # ↑ if recent hits should count morecold_threshold = 0.20 # below this → remove file + retain tombstonehard_delete_after_days = 180breadth_weight = 0.0 # opt-in reward for distinct operators+observation_retention_days = 0 # 0 = never prune raw observations+observation_prune_batch = 5000 # rows per prune transaction[slots] # optional shared-server injection boundaryper_user = false # shared + own slots in agent context