Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(sweep): opt-in bounded retention for raw observations (#508)

Djalma Júnior4756e153b927
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7070993f..275c57d0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Added
+- Bounded raw-observation retention as an opt-in fourth pass of the M8 forget
+ sweep, disabled by default. Nothing in the tree could delete an `observations`
+ row for age: the sweep acts on pages only, so raw capture grew without limit.
+ Measured on a three-month-old install, `observations` plus its FTS shadow and
+ four indexes were ~5.4 GB of a 5.53 GB store (5,236,232 rows), against 0.03 GB
+ of `pages` — the 2,365 compiled pages that hold the durable value. The nightly
+ backup tarball was 2.7 GB, almost all of it capture already distilled into
+ 30 MB of Markdown. `decay.observation_retention_days` (default `0` = disabled)
+ now lets an operator bound that, and `decay.observation_prune_batch` (default
+ `5000`) bounds each transaction. (#508)
+
+ The pass deletes an observation only when its session was already consolidated
+ into a summary page that is still live, and the row is older than the
+ configured age. That is three gates on columns the schema already maintains:
+ `sessions.summary_page_id` is written only by the session-end path, so it
+ already implies `ended_at IS NOT NULL`; `pages.superseded_at IS NULL` excludes
+ a session whose page decay has evicted, because that session's raw rows are
+ now the only surviving copy; and a hard-deleted page has already NULLed the
+ pointer through `ON DELETE SET NULL`, so the join finds nothing. The prune
+ therefore runs LAST, after this same run's evictions and hard-deletes, rather
+ than letting raw capture outlive its distillation by one sweep. A session with
+ no `sessions` row, or one still running, matches nothing and can never lose a
+ row.
+
+ Each batch is one transaction sent as its own message to the single-writer
+ actor, so every other pending write interleaves between batches and a
+ multi-million row prune never holds the write lock across the run. Measured on
+ a 300,000-row store built from the shipped migrations: ~23-27 ms per 5,000-row
+ batch including the FTS trigger work, plus ~2-4 ms to commit. The session-end
+ observation watermark (`sessions.ended_observation_count`) is repaired in the
+ same transaction and only downward, so a resumed session's genuinely new work
+ is still read as new work instead of `AlreadyEnded`. One `prune_observations`
+ audit row is written per batch that actually deleted, inside that batch's
+ transaction. Zero migrations: every access is an index seek on
+ `idx_observations_project_created` and `idx_observations_session`, both of
+ which predate this change.
+
+ The cost is stated rather than hidden, and it is not just disk: observations
+ are the input to consolidation, so pruning is irreversible — a pruned session
+ can never be re-consolidated (not with a better model, a better prompt, or a
+ fixed consolidator bug) and its summary page becomes the only surviving
+ account of that session. Concretely: the raw transcript
+ view returns empty, `raw_hits` can no longer match the exact original wording,
+ a manual auto-improve rerun rejects with `too_few_observations`, and a
+ `move_session` with `PagesMode::Regenerate` has nothing left to rebuild the
+ page from. All of it is reachable only by setting a positive age. Note that
+ SQLite does not return freed pages to the OS without `VACUUM`, so the `.db`
+ file will not shrink after the first prune — the `ai-memory backup` tarball
+ will, because it is a fresh online copy.
+
### Fixed
- The CHANGELOG frozen-section check no longer fires on a branch that merged
`main` after a release. It compared line ranges since the merge base, so a
diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rs
index e4fce548..bf5a32e1 100644
--- a/crates/ai-memory-cli/src/commands/serve.rs
+++ b/crates/ai-memory-cli/src/commands/serve.rs
@@ -7,8 +7,9 @@ use std::sync::Arc;
use std::time::Duration;
use ai_memory_consolidate::{
- AutoImproveReviewConfig, Consolidator, EmbedBackfillOptions, ScheduledAutoImproveSettings,
- run_auto_improve_scheduler_tick, run_embedding_backfill, run_lint, run_sweep_with_breadth,
+ AutoImproveReviewConfig, Consolidator, EmbedBackfillOptions, ObservationRetention,
+ ScheduledAutoImproveSettings, run_auto_improve_scheduler_tick, run_embedding_backfill,
+ run_lint, run_sweep_with_options,
};
use ai_memory_core::{ActiveProject, ProjectId, Sanitizer, WorkspaceId};
use ai_memory_hooks::{
@@ -17,7 +18,7 @@ use ai_memory_hooks::{
};
use ai_memory_llm::{Embedder, LlmProvider, ProviderHealth, build_embedder, build_provider};
use ai_memory_mcp::{
- AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_decay_breadth,
+ AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_sweep_tuning,
};
use ai_memory_store::{ReaderPool, Store, WriterHandle};
use ai_memory_web::{WebMountSpec, mount_web_router, normalize_prefix, web_base_href};
@@ -525,6 +526,7 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {
.with_wiki(wiki.clone())
.with_decay_params(decay_params)
.with_decay_breadth_weight(config.decay.breadth_weight)
+ .with_observation_retention(config.decay.observation_retention())
.with_auto_improve_require_approval(config.auto_improve.require_approval)
.with_auto_improve_review_config(auto_improve_review_config_from_settings(
&config.auto_improve,
@@ -687,7 +689,7 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {
sanitizer: sanitizer.clone(),
data_dir: config.data_dir.clone(),
});
- let admin = admin_router_with_decay_breadth(
+ let admin = admin_router_with_sweep_tuning(
AdminState {
ingest_metrics: ingest_metrics.clone(),
writer: store.writer.clone(),
@@ -717,6 +719,7 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {
trusted_proxy_identity: trusted_proxy_identity_enabled(&config.auth),
},
config.decay.breadth_weight,
+ config.decay.observation_retention(),
);
// Multi-rung auth assembly:
// - rung 0 (no bearer_token configured) → AuthState::new
@@ -966,6 +969,7 @@ async fn start_maintenance_scheduler(
&wiki,
&decay.decay_params(),
decay.breadth_weight,
+ decay.observation_retention(),
)
.await?;
if outcome.errors > 0 {
@@ -980,6 +984,7 @@ async fn start_maintenance_scheduler(
evicted = outcome.evicted,
expired = outcome.expired,
hard_deleted = outcome.hard_deleted,
+ observations_pruned = outcome.observations_pruned,
errors = outcome.errors,
elapsed_ms = started.elapsed().as_millis(),
"scheduled forget sweep completed"
@@ -1195,6 +1200,7 @@ struct ScheduledSweepTickOutcome {
evicted: usize,
expired: usize,
hard_deleted: usize,
+ observations_pruned: usize,
errors: usize,
}
@@ -1204,6 +1210,7 @@ async fn run_scheduled_sweep_tick(
wiki: &Wiki,
decay: &ai_memory_store::DecayParams,
breadth_weight: f64,
+ retention: ObservationRetention,
) -> Result<ScheduledSweepTickOutcome> {
let scopes = reader.list_all_scopes().await?;
let mut outcome = ScheduledSweepTickOutcome {
@@ -1212,7 +1219,7 @@ async fn run_scheduled_sweep_tick(
};
for scope in scopes {
- match run_sweep_with_breadth(
+ match run_sweep_with_options(
reader,
writer,
Some(wiki),
@@ -1220,6 +1227,7 @@ async fn run_scheduled_sweep_tick(
scope.project_id,
decay,
breadth_weight,
+ retention,
false,
)
.await
@@ -1229,6 +1237,7 @@ async fn run_scheduled_sweep_tick(
outcome.evicted += report.evicted.iter().filter(|page| page.deleted).count();
outcome.expired += report.expired.len();
outcome.hard_deleted += report.hard_deleted;
+ outcome.observations_pruned += report.observations_pruned;
}
Err(e) => {
outcome.errors += 1;
@@ -2521,9 +2530,16 @@ mod tests {
cold_threshold: 2.0,
..ai_memory_store::DecayParams::default()
};
- let outcome = run_scheduled_sweep_tick(&store.reader, &store.writer, &wiki, &decay, 0.0)
- .await
- .unwrap();
+ let outcome = run_scheduled_sweep_tick(
+ &store.reader,
+ &store.writer,
+ &wiki,
+ &decay,
+ 0.0,
+ ObservationRetention::default(),
+ )
+ .await
+ .unwrap();
assert_eq!(outcome.scopes, 2);
assert_eq!(outcome.errors, 0);
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index 8634d79e..47638b01 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -61,6 +61,11 @@ pub struct DecaySettings {
pub hard_delete_after_days: i64,
/// Optional weight for the number of distinct authenticated readers.
pub breadth_weight: f64,
+ /// Age past which a consolidated session's raw observations may be pruned.
+ /// `0` disables the pass; nothing is deleted until an operator opts in.
+ pub observation_retention_days: i64,
+ /// Observation rows deleted per prune transaction.
+ pub observation_prune_batch: usize,
}
impl Default for DecaySettings {
@@ -74,6 +79,8 @@ impl Default for DecaySettings {
cold_threshold: base.cold_threshold,
hard_delete_after_days: base.hard_delete_after_days,
breadth_weight: 0.0,
+ observation_retention_days: 0,
+ observation_prune_batch: ai_memory_consolidate::DEFAULT_OBSERVATION_PRUNE_BATCH,
}
}
}
@@ -91,6 +98,19 @@ impl DecaySettings {
hard_delete_after_days: self.hard_delete_after_days,
}
}
+
+ /// Opt-in observation prune bound consumed by the M8 sweep.
+ ///
+ /// Deliberately separate from [`Self::decay_params`]: keeping it out of the
+ /// public `DecayParams` struct is what lets every downstream Rust caller
+ /// that builds one directly keep compiling — and keep today's behaviour.
+ #[must_use]
+ pub fn observation_retention(self) -> ai_memory_consolidate::ObservationRetention {
+ ai_memory_consolidate::ObservationRetention {
+ days: self.observation_retention_days,
+ batch: self.observation_prune_batch,
+ }
+ }
}
/// Top-level runtime configuration.
@@ -929,6 +949,19 @@ impl Config {
);
}
+ // Fail closed at load rather than at 3am inside a destructive pass: a
+ // negative age would be a nonsensical cutoff, and a zero batch would
+ // spin the prune loop forever without deleting anything.
+ if config.decay.observation_retention_days < 0 {
+ anyhow::bail!(
+ "decay.observation_retention_days must be greater than or equal to zero \
+ (0 disables observation pruning)"
+ );
+ }
+ if config.decay.observation_prune_batch == 0 {
+ anyhow::bail!("decay.observation_prune_batch must be greater than zero");
+ }
+
// Fail at startup rather than shipping a prompt that is all scaffolding
// and no observations: below this floor the fixed system prompt and page
// conventions consume the entire budget, so every consolidation would
@@ -1365,6 +1398,11 @@ mod tests {
assert_eq!(cfg.maintenance.lint_interval_secs, 86_400);
assert_eq!(cfg.maintenance.embedding_backfill_interval_secs, 0);
assert_eq!(cfg.decay.breadth_weight, 0.0);
+ // Observation pruning must stay OFF by default: an install that never
+ // opts in keeps every raw observation it has today.
+ assert_eq!(cfg.decay.observation_retention_days, 0);
+ assert_eq!(cfg.decay.observation_prune_batch, 5_000);
+ assert!(!cfg.decay.observation_retention().is_enabled());
assert!(!cfg.slots.per_user);
assert!(cfg.auto_improve.scheduler.enabled);
assert_eq!(cfg.auto_improve.scheduler.interval_secs, 3_600);
@@ -1433,6 +1471,27 @@ mod tests {
}
}
+ /// A negative retention age or a zero batch is rejected at load, beside
+ /// the breadth-weight guard, so a destructive pass can never be configured
+ /// into a nonsensical shape.
+ #[test]
+ fn load_rejects_destructive_invalid_observation_retention() {
+ for (key, value) in [
+ ("observation_retention_days", "-1"),
+ ("observation_prune_batch", "0"),
+ ] {
+ let tmp = TempDir::new().unwrap();
+ let config_path = tmp.path().join("config.toml");
+ std::fs::write(&config_path, format!("[decay]\n{key} = {value}\n")).unwrap();
+ let error = Config::load(Some(&config_path), Some(tmp.path().to_path_buf()))
+ .expect_err("invalid observation retention must fail closed");
+ assert!(
+ error.to_string().contains(key),
+ "unexpected error for {key} = {value}: {error:#}"
+ );
+ }
+ }
+
/// The consolidation budget must be big enough to leave room for
/// observations after the fixed system prompt and page conventions.
/// Below the floor every consolidation would be evidence-free, so it
diff --git a/crates/ai-memory-cli/templates/config.default.toml b/crates/ai-memory-cli/templates/config.default.toml
index 289fda9f..96196301 100644
--- a/crates/ai-memory-cli/templates/config.default.toml
+++ b/crates/ai-memory-cli/templates/config.default.toml
@@ -80,6 +80,22 @@ log_level = "info"
# - Bigger access boost (frequently-used stuff lasts longer): sigma = 1.0
# - Bigger time-pressure on access: mu = 0.08
# - Reward pages used by more identified operators: breadth_weight = 0.2
+#
+# Raw observations are the store's growth driver (a busy install can add
+# gigabytes a year), while the durable value is the compiled pages. Set
+# observation_retention_days to a positive number to let the sweep delete raw
+# observations older than that — but ONLY for sessions already consolidated into
+# a page that is still live. Unconsolidated sessions, and sessions whose page was
+# evicted by decay, keep every row. 0 (the default) disables the pass entirely.
+# Pruning is IRREVERSIBLE: observations are the input to consolidation, so a
+# pruned session can never be re-consolidated (a better model, a better prompt,
+# or a fixed consolidator bug make no difference) and its summary page becomes
+# the only surviving account of that session. The 0 default is a contract: this
+# pass never runs unless an operator sets a positive age here.
+# SQLite does not return freed pages to the OS without VACUUM, so the .db file
+# will not shrink after the first prune; the `ai-memory backup` tarball will.
+# - A quarter of raw capture: observation_retention_days = 90
+# - Smaller transactions on a slow disk: observation_prune_batch = 2000
[decay]
lambda = 0.02
sigma = 0.6
@@ -88,6 +104,8 @@ salience_default = 1.0
cold_threshold = 0.20
hard_delete_after_days = 180
breadth_weight = 0.0
+observation_retention_days = 0 # 0 = never prune raw observations
+observation_prune_batch = 5000 # rows per prune transaction
# Privacy strip.
#
diff --git a/crates/ai-memory-consolidate/src/lib.rs b/crates/ai-memory-consolidate/src/lib.rs
index 8345b3cd..f58e9155 100644
--- a/crates/ai-memory-consolidate/src/lib.rs
+++ b/crates/ai-memory-consolidate/src/lib.rs
@@ -64,7 +64,10 @@ pub use embed::{
EmbedBackfillCounts, EmbedBackfillError, EmbedBackfillOptions, run_embedding_backfill,
};
pub use lint::{LintError, LintFinding, LintOptions, LintReport, run_lint, stale_days_for};
-pub use sweep::{EvictedPage, SweepError, SweepReport, run_sweep, run_sweep_with_breadth};
+pub use sweep::{
+ DEFAULT_OBSERVATION_PRUNE_BATCH, EvictedPage, ObservationRetention, SweepError, SweepReport,
+ run_sweep, run_sweep_with_breadth, run_sweep_with_options,
+};
pub use types::{
ConsolidatedBatch, ConsolidatedPage, ConsolidatedPageUpdate, ConsolidationOutcome, PageKind,
SlotKind,
diff --git a/crates/ai-memory-consolidate/src/sweep.rs b/crates/ai-memory-consolidate/src/sweep.rs
index e48224ab..b7dcfc19 100644
--- a/crates/ai-memory-consolidate/src/sweep.rs
+++ b/crates/ai-memory-consolidate/src/sweep.rs
@@ -18,8 +18,16 @@
//! `hard_delete_after_days` and their supersession ancestry. A page recreated
//! at the same path is a separate live chain and is preserved. Ordinary
//! supersession rows are safe because only decay writes `superseded_at`.
+//!
+//! Observation prune pass (opt-in, disabled unless
+//! `ObservationRetention::days > 0`) deletes raw observations older than the
+//! configured age, and only for sessions already distilled into a page that is
+//! still live. It runs LAST so both of this run's page deletions are already
+//! visible to its predicate: a session whose summary page was evicted or
+//! hard-deleted by the passes above keeps its raw capture, because that capture
+//! is now the only surviving copy.
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
use ai_memory_core::{PageId, ProjectId, Tier, WorkspaceId};
use ai_memory_store::{
@@ -79,8 +87,56 @@ pub struct SweepReport {
pub expired: Vec<ExpiredPage>,
/// Number of page-version rows permanently deleted on this pass.
pub hard_deleted: usize,
+ /// Observations the prune predicate matched. Reported in both modes; `0`
+ /// when observation retention is disabled, which is the default.
+ pub observations_prunable: usize,
+ /// Observation rows permanently deleted. Always `0` on `dry_run`, exactly
+ /// like the wiki-routed passes above.
+ pub observations_pruned: usize,
+ /// Distinct consolidated sessions that lost rows, so the blast radius is a
+ /// number in the response rather than a claim in a changelog.
+ pub observation_prune_sessions: usize,
+ /// Transactions the prune spent. Makes the batching bound observable.
+ pub observation_prune_batches: usize,
+}
+
+/// Opt-in bound on how long raw observations outlive their consolidation.
+///
+/// Deliberately a type of its own rather than two more
+/// [`DecayParams`] fields: the retention coefficients are a public struct that
+/// downstream Rust callers construct directly, and widening it would break
+/// them. Same reasoning the access-breadth coefficient already carries.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ObservationRetention {
+ /// Age in days past which a consolidated session's observations may be
+ /// pruned. `0` disables the pass entirely — the default, so an existing
+ /// install behaves exactly as it did before this pass existed.
+ pub days: i64,
+ /// Rows deleted per transaction.
+ pub batch: usize,
+}
+
+/// Disabled: nothing is ever pruned until an operator sets a positive age.
+impl Default for ObservationRetention {
+ fn default() -> Self {
+ Self {
+ days: 0,
+ batch: DEFAULT_OBSERVATION_PRUNE_BATCH,
+ }
+ }
+}
+
+impl ObservationRetention {
+ /// `true` when the pass is enabled and can delete something.
+ #[must_use]
+ pub const fn is_enabled(self) -> bool {
+ self.days > 0 && self.batch > 0
+ }
}
+/// Default rows per prune transaction.
+pub const DEFAULT_OBSERVATION_PRUNE_BATCH: usize = 5_000;
+
/// Errors raised by the sweep.
#[derive(Debug, Error)]
#[non_exhaustive]
@@ -91,6 +147,9 @@ pub enum SweepError {
/// The optional access-breadth coefficient was negative or non-finite.
#[error("decay breadth_weight must be a finite number greater than or equal to zero")]
InvalidBreadthWeight,
+ /// The opt-in observation retention age was negative.
+ #[error("decay observation_retention_days must be greater than or equal to zero")]
+ InvalidObservationRetention,
}
const US_PER_DAY: f64 = 86_400_000_000.0;
@@ -139,17 +198,54 @@ pub async fn run_sweep(
#[allow(clippy::too_many_arguments)]
pub async fn run_sweep_with_breadth(
reader: &ReaderPool,
- _writer: &WriterHandle,
+ writer: &WriterHandle,
+ wiki: Option<&Wiki>,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ params: &DecayParams,
+ breadth_weight: f64,
+ dry_run: bool,
+) -> Result<SweepReport, SweepError> {
+ run_sweep_with_options(
+ reader,
+ writer,
+ wiki,
+ workspace_id,
+ project_id,
+ params,
+ breadth_weight,
+ ObservationRetention::default(),
+ dry_run,
+ )
+ .await
+}
+
+/// Run a sweep with the breadth coefficient and the opt-in observation prune.
+///
+/// The prune is the only pass that can delete raw capture, and it is off unless
+/// `retention.days` is positive.
+///
+/// # Errors
+/// Returns [`SweepError::InvalidObservationRetention`] for a negative age, in
+/// addition to the errors documented by [`run_sweep_with_breadth`].
+#[allow(clippy::too_many_arguments)]
+pub async fn run_sweep_with_options(
+ reader: &ReaderPool,
+ writer: &WriterHandle,
wiki: Option<&Wiki>,
workspace_id: WorkspaceId,
project_id: ProjectId,
params: &DecayParams,
breadth_weight: f64,
+ retention: ObservationRetention,
dry_run: bool,
) -> Result<SweepReport, SweepError> {
if !breadth_weight.is_finite() || breadth_weight < 0.0 {
return Err(SweepError::InvalidBreadthWeight);
}
+ if retention.days < 0 {
+ return Err(SweepError::InvalidObservationRetention);
+ }
let candidates = reader.decay_candidates(workspace_id, project_id).await?;
let breadth =
access_breadth_for_scoring(reader, workspace_id, project_id, breadth_weight).await?;
@@ -199,6 +295,18 @@ pub async fn run_sweep_with_breadth(
}
let mut hard_deleted = 0usize;
+ let mut observations_prunable = 0usize;
+ let mut observations_pruned = 0usize;
+ let mut observation_prune_sessions = 0usize;
+ let mut observation_prune_batches = 0usize;
+ let observation_cutoff_us = Timestamp::now()
+ .as_microsecond()
+ .saturating_sub(retention.days.saturating_mul(US_PER_DAY_I64));
+ if retention.is_enabled() {
+ observations_prunable = reader
+ .prunable_observation_count(workspace_id, project_id, observation_cutoff_us)
+ .await?;
+ }
if !dry_run {
for page in &mut expired {
let path = match ai_memory_core::PagePath::new(page.path.clone()) {
@@ -284,6 +392,37 @@ pub async fn run_sweep_with_breadth(
}
}
}
+
+ // LAST on purpose. Both passes above can strip a session of its
+ // distillation — decay stamps `superseded_at`, hard-delete removes the
+ // row and the `ON DELETE SET NULL` foreign key clears the pointer — and
+ // the prune predicate reads exactly those two columns. Running here
+ // means this run's own evictions already exclude their sessions,
+ // instead of a batch of raw capture outliving its page by one sweep.
+ //
+ // Unlike the wiki-routed passes this one runs without a `wiki` handle:
+ // observations have no Markdown file, so there is no source of truth
+ // for reconciliation to re-index and no store-only-mutation hazard.
+ if retention.is_enabled() {
+ let mut touched: HashSet<ai_memory_core::SessionId> = HashSet::new();
+ loop {
+ let outcome = writer
+ .prune_consolidated_observations(
+ workspace_id,
+ project_id,
+ observation_cutoff_us,
+ retention.batch,
+ )
+ .await?;
+ observation_prune_batches += 1;
+ observations_pruned += outcome.deleted;
+ touched.extend(outcome.sessions_touched.iter().copied());
+ if outcome.deleted < retention.batch {
+ break;
+ }
+ }
+ observation_prune_sessions = touched.len();
+ }
}
Ok(SweepReport {
@@ -292,6 +431,10 @@ pub async fn run_sweep_with_breadth(
evicted,
expired,
hard_deleted,
+ observations_prunable,
+ observations_pruned,
+ observation_prune_sessions,
+ observation_prune_batches,
})
}
diff --git a/crates/ai-memory-consolidate/tests/observation_retention.rs b/crates/ai-memory-consolidate/tests/observation_retention.rs
new file mode 100644
index 00000000..a6e9f545
--- /dev/null
+++ b/crates/ai-memory-consolidate/tests/observation_retention.rs
@@ -0,0 +1,554 @@
+//! The opt-in observation prune, end to end.
+//!
+//! Raw observations are ~98% of a mature store's bytes, and their durable value
+//! was already distilled into pages. Deleting them is safe exactly when that
+//! distillation still exists — so these tests hold the pass to the only
+//! discrimination that makes it safe: a session consolidated into a live page
+//! loses its old raw capture; a session with equally old capture and no page,
+//! or with a page decay already evicted, keeps every row.
+//!
+//! They also pin the two properties an operator has to be able to trust without
+//! reading the SQL: nothing is deleted at the shipped defaults, and the FTS
+//! index stops matching what the prune removed.
+
+use ai_memory_consolidate::{ObservationRetention, run_sweep, run_sweep_with_options};
+use ai_memory_core::{
+ AgentKind, NewObservation, NewPage, NewSession, ObservationKind, PageId, PagePath, ProjectId,
+ Sanitized, Sanitizer, SessionId, Tier, WorkspaceId,
+};
+use ai_memory_store::{DecayParams, Store, WriterHandle};
+use rusqlite::{Connection, params};
+use tempfile::TempDir;
+
+const US_PER_DAY: i64 = 86_400_000_000;
+/// Well past any retention age used below, so age is never what separates the
+/// fixtures — only whether the session was consolidated.
+const OBS_AGE_DAYS: i64 = 400;
+const RETENTION_DAYS: i64 = 90;
+
+/// A session whose observations the prune is allowed to delete: ended, with a
+/// live summary page.
+const CONSOLIDATED_BODY: &str = "consolidated capture zebrafinch";
+/// A session with identical, equally old capture that was never consolidated.
+const RAW_BODY: &str = "unconsolidated capture zebrafinch";
+/// A session that WAS consolidated, but whose page decay has since evicted —
+/// the raw rows are now the only surviving copy.
+const EVICTED_BODY: &str = "evicted capture zebrafinch";
+
+struct Fixture {
+ ws: WorkspaceId,
+ proj: ProjectId,
+ consolidated: SessionId,
+ unconsolidated: SessionId,
+ evicted: SessionId,
+}
+
+/// Three sessions x three observations each, all backdated to the same age.
+async fn seed(store: &Store) -> Fixture {
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .expect("ws");
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "retention".to_string(), None)
+ .await
+ .expect("proj");
+
+ let consolidated = session(store, ws, proj, 1, CONSOLIDATED_BODY).await;
+ let unconsolidated = session(store, ws, proj, 2, RAW_BODY).await;
+ let evicted = session(store, ws, proj, 3, EVICTED_BODY).await;
+
+ // Only the first two ends link a page; the third's page is evicted below.
+ let page = write_page(&store.writer, ws, proj, "sessions/consolidated.md").await;
+ store
+ .writer
+ .end_session(consolidated, Some(page))
+ .await
+ .expect("end consolidated");
+ let evicted_page = write_page(&store.writer, ws, proj, "sessions/evicted.md").await;
+ store
+ .writer
+ .end_session(evicted, Some(evicted_page))
+ .await
+ .expect("end evicted");
+
+ let conn = aux(store);
+ // Decay eviction, exactly as `soft_delete_for_decay_if_latest` writes it:
+ // the row survives as a tombstone but the Markdown is gone.
+ conn.execute(
+ "UPDATE pages SET is_latest = 0, superseded_at = ?1 WHERE id = ?2",
+ params![
+ jiff::Timestamp::now().as_microsecond(),
+ evicted_page.as_bytes()
+ ],
+ )
+ .expect("evict page");
+ let then_us = jiff::Timestamp::now().as_microsecond() - OBS_AGE_DAYS * US_PER_DAY;
+ conn.execute("UPDATE observations SET created_at = ?1", params![then_us])
+ .expect("backdate observations");
+
+ Fixture {
+ ws,
+ proj,
+ consolidated,
+ unconsolidated,
+ evicted,
+ }
+}
+
+async fn session(store: &Store, ws: WorkspaceId, proj: ProjectId, n: u8, body: &str) -> SessionId {
+ let mut raw = [0u8; 16];
+ raw[15] = n;
+ let id = SessionId::from_slice(&raw).expect("session id");
+ store
+ .writer
+ .begin_session(NewSession {
+ id,
+ workspace_id: ws,
+ project_id: proj,
+ agent_kind: AgentKind::ClaudeCode,
+ cwd: None,
+ actor_user: None,
+ })
+ .await
+ .expect("begin session");
+ let sanitizer = Sanitizer::builtin();
+ for i in 0..3 {
+ store
+ .writer
+ .insert_observation(Sanitized::new(
+ NewObservation {
+ session_id: id,
+ workspace_id: ws,
+ project_id: proj,
+ kind: ObservationKind::UserPrompt,
+ extension: None,
+ source_event: None,
+ title: format!("obs {n}-{i}"),
+ body: body.to_string(),
+ importance: 5,
+ },
+ &sanitizer,
+ ))
+ .await
+ .expect("insert observation");
+ }
+ id
+}
+
+async fn write_page(writer: &WriterHandle, ws: WorkspaceId, proj: ProjectId, path: &str) -> PageId {
+ writer
+ .upsert_page(NewPage {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new(path.to_string()).expect("path"),
+ title: path.into(),
+ body: "distilled".into(),
+ tier: Tier::Episodic,
+ frontmatter_json: serde_json::json!({}),
+ pinned: false,
+ links: Vec::new(),
+ author_id: None,
+ expires_at: None,
+ entities: Vec::new(),
+ })
+ .await
+ .expect("upsert page")
+}
+
+fn aux(store: &Store) -> Connection {
+ let conn = Connection::open(store.db_path()).expect("aux conn");
+ conn.pragma_update(None, "busy_timeout", 5_000).unwrap();
+ conn
+}
+
+fn observations_for(conn: &Connection, session: SessionId) -> i64 {
+ conn.query_row(
+ "SELECT COUNT(*) FROM observations WHERE session_id = ?1",
+ params![session.as_bytes()],
+ |row| row.get(0),
+ )
+ .expect("count observations")
+}
+
+fn fts_matches(conn: &Connection, needle: &str) -> i64 {
+ conn.query_row(
+ "SELECT COUNT(*) FROM observations_fts WHERE observations_fts MATCH ?1",
+ params![needle],
+ |row| row.get(0),
+ )
+ .expect("fts match")
+}
+
+fn enabled() -> ObservationRetention {
+ ObservationRetention {
+ days: RETENTION_DAYS,
+ batch: 5_000,
+ }
+}
+
+/// The no-change proof. `run_sweep` is what every existing caller reaches, and
+/// `ObservationRetention::default()` is what the shipped config produces. On a
+/// store full of ancient consolidated capture, both must delete nothing at all.
+#[tokio::test]
+async fn the_default_configuration_prunes_no_observation() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("store");
+ let f = seed(&store).await;
+ let params = DecayParams::default();
+
+ let report = run_sweep(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &params,
+ false,
+ )
+ .await
+ .expect("sweep");
+ assert_eq!(report.observations_pruned, 0);
+ assert_eq!(
+ report.observations_prunable, 0,
+ "disabled must not even count"
+ );
+ assert_eq!(report.observation_prune_batches, 0);
+
+ let report = run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &params,
+ 0.0,
+ ObservationRetention::default(),
+ false,
+ )
+ .await
+ .expect("sweep");
+ assert_eq!(report.observations_pruned, 0);
+
+ let conn = aux(&store);
+ for s in [f.consolidated, f.unconsolidated, f.evicted] {
+ assert_eq!(
+ observations_for(&conn, s),
+ 3,
+ "the default configuration must leave every observation in place"
+ );
+ }
+}
+
+/// The discrimination the whole feature rests on. Same scope, same age, same
+/// body length: the ONLY difference is whether the session's work still exists
+/// as a page. Consolidated loses its rows; unconsolidated and decay-evicted
+/// keep theirs.
+#[tokio::test]
+async fn only_a_consolidated_session_with_a_live_page_loses_its_observations() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("store");
+ let f = seed(&store).await;
+
+ let report = run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &DecayParams::default(),
+ 0.0,
+ enabled(),
+ false,
+ )
+ .await
+ .expect("sweep");
+
+ assert_eq!(report.observations_pruned, 3);
+ assert_eq!(report.observations_prunable, 3);
+ assert_eq!(report.observation_prune_sessions, 1);
+
+ let conn = aux(&store);
+ assert_eq!(
+ observations_for(&conn, f.consolidated),
+ 0,
+ "a consolidated session's old raw capture must be pruned"
+ );
+ assert_eq!(
+ observations_for(&conn, f.unconsolidated),
+ 3,
+ "a session never consolidated into a page must keep every row"
+ );
+ assert_eq!(
+ observations_for(&conn, f.evicted),
+ 3,
+ "raw capture whose summary page decay already evicted is the last copy"
+ );
+ assert_eq!(
+ conn.query_row(
+ "SELECT detail FROM audit_log WHERE op = 'prune_observations'",
+ [],
+ |r| r.get::<_, String>(0)
+ )
+ .expect("exactly one audit row for the single deleting batch"),
+ "{\"deleted\":3}",
+ "the audit row must record how many rows the batch deleted"
+ );
+}
+
+/// `observations_fts` is `content='observations'`, so a stale index would keep
+/// returning hits for rows that no longer exist. The `observations_fts_ad`
+/// trigger is supposed to prevent that; assert it rather than trust it.
+#[tokio::test]
+async fn the_fts_index_stops_matching_pruned_bodies() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("store");
+ let f = seed(&store).await;
+ let conn = aux(&store);
+
+ assert_eq!(fts_matches(&conn, "consolidated"), 3);
+ assert_eq!(fts_matches(&conn, "zebrafinch"), 9);
+
+ run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &DecayParams::default(),
+ 0.0,
+ enabled(),
+ false,
+ )
+ .await
+ .expect("sweep");
+
+ assert_eq!(
+ fts_matches(&conn, "consolidated"),
+ 0,
+ "the FTS index must no longer match a pruned body"
+ );
+ assert_eq!(
+ fts_matches(&conn, "zebrafinch"),
+ 6,
+ "only the pruned rows may leave the index"
+ );
+ // The drift check behind `fts_drift_status` compares these two directly.
+ let rows: i64 = conn
+ .query_row("SELECT COUNT(*) FROM observations", [], |r| r.get(0))
+ .unwrap();
+ let docsize: i64 = conn
+ .query_row("SELECT COUNT(*) FROM observations_fts_docsize", [], |r| {
+ r.get(0)
+ })
+ .unwrap();
+ assert_eq!(rows, docsize, "FTS docsize must track the delete exactly");
+}
+
+/// A dry run must be able to answer "how much would this delete?" without
+/// opening a write transaction. Counting is the whole point; deleting is not.
+#[tokio::test]
+async fn a_dry_run_counts_without_deleting() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("store");
+ let f = seed(&store).await;
+
+ let report = run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &DecayParams::default(),
+ 0.0,
+ enabled(),
+ true,
+ )
+ .await
+ .expect("sweep");
+
+ assert_eq!(report.observations_prunable, 3);
+ assert_eq!(report.observations_pruned, 0);
+ assert_eq!(report.observation_prune_batches, 0);
+ let conn = aux(&store);
+ assert_eq!(observations_for(&conn, f.consolidated), 3);
+ assert_eq!(
+ conn.query_row(
+ "SELECT COUNT(*) FROM audit_log WHERE op = 'prune_observations'",
+ [],
+ |r| r.get::<_, i64>(0)
+ )
+ .unwrap(),
+ 0,
+ "a dry run must not write an audit row either"
+ );
+}
+
+/// The bound that keeps a multi-million row delete off the write lock: one
+/// transaction per batch, looping until a short batch. With `batch = 1` the
+/// three eligible rows must cost four batches (three full, one short) and still
+/// land completely.
+#[tokio::test]
+async fn the_prune_deletes_in_bounded_batches() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("store");
+ let f = seed(&store).await;
+
+ let report = run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &DecayParams::default(),
+ 0.0,
+ ObservationRetention {
+ days: RETENTION_DAYS,
+ batch: 1,
+ },
+ false,
+ )
+ .await
+ .expect("sweep");
+
+ assert_eq!(report.observations_pruned, 3);
+ assert_eq!(
+ report.observation_prune_batches, 4,
+ "each row is its own transaction, plus the short batch that ends the loop"
+ );
+ assert_eq!(
+ report.observation_prune_sessions, 1,
+ "a session split across batches must be counted once"
+ );
+ let conn = aux(&store);
+ assert_eq!(
+ conn.query_row(
+ "SELECT COUNT(*) FROM audit_log WHERE op = 'prune_observations'",
+ [],
+ |r| r.get::<_, i64>(0)
+ )
+ .unwrap(),
+ 3,
+ "one audit row per batch that actually deleted, none for the empty one"
+ );
+}
+
+/// `session_end_disposition` reads `sessions.ended_observation_count` against a
+/// live `COUNT(*)`. Prune without repairing that watermark and a resumed
+/// session's genuinely new work reads as `AlreadyEnded` forever.
+#[tokio::test]
+async fn the_session_end_watermark_follows_the_prune() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("store");
+ let f = seed(&store).await;
+ let conn = aux(&store);
+
+ let watermark = |conn: &Connection, s: SessionId| -> i64 {
+ conn.query_row(
+ "SELECT ended_observation_count FROM sessions WHERE id = ?1",
+ params![s.as_bytes()],
+ |row| row.get(0),
+ )
+ .expect("watermark")
+ };
+ assert_eq!(watermark(&conn, f.consolidated), 3);
+
+ run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &DecayParams::default(),
+ 0.0,
+ enabled(),
+ false,
+ )
+ .await
+ .expect("sweep");
+
+ assert_eq!(
+ watermark(&conn, f.consolidated),
+ 0,
+ "the watermark must come back down to the surviving row count"
+ );
+ // Idempotent and one-directional: a second sweep with nothing left to
+ // delete must not touch it again.
+ run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &DecayParams::default(),
+ 0.0,
+ enabled(),
+ false,
+ )
+ .await
+ .expect("second sweep");
+ assert_eq!(watermark(&conn, f.consolidated), 0);
+}
+
+/// Observations younger than the retention age are never eligible, however
+/// thoroughly their session was consolidated — the age gate is what keeps
+/// consolidation, auto-improve review and the `raw_hits` fallback whole.
+#[tokio::test]
+async fn recent_observations_survive_a_consolidated_session() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("store");
+ let f = seed(&store).await;
+ let conn = aux(&store);
+ let recent = jiff::Timestamp::now().as_microsecond() - US_PER_DAY;
+ conn.execute(
+ "UPDATE observations SET created_at = ?1 WHERE session_id = ?2",
+ params![recent, f.consolidated.as_bytes()],
+ )
+ .expect("freshen");
+
+ let report = run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &DecayParams::default(),
+ 0.0,
+ enabled(),
+ false,
+ )
+ .await
+ .expect("sweep");
+
+ assert_eq!(report.observations_pruned, 0);
+ assert_eq!(observations_for(&conn, f.consolidated), 3);
+}
+
+/// A negative age is a nonsensical cutoff and must stop the sweep before it
+/// mutates anything, exactly as an invalid breadth coefficient does.
+#[tokio::test]
+async fn a_negative_retention_age_stops_the_sweep() {
+ let tmp = TempDir::new().expect("tempdir");
+ let store = Store::open(tmp.path()).expect("store");
+ let f = seed(&store).await;
+
+ let error = run_sweep_with_options(
+ &store.reader,
+ &store.writer,
+ None,
+ f.ws,
+ f.proj,
+ &DecayParams::default(),
+ 0.0,
+ ObservationRetention {
+ days: -1,
+ batch: 5_000,
+ },
+ false,
+ )
+ .await
+ .expect_err("a negative retention age must fail closed");
+ assert!(error.to_string().contains("observation_retention_days"));
+}
diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rs
index 1c348de9..2f05f4a6 100644
--- a/crates/ai-memory-mcp/src/admin.rs
+++ b/crates/ai-memory-mcp/src/admin.rs
@@ -46,10 +46,10 @@ use std::pin::Pin;
use ai_memory_consolidate::{
AutoImproveReviewConfig, AutoImproveTelemetryParams, AutoImproveTelemetryReport, Bootstrap,
BootstrapConfig, BootstrapOutcome, BootstrapSource, CuratorParams, CuratorReport,
- EmbedBackfillCounts, EmbedBackfillOptions, SourceCounts, prune_sources_to_budget,
- render_auto_improve_telemetry_report_markdown, render_curator_report_markdown,
- run_auto_improve_review, run_auto_improve_telemetry_report, run_curator_report_with_breadth,
- run_embedding_backfill, run_lint, run_sweep_with_breadth,
+ EmbedBackfillCounts, EmbedBackfillOptions, ObservationRetention, SourceCounts,
+ prune_sources_to_budget, render_auto_improve_telemetry_report_markdown,
+ render_curator_report_markdown, run_auto_improve_review, run_auto_improve_telemetry_report,
+ run_curator_report_with_breadth, run_embedding_backfill, run_lint, run_sweep_with_options,
};
use ai_memory_core::{
ActiveProject, AgentKind, AutoImproveProposalId, Capability, DEFAULT_PROJECT_NAME,
@@ -81,8 +81,13 @@ use tracing::{info, warn};
const CONTRIBUTORS_WEBHOOK_NAME: &str = "contributors";
-#[derive(Clone, Copy)]
-struct DecayBreadthWeight(f64);
+/// Sweep knobs that live beside `DecayParams` rather than inside it, injected
+/// as one Extension so adding the next one is not a third router constructor.
+#[derive(Clone, Copy, Default)]
+struct SweepTuning {
+ breadth_weight: f64,
+ retention: ObservationRetention,
+}
/// Shared state for the admin router.
#[derive(Clone)]
@@ -554,6 +559,16 @@ pub fn admin_router(state: AdminState) -> Router {
/// Build the admin router with the optional distinct-reader retention weight.
pub fn admin_router_with_decay_breadth(state: AdminState, breadth_weight: f64) -> Router {
+ admin_router_with_sweep_tuning(state, breadth_weight, ObservationRetention::default())
+}
+
+/// Build the admin router with every sweep knob that lives outside
+/// `DecayParams`, including the opt-in observation prune (disabled by default).
+pub fn admin_router_with_sweep_tuning(
+ state: AdminState,
+ breadth_weight: f64,
+ retention: ObservationRetention,
+) -> Router {
let state = Arc::new(state);
let operational = Router::new()
.route("/admin/backup", post(handle_backup))
@@ -627,7 +642,10 @@ pub fn admin_router_with_decay_breadth(state: AdminState, breadth_weight: f64) -
require_root_for_multiuser_admin,
))
.with_state(state)
- .layer(axum::Extension(DecayBreadthWeight(breadth_weight)))
+ .layer(axum::Extension(SweepTuning {
+ breadth_weight,
+ retention,
+ }))
}
async fn require_root_for_multiuser_admin(
@@ -2110,7 +2128,7 @@ async fn handle_auto_improve_report(
async fn handle_curator(
State(state): State<Arc<AdminState>>,
- axum::Extension(breadth): axum::Extension<DecayBreadthWeight>,
+ axum::Extension(tuning): axum::Extension<SweepTuning>,
Json(req): Json<CuratorRequest>,
) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {
let mode = req.mode.as_deref().map(str::trim).filter(|s| !s.is_empty());
@@ -2141,7 +2159,7 @@ async fn handle_curator(
&req.workspace,
&req.project,
params.clone(),
- breadth.0,
+ tuning.breadth_weight,
)
.await
.map_err(|e| internal_err(e.to_string()))?;
@@ -2803,19 +2821,20 @@ struct ForgetSweepRequest {
async fn handle_forget_sweep(
State(state): State<Arc<AdminState>>,
- axum::Extension(breadth): axum::Extension<DecayBreadthWeight>,
+ axum::Extension(tuning): axum::Extension<SweepTuning>,
Json(req): Json<ForgetSweepRequest>,
) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {
let (ws, proj) = lookup_ws_proj_no_create(&state, &req.workspace, &req.project).await?;
- run_sweep_with_breadth(
+ run_sweep_with_options(
&state.reader,
&state.writer,
Some(&state.wiki),
ws,
proj,
&state.decay_params,
- breadth.0,
+ tuning.breadth_weight,
+ tuning.retention,
req.dry_run,
)
.await
diff --git a/crates/ai-memory-mcp/src/lib.rs b/crates/ai-memory-mcp/src/lib.rs
index 9176704e..24044bcd 100644
--- a/crates/ai-memory-mcp/src/lib.rs
+++ b/crates/ai-memory-mcp/src/lib.rs
@@ -16,5 +16,6 @@ mod server;
pub use actor::actor_from_headers;
pub use admin::{
AdminState, ScopeInvalidation, ScopeInvalidator, admin_router, admin_router_with_decay_breadth,
+ admin_router_with_sweep_tuning,
};
pub use server::{AiMemoryServer, MEMORY_INSTRUCTIONS};
diff --git a/crates/ai-memory-mcp/src/server.rs b/crates/ai-memory-mcp/src/server.rs
index 693b2c3f..7ddf0a89 100644
--- a/crates/ai-memory-mcp/src/server.rs
+++ b/crates/ai-memory-mcp/src/server.rs
@@ -6,8 +6,8 @@ use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use ai_memory_consolidate::{
- AutoImproveReviewConfig, Consolidator, projection::cap_text_with_marker,
- run_auto_improve_review, run_lint, run_sweep_with_breadth,
+ AutoImproveReviewConfig, Consolidator, ObservationRetention, projection::cap_text_with_marker,
+ run_auto_improve_review, run_lint, run_sweep_with_options,
};
use ai_memory_core::{
ActiveProject, AgentKind, FeedbackKind, HandoffId, HandoffState, NewHandoff, PageId, PagePath,
@@ -381,6 +381,9 @@ pub struct AiMemoryServer {
decay_params: DecayParams,
/// Optional distinct-reader reinforcement coefficient.
decay_breadth_weight: f64,
+ /// Opt-in bound on how long raw observations outlive their consolidation.
+ /// Default is disabled, so `memory_forget_sweep` deletes no raw capture.
+ observation_retention: ObservationRetention,
/// M9 embedder for hybrid query. When `None`, `memory_query`
/// still fuses FTS5 with entity matches and graph-neighbour expansion.
embedder: Option<Arc<dyn Embedder>>,
@@ -1258,6 +1261,7 @@ impl AiMemoryServer {
wiki: None,
decay_params: DecayParams::default(),
decay_breadth_weight: 0.0,
+ observation_retention: ObservationRetention::default(),
embedder: None,
reranker: None,
client_activity: Arc::new(std::sync::Mutex::new(ClientActivityBuffer::new())),
@@ -1770,6 +1774,14 @@ impl AiMemoryServer {
self
}
+ /// Set the opt-in observation prune bound used by `memory_forget_sweep`.
+ /// Unset, the sweep never deletes a raw observation.
+ #[must_use]
+ pub fn with_observation_retention(mut self, retention: ObservationRetention) -> Self {
+ self.observation_retention = retention;
+ self
+ }
+
/// Attach the wiki handle. Without this, `memory_forget_sweep`
/// and `memory_lint` cannot write their report pages.
#[must_use]
@@ -2375,7 +2387,7 @@ impl AiMemoryServer {
}
/// Run the M8 forget sweep over episodic pages.
- #[tool(description = "Run the retention sweep. THREE passes, and they \
+ #[tool(description = "Run the retention sweep. FOUR passes, and they \
differ on what they will delete. (1) TTL: pages whose frontmatter \
expires_at is in the past are hard-deleted (file + rows) REGARDLESS \
OF TIER OR PIN — an explicit expiry overrides a pin, so a pinned \
@@ -2386,8 +2398,15 @@ impl AiMemoryServer {
evicted to a tombstone; only this pass exempts semantic / procedural \
/ pinned pages. (3) Hard-delete: tombstones older than \
hard_delete_after_days and their supersession ancestry are removed \
- permanently. The report's expired / hard_deleted counts come from \
- passes 1 and 3. Pass dry_run=true to preview.")]
+ permanently. (4) Observation prune: raw observations older than \
+ decay.observation_retention_days are deleted permanently, in bounded \
+ batches, but ONLY for sessions already consolidated into a summary \
+ page that is still live — an unconsolidated session, or one whose \
+ page pass 2 or 3 just removed, keeps every row. This pass is DISABLED \
+ by default (observation_retention_days = 0) and deletes nothing until \
+ an operator opts in; when off, observations_pruned is 0. The report's \
+ expired / hard_deleted / observations_pruned counts come from passes \
+ 1, 3 and 4. Pass dry_run=true to preview.")]
async fn memory_forget_sweep(
&self,
Parameters(args): Parameters<SweepArgs>,
@@ -2405,7 +2424,7 @@ impl AiMemoryServer {
&aps_actor,
)
.await?;
- let report = run_sweep_with_breadth(
+ let report = run_sweep_with_options(
&self.reader,
&self.writer,
self.wiki.as_ref(),
@@ -2413,6 +2432,7 @@ impl AiMemoryServer {
proj,
&self.decay_params,
self.decay_breadth_weight,
+ self.observation_retention,
args.dry_run.unwrap_or(false),
)
.await
@@ -5953,7 +5973,7 @@ mod tests {
/// `memory_forget_sweep` is admin-gated and destructive, and the surface
/// an agent reads is the only place it can learn what the tool will
- /// delete. Both surfaces must name all three passes and must not claim a
+ /// delete. Both surfaces must name all four passes and must not claim a
/// blanket pin exemption.
///
/// The description previously said "Semantic / procedural / pinned pages
@@ -5997,6 +6017,28 @@ mod tests {
desc.contains("hard_delete_after_days"),
"description must name the hard-delete pass; got: {desc}"
);
+ // The fourth pass deletes raw capture, not pages, and can remove
+ // millions of rows on a real install. An agent that cannot read it
+ // here cannot know the tool touches observations at all — which is the
+ // exact regression this test was written to stop.
+ assert!(
+ desc.contains("observation_retention_days"),
+ "description must name the observation prune pass; got: {desc}"
+ );
+ assert!(
+ desc.contains("consolidated") && desc.contains("still live"),
+ "description must state that only consolidated sessions with a live \
+ page are pruned; got: {desc}"
+ );
+ assert!(
+ desc.contains("DISABLED"),
+ "description must state that observation pruning is off by default; \
+ got: {desc}"
+ );
+ assert!(
+ !desc.contains("THREE passes"),
+ "the three-pass wording must not come back now there are four; got: {desc}"
+ );
// A bare unqualified exemption sentence is exactly the regression
// this guards, so reject the old wording verbatim.
assert!(
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index 4792d07f..c1ee768f 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -46,8 +46,8 @@ pub use error::{StoreError, StoreResult};
pub use maintenance::MaintenanceJob;
pub use ops::{
AdmittedSession, DeleteWorkspaceSummary, EmbeddingWrite, HookSessionAdmission,
- IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary, PagesMode,
- PurgeSummary, ReorgSummary,
+ IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary,
+ ObservationPruneOutcome, PagesMode, PurgeSummary, ReorgSummary,
};
pub use reader::{
ActivityWindow, AgentSessionCount, AutoImproveCandidateSession, BriefPageBody, BriefingPage,
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index 41ef311c..9cb690f4 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -1957,6 +1957,130 @@ pub fn hard_delete_decayed_page_chain(
Ok(n)
}
+/// One batch of an observation prune, as reported by
+/// [`prune_consolidated_observations`].
+#[derive(Debug, Default, Clone, PartialEq, Eq)]
+pub struct ObservationPruneOutcome {
+ /// Observation rows permanently deleted by this batch.
+ pub deleted: usize,
+ /// Distinct sessions that lost rows in this batch. Returned rather than
+ /// counted so a caller looping over batches can deduplicate a session whose
+ /// rows straddle a batch boundary.
+ pub sessions_touched: Vec<SessionId>,
+ /// Sessions whose `ended_observation_count` watermark was lowered back onto
+ /// the surviving row count.
+ pub sessions_resynced: usize,
+}
+
+/// Permanently delete one bounded batch of raw observations belonging to
+/// sessions that were already distilled into a live summary page.
+///
+/// Raw capture is the store's growth driver, but it is also the only input
+/// consolidation, auto-improvement and the `raw_hits` fallback have. Deleting
+/// it is safe exactly when its durable distillation still exists, so the
+/// predicate joins through `sessions.summary_page_id` — the marker
+/// [`end_session_row`] writes — and requires the page it points at to still be
+/// live. `superseded_at IS NOT NULL` means decay already evicted that page, so
+/// its session's raw capture is the last copy and must stay; a page hard-deleted
+/// by the same sweep is excluded for free, because the `ON DELETE SET NULL`
+/// foreign key has already cleared the pointer. A session with no `sessions`
+/// row, or one that never ended, matches nothing and can never be pruned.
+///
+/// `ORDER BY created_at` makes the batching deterministic: a run interrupted
+/// halfway leaves a clean age boundary rather than holes, so the next run
+/// resumes on a store whose invariant ("everything older than X is gone for
+/// consolidated sessions") is still one comparison.
+///
+/// The watermark resync is not optional. `session_end_disposition` compares a
+/// live `COUNT(*)` against the persisted `sessions.ended_observation_count`;
+/// leaving the watermark above reality would make a resumed session's genuinely
+/// new work read as `AlreadyEnded` and never be re-consolidated. The
+/// `ended_observation_count > (…)` guard keeps the repair idempotent and
+/// one-directional — it can only lower a watermark that is now above reality.
+/// Bounding it to the ids the delete returned also repairs sessions whose row
+/// lives in a different scope than the observations being pruned.
+pub fn prune_consolidated_observations(
+ conn: &mut Connection,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ cutoff_us: i64,
+ batch: usize,
+) -> StoreResult<ObservationPruneOutcome> {
+ if batch == 0 {
+ return Ok(ObservationPruneOutcome::default());
+ }
+ let tx = conn.transaction()?;
+ let mut deleted = 0usize;
+ let mut sessions: BTreeSet<Vec<u8>> = BTreeSet::new();
+ {
+ let mut stmt = tx.prepare(
+ "DELETE FROM observations \
+ WHERE id IN ( \
+ SELECT o.id FROM observations o \
+ WHERE o.workspace_id = ?1 \
+ AND o.project_id = ?2 \
+ AND o.created_at < ?3 \
+ AND EXISTS ( \
+ SELECT 1 FROM sessions s \
+ JOIN pages p ON p.id = s.summary_page_id \
+ WHERE s.id = o.session_id \
+ AND p.superseded_at IS NULL \
+ ) \
+ ORDER BY o.created_at \
+ LIMIT ?4 \
+ ) \
+ RETURNING session_id",
+ )?;
+ let rows = stmt.query_map(
+ params![
+ workspace_id.as_bytes(),
+ project_id.as_bytes(),
+ cutoff_us,
+ u64::try_from(batch).unwrap_or(u64::MAX),
+ ],
+ |row| row.get::<_, Vec<u8>>(0),
+ )?;
+ for row in rows {
+ sessions.insert(row?);
+ deleted += 1;
+ }
+ }
+ let mut sessions_touched = Vec::with_capacity(sessions.len());
+ let mut sessions_resynced = 0usize;
+ if deleted > 0 {
+ for session_id in &sessions {
+ sessions_touched.push(SessionId::from_slice(session_id)?);
+ sessions_resynced += tx.execute(
+ "UPDATE sessions \
+ SET ended_observation_count = ( \
+ SELECT COUNT(*) FROM observations o WHERE o.session_id = sessions.id \
+ ) \
+ WHERE id = ?1 \
+ AND ended_observation_count > ( \
+ SELECT COUNT(*) FROM observations o WHERE o.session_id = sessions.id \
+ )",
+ params![session_id],
+ )?;
+ }
+ audit_with_detail(
+ &tx,
+ "prune_observations",
+ Some(workspace_id.as_bytes()),
+ Some(project_id.as_bytes()),
+ None,
+ None,
+ Timestamp::now().as_microsecond(),
+ &format!("{{\"deleted\":{deleted}}}"),
+ )?;
+ }
+ tx.commit()?;
+ Ok(ObservationPruneOutcome {
+ deleted,
+ sessions_touched,
+ sessions_resynced,
+ })
+}
+
/// Insert a new handoff in state=open.
pub fn insert_handoff(conn: &mut Connection, h: &NewHandoff) -> StoreResult<HandoffId> {
let tx = conn.transaction()?;
@@ -2458,10 +2582,36 @@ fn audit(
page_id: Option<&[u8; 16]>,
author_id: Option<&[u8; 16]>,
at: i64,
+) -> StoreResult<()> {
+ audit_with_detail(
+ tx,
+ op,
+ workspace_id,
+ project_id,
+ page_id,
+ author_id,
+ at,
+ "{}",
+ )
+}
+
+/// `audit`, but with a caller-supplied JSON `detail` payload. Exists so a
+/// destructive batch can record what it did (e.g. the observation prune's
+/// deleted-row count) instead of leaving a gap only inferable from timestamps.
+#[allow(clippy::too_many_arguments)]
+fn audit_with_detail(
+ tx: &rusqlite::Transaction<'_>,
+ op: &str,
+ workspace_id: Option<&[u8; 16]>,
+ project_id: Option<&[u8; 16]>,
+ page_id: Option<&[u8; 16]>,
+ author_id: Option<&[u8; 16]>,
+ at: i64,
+ detail: &str,
) -> StoreResult<()> {
tx.execute(
"INSERT INTO audit_log (at, op, workspace_id, project_id, page_id, author_id, detail) \
- VALUES (?1, ?2, ?3, ?4, ?5, ?6, '{}')",
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",
params![
at,
op,
@@ -2469,6 +2619,7 @@ fn audit(
project_id.map(|b| &b[..]),
page_id.map(|b| &b[..]),
author_id.map(|b| &b[..]),
+ detail,
],
)?;
Ok(())
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index 908e781b..eeb013b6 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -3205,6 +3205,42 @@ impl ReaderPool {
.await
}
+ /// Count the observations the prune pass would delete for this scope.
+ ///
+ /// Same predicate as
+ /// [`prune_consolidated_observations`](crate::ops::prune_consolidated_observations),
+ /// read-only, so a dry run can report a number without any chance of a
+ /// write. Kept beside the delete rather than derived from it because the
+ /// dry run must never open a write transaction at all.
+ ///
+ /// # Errors
+ /// Propagates SQL errors.
+ pub async fn prunable_observation_count(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ cutoff_us: i64,
+ ) -> StoreResult<usize> {
+ self.with_conn(move |conn| {
+ let n: i64 = conn.query_row(
+ "SELECT COUNT(*) FROM observations o \
+ WHERE o.workspace_id = ?1 \
+ AND o.project_id = ?2 \
+ AND o.created_at < ?3 \
+ AND EXISTS ( \
+ SELECT 1 FROM sessions s \
+ JOIN pages p ON p.id = s.summary_page_id \
+ WHERE s.id = o.session_id \
+ AND p.superseded_at IS NULL \
+ )",
+ params![workspace_id.as_bytes(), project_id.as_bytes(), cutoff_us],
+ |row| row.get(0),
+ )?;
+ Ok(usize::try_from(n).unwrap_or(0))
+ })
+ .await
+ }
+
/// Return the number of DISTINCT operators that reinforced each
/// `is_latest = 1` page of a project, for the sweep's breadth term.
///
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index a5ba035b..3d778629 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -26,8 +26,8 @@ use crate::auto_improve::{
use crate::error::{StoreError, StoreResult};
use crate::ops::{
self, AdmittedSession, DeleteWorkspaceSummary, EmbeddingWrite, HookSessionAdmission,
- IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary, PagesMode,
- PurgeSummary, ReorgSummary,
+ IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary, MoveSummary,
+ ObservationPruneOutcome, PagesMode, PurgeSummary, ReorgSummary,
};
use crate::session_consolidation::SessionConsolidationJob;
use crate::users::{self, TOKEN_HASH_LEN};
@@ -246,6 +246,13 @@ pub(crate) enum WriteCmd {
cutoff_us: i64,
reply: oneshot::Sender<StoreResult<usize>>,
},
+ PruneConsolidatedObservations {
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ cutoff_us: i64,
+ batch: usize,
+ reply: oneshot::Sender<StoreResult<ObservationPruneOutcome>>,
+ },
HealCatchAllRepoPaths {
home: Option<String>,
reply: oneshot::Sender<StoreResult<u64>>,
@@ -1136,6 +1143,35 @@ impl WriterHandle {
rx.await.map_err(|_| StoreError::WriterClosed)?
}
+ /// Delete one bounded batch of raw observations whose session was already
+ /// consolidated into a live summary page, repairing the session end
+ /// watermark in the same transaction.
+ ///
+ /// One batch is one transaction and one mailbox message, so a multi-million
+ /// row prune never holds the write lock across the whole run — every other
+ /// pending write interleaves between batches.
+ ///
+ /// # Errors
+ /// Returns [`StoreError::WriterClosed`] or propagates SQL errors.
+ pub async fn prune_consolidated_observations(
+ &self,
+ workspace_id: WorkspaceId,
+ project_id: ProjectId,
+ cutoff_us: i64,
+ batch: usize,
+ ) -> StoreResult<ObservationPruneOutcome> {
+ let (tx, rx) = oneshot::channel();
+ self.send(WriteCmd::PruneConsolidatedObservations {
+ workspace_id,
+ project_id,
+ cutoff_us,
+ batch,
+ reply: tx,
+ })
+ .await?;
+ rx.await.map_err(|_| StoreError::WriterClosed)?
+ }
+
/// NULL out catch-all project `repo_path` rows so existing installs
/// self-heal on upgrade: the broad sentinels (`$HOME` and the filesystem
/// root) plus any path that exists locally but is not a git work-tree
@@ -2097,6 +2133,22 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
);
send_or_warn(reply, result, "hard_delete_decayed_page_chain");
}
+ WriteCmd::PruneConsolidatedObservations {
+ workspace_id,
+ project_id,
+ cutoff_us,
+ batch,
+ reply,
+ } => {
+ let result = ops::prune_consolidated_observations(
+ &mut conn,
+ workspace_id,
+ project_id,
+ cutoff_us,
+ batch,
+ );
+ send_or_warn(reply, result, "prune_consolidated_observations");
+ }
WriteCmd::HealCatchAllRepoPaths { home, reply } => {
let result = ops::heal_catch_all_repo_paths(&mut conn, home.as_deref());
send_or_warn(reply, result, "heal_catch_all_repo_paths");
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index dc9164a1..cd997219 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -146,7 +146,20 @@ from hook paths.
version ancestry only within that sweep's resolved workspace/project,
together with entity-index rows orphaned by the purge. A newer page recreated
at the same path is preserved. Semantic / pinned / freshly-touched pages
- survive.
+ survive. A fourth pass, disabled unless `[decay] observation_retention_days`
+ is positive, then deletes raw `observations` older than that age — but only
+ for sessions already consolidated into a summary page that is still live, so
+ raw capture is never removed while it is the last copy of that session's
+ work. It runs last so this run's own evictions and hard-deletes already
+ exclude their sessions, deletes in `observation_prune_batch` transactions so
+ a multi-million row prune cannot hold the write lock, and repairs
+ `sessions.ended_observation_count` downward in the same transaction. The
+ prune is irreversible in a specific sense: observations are the input to
+ consolidation, so a pruned session can never be re-consolidated — not with
+ a better model, a better prompt, or a fixed consolidator bug — and its
+ summary page becomes the only surviving account of that session. Freed
+ SQLite pages are reused, not returned to the OS: the `.db` file does not
+ shrink, the backup tarball does.
Scheduled sweep, rule-based lint, and opt-in embedding backfill ticks
enumerate every existing workspace/project scope before doing per-project
work, matching the auto-improvement scheduler's store-wide scope model. A
@@ -513,6 +526,8 @@ mu = 0.04 # ↑ if recent hits should count more
cold_threshold = 0.20 # below this → remove file + retain tombstone
hard_delete_after_days = 180
breadth_weight = 0.0 # opt-in reward for distinct operators
+observation_retention_days = 0 # 0 = never prune raw observations
+observation_prune_batch = 5000 # rows per prune transaction
[slots] # optional shared-server injection boundary
per_user = false # shared + own slots in agent context