Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(compact): reclaim database space on demand, and report whether it is worth it (#549) (#556)
Fabio Akitab3dfab8ad709
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b0234f72..07390195 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -83,6 +83,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`VACUUM` alone does not clear them. A test pins this: removing the thirdrebuild leaves a purged agent transcript's text in the database file.+- `ai-memory compact --confirm` and `POST /admin/compact` reclaim free+ database pages on demand, deleting nothing, and `ai-memory status` now+ reports the figure that makes the decision possible: database size,+ reclaimable bytes, and the share of the file they represent (#549).++ Until now compaction was only reachable by *deleting* something — it existed+ solely as `--compact` on the destructive commands — so a store that had+ accumulated free pages through a retention sweep or months of superseded page+ versions had no way to return the space except by purging data worth keeping.+ There was also no figure anywhere saying whether a `VACUUM` would reclaim+ anything at all, which made scheduling one guesswork.++ `docs/deploy.md` documents the operational side, including why an+ unconditional nightly `VACUUM` is the wrong default: it takes an exclusive+ lock and rewrites the whole database, so every write blocks — and because+ SQLite reuses free pages, a store in steady use usually has almost nothing to+ reclaim, paying the full cost for no benefit. The recipe there is conditional+ on the reported figure and runs in off hours.++ `status` only advises compaction once the backlog is worth the stall (≥20% of+ the file *and* ≥64 MiB); the raw numbers are always reported.+### Fixed- A failed or skipped embed now leaves a durable record. `page_embeddings`stores only successes, and its upsert rewrites `created_at`, so a globaldiff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rsindex 88973a72..815903ac 100644--- a/crates/ai-memory-cli/src/cli.rs+++ b/crates/ai-memory-cli/src/cli.rs@@ -70,6 +70,14 @@ pub enum Command {Serve(ServeArgs),/// Wipe the data directory's wiki/, db/, raw/ contents.Reset(ResetArgs),+ /// Reclaim free database pages. Deletes nothing.+ ///+ /// Rebuilds the FTS indexes and VACUUMs, returning free pages to the+ /// filesystem. `ai-memory status` reports how much there is to reclaim —+ /// check it first: this takes an exclusive lock and rewrites the whole+ /// database, so every write blocks until it finishes and it needs free+ /// disk space of roughly the database's own size.+ Compact(CompactArgs),/// Snapshot wiki/, db/, and config.toml into a gzipped tarball.Backup(BackupArgs),/// Restore a backup tarball into the data directory.@@ -613,6 +621,15 @@ pub struct ReorgArgs {pub dry_run: bool,}+/// Arguments for `compact`.+#[derive(Debug, Args)]+pub struct CompactArgs {+ /// REQUIRED. Not because compaction destroys anything — it deletes+ /// nothing — but because it blocks every write for as long as it runs.+ #[arg(long)]+ pub confirm: bool,+}+/// Arguments for `purge-project`.#[derive(Debug, Args)]pub struct PurgeProjectArgs {diff --git a/crates/ai-memory-cli/src/commands/compact.rs b/crates/ai-memory-cli/src/commands/compact.rsnew file mode 100644index 00000000..68041f71--- /dev/null+++ b/crates/ai-memory-cli/src/commands/compact.rs@@ -0,0 +1,93 @@+//! `ai-memory compact` — thin HTTP client for on-demand space reclamation.++use anyhow::{Result, bail};+use serde::Serialize;++use crate::cli::CompactArgs;+use crate::config::Config;+use crate::http_client::{ServerEndpoint, post_json};++/// Request sent to `POST /admin/compact`.+#[derive(Serialize)]+struct CompactRequest {+ confirm: bool,+}++/// Run the `compact` subcommand.+///+/// Reclaims free pages without deleting anything. Requires `--confirm`+/// because it holds an exclusive lock for the duration, not because it is+/// destructive.+///+/// # Errors+/// Returns an error when `--confirm` is absent, the server is unreachable, or+/// the server returns a non-2xx response.+pub async fn run(config: &Config, args: CompactArgs) -> Result<()> {+ if !args.confirm {+ bail!(+ "compact rewrites the whole database under an exclusive lock: every \+ write blocks until it finishes, and it needs free disk space of \+ roughly the database's own size.\n\n\+ Check `ai-memory status` for the reclaimable figure first, then:\n\n \+ ai-memory compact --confirm"+ );+ }++ let endpoint = ServerEndpoint::from_config_resolving_auth(config).await;+ let report: serde_json::Value = post_json(+ &endpoint,+ "/admin/compact",+ &CompactRequest { confirm: true },+ )+ .await?;++ let n = |key: &str| report[key].as_u64().unwrap_or(0);+ let reclaimed = n("bytes_reclaimed");+ println!(+ "Compacted: {} → {} ({} reclaimed).",+ human_bytes(n("bytes_before")),+ human_bytes(n("bytes_after")),+ human_bytes(reclaimed),+ );+ if reclaimed == 0 {+ // Not a failure, and worth saying plainly: SQLite reuses free pages,+ // so a healthy store that is not carrying a large deletion has+ // genuinely nothing to give back.+ println!(+ "Nothing to reclaim — the database had no meaningful free-page \+ backlog. This is the normal result for a store that has not just \+ had a large deletion."+ );+ }+ Ok(())+}++/// Render a byte count for humans. Binary units, one decimal above KiB.+pub(crate) fn human_bytes(bytes: u64) -> String {+ const UNITS: [&str; 5] = ["B", "KiB", "MiB", "GiB", "TiB"];+ if bytes < 1024 {+ return format!("{bytes} B");+ }+ let mut value = bytes as f64;+ let mut unit = 0;+ while value >= 1024.0 && unit < UNITS.len() - 1 {+ value /= 1024.0;+ unit += 1;+ }+ format!("{value:.1} {}", UNITS[unit])+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn human_bytes_scales_and_keeps_exact_byte_counts_exact() {+ assert_eq!(human_bytes(0), "0 B");+ assert_eq!(human_bytes(1023), "1023 B");+ assert_eq!(human_bytes(1024), "1.0 KiB");+ assert_eq!(human_bytes(1536), "1.5 KiB");+ assert_eq!(human_bytes(1024 * 1024), "1.0 MiB");+ assert_eq!(human_bytes(3 * 1024 * 1024 * 1024), "3.0 GiB");+ }+}diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rsindex b3ec6416..6b7fcd7d 100644--- a/crates/ai-memory-cli/src/commands/mod.rs+++ b/crates/ai-memory-cli/src/commands/mod.rs@@ -13,6 +13,7 @@ pub mod backup;pub mod bootstrap;pub mod checkpoints;pub mod commit;+pub mod compact;pub mod completions;pub mod continue_session;pub mod curator;diff --git a/crates/ai-memory-cli/src/commands/status.rs b/crates/ai-memory-cli/src/commands/status.rsindex 7cb29220..6cec019c 100644--- a/crates/ai-memory-cli/src/commands/status.rs+++ b/crates/ai-memory-cli/src/commands/status.rs@@ -41,6 +41,9 @@ struct Report {/// Derived-index diagnostics.#[serde(default)]derived: Derived,+ /// Physical storage figures (absent from pre-#549 servers).+ #[serde(default)]+ storage: Storage,/// Hook-ingestion counters from the server process.#[serde(default)]ingest: Option<IngestReport>,@@ -75,6 +78,24 @@ struct Derived {stale_links_from_latest_pages: u64,}+/// Suggest compaction only above this share of the file. Below it the+/// exclusive lock costs more than the space is worth, and SQLite will reuse+/// those pages on its own as the store grows.+const RECLAIM_ADVICE_PCT: f64 = 20.0;++/// …and only when the absolute figure is worth a stall. 20% of a 4 MiB+/// database is not a reason to block every write.+const RECLAIM_ADVICE_MIN_BYTES: u64 = 64 * 1024 * 1024;++#[derive(Debug, Default, Deserialize, Serialize)]+struct Storage {+ page_size: u64,+ page_count: u64,+ freelist_count: u64,+ database_bytes: u64,+ reclaimable_bytes: u64,+}+#[derive(Debug, Deserialize, Serialize)]struct EmbeddingTriple {provider: String,@@ -178,6 +199,7 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {"observations": report.counts.observations,},"derived": report.derived,+ "storage": report.storage,"providers": report.providers,"spool": spool,"capture_mode": capture_mode,@@ -216,6 +238,27 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {report.derived.embed_failures_unresolved, report.derived.embed_failures_recovered);}+ // The figure that makes a `compact` decision possible. Reported+ // always, so "should I VACUUM?" has an answer other than a guess; the+ // advisory line only appears once it is worth the exclusive lock.+ if report.storage.database_bytes > 0 {+ let pct = (report.storage.reclaimable_bytes as f64+ / report.storage.database_bytes as f64)+ * 100.0;+ println!(+ " storage: {} on disk, {} reclaimable ({pct:.1}%)",+ super::compact::human_bytes(report.storage.database_bytes),+ super::compact::human_bytes(report.storage.reclaimable_bytes),+ );+ if pct >= RECLAIM_ADVICE_PCT+ && report.storage.reclaimable_bytes >= RECLAIM_ADVICE_MIN_BYTES+ {+ println!(+ " `ai-memory compact --confirm` would return it \+ (blocks writes while it runs)"+ );+ }+ }println!(" links: {} latest-page links (unresolved: {}, stale: {})",report.derived.links_from_latest_pages,diff --git a/crates/ai-memory-cli/src/main.rs b/crates/ai-memory-cli/src/main.rsindex b48d8756..cb7a0674 100644--- a/crates/ai-memory-cli/src/main.rs+++ b/crates/ai-memory-cli/src/main.rs@@ -107,6 +107,7 @@ async fn main() -> Result<()> {Command::DeletePage(args) => commands::delete_page::run(&config, args).await,Command::Serve(args) => commands::serve::run(&config, args).await,Command::Reset(args) => commands::reset::run(&config, args),+ Command::Compact(args) => commands::compact::run(&config, args).await,Command::Backup(args) => commands::backup::run(&config, args).await,Command::Restore(args) => commands::restore::run(&config, args),Command::Reindex(args) => commands::reindex::run(&config, args).await,diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rsindex 05e27f59..b0391e50 100644--- a/crates/ai-memory-mcp/src/admin.rs+++ b/crates/ai-memory-mcp/src/admin.rs@@ -26,6 +26,7 @@//! - `POST /admin/purge-session` — delete one session and what it derived.//! - `POST /admin/rename-project` — rename a project (column-only; no files move).//! - `POST /admin/rename-workspace` — rename a workspace and refresh scope manifests.+//! - `POST /admin/compact` — reclaim free pages; deletes nothing.//! - `POST /admin/delete-workspace` — delete a workspace and its projects//! (logical unless `compact` is set; see `ai_memory_store::Compaction`).//! - `POST /admin/merge-workspace` — fold every project of one workspace into@@ -628,6 +629,7 @@ pub fn admin_router_with_sweep_tuning(.route("/admin/move-project", post(handle_move_project)).route("/admin/move-session", post(handle_move_session)).route("/admin/delete-workspace", post(handle_delete_workspace))+ .route("/admin/compact", post(handle_compact)).route("/admin/rename-workspace", post(handle_rename_workspace)).route("/admin/merge-workspace", post(handle_merge_workspace)).route("/admin/write-page", post(handle_write_page))@@ -923,6 +925,9 @@ pub struct StatusReport {pub counts: ai_memory_store::StatusCounts,/// Derived-index and retrieval-readiness diagnostics.pub derived: ai_memory_store::DerivedIndexStatus,+ /// Physical storage figures — file size and reclaimable free pages — so an+ /// operator can decide whether a `VACUUM` is worth its exclusive lock.+ pub storage: ai_memory_store::StorageStatus,/// Passive process-scoped provider health.pub providers: ProviderHealthSnapshot,/// Hook-ingestion counters for this server process. Counts and one@@ -952,6 +957,9 @@ async fn handle_status(State(state): State<Arc<AdminState>>) -> impl IntoResponsmatch state.reader.status_counts().await {Ok(counts) => match state.reader.derived_index_status().await {Ok(derived) => {+ // Three pragma reads; a failure here must not take down the+ // whole status response, which is also the health probe.+ let storage = state.reader.storage_status().await.unwrap_or_default();let report = StatusReport {version: env!("CARGO_PKG_VERSION").to_string(),data_dir: state.data_dir.display().to_string(),@@ -959,6 +967,7 @@ async fn handle_status(State(state): State<Arc<AdminState>>) -> impl IntoResponsdb_path: state.db_path.display().to_string(),counts,derived,+ storage,providers: state.provider_health.snapshot(),ingest: state.ingest_metrics.snapshot(),};@@ -3845,6 +3854,62 @@ pub struct DeleteWorkspaceResult {/// `POST /admin/delete-workspace` — remove a workspace and, via cascade, every/// project/page under it. Guarded: refuses a non-empty workspace unless/// `force` (a typo shouldn't wipe live data). Orphan-workspace cleanup.+/// JSON request body for `POST /admin/compact`.+#[derive(Deserialize)]+struct CompactRequest {+ /// Mandatory confirmation. Not because compaction destroys anything — it+ /// deletes nothing — but because it takes an exclusive lock and rewrites+ /// the whole database, so every write blocks until it finishes. That is an+ /// availability decision, and it should be deliberate.+ confirm: bool,+}++/// Wire-format summary returned by `POST /admin/compact`.+#[derive(Debug, Serialize)]+pub struct CompactReport {+ /// Database size in bytes before the rebuild + `VACUUM`.+ pub bytes_before: u64,+ /// Database size in bytes afterwards.+ pub bytes_after: u64,+ /// Bytes returned to the filesystem (saturating at zero).+ pub bytes_reclaimed: u64,+}++/// `POST /admin/compact` — rebuild the FTS indexes and `VACUUM`, deleting+/// nothing. The on-demand form of what the destructive commands offer as+/// `--compact`.+async fn handle_compact(+ State(state): State<Arc<AdminState>>,+ Json(req): Json<CompactRequest>,+) -> impl IntoResponse {+ if !req.confirm {+ return (+ StatusCode::BAD_REQUEST,+ Json(serde_json::json!({+ "error": "compact rewrites the whole database under an exclusive \+ lock; pass confirm: true to proceed"+ })),+ );+ }+ match state.writer.compact().await {+ Ok(summary) => {+ let report = CompactReport {+ bytes_before: summary.bytes_before,+ bytes_after: summary.bytes_after,+ bytes_reclaimed: summary.bytes_reclaimed(),+ };+ (+ StatusCode::OK,+ Json(serde_json::to_value(&report).unwrap_or_else(|_| serde_json::json!({}))),+ )+ }+ Err(e) => (+ StatusCode::INTERNAL_SERVER_ERROR,+ Json(serde_json::json!({ "error": e.to_string() })),+ ),+ }+}+async fn handle_delete_workspace(State(state): State<Arc<AdminState>>,actor_ext: Option<axum::Extension<ai_memory_core::ActorContext>>,diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex 43a61c3e..de4d00a9 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -45,10 +45,10 @@ pub use decay::{pub use error::{StoreError, StoreResult};pub use maintenance::MaintenanceJob;pub use ops::{- AdmittedSession, Compaction, DeleteWorkspaceSummary, EmbedOutcome, EmbeddingWrite,- HookSessionAdmission, IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary,- MoveSummary, ObservationPruneOutcome, PagesMode, PurgeSessionSummary, PurgeSummary,- ReorgSummary, purge_session, record_embed_failure,+ AdmittedSession, CompactSummary, Compaction, DeleteWorkspaceSummary, EmbedOutcome,+ EmbeddingWrite, HookSessionAdmission, IngestObservationOutcome, LifecycleOnlyEndOutcome,+ MoveSessionSummary, MoveSummary, ObservationPruneOutcome, PagesMode, PurgeSessionSummary,+ PurgeSummary, ReorgSummary, purge_session, record_embed_failure,};pub use reader::{ActivityWindow, AgentSessionCount, AuditEvent, AuditLogFilter, AutoImproveCandidateSession,@@ -58,8 +58,8 @@ pub use reader::{ObservationOrder, ObservationPage, ObservationPageResult, ObservationRecord, OpenSession,PageAuthor, PageHit, PageHitWithMeta, PageLinks, PageMeta, PageSummary, ProjectSummary,ReaderPool, ReindexTargetStatus, RelatedPage, RrfContributions, ScopeRow, SearchExplain,- SessionDependentRows, SessionEndDisposition, SessionSummary, StatusCounts, StoredEmbedding,- StoredPageBody, WorkspaceScopeRow, WorkspaceSummary, f32_vec_to_bytes,+ SessionDependentRows, SessionEndDisposition, SessionSummary, StatusCounts, StorageStatus,+ StoredEmbedding, StoredPageBody, WorkspaceScopeRow, WorkspaceSummary, f32_vec_to_bytes,};pub use scope::{ResolvedScope, ScopeName, ScopeResolutionError, ScopeResolver, WORKSPACE_PROJECT_PAIR_REQUIRED,diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex 0ba4ee09..9d947496 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -3016,7 +3016,7 @@ const ALL_FTS_INDEXES: &[&str] = &["observations_fts", "pages_fts", "workstream_/// searchable after a purge, and the cost is dominated by the `VACUUM` that/// follows in every case. A `rebuild` is idempotent, so the extra work is a/// no-op for a scope that deleted nothing from that table.-fn reclaim_freed_pages(conn: &Connection) -> StoreResult<()> {+pub(crate) fn reclaim_freed_pages(conn: &Connection) -> StoreResult<()> {let mut batch = String::new();for index in ALL_FTS_INDEXES {// Compile-time constants, never caller input; FTS5 has no@@ -3028,6 +3028,62 @@ fn reclaim_freed_pages(conn: &Connection) -> StoreResult<()> {Ok(())}+/// What one [`compact`] run reclaimed.+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]+pub struct CompactSummary {+ /// Database size in bytes before the rebuild + `VACUUM`.+ pub bytes_before: u64,+ /// Database size in bytes afterwards.+ pub bytes_after: u64,+}++impl CompactSummary {+ /// Bytes returned to the filesystem. Saturating: a `VACUUM` can leave the+ /// file marginally larger when defragmentation costs more than the free+ /// pages it released, and that is a zero, not a negative.+ #[must_use]+ pub fn bytes_reclaimed(&self) -> u64 {+ self.bytes_before.saturating_sub(self.bytes_after)+ }+}++/// Reclaim free pages on demand, deleting nothing.+///+/// Same operation the destructive commands offer as `--compact`, reachable+/// without first deleting something. Until this existed, an operator whose+/// database had accumulated free pages — a large forget-sweep, a retention+/// pass, months of superseded page versions — had no way to return the space+/// except by purging data they wanted to keep.+///+/// # Cost+///+/// `VACUUM` takes an exclusive lock and rewrites the whole file, so every+/// write blocks for the duration and it needs free disk space of roughly the+/// database's own size. Read [`crate::reader::StorageStatus::reclaimable_bytes`]+/// first and run this when it is worth the stall — not on a blind schedule.+///+/// # Errors+/// Propagates the SQL error from the rebuild or the `VACUUM`.+pub fn compact(conn: &mut Connection) -> StoreResult<CompactSummary> {+ let bytes_before = database_bytes(conn)?;+ reclaim_freed_pages(conn)?;+ let bytes_after = database_bytes(conn)?;+ Ok(CompactSummary {+ bytes_before,+ bytes_after,+ })+}++/// `page_count * page_size` — the database's own view of its size, which is+/// what `VACUUM` acts on. Deliberately not the filesystem size: a `-wal`+/// sidecar holds committed pages not yet checkpointed into the main file, so+/// `metadata().len()` would move for reasons compaction has nothing to do with.+pub(crate) fn database_bytes(conn: &Connection) -> StoreResult<u64> {+ let page_count: i64 = conn.query_row("PRAGMA page_count", [], |r| r.get(0))?;+ let page_size: i64 = conn.query_row("PRAGMA page_size", [], |r| r.get(0))?;+ Ok((page_count.max(0) as u64).saturating_mul(page_size.max(0) as u64))+}+/// What [`purge_session`] removed. All counts are rows actually deleted.#[derive(Debug, Default, Clone, PartialEq, Eq)]pub struct PurgeSessionSummary {@@ -4681,6 +4737,100 @@ pub(crate) mod tests {);}+ /// The premise of #549: after a large delete the space is *not* returned+ /// to the filesystem — SQLite keeps the pages on its freelist for reuse.+ /// `compact` is what gives them back, without deleting anything itself.+ ///+ /// Deliberately paired assertions rather than one: the first pins that+ /// there is a backlog to reclaim (otherwise the second could pass on an+ /// empty database and prove nothing), the second that compaction reclaims+ /// it.+ #[test]+ fn compact_returns_free_pages_that_a_delete_left_behind() {+ let (_tmp, mut conn, ws, proj) = fresh_db();++ // Enough rows that the freelist is unambiguous rather than noise.+ for i in 0..400 {+ upsert_page(+ &mut conn,+ &page(ws, proj, &format!("notes/{i}.md"), &"padding ".repeat(256)),+ )+ .unwrap();+ }+ conn.execute("DELETE FROM pages", []).unwrap();++ let before = database_bytes(&conn).unwrap();+ let freelist: i64 = conn+ .query_row("PRAGMA freelist_count", [], |r| r.get(0))+ .unwrap();+ assert!(+ freelist > 0,+ "the delete must leave a freelist, or this test proves nothing"+ );++ let summary = compact(&mut conn).unwrap();++ assert_eq!(summary.bytes_before, before);+ assert!(+ summary.bytes_after < summary.bytes_before,+ "compaction must shrink the file: {} -> {}",+ summary.bytes_before,+ summary.bytes_after+ );+ assert_eq!(+ summary.bytes_reclaimed(),+ summary.bytes_before - summary.bytes_after+ );+ let after_freelist: i64 = conn+ .query_row("PRAGMA freelist_count", [], |r| r.get(0))+ .unwrap();+ assert_eq!(+ after_freelist, 0,+ "the freelist is gone, not merely smaller"+ );+ }++ /// Compaction deletes nothing. Worth a test of its own: the operation is+ /// reachable from an admin route and shares its implementation with the+ /// destructive commands' `--compact`, so "it only reclaims" is a property+ /// someone could break without noticing.+ #[test]+ fn compact_preserves_every_row_and_keeps_the_index_searchable() {+ let (_tmp, mut conn, ws, proj) = fresh_db();+ seed_session(&mut conn, ws, proj, "survivor");+ let pages_before = count(&conn, "SELECT COUNT(*) FROM pages");+ let obs_before = count(&conn, "SELECT COUNT(*) FROM observations");++ compact(&mut conn).unwrap();++ assert_eq!(count(&conn, "SELECT COUNT(*) FROM pages"), pages_before);+ assert_eq!(+ count(&conn, "SELECT COUNT(*) FROM observations"),+ obs_before+ );+ assert_eq!(+ count(+ &conn,+ "SELECT COUNT(*) FROM observations_fts \+ WHERE observations_fts MATCH '\"obs-survivor\"'"+ ),+ 1,+ "the rebuilt index still finds the surviving session"+ );+ }++ /// A `VACUUM` can leave the file marginally larger when defragmentation+ /// costs more than the pages it released. That is zero reclaimed, not a+ /// wrapped-around u64.+ #[test]+ fn bytes_reclaimed_saturates_instead_of_underflowing() {+ let grew = CompactSummary {+ bytes_before: 1_000,+ bytes_after: 1_200,+ };+ assert_eq!(grew.bytes_reclaimed(), 0);+ }+/// #387, the reporter's reproduction: after purging one session by UUID,/// its rows are gone from every table that held them.#[test]diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rsindex 216a2ca7..469fd2ff 100644--- a/crates/ai-memory-store/src/reader.rs+++ b/crates/ai-memory-store/src/reader.rs@@ -967,6 +967,42 @@ pub struct DerivedIndexStatus {pub stale_links_from_latest_pages: u64,}+/// Physical storage figures, so an operator can decide whether a `VACUUM` is+/// worth its exclusive lock instead of scheduling one blindly.+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]+pub struct StorageStatus {+ /// SQLite page size in bytes.+ pub page_size: u64,+ /// Total pages in the database.+ pub page_count: u64,+ /// Pages on the freelist — allocated, holding no live data, reusable by+ /// SQLite without growing the file.+ pub freelist_count: u64,+ /// `page_count * page_size`. The database's own view of its size, which is+ /// what `VACUUM` acts on.+ pub database_bytes: u64,+ /// `freelist_count * page_size` — an *estimate* of what a `VACUUM` would+ /// return to the filesystem.+ ///+ /// An estimate in both directions: `VACUUM` also defragments, so it can+ /// release more than the freelist, and it rewrites page headers, so it can+ /// release slightly less. Treat it as the signal for "is this worth an+ /// exclusive lock", not as an exact figure.+ pub reclaimable_bytes: u64,+}++impl StorageStatus {+ /// Reclaimable share of the file, 0.0–100.0. Zero when the database is+ /// empty rather than a division by zero.+ #[must_use]+ pub fn reclaimable_pct(&self) -> f64 {+ if self.database_bytes == 0 {+ return 0.0;+ }+ (self.reclaimable_bytes as f64 / self.database_bytes as f64) * 100.0+ }+}+/// Count of embedding rows sharing one `(provider, model, dim)` triple.#[derive(Debug, Clone, Serialize)]pub struct EmbeddingTripleCount {@@ -6836,6 +6872,32 @@ impl ReaderPool {.await}+ /// Physical storage figures for the database file.+ ///+ /// Three `PRAGMA` reads, no table scan, so it is cheap enough to sit in+ /// `status` next to the row counts.+ ///+ /// # Errors+ /// Propagates the SQL error from the pragma reads.+ pub async fn storage_status(&self) -> StoreResult<StorageStatus> {+ self.with_conn(|conn| {+ let page_size: i64 = conn.query_row("PRAGMA page_size", [], |r| r.get(0))?;+ let page_count: i64 = conn.query_row("PRAGMA page_count", [], |r| r.get(0))?;+ let freelist_count: i64 = conn.query_row("PRAGMA freelist_count", [], |r| r.get(0))?;+ let page_size = u64::try_from(page_size.max(0)).unwrap_or(0);+ let page_count = u64::try_from(page_count.max(0)).unwrap_or(0);+ let freelist_count = u64::try_from(freelist_count.max(0)).unwrap_or(0);+ Ok(StorageStatus {+ page_size,+ page_count,+ freelist_count,+ database_bytes: page_count.saturating_mul(page_size),+ reclaimable_bytes: freelist_count.saturating_mul(page_size),+ })+ })+ .await+ }+/// Return health counters for derived indexes and link/embedding state.////// These checks are intentionally read-only and derived-index-safe: they@@ -8123,8 +8185,27 @@ fn open_read_only(path: &Path) -> StoreResult<Connection> {#[cfg(test)]mod tests {++ /// The percentage is what an operator actually reads, and it divides by a+ /// figure that is zero on a fresh database.+ #[test]+ fn reclaimable_pct_is_zero_on_an_empty_database_rather_than_nan() {+ let empty = StorageStatus::default();+ assert_eq!(empty.database_bytes, 0);+ assert_eq!(empty.reclaimable_pct(), 0.0);+ assert!(empty.reclaimable_pct().is_finite(), "never NaN or inf");++ let quarter = StorageStatus {+ page_size: 4096,+ page_count: 400,+ freelist_count: 100,+ database_bytes: 400 * 4096,+ reclaimable_bytes: 100 * 4096,+ };+ assert!((quarter.reclaimable_pct() - 25.0).abs() < f64::EPSILON);+ }use super::{- DESCRIPTOR_MAX_CHARS, entity_query_tokens, handoff_listing_sql, like_escape,+ DESCRIPTOR_MAX_CHARS, StorageStatus, entity_query_tokens, handoff_listing_sql, like_escape,page_descriptor, page_descriptor_expr,};use crate::Store;diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rsindex 433659dc..c6da36c9 100644--- a/crates/ai-memory-store/src/writer.rs+++ b/crates/ai-memory-store/src/writer.rs@@ -327,6 +327,11 @@ pub(crate) enum WriteCmd {compaction: crate::ops::Compaction,reply: oneshot::Sender<StoreResult<crate::ops::PurgeSessionSummary>>,},+ /// Reclaim free pages on demand: rebuild the FTS indexes and `VACUUM`,+ /// deleting nothing. See [`ops::compact`].+ Compact {+ reply: oneshot::Sender<StoreResult<crate::ops::CompactSummary>>,+ },/// Delete a workspace row (its `workspace_id` FKs cascade projects/pages//// sessions/…). Refused when non-empty unless `force`.DeleteWorkspace {@@ -1370,6 +1375,21 @@ impl WriterHandle {rx.await.map_err(|_| StoreError::WriterClosed)?}+ /// Reclaim free pages on demand, deleting nothing.+ ///+ /// Runs through the writer actor like every other exclusive operation, so+ /// it cannot overlap a write: `VACUUM` takes an exclusive lock and would+ /// otherwise contend with one.+ ///+ /// # Errors+ /// Returns [`StoreError::WriterClosed`] if the actor has shut down, or+ /// propagates the SQL error from the rebuild or `VACUUM`.+ pub async fn compact(&self) -> StoreResult<crate::ops::CompactSummary> {+ let (tx, rx) = oneshot::channel();+ self.send(WriteCmd::Compact { reply: tx }).await?;+ rx.await.map_err(|_| StoreError::WriterClosed)?+ }+/// Rename a workspace (column-only; the on-disk dir is UUID-keyed).////// # Errors@@ -2389,6 +2409,10 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {);send_or_warn(reply, result, "purge_session");}+ WriteCmd::Compact { reply } => {+ let result = ops::compact(&mut conn);+ send_or_warn(reply, result, "compact");+ }WriteCmd::DeleteWorkspace {workspace_id,force,diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 364f5daf..707dc362 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -457,6 +457,7 @@ install-skills reorg purge-projectrename-project move-project move-sessionuninstall auth usercompletions handoffs purge-session+compact```Run `ai-memory --help` for the full tree.diff --git a/docs/deploy.md b/docs/deploy.mdindex 53b6b94e..e90b7272 100644--- a/docs/deploy.md+++ b/docs/deploy.md@@ -210,6 +210,75 @@ scp "$SERVER:$DEPLOY_DIR/data/snapshot-$(date +%F).tar.gz" ./backups/The `ai-memory backup` command uses SQLite's online backup API sowrites during the snapshot are coherent.+## Reclaiming disk space++SQLite does not return deleted space to the filesystem. Deleted rows leave+their pages on a **freelist**, which SQLite reuses as the database grows again.+For a store in steady use that is the right behaviour and needs no attention.++`ai-memory status` reports the figure that decides it:++```+ storage: 2.4 GiB on disk, 610.0 MiB reclaimable (25.4%)+ `ai-memory compact --confirm` would return it (blocks writes while it runs)+```++The advisory line only appears once the backlog is worth acting on. When it+does:++```bash+ssh "$SERVER" "docker exec ai-memory /usr/local/bin/ai-memory compact --confirm"+```++Compaction deletes nothing — it rebuilds the FTS indexes and `VACUUM`s.++### Do not schedule an unconditional nightly VACUUM++The obvious move is a nightly cron entry. Resist it:++- `VACUUM` takes an **exclusive lock** and rewrites the entire database. Every+ write blocks for the duration, which on a large store is minutes — and this+ server's job is answering hook traffic, so blocking writes drops captures.+- It needs free disk space of roughly the database's own size, so the nightly+ job also sets a permanent floor on free space.+- Because SQLite reuses free pages, a store in steady use usually has almost+ nothing to reclaim. The nightly run pays the full cost for no benefit on most+ nights.++The case that genuinely leaves a large freelist is a **one-off deletion** — a+`purge-project`, a big retention sweep, a `forget-sweep` over months of+episodic pages. Those are events, not a schedule, and the destructive commands+already offer `--compact` inline for exactly that moment.++If you do want it automated, make it **conditional** on the figure above and+put it in off hours:++```bash+#!/usr/bin/env bash+# /usr/local/bin/ai-memory-compact-if-worthwhile+set -euo pipefail++RECLAIMABLE=$(ai-memory status --json | jq '.storage.reclaimable_bytes')+THRESHOLD=$((1024 * 1024 * 1024)) # 1 GiB — tune to your store++if [ "$RECLAIMABLE" -ge "$THRESHOLD" ]; then+ ai-memory compact --confirm+fi+```++Drive it from a systemd timer (`OnCalendar=Sun 04:00`, `Persistent=true`) or a+host cron entry. A weekly check that usually does nothing costs one cheap+`status` call; a nightly `VACUUM` that usually does nothing costs a write stall+every night.++### What compaction does not do++It is not erasure. The wiki git history keeps page content in its objects and+commit messages, and any backup taken earlier still holds everything. Compaction+returns bytes from the live SQLite file — worth doing on its own terms, and not+a guarantee that content is unrecoverable. See+[lifecycle-ops.md](lifecycle-ops.md) for the full boundary.+## Rolling back```bash