Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(compact): reclaim database space on demand, and report whether it is worth it (#549) (#556)

Fabio Akitab3dfab8ad709
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b0234f72..07390195 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -83,6 +83,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`VACUUM` alone does not clear them. A test pins this: removing the third
rebuild leaves a purged agent transcript's text in the database file.
+- `ai-memory compact --confirm` and `POST /admin/compact` reclaim free
+ database pages on demand, deleting nothing, and `ai-memory status` now
+ reports the figure that makes the decision possible: database size,
+ reclaimable bytes, and the share of the file they represent (#549).
+
+ Until now compaction was only reachable by *deleting* something — it existed
+ solely as `--compact` on the destructive commands — so a store that had
+ accumulated free pages through a retention sweep or months of superseded page
+ versions had no way to return the space except by purging data worth keeping.
+ There was also no figure anywhere saying whether a `VACUUM` would reclaim
+ anything at all, which made scheduling one guesswork.
+
+ `docs/deploy.md` documents the operational side, including why an
+ unconditional nightly `VACUUM` is the wrong default: it takes an exclusive
+ lock and rewrites the whole database, so every write blocks — and because
+ SQLite reuses free pages, a store in steady use usually has almost nothing to
+ reclaim, paying the full cost for no benefit. The recipe there is conditional
+ on the reported figure and runs in off hours.
+
+ `status` only advises compaction once the backlog is worth the stall (≥20% of
+ the file *and* ≥64 MiB); the raw numbers are always reported.
+
### Fixed
- A failed or skipped embed now leaves a durable record. `page_embeddings`
stores only successes, and its upsert rewrites `created_at`, so a global
diff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rs
index 88973a72..815903ac 100644
--- a/crates/ai-memory-cli/src/cli.rs
+++ b/crates/ai-memory-cli/src/cli.rs
@@ -70,6 +70,14 @@ pub enum Command {
Serve(ServeArgs),
/// Wipe the data directory's wiki/, db/, raw/ contents.
Reset(ResetArgs),
+ /// Reclaim free database pages. Deletes nothing.
+ ///
+ /// Rebuilds the FTS indexes and VACUUMs, returning free pages to the
+ /// filesystem. `ai-memory status` reports how much there is to reclaim —
+ /// check it first: this takes an exclusive lock and rewrites the whole
+ /// database, so every write blocks until it finishes and it needs free
+ /// disk space of roughly the database's own size.
+ Compact(CompactArgs),
/// Snapshot wiki/, db/, and config.toml into a gzipped tarball.
Backup(BackupArgs),
/// Restore a backup tarball into the data directory.
@@ -613,6 +621,15 @@ pub struct ReorgArgs {
pub dry_run: bool,
}
+/// Arguments for `compact`.
+#[derive(Debug, Args)]
+pub struct CompactArgs {
+ /// REQUIRED. Not because compaction destroys anything — it deletes
+ /// nothing — but because it blocks every write for as long as it runs.
+ #[arg(long)]
+ pub confirm: bool,
+}
+
/// Arguments for `purge-project`.
#[derive(Debug, Args)]
pub struct PurgeProjectArgs {
diff --git a/crates/ai-memory-cli/src/commands/compact.rs b/crates/ai-memory-cli/src/commands/compact.rs
new file mode 100644
index 00000000..68041f71
--- /dev/null
+++ b/crates/ai-memory-cli/src/commands/compact.rs
@@ -0,0 +1,93 @@
+//! `ai-memory compact` — thin HTTP client for on-demand space reclamation.
+
+use anyhow::{Result, bail};
+use serde::Serialize;
+
+use crate::cli::CompactArgs;
+use crate::config::Config;
+use crate::http_client::{ServerEndpoint, post_json};
+
+/// Request sent to `POST /admin/compact`.
+#[derive(Serialize)]
+struct CompactRequest {
+ confirm: bool,
+}
+
+/// Run the `compact` subcommand.
+///
+/// Reclaims free pages without deleting anything. Requires `--confirm`
+/// because it holds an exclusive lock for the duration, not because it is
+/// destructive.
+///
+/// # Errors
+/// Returns an error when `--confirm` is absent, the server is unreachable, or
+/// the server returns a non-2xx response.
+pub async fn run(config: &Config, args: CompactArgs) -> Result<()> {
+ if !args.confirm {
+ bail!(
+ "compact rewrites the whole database under an exclusive lock: every \
+ write blocks until it finishes, and it needs free disk space of \
+ roughly the database's own size.\n\n\
+ Check `ai-memory status` for the reclaimable figure first, then:\n\n \
+ ai-memory compact --confirm"
+ );
+ }
+
+ let endpoint = ServerEndpoint::from_config_resolving_auth(config).await;
+ let report: serde_json::Value = post_json(
+ &endpoint,
+ "/admin/compact",
+ &CompactRequest { confirm: true },
+ )
+ .await?;
+
+ let n = |key: &str| report[key].as_u64().unwrap_or(0);
+ let reclaimed = n("bytes_reclaimed");
+ println!(
+ "Compacted: {} → {} ({} reclaimed).",
+ human_bytes(n("bytes_before")),
+ human_bytes(n("bytes_after")),
+ human_bytes(reclaimed),
+ );
+ if reclaimed == 0 {
+ // Not a failure, and worth saying plainly: SQLite reuses free pages,
+ // so a healthy store that is not carrying a large deletion has
+ // genuinely nothing to give back.
+ println!(
+ "Nothing to reclaim — the database had no meaningful free-page \
+ backlog. This is the normal result for a store that has not just \
+ had a large deletion."
+ );
+ }
+ Ok(())
+}
+
+/// Render a byte count for humans. Binary units, one decimal above KiB.
+pub(crate) fn human_bytes(bytes: u64) -> String {
+ const UNITS: [&str; 5] = ["B", "KiB", "MiB", "GiB", "TiB"];
+ if bytes < 1024 {
+ return format!("{bytes} B");
+ }
+ let mut value = bytes as f64;
+ let mut unit = 0;
+ while value >= 1024.0 && unit < UNITS.len() - 1 {
+ value /= 1024.0;
+ unit += 1;
+ }
+ format!("{value:.1} {}", UNITS[unit])
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn human_bytes_scales_and_keeps_exact_byte_counts_exact() {
+ assert_eq!(human_bytes(0), "0 B");
+ assert_eq!(human_bytes(1023), "1023 B");
+ assert_eq!(human_bytes(1024), "1.0 KiB");
+ assert_eq!(human_bytes(1536), "1.5 KiB");
+ assert_eq!(human_bytes(1024 * 1024), "1.0 MiB");
+ assert_eq!(human_bytes(3 * 1024 * 1024 * 1024), "3.0 GiB");
+ }
+}
diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rs
index b3ec6416..6b7fcd7d 100644
--- a/crates/ai-memory-cli/src/commands/mod.rs
+++ b/crates/ai-memory-cli/src/commands/mod.rs
@@ -13,6 +13,7 @@ pub mod backup;
pub mod bootstrap;
pub mod checkpoints;
pub mod commit;
+pub mod compact;
pub mod completions;
pub mod continue_session;
pub mod curator;
diff --git a/crates/ai-memory-cli/src/commands/status.rs b/crates/ai-memory-cli/src/commands/status.rs
index 7cb29220..6cec019c 100644
--- a/crates/ai-memory-cli/src/commands/status.rs
+++ b/crates/ai-memory-cli/src/commands/status.rs
@@ -41,6 +41,9 @@ struct Report {
/// Derived-index diagnostics.
#[serde(default)]
derived: Derived,
+ /// Physical storage figures (absent from pre-#549 servers).
+ #[serde(default)]
+ storage: Storage,
/// Hook-ingestion counters from the server process.
#[serde(default)]
ingest: Option<IngestReport>,
@@ -75,6 +78,24 @@ struct Derived {
stale_links_from_latest_pages: u64,
}
+/// Suggest compaction only above this share of the file. Below it the
+/// exclusive lock costs more than the space is worth, and SQLite will reuse
+/// those pages on its own as the store grows.
+const RECLAIM_ADVICE_PCT: f64 = 20.0;
+
+/// …and only when the absolute figure is worth a stall. 20% of a 4 MiB
+/// database is not a reason to block every write.
+const RECLAIM_ADVICE_MIN_BYTES: u64 = 64 * 1024 * 1024;
+
+#[derive(Debug, Default, Deserialize, Serialize)]
+struct Storage {
+ page_size: u64,
+ page_count: u64,
+ freelist_count: u64,
+ database_bytes: u64,
+ reclaimable_bytes: u64,
+}
+
#[derive(Debug, Deserialize, Serialize)]
struct EmbeddingTriple {
provider: String,
@@ -178,6 +199,7 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {
"observations": report.counts.observations,
},
"derived": report.derived,
+ "storage": report.storage,
"providers": report.providers,
"spool": spool,
"capture_mode": capture_mode,
@@ -216,6 +238,27 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {
report.derived.embed_failures_unresolved, report.derived.embed_failures_recovered
);
}
+ // The figure that makes a `compact` decision possible. Reported
+ // always, so "should I VACUUM?" has an answer other than a guess; the
+ // advisory line only appears once it is worth the exclusive lock.
+ if report.storage.database_bytes > 0 {
+ let pct = (report.storage.reclaimable_bytes as f64
+ / report.storage.database_bytes as f64)
+ * 100.0;
+ println!(
+ " storage: {} on disk, {} reclaimable ({pct:.1}%)",
+ super::compact::human_bytes(report.storage.database_bytes),
+ super::compact::human_bytes(report.storage.reclaimable_bytes),
+ );
+ if pct >= RECLAIM_ADVICE_PCT
+ && report.storage.reclaimable_bytes >= RECLAIM_ADVICE_MIN_BYTES
+ {
+ println!(
+ " `ai-memory compact --confirm` would return it \
+ (blocks writes while it runs)"
+ );
+ }
+ }
println!(
" links: {} latest-page links (unresolved: {}, stale: {})",
report.derived.links_from_latest_pages,
diff --git a/crates/ai-memory-cli/src/main.rs b/crates/ai-memory-cli/src/main.rs
index b48d8756..cb7a0674 100644
--- a/crates/ai-memory-cli/src/main.rs
+++ b/crates/ai-memory-cli/src/main.rs
@@ -107,6 +107,7 @@ async fn main() -> Result<()> {
Command::DeletePage(args) => commands::delete_page::run(&config, args).await,
Command::Serve(args) => commands::serve::run(&config, args).await,
Command::Reset(args) => commands::reset::run(&config, args),
+ Command::Compact(args) => commands::compact::run(&config, args).await,
Command::Backup(args) => commands::backup::run(&config, args).await,
Command::Restore(args) => commands::restore::run(&config, args),
Command::Reindex(args) => commands::reindex::run(&config, args).await,
diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rs
index 05e27f59..b0391e50 100644
--- a/crates/ai-memory-mcp/src/admin.rs
+++ b/crates/ai-memory-mcp/src/admin.rs
@@ -26,6 +26,7 @@
//! - `POST /admin/purge-session` — delete one session and what it derived.
//! - `POST /admin/rename-project` — rename a project (column-only; no files move).
//! - `POST /admin/rename-workspace` — rename a workspace and refresh scope manifests.
+//! - `POST /admin/compact` — reclaim free pages; deletes nothing.
//! - `POST /admin/delete-workspace` — delete a workspace and its projects
//! (logical unless `compact` is set; see `ai_memory_store::Compaction`).
//! - `POST /admin/merge-workspace` — fold every project of one workspace into
@@ -628,6 +629,7 @@ pub fn admin_router_with_sweep_tuning(
.route("/admin/move-project", post(handle_move_project))
.route("/admin/move-session", post(handle_move_session))
.route("/admin/delete-workspace", post(handle_delete_workspace))
+ .route("/admin/compact", post(handle_compact))
.route("/admin/rename-workspace", post(handle_rename_workspace))
.route("/admin/merge-workspace", post(handle_merge_workspace))
.route("/admin/write-page", post(handle_write_page))
@@ -923,6 +925,9 @@ pub struct StatusReport {
pub counts: ai_memory_store::StatusCounts,
/// Derived-index and retrieval-readiness diagnostics.
pub derived: ai_memory_store::DerivedIndexStatus,
+ /// Physical storage figures — file size and reclaimable free pages — so an
+ /// operator can decide whether a `VACUUM` is worth its exclusive lock.
+ pub storage: ai_memory_store::StorageStatus,
/// Passive process-scoped provider health.
pub providers: ProviderHealthSnapshot,
/// Hook-ingestion counters for this server process. Counts and one
@@ -952,6 +957,9 @@ async fn handle_status(State(state): State<Arc<AdminState>>) -> impl IntoRespons
match state.reader.status_counts().await {
Ok(counts) => match state.reader.derived_index_status().await {
Ok(derived) => {
+ // Three pragma reads; a failure here must not take down the
+ // whole status response, which is also the health probe.
+ let storage = state.reader.storage_status().await.unwrap_or_default();
let report = StatusReport {
version: env!("CARGO_PKG_VERSION").to_string(),
data_dir: state.data_dir.display().to_string(),
@@ -959,6 +967,7 @@ async fn handle_status(State(state): State<Arc<AdminState>>) -> impl IntoRespons
db_path: state.db_path.display().to_string(),
counts,
derived,
+ storage,
providers: state.provider_health.snapshot(),
ingest: state.ingest_metrics.snapshot(),
};
@@ -3845,6 +3854,62 @@ pub struct DeleteWorkspaceResult {
/// `POST /admin/delete-workspace` — remove a workspace and, via cascade, every
/// project/page under it. Guarded: refuses a non-empty workspace unless
/// `force` (a typo shouldn't wipe live data). Orphan-workspace cleanup.
+/// JSON request body for `POST /admin/compact`.
+#[derive(Deserialize)]
+struct CompactRequest {
+ /// Mandatory confirmation. Not because compaction destroys anything — it
+ /// deletes nothing — but because it takes an exclusive lock and rewrites
+ /// the whole database, so every write blocks until it finishes. That is an
+ /// availability decision, and it should be deliberate.
+ confirm: bool,
+}
+
+/// Wire-format summary returned by `POST /admin/compact`.
+#[derive(Debug, Serialize)]
+pub struct CompactReport {
+ /// Database size in bytes before the rebuild + `VACUUM`.
+ pub bytes_before: u64,
+ /// Database size in bytes afterwards.
+ pub bytes_after: u64,
+ /// Bytes returned to the filesystem (saturating at zero).
+ pub bytes_reclaimed: u64,
+}
+
+/// `POST /admin/compact` — rebuild the FTS indexes and `VACUUM`, deleting
+/// nothing. The on-demand form of what the destructive commands offer as
+/// `--compact`.
+async fn handle_compact(
+ State(state): State<Arc<AdminState>>,
+ Json(req): Json<CompactRequest>,
+) -> impl IntoResponse {
+ if !req.confirm {
+ return (
+ StatusCode::BAD_REQUEST,
+ Json(serde_json::json!({
+ "error": "compact rewrites the whole database under an exclusive \
+ lock; pass confirm: true to proceed"
+ })),
+ );
+ }
+ match state.writer.compact().await {
+ Ok(summary) => {
+ let report = CompactReport {
+ bytes_before: summary.bytes_before,
+ bytes_after: summary.bytes_after,
+ bytes_reclaimed: summary.bytes_reclaimed(),
+ };
+ (
+ StatusCode::OK,
+ Json(serde_json::to_value(&report).unwrap_or_else(|_| serde_json::json!({}))),
+ )
+ }
+ Err(e) => (
+ StatusCode::INTERNAL_SERVER_ERROR,
+ Json(serde_json::json!({ "error": e.to_string() })),
+ ),
+ }
+}
+
async fn handle_delete_workspace(
State(state): State<Arc<AdminState>>,
actor_ext: Option<axum::Extension<ai_memory_core::ActorContext>>,
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index 43a61c3e..de4d00a9 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -45,10 +45,10 @@ pub use decay::{
pub use error::{StoreError, StoreResult};
pub use maintenance::MaintenanceJob;
pub use ops::{
- AdmittedSession, Compaction, DeleteWorkspaceSummary, EmbedOutcome, EmbeddingWrite,
- HookSessionAdmission, IngestObservationOutcome, LifecycleOnlyEndOutcome, MoveSessionSummary,
- MoveSummary, ObservationPruneOutcome, PagesMode, PurgeSessionSummary, PurgeSummary,
- ReorgSummary, purge_session, record_embed_failure,
+ AdmittedSession, CompactSummary, Compaction, DeleteWorkspaceSummary, EmbedOutcome,
+ EmbeddingWrite, HookSessionAdmission, IngestObservationOutcome, LifecycleOnlyEndOutcome,
+ MoveSessionSummary, MoveSummary, ObservationPruneOutcome, PagesMode, PurgeSessionSummary,
+ PurgeSummary, ReorgSummary, purge_session, record_embed_failure,
};
pub use reader::{
ActivityWindow, AgentSessionCount, AuditEvent, AuditLogFilter, AutoImproveCandidateSession,
@@ -58,8 +58,8 @@ pub use reader::{
ObservationOrder, ObservationPage, ObservationPageResult, ObservationRecord, OpenSession,
PageAuthor, PageHit, PageHitWithMeta, PageLinks, PageMeta, PageSummary, ProjectSummary,
ReaderPool, ReindexTargetStatus, RelatedPage, RrfContributions, ScopeRow, SearchExplain,
- SessionDependentRows, SessionEndDisposition, SessionSummary, StatusCounts, StoredEmbedding,
- StoredPageBody, WorkspaceScopeRow, WorkspaceSummary, f32_vec_to_bytes,
+ SessionDependentRows, SessionEndDisposition, SessionSummary, StatusCounts, StorageStatus,
+ StoredEmbedding, StoredPageBody, WorkspaceScopeRow, WorkspaceSummary, f32_vec_to_bytes,
};
pub use scope::{
ResolvedScope, ScopeName, ScopeResolutionError, ScopeResolver, WORKSPACE_PROJECT_PAIR_REQUIRED,
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index 0ba4ee09..9d947496 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -3016,7 +3016,7 @@ const ALL_FTS_INDEXES: &[&str] = &["observations_fts", "pages_fts", "workstream_
/// searchable after a purge, and the cost is dominated by the `VACUUM` that
/// follows in every case. A `rebuild` is idempotent, so the extra work is a
/// no-op for a scope that deleted nothing from that table.
-fn reclaim_freed_pages(conn: &Connection) -> StoreResult<()> {
+pub(crate) fn reclaim_freed_pages(conn: &Connection) -> StoreResult<()> {
let mut batch = String::new();
for index in ALL_FTS_INDEXES {
// Compile-time constants, never caller input; FTS5 has no
@@ -3028,6 +3028,62 @@ fn reclaim_freed_pages(conn: &Connection) -> StoreResult<()> {
Ok(())
}
+/// What one [`compact`] run reclaimed.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
+pub struct CompactSummary {
+ /// Database size in bytes before the rebuild + `VACUUM`.
+ pub bytes_before: u64,
+ /// Database size in bytes afterwards.
+ pub bytes_after: u64,
+}
+
+impl CompactSummary {
+ /// Bytes returned to the filesystem. Saturating: a `VACUUM` can leave the
+ /// file marginally larger when defragmentation costs more than the free
+ /// pages it released, and that is a zero, not a negative.
+ #[must_use]
+ pub fn bytes_reclaimed(&self) -> u64 {
+ self.bytes_before.saturating_sub(self.bytes_after)
+ }
+}
+
+/// Reclaim free pages on demand, deleting nothing.
+///
+/// Same operation the destructive commands offer as `--compact`, reachable
+/// without first deleting something. Until this existed, an operator whose
+/// database had accumulated free pages — a large forget-sweep, a retention
+/// pass, months of superseded page versions — had no way to return the space
+/// except by purging data they wanted to keep.
+///
+/// # Cost
+///
+/// `VACUUM` takes an exclusive lock and rewrites the whole file, so every
+/// write blocks for the duration and it needs free disk space of roughly the
+/// database's own size. Read [`crate::reader::StorageStatus::reclaimable_bytes`]
+/// first and run this when it is worth the stall — not on a blind schedule.
+///
+/// # Errors
+/// Propagates the SQL error from the rebuild or the `VACUUM`.
+pub fn compact(conn: &mut Connection) -> StoreResult<CompactSummary> {
+ let bytes_before = database_bytes(conn)?;
+ reclaim_freed_pages(conn)?;
+ let bytes_after = database_bytes(conn)?;
+ Ok(CompactSummary {
+ bytes_before,
+ bytes_after,
+ })
+}
+
+/// `page_count * page_size` — the database's own view of its size, which is
+/// what `VACUUM` acts on. Deliberately not the filesystem size: a `-wal`
+/// sidecar holds committed pages not yet checkpointed into the main file, so
+/// `metadata().len()` would move for reasons compaction has nothing to do with.
+pub(crate) fn database_bytes(conn: &Connection) -> StoreResult<u64> {
+ let page_count: i64 = conn.query_row("PRAGMA page_count", [], |r| r.get(0))?;
+ let page_size: i64 = conn.query_row("PRAGMA page_size", [], |r| r.get(0))?;
+ Ok((page_count.max(0) as u64).saturating_mul(page_size.max(0) as u64))
+}
+
/// What [`purge_session`] removed. All counts are rows actually deleted.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub struct PurgeSessionSummary {
@@ -4681,6 +4737,100 @@ pub(crate) mod tests {
);
}
+ /// The premise of #549: after a large delete the space is *not* returned
+ /// to the filesystem — SQLite keeps the pages on its freelist for reuse.
+ /// `compact` is what gives them back, without deleting anything itself.
+ ///
+ /// Deliberately paired assertions rather than one: the first pins that
+ /// there is a backlog to reclaim (otherwise the second could pass on an
+ /// empty database and prove nothing), the second that compaction reclaims
+ /// it.
+ #[test]
+ fn compact_returns_free_pages_that_a_delete_left_behind() {
+ let (_tmp, mut conn, ws, proj) = fresh_db();
+
+ // Enough rows that the freelist is unambiguous rather than noise.
+ for i in 0..400 {
+ upsert_page(
+ &mut conn,
+ &page(ws, proj, &format!("notes/{i}.md"), &"padding ".repeat(256)),
+ )
+ .unwrap();
+ }
+ conn.execute("DELETE FROM pages", []).unwrap();
+
+ let before = database_bytes(&conn).unwrap();
+ let freelist: i64 = conn
+ .query_row("PRAGMA freelist_count", [], |r| r.get(0))
+ .unwrap();
+ assert!(
+ freelist > 0,
+ "the delete must leave a freelist, or this test proves nothing"
+ );
+
+ let summary = compact(&mut conn).unwrap();
+
+ assert_eq!(summary.bytes_before, before);
+ assert!(
+ summary.bytes_after < summary.bytes_before,
+ "compaction must shrink the file: {} -> {}",
+ summary.bytes_before,
+ summary.bytes_after
+ );
+ assert_eq!(
+ summary.bytes_reclaimed(),
+ summary.bytes_before - summary.bytes_after
+ );
+ let after_freelist: i64 = conn
+ .query_row("PRAGMA freelist_count", [], |r| r.get(0))
+ .unwrap();
+ assert_eq!(
+ after_freelist, 0,
+ "the freelist is gone, not merely smaller"
+ );
+ }
+
+ /// Compaction deletes nothing. Worth a test of its own: the operation is
+ /// reachable from an admin route and shares its implementation with the
+ /// destructive commands' `--compact`, so "it only reclaims" is a property
+ /// someone could break without noticing.
+ #[test]
+ fn compact_preserves_every_row_and_keeps_the_index_searchable() {
+ let (_tmp, mut conn, ws, proj) = fresh_db();
+ seed_session(&mut conn, ws, proj, "survivor");
+ let pages_before = count(&conn, "SELECT COUNT(*) FROM pages");
+ let obs_before = count(&conn, "SELECT COUNT(*) FROM observations");
+
+ compact(&mut conn).unwrap();
+
+ assert_eq!(count(&conn, "SELECT COUNT(*) FROM pages"), pages_before);
+ assert_eq!(
+ count(&conn, "SELECT COUNT(*) FROM observations"),
+ obs_before
+ );
+ assert_eq!(
+ count(
+ &conn,
+ "SELECT COUNT(*) FROM observations_fts \
+ WHERE observations_fts MATCH '\"obs-survivor\"'"
+ ),
+ 1,
+ "the rebuilt index still finds the surviving session"
+ );
+ }
+
+ /// A `VACUUM` can leave the file marginally larger when defragmentation
+ /// costs more than the pages it released. That is zero reclaimed, not a
+ /// wrapped-around u64.
+ #[test]
+ fn bytes_reclaimed_saturates_instead_of_underflowing() {
+ let grew = CompactSummary {
+ bytes_before: 1_000,
+ bytes_after: 1_200,
+ };
+ assert_eq!(grew.bytes_reclaimed(), 0);
+ }
+
/// #387, the reporter's reproduction: after purging one session by UUID,
/// its rows are gone from every table that held them.
#[test]
diff --git a/crates/ai-memory-store/src/reader.rs b/crates/ai-memory-store/src/reader.rs
index 216a2ca7..469fd2ff 100644
--- a/crates/ai-memory-store/src/reader.rs
+++ b/crates/ai-memory-store/src/reader.rs
@@ -967,6 +967,42 @@ pub struct DerivedIndexStatus {
pub stale_links_from_latest_pages: u64,
}
+/// Physical storage figures, so an operator can decide whether a `VACUUM` is
+/// worth its exclusive lock instead of scheduling one blindly.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
+pub struct StorageStatus {
+ /// SQLite page size in bytes.
+ pub page_size: u64,
+ /// Total pages in the database.
+ pub page_count: u64,
+ /// Pages on the freelist — allocated, holding no live data, reusable by
+ /// SQLite without growing the file.
+ pub freelist_count: u64,
+ /// `page_count * page_size`. The database's own view of its size, which is
+ /// what `VACUUM` acts on.
+ pub database_bytes: u64,
+ /// `freelist_count * page_size` — an *estimate* of what a `VACUUM` would
+ /// return to the filesystem.
+ ///
+ /// An estimate in both directions: `VACUUM` also defragments, so it can
+ /// release more than the freelist, and it rewrites page headers, so it can
+ /// release slightly less. Treat it as the signal for "is this worth an
+ /// exclusive lock", not as an exact figure.
+ pub reclaimable_bytes: u64,
+}
+
+impl StorageStatus {
+ /// Reclaimable share of the file, 0.0–100.0. Zero when the database is
+ /// empty rather than a division by zero.
+ #[must_use]
+ pub fn reclaimable_pct(&self) -> f64 {
+ if self.database_bytes == 0 {
+ return 0.0;
+ }
+ (self.reclaimable_bytes as f64 / self.database_bytes as f64) * 100.0
+ }
+}
+
/// Count of embedding rows sharing one `(provider, model, dim)` triple.
#[derive(Debug, Clone, Serialize)]
pub struct EmbeddingTripleCount {
@@ -6836,6 +6872,32 @@ impl ReaderPool {
.await
}
+ /// Physical storage figures for the database file.
+ ///
+ /// Three `PRAGMA` reads, no table scan, so it is cheap enough to sit in
+ /// `status` next to the row counts.
+ ///
+ /// # Errors
+ /// Propagates the SQL error from the pragma reads.
+ pub async fn storage_status(&self) -> StoreResult<StorageStatus> {
+ self.with_conn(|conn| {
+ let page_size: i64 = conn.query_row("PRAGMA page_size", [], |r| r.get(0))?;
+ let page_count: i64 = conn.query_row("PRAGMA page_count", [], |r| r.get(0))?;
+ let freelist_count: i64 = conn.query_row("PRAGMA freelist_count", [], |r| r.get(0))?;
+ let page_size = u64::try_from(page_size.max(0)).unwrap_or(0);
+ let page_count = u64::try_from(page_count.max(0)).unwrap_or(0);
+ let freelist_count = u64::try_from(freelist_count.max(0)).unwrap_or(0);
+ Ok(StorageStatus {
+ page_size,
+ page_count,
+ freelist_count,
+ database_bytes: page_count.saturating_mul(page_size),
+ reclaimable_bytes: freelist_count.saturating_mul(page_size),
+ })
+ })
+ .await
+ }
+
/// Return health counters for derived indexes and link/embedding state.
///
/// These checks are intentionally read-only and derived-index-safe: they
@@ -8123,8 +8185,27 @@ fn open_read_only(path: &Path) -> StoreResult<Connection> {
#[cfg(test)]
mod tests {
+
+ /// The percentage is what an operator actually reads, and it divides by a
+ /// figure that is zero on a fresh database.
+ #[test]
+ fn reclaimable_pct_is_zero_on_an_empty_database_rather_than_nan() {
+ let empty = StorageStatus::default();
+ assert_eq!(empty.database_bytes, 0);
+ assert_eq!(empty.reclaimable_pct(), 0.0);
+ assert!(empty.reclaimable_pct().is_finite(), "never NaN or inf");
+
+ let quarter = StorageStatus {
+ page_size: 4096,
+ page_count: 400,
+ freelist_count: 100,
+ database_bytes: 400 * 4096,
+ reclaimable_bytes: 100 * 4096,
+ };
+ assert!((quarter.reclaimable_pct() - 25.0).abs() < f64::EPSILON);
+ }
use super::{
- DESCRIPTOR_MAX_CHARS, entity_query_tokens, handoff_listing_sql, like_escape,
+ DESCRIPTOR_MAX_CHARS, StorageStatus, entity_query_tokens, handoff_listing_sql, like_escape,
page_descriptor, page_descriptor_expr,
};
use crate::Store;
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index 433659dc..c6da36c9 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -327,6 +327,11 @@ pub(crate) enum WriteCmd {
compaction: crate::ops::Compaction,
reply: oneshot::Sender<StoreResult<crate::ops::PurgeSessionSummary>>,
},
+ /// Reclaim free pages on demand: rebuild the FTS indexes and `VACUUM`,
+ /// deleting nothing. See [`ops::compact`].
+ Compact {
+ reply: oneshot::Sender<StoreResult<crate::ops::CompactSummary>>,
+ },
/// Delete a workspace row (its `workspace_id` FKs cascade projects/pages/
/// sessions/…). Refused when non-empty unless `force`.
DeleteWorkspace {
@@ -1370,6 +1375,21 @@ impl WriterHandle {
rx.await.map_err(|_| StoreError::WriterClosed)?
}
+ /// Reclaim free pages on demand, deleting nothing.
+ ///
+ /// Runs through the writer actor like every other exclusive operation, so
+ /// it cannot overlap a write: `VACUUM` takes an exclusive lock and would
+ /// otherwise contend with one.
+ ///
+ /// # Errors
+ /// Returns [`StoreError::WriterClosed`] if the actor has shut down, or
+ /// propagates the SQL error from the rebuild or `VACUUM`.
+ pub async fn compact(&self) -> StoreResult<crate::ops::CompactSummary> {
+ let (tx, rx) = oneshot::channel();
+ self.send(WriteCmd::Compact { reply: tx }).await?;
+ rx.await.map_err(|_| StoreError::WriterClosed)?
+ }
+
/// Rename a workspace (column-only; the on-disk dir is UUID-keyed).
///
/// # Errors
@@ -2389,6 +2409,10 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
);
send_or_warn(reply, result, "purge_session");
}
+ WriteCmd::Compact { reply } => {
+ let result = ops::compact(&mut conn);
+ send_or_warn(reply, result, "compact");
+ }
WriteCmd::DeleteWorkspace {
workspace_id,
force,
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 364f5daf..707dc362 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -457,6 +457,7 @@ install-skills reorg purge-project
rename-project move-project move-session
uninstall auth user
completions handoffs purge-session
+compact
```
Run `ai-memory --help` for the full tree.
diff --git a/docs/deploy.md b/docs/deploy.md
index 53b6b94e..e90b7272 100644
--- a/docs/deploy.md
+++ b/docs/deploy.md
@@ -210,6 +210,75 @@ scp "$SERVER:$DEPLOY_DIR/data/snapshot-$(date +%F).tar.gz" ./backups/
The `ai-memory backup` command uses SQLite's online backup API so
writes during the snapshot are coherent.
+## Reclaiming disk space
+
+SQLite does not return deleted space to the filesystem. Deleted rows leave
+their pages on a **freelist**, which SQLite reuses as the database grows again.
+For a store in steady use that is the right behaviour and needs no attention.
+
+`ai-memory status` reports the figure that decides it:
+
+```
+ storage: 2.4 GiB on disk, 610.0 MiB reclaimable (25.4%)
+ `ai-memory compact --confirm` would return it (blocks writes while it runs)
+```
+
+The advisory line only appears once the backlog is worth acting on. When it
+does:
+
+```bash
+ssh "$SERVER" "docker exec ai-memory /usr/local/bin/ai-memory compact --confirm"
+```
+
+Compaction deletes nothing — it rebuilds the FTS indexes and `VACUUM`s.
+
+### Do not schedule an unconditional nightly VACUUM
+
+The obvious move is a nightly cron entry. Resist it:
+
+- `VACUUM` takes an **exclusive lock** and rewrites the entire database. Every
+ write blocks for the duration, which on a large store is minutes — and this
+ server's job is answering hook traffic, so blocking writes drops captures.
+- It needs free disk space of roughly the database's own size, so the nightly
+ job also sets a permanent floor on free space.
+- Because SQLite reuses free pages, a store in steady use usually has almost
+ nothing to reclaim. The nightly run pays the full cost for no benefit on most
+ nights.
+
+The case that genuinely leaves a large freelist is a **one-off deletion** — a
+`purge-project`, a big retention sweep, a `forget-sweep` over months of
+episodic pages. Those are events, not a schedule, and the destructive commands
+already offer `--compact` inline for exactly that moment.
+
+If you do want it automated, make it **conditional** on the figure above and
+put it in off hours:
+
+```bash
+#!/usr/bin/env bash
+# /usr/local/bin/ai-memory-compact-if-worthwhile
+set -euo pipefail
+
+RECLAIMABLE=$(ai-memory status --json | jq '.storage.reclaimable_bytes')
+THRESHOLD=$((1024 * 1024 * 1024)) # 1 GiB — tune to your store
+
+if [ "$RECLAIMABLE" -ge "$THRESHOLD" ]; then
+ ai-memory compact --confirm
+fi
+```
+
+Drive it from a systemd timer (`OnCalendar=Sun 04:00`, `Persistent=true`) or a
+host cron entry. A weekly check that usually does nothing costs one cheap
+`status` call; a nightly `VACUUM` that usually does nothing costs a write stall
+every night.
+
+### What compaction does not do
+
+It is not erasure. The wiki git history keeps page content in its objects and
+commit messages, and any backup taken earlier still holds everything. Compaction
+returns bytes from the live SQLite file — worth doing on its own terms, and not
+a guarantee that content is unrecoverable. See
+[lifecycle-ops.md](lifecycle-ops.md) for the full boundary.
+
## Rolling back
```bash