Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(purge): stop overstating what purge-project and delete-workspace delete (#540) (#548)

Fabio Akita3f15ae8294c7
diff --git a/CHANGELOG.md b/CHANGELOG.md
index dfb062b4..953423fb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -70,6 +70,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
earlier backups still contain it. `docs/lifecycle-ops.md` states that
boundary in a table so it is not inferred from the command name.
+- `ai-memory purge-project --compact` and `{"compact": true}` on
+ `POST /admin/delete-workspace` reclaim the bytes a delete frees, reusing the
+ `Compaction` opt-in that `purge-session` gained in the same cycle (#540). Both
+ responses now report `compacted`.
+
+ Compaction rebuilds **all three** FTS5 indexes rather than the two a session
+ purge needs. A managed workstream's `workstream_events` rows leave through the
+ `projects → workstreams → workstream_events` cascade, and a cascade does not
+ fire the `AFTER DELETE` trigger that would drop them from
+ `workstream_events_fts` — so the tokens survive an ordinary delete, and a
+ `VACUUM` alone does not clear them. A test pins this: removing the third
+ rebuild leaves a purged agent transcript's text in the database file.
+
### Fixed
- A failed or skipped embed now leaves a durable record. `page_embeddings`
stores only successes, and its upsert rewrites `created_at`, so a global
@@ -126,6 +139,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
drain honouring `AI_MEMORY_SERVER_URL` would send captured events to whatever
address happened to be exported into the process that ran it.
+- `purge-project` and `delete-workspace` no longer overstate what they delete
+ (#540). The help text promised "Permanently delete a project and ALL its
+ data", which reads as byte-level removal neither command performed.
+
+ The deletion was never the defect. Rows go and bytes stay in free pages until
+ the file is rewritten — ordinary SQLite behaviour, and the same boundary
+ `purge-session` already documented. Measured on a canary token: delete alone
+ leaves the text on disk, delete + `VACUUM` still leaves it, and only
+ delete + FTS `rebuild` + `VACUUM` removes it. The claim was what needed
+ fixing, so both commands now describe the boundary they actually enforce and
+ offer the same `--compact` opt-in, and `docs/lifecycle-ops.md` covers all
+ three destructive commands in one place instead of documenting the limits of
+ one and overstating the other two.
+
### Docs
- `docs/windows.md` gains **Scenario E**, a persistent-server story for native
diff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rs
index 1562595e..88973a72 100644
--- a/crates/ai-memory-cli/src/cli.rs
+++ b/crates/ai-memory-cli/src/cli.rs
@@ -145,9 +145,15 @@ pub enum Command {
/// `is_latest=false` (they were a multi-project mash-up) so the
/// next consolidation can regenerate them per-project. Idempotent.
Reorg(ReorgArgs),
- /// Permanently delete a project and ALL its data (pages, sessions,
- /// observations, handoffs, embeddings, on-disk wiki files).
- /// This is irreversible — requires `--confirm`.
+ /// Delete a project: its pages, sessions, observations, handoffs,
+ /// embeddings, managed workstreams and on-disk wiki files.
+ ///
+ /// Irreversible — requires `--confirm`. By default this is a logical
+ /// delete: the rows are gone and nothing can reach them through the API
+ /// or search, but their bytes remain in free pages of the database file
+ /// until it is rewritten, as with any SQLite delete. `--compact`
+ /// reclaims them; neither mode is forensic erasure, because the wiki git
+ /// history and any earlier backup still hold the content.
PurgeProject(PurgeProjectArgs),
/// Permanently delete ONE session and everything derived from it.
///
@@ -627,6 +633,19 @@ pub struct PurgeProjectArgs {
/// out — purging is destructive and irreversible.
#[arg(long)]
pub confirm: bool,
+ /// Also reclaim the freed bytes: rebuild the FTS indexes and VACUUM the
+ /// database after the delete commits.
+ ///
+ /// Without this the purge is a logical delete — the project is gone from
+ /// the API and from search, but its bytes stay in free pages of the
+ /// database file until it is next rewritten, as with any SQLite delete.
+ ///
+ /// This rewrites the WHOLE database and needs free disk space of about
+ /// its size, so it takes minutes on a large store. It also does NOT make
+ /// the content forensically unrecoverable: the wiki git history and any
+ /// backup taken before the purge still contain it.
+ #[arg(long)]
+ pub compact: bool,
}
/// Arguments for `purge-session`.
diff --git a/crates/ai-memory-cli/src/commands/purge_project.rs b/crates/ai-memory-cli/src/commands/purge_project.rs
index 6ffca85a..9e3a9686 100644
--- a/crates/ai-memory-cli/src/commands/purge_project.rs
+++ b/crates/ai-memory-cli/src/commands/purge_project.rs
@@ -15,6 +15,8 @@ struct PurgeProjectRequest {
confirm: bool,
/// Purge even when a managed workstream still holds a live run lease.
force: bool,
+ /// Rebuild the FTS indexes and VACUUM after the delete commits.
+ compact: bool,
}
/// Run the `purge-project` subcommand.
@@ -49,6 +51,7 @@ pub async fn run(config: &Config, args: PurgeProjectArgs) -> Result<()> {
project: project.clone(),
confirm: true,
force: args.force,
+ compact: args.compact,
},
)
.await?;
@@ -90,5 +93,15 @@ pub async fn run(config: &Config, args: PurgeProjectArgs) -> Result<()> {
failed.len()
);
}
+ if report["compacted"].as_bool().unwrap_or(false) {
+ println!("Database compacted: freed bytes reclaimed.");
+ } else {
+ println!(
+ "Note: this was a logical delete. The project is unreachable through \
+ the API and search, but its bytes remain in the database file until \
+ it is rewritten. Re-run with --compact to reclaim them (slow), and \
+ see docs/lifecycle-ops.md for what that does and does not guarantee."
+ );
+ }
Ok(())
}
diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rs
index 327517bd..386dfb96 100644
--- a/crates/ai-memory-hooks/src/router.rs
+++ b/crates/ai-memory-hooks/src/router.rs
@@ -6841,7 +6841,14 @@ mod tests {
// points at it (exactly the purge-on-live-server scenario).
state
.writer
- .purge_project(ws, proj, "default/heal-project", None, false)
+ .purge_project(
+ ws,
+ proj,
+ "default/heal-project",
+ None,
+ false,
+ ai_memory_store::Compaction::Skip,
+ )
.await
.unwrap();
assert!(
@@ -7643,7 +7650,14 @@ mod tests {
state
.writer
- .purge_project(ws, proj, "default/repo-root-project", None, false)
+ .purge_project(
+ ws,
+ proj,
+ "default/repo-root-project",
+ None,
+ false,
+ ai_memory_store::Compaction::Skip,
+ )
.await
.unwrap();
diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rs
index 40f70c96..05e27f59 100644
--- a/crates/ai-memory-mcp/src/admin.rs
+++ b/crates/ai-memory-mcp/src/admin.rs
@@ -21,11 +21,13 @@
//! - `POST /admin/commit` — stage + commit the wiki tree via git.
//! - `GET /admin/checkpoints` — list recent wiki git checkpoints.
//! - `POST /admin/restore-page` — restore one page from a checkpoint.
-//! - `POST /admin/purge-project` — delete a project and all its data.
+//! - `POST /admin/purge-project` — delete a project's rows and wiki files
+//! (logical unless `compact` is set; see `ai_memory_store::Compaction`).
//! - `POST /admin/purge-session` — delete one session and what it derived.
//! - `POST /admin/rename-project` — rename a project (column-only; no files move).
//! - `POST /admin/rename-workspace` — rename a workspace and refresh scope manifests.
-//! - `POST /admin/delete-workspace` — delete a workspace and all of its projects.
+//! - `POST /admin/delete-workspace` — delete a workspace and its projects
+//! (logical unless `compact` is set; see `ai_memory_store::Compaction`).
//! - `POST /admin/merge-workspace` — fold every project of one workspace into
//! another, then delete the emptied source workspace.
//! - `POST /admin/move-project` — move a project into another workspace
@@ -3468,6 +3470,11 @@ struct PurgeProjectRequest {
/// out from under a running agent, which then cannot save its history.
#[serde(default)]
force: bool,
+ /// Reclaim freed bytes: rebuild the FTS indexes and `VACUUM` after the
+ /// delete commits. Off by default; see `ai_memory_store::Compaction` for
+ /// the cost and for what it does not guarantee.
+ #[serde(default)]
+ compact: bool,
}
/// Wire-format summary returned by `POST /admin/purge-project`.
@@ -3496,6 +3503,10 @@ pub struct PurgeProjectReport {
pub files_deleted: Vec<String>,
/// Paths that could not be removed from disk (non-fatal; DB rows are gone).
pub files_failed: Vec<String>,
+ /// Whether the freed bytes were reclaimed (`VACUUM` ran). False for a
+ /// plain logical delete: the rows are gone from the API and from search,
+ /// but their bytes stay in free pages until the file is next rewritten.
+ pub compacted: bool,
/// Pre-purge checkpoint, if the tree had uncommitted changes.
#[serde(skip_serializing_if = "Option::is_none")]
pub pre_checkpoint: Option<String>,
@@ -3630,9 +3641,15 @@ async fn handle_purge_project(
Err(e) => return e,
};
+ let compaction = if req.compact {
+ ai_memory_store::Compaction::Reclaim
+ } else {
+ ai_memory_store::Compaction::Skip
+ };
+
let summary = match state
.writer
- .purge_project(ws_id, proj_id, &label, author_id, req.force)
+ .purge_project(ws_id, proj_id, &label, author_id, req.force, compaction)
.await
{
Ok(s) => s,
@@ -3688,6 +3705,7 @@ async fn handle_purge_project(
workstreams_deleted: summary.workstreams_deleted,
managed_runs_deleted: summary.managed_runs_deleted,
workstream_ids: summary.workstream_ids,
+ compacted: summary.compacted,
files_deleted,
files_failed,
pre_checkpoint,
@@ -3785,6 +3803,11 @@ struct DeleteWorkspaceRequest {
/// non-empty workspace is refused so a typo can't wipe live data.
#[serde(default)]
force: bool,
+ /// Reclaim freed bytes: rebuild the FTS indexes and `VACUUM` after the
+ /// delete commits. Off by default; see `ai_memory_store::Compaction` for
+ /// the cost and for what it does not guarantee.
+ #[serde(default)]
+ compact: bool,
}
/// Wire-format summary returned by `POST /admin/delete-workspace`.
@@ -3807,6 +3830,10 @@ pub struct DeleteWorkspaceResult {
pub files_deleted: Vec<String>,
/// Paths that could not be removed from disk (non-fatal; DB rows are gone).
pub files_failed: Vec<String>,
+ /// Whether the freed bytes were reclaimed (`VACUUM` ran). False for a
+ /// plain logical delete: the rows are gone from the API and from search,
+ /// but their bytes stay in free pages until the file is next rewritten.
+ pub compacted: bool,
/// Pre-delete checkpoint, if the tree had uncommitted changes.
#[serde(skip_serializing_if = "Option::is_none")]
pub pre_checkpoint: Option<String>,
@@ -3826,7 +3853,12 @@ async fn handle_delete_workspace(
let actor = actor_ext
.map(|axum::Extension(a)| a)
.unwrap_or_else(ai_memory_core::ActorContext::anonymous);
- match delete_workspace_core(&state, &req.workspace, req.force, actor).await {
+ let compaction = if req.compact {
+ ai_memory_store::Compaction::Reclaim
+ } else {
+ ai_memory_store::Compaction::Skip
+ };
+ match delete_workspace_core(&state, &req.workspace, req.force, actor, compaction).await {
Ok(result) => (
StatusCode::OK,
Json(serde_json::to_value(&result).unwrap_or(serde_json::Value::Null)),
@@ -3845,6 +3877,7 @@ async fn delete_workspace_core(
workspace: &str,
force: bool,
actor: ai_memory_core::ActorContext,
+ compaction: ai_memory_store::Compaction,
) -> Result<DeleteWorkspaceResult, MoveErr> {
let ws_id = lookup_ws_no_create(state, workspace).await?;
@@ -3886,7 +3919,11 @@ async fn delete_workspace_core(
let pre_checkpoint =
checkpoint_or_500(&state.wiki, format!("pre-delete-workspace {workspace}"))?;
- let summary = match state.writer.delete_workspace(ws_id, force).await {
+ let summary = match state
+ .writer
+ .delete_workspace(ws_id, force, compaction)
+ .await
+ {
Ok(s) => s,
Err(e) => {
let status = match &e {
@@ -3936,6 +3973,7 @@ async fn delete_workspace_core(
workstreams_deleted: summary.workstreams_deleted,
managed_runs_deleted: summary.managed_runs_deleted,
workstream_ids: summary.workstream_ids,
+ compacted: summary.compacted,
files_deleted,
files_failed,
pre_checkpoint,
@@ -5691,9 +5729,20 @@ async fn copy_purge_merge(
// The source purge is an internal step of move-project (a distinct op that
// records its own move report); it is not attributed as a standalone
// `purge_project` here, so the audit author is left NULL.
+ // `Skip`: compaction is an operator's explicit choice on a destructive
+ // command, not a side effect of moving a project. A `VACUUM` here would
+ // rewrite the whole database in the middle of a move the caller asked to
+ // be cheap.
let summary = match state
.writer
- .purge_project(src_ws, src_proj, &label, None, false)
+ .purge_project(
+ src_ws,
+ src_proj,
+ &label,
+ None,
+ false,
+ ai_memory_store::Compaction::Skip,
+ )
.await
{
Ok(s) => s,
@@ -5914,21 +5963,29 @@ async fn handle_merge_workspace(
// Every project moved → the source workspace is now empty. Delete the shell
// with force=false: it must be empty, so a race that repopulated it aborts
// safely without wiping fresh data.
- let source_workspace_deleted =
- match delete_workspace_core(&state, &req.from, false, actor).await {
- Ok(_) => true,
- Err((_status, body)) => {
- // The moves succeeded; only the final empty-shell delete failed
- // (e.g. a concurrent write recreated a project). No data was
- // lost, so report success-with-caveat instead of a hard error.
- warn!(
- workspace = %req.from,
- "merge-workspace: source drained but shell delete failed: {:?}",
- body
- );
- false
- }
- };
+ let source_workspace_deleted = match delete_workspace_core(
+ &state,
+ &req.from,
+ false,
+ actor,
+ // The shell is empty by this point; there is nothing to reclaim.
+ ai_memory_store::Compaction::Skip,
+ )
+ .await
+ {
+ Ok(_) => true,
+ Err((_status, body)) => {
+ // The moves succeeded; only the final empty-shell delete failed
+ // (e.g. a concurrent write recreated a project). No data was
+ // lost, so report success-with-caveat instead of a hard error.
+ warn!(
+ workspace = %req.from,
+ "merge-workspace: source drained but shell delete failed: {:?}",
+ body
+ );
+ false
+ }
+ };
let report = MergeWorkspaceReport {
from: req.from,
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index bffb0ae1..db22fc82 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -148,6 +148,9 @@ pub struct PurgeSummary {
/// pre-delete identifiers to remove `raw/workstreams/<id>/` after the SQL
/// transaction commits and to report any filesystem partial failure.
pub workstream_ids: Vec<String>,
+ /// Whether the freed bytes were reclaimed (`VACUUM` ran). False for a
+ /// plain logical delete.
+ pub compacted: bool,
}
use jiff::Timestamp;
use rusqlite::{Connection, OptionalExtension, Transaction, params};
@@ -2956,7 +2959,13 @@ pub fn insert_wiki_migration(
Ok(())
}
-/// Delete a project and all its data inside one transaction.
+/// Delete a project's rows inside one transaction.
+///
+/// This is a *logical* delete unless `compaction` is [`Compaction::Reclaim`].
+/// The rows are gone and nothing reaches them through the API or search, but
+/// their bytes remain in free pages of the database file until it is
+/// rewritten — as with any SQLite delete. Neither mode is forensic erasure:
+/// the wiki git history and any earlier backup still hold the content.
///
/// Execution order:
/// 1. Count rows in each dependent table (pages/all versions, sessions,
@@ -2981,7 +2990,9 @@ pub fn insert_wiki_migration(
/// Returns [`StoreError::ManagedRunActive`] when a managed run's lease is
/// still live and `force` is false, or [`StoreError`] if any SQL statement
/// fails. The transaction is rolled back automatically on error.
-/// Whether [`purge_session`] should reclaim the freed bytes afterwards.
+/// Whether a destructive scope operation should reclaim the freed bytes
+/// afterwards. Shared by [`purge_session`], [`purge_project`] and
+/// [`delete_workspace`] so the family makes one promise, not three.
///
/// A purge is a logical delete: the rows are gone and nothing can reach them
/// through the API or through search, but their bytes remain in free pages of
@@ -3011,6 +3022,30 @@ pub enum Compaction {
Reclaim,
}
+/// Rebuild every FTS5 index, then `VACUUM`. Runs after the caller's
+/// transaction has committed: `VACUUM` cannot run inside one, and a rebuild is
+/// a whole-index operation we do not want holding the write lock for the
+/// duration of a delete.
+///
+/// All three indexes are rebuilt regardless of which scope was deleted. They
+/// are `content=`-backed external-content tables, so an ordinary delete leaves
+/// the tokens behind inside the index segments and only a `rebuild` clears
+/// them — and a project or workspace delete cascades `workstream_events` just
+/// as it does `pages` and `observations`. Rebuilding only the indexes a given
+/// caller "should" have touched is exactly the reasoning that leaves text
+/// searchable after a purge, and the cost is dominated by the `VACUUM` that
+/// follows in every case. A `rebuild` is idempotent, so the extra work is a
+/// no-op for a scope that deleted nothing from that table.
+fn reclaim_freed_pages(conn: &Connection) -> StoreResult<()> {
+ conn.execute_batch(
+ "INSERT INTO observations_fts(observations_fts) VALUES('rebuild'); \
+ INSERT INTO pages_fts(pages_fts) VALUES('rebuild'); \
+ INSERT INTO workstream_events_fts(workstream_events_fts) VALUES('rebuild'); \
+ VACUUM;",
+ )?;
+ Ok(())
+}
+
/// What [`purge_session`] removed. All counts are rows actually deleted.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub struct PurgeSessionSummary {
@@ -3220,15 +3255,8 @@ pub fn purge_session(
tx.commit()?;
- // Outside the transaction: `VACUUM` cannot run inside one, and a rebuild
- // is a whole-index operation we do not want holding the write lock for
- // the duration of the delete.
if compaction == Compaction::Reclaim {
- conn.execute_batch(
- "INSERT INTO observations_fts(observations_fts) VALUES('rebuild'); \
- INSERT INTO pages_fts(pages_fts) VALUES('rebuild'); \
- VACUUM;",
- )?;
+ reclaim_freed_pages(conn)?;
}
Ok(PurgeSessionSummary {
@@ -3248,6 +3276,7 @@ pub fn purge_project(
workspace_project_label: &str,
author_id: Option<ai_memory_core::UserId>,
force: bool,
+ compaction: Compaction,
) -> StoreResult<PurgeSummary> {
let tx = conn.transaction()?;
@@ -3376,6 +3405,11 @@ pub fn purge_project(
)?;
tx.commit()?;
+
+ if compaction == Compaction::Reclaim {
+ reclaim_freed_pages(conn)?;
+ }
+
Ok(PurgeSummary {
label: workspace_project_label.to_string(),
page_paths,
@@ -3387,6 +3421,7 @@ pub fn purge_project(
workstreams_deleted,
managed_runs_deleted,
workstream_ids,
+ compacted: compaction == Compaction::Reclaim,
})
}
@@ -3403,6 +3438,9 @@ pub struct DeleteWorkspaceSummary {
pub managed_runs_deleted: u64,
/// Pre-delete identifiers for post-commit raw-segment cleanup.
pub workstream_ids: Vec<String>,
+ /// Whether the freed bytes were reclaimed (`VACUUM` ran). False for a
+ /// plain logical delete.
+ pub compacted: bool,
}
/// Delete a workspace row. Refuses a workspace that still holds projects
@@ -3419,6 +3457,7 @@ pub fn delete_workspace(
conn: &mut Connection,
workspace_id: &WorkspaceId,
force: bool,
+ compaction: Compaction,
) -> StoreResult<DeleteWorkspaceSummary> {
let tx = conn.transaction()?;
let wid = workspace_id.as_bytes();
@@ -3458,12 +3497,18 @@ pub fn delete_workspace(
return Err(StoreError::NotFound("workspace".into()));
}
tx.commit()?;
+
+ if compaction == Compaction::Reclaim {
+ reclaim_freed_pages(conn)?;
+ }
+
Ok(DeleteWorkspaceSummary {
projects_deleted,
pages_deleted,
workstreams_deleted,
managed_runs_deleted,
workstream_ids,
+ compacted: compaction == Compaction::Reclaim,
})
}
@@ -4295,7 +4340,7 @@ pub(crate) mod tests {
let prepared = open_managed_run(&mut conn, &ws, &proj);
// Non-empty (holds the "scratch" project + a page) → refused w/o force.
- let err = delete_workspace(&mut conn, &ws, false).unwrap_err();
+ let err = delete_workspace(&mut conn, &ws, false, Compaction::Skip).unwrap_err();
assert!(
matches!(err, StoreError::WorkspaceNotEmpty(n) if n >= 1),
"expected WorkspaceNotEmpty, got {err:?}"
@@ -4310,7 +4355,7 @@ pub(crate) mod tests {
assert_eq!(ws_still, 1, "refused delete must not touch the row");
// Force cascades: project + page gone, workspace row gone.
- let summary = delete_workspace(&mut conn, &ws, true).unwrap();
+ let summary = delete_workspace(&mut conn, &ws, true, Compaction::Skip).unwrap();
assert!(
summary.projects_deleted >= 1 && summary.pages_deleted >= 1,
"{summary:?}"
@@ -4332,7 +4377,7 @@ pub(crate) mod tests {
// Deleting again → NotFound.
assert!(matches!(
- delete_workspace(&mut conn, &ws, true).unwrap_err(),
+ delete_workspace(&mut conn, &ws, true, Compaction::Skip).unwrap_err(),
StoreError::NotFound(_)
));
}
@@ -4341,7 +4386,7 @@ pub(crate) mod tests {
fn delete_workspace_empty_succeeds_without_force() {
let (_tmp, mut conn, _ws, _proj) = fresh_db();
let empty = get_or_create_workspace(&mut conn, "orphan-ws").unwrap();
- let summary = delete_workspace(&mut conn, &empty, false).unwrap();
+ let summary = delete_workspace(&mut conn, &empty, false, Compaction::Skip).unwrap();
assert_eq!(summary.projects_deleted, 0);
assert_eq!(summary.pages_deleted, 0);
assert_eq!(summary.workstreams_deleted, 0);
@@ -4451,6 +4496,178 @@ pub(crate) mod tests {
conn.query_row(sql, [], |r| r.get(0)).unwrap()
}
+ /// Insert one managed workstream event carrying a distinctive marker.
+ /// `workstream_events` cascades out of `workstreams`, which cascades out
+ /// of `projects` — so a project purge removes these rows without ever
+ /// running a `DELETE` statement against the table itself.
+ fn seed_workstream_event(
+ conn: &Connection,
+ workstream_id: &ai_memory_core::WorkstreamId,
+ marker: &str,
+ ) {
+ conn.execute(
+ "INSERT INTO workstream_events (workstream_id, sequence, event_id, agent_kind, \
+ native_session_id, kind, role, content, created_at) \
+ VALUES (?1, 1, ?2, 'claude-code', 'native-1', 'message', 'user', ?3, ?4)",
+ rusqlite::params![
+ workstream_id.as_bytes(),
+ format!("evt-{marker}"),
+ format!("evt-body-{marker}"),
+ Timestamp::now().as_microsecond(),
+ ],
+ )
+ .unwrap();
+ }
+
+ /// The default for a project purge is the same logical delete `#387`
+ /// documented for a session purge. Pinned so the CLI help, the admin route
+ /// docs and `docs/lifecycle-ops.md` cannot drift away from the behaviour:
+ /// if this starts failing, byte-level removal became the default and all
+ /// three need updating together.
+ #[test]
+ fn purge_project_is_a_logical_delete_by_default() {
+ let (tmp, mut conn, ws, proj) = fresh_db();
+ let db_path = tmp.path().join("test.sqlite");
+ seed_session(&mut conn, ws, proj, "canaryproj");
+
+ let summary = purge_project(
+ &mut conn,
+ &ws,
+ &proj,
+ "default/scratch",
+ None,
+ false,
+ Compaction::Skip,
+ )
+ .unwrap();
+ assert!(!summary.compacted, "the default does not VACUUM");
+
+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();
+ let bytes = std::fs::read(&db_path).unwrap();
+ assert!(
+ bytes.windows(14).any(|w| w == b"obs-canaryproj"),
+ "documented boundary: purge-project is a logical delete. If this now \
+ fails, byte-level removal became the default and the #540 note, the \
+ CLI help and docs/lifecycle-ops.md all need updating to match."
+ );
+ }
+
+ /// The opt-in half. Paired with `purge_project_is_a_logical_delete_by_default`
+ /// so the two together pin the difference between what the command promises
+ /// and what it offers.
+ #[test]
+ fn purge_project_with_reclaim_removes_the_bytes_from_the_file() {
+ let (tmp, mut conn, ws, proj) = fresh_db();
+ let db_path = tmp.path().join("test.sqlite");
+ seed_session(&mut conn, ws, proj, "canaryproj");
+ // A second project in the same workspace must survive untouched.
+ let other = get_or_create_project(&mut conn, &ws, "keeper", None).unwrap();
+ seed_session(&mut conn, ws, other, "survivor");
+
+ let summary = purge_project(
+ &mut conn,
+ &ws,
+ &proj,
+ "default/scratch",
+ None,
+ false,
+ Compaction::Reclaim,
+ )
+ .unwrap();
+ assert!(summary.compacted, "the summary reports that VACUUM ran");
+
+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();
+ let bytes = std::fs::read(&db_path).unwrap();
+ assert!(
+ !bytes.windows(14).any(|w| w == b"obs-canaryproj"),
+ "Reclaim must remove the purged project's text from the database file"
+ );
+ assert!(
+ bytes.windows(12).any(|w| w == b"obs-survivor"),
+ "the sibling project's text is untouched by the compaction"
+ );
+ }
+
+ /// The reason `reclaim_freed_pages` rebuilds all three FTS indexes rather
+ /// than the two a session purge needs.
+ ///
+ /// `workstream_events` is an external-content FTS5 table whose rows leave
+ /// via the `projects` → `workstreams` → `workstream_events` cascade. A
+ /// cascade does not fire the `AFTER DELETE` trigger that would remove the
+ /// row from the index (`recursive_triggers` is off), so both the index
+ /// entry and its tokens survive the delete. Rebuilding only `pages_fts`
+ /// and `observations_fts` — the set a session purge needs — would leave a
+ /// managed agent's transcript text in the file after an operator asked for
+ /// it to be reclaimed.
+ #[test]
+ fn reclaim_clears_workstream_event_text_that_left_via_cascade() {
+ let (tmp, mut conn, ws, proj) = fresh_db();
+ let db_path = tmp.path().join("test.sqlite");
+ let prepared = open_managed_run(&mut conn, &ws, &proj);
+ seed_workstream_event(&conn, &prepared.workstream_id, "canaryevt");
+
+ // Present before the purge, both in the file and in the index.
+ let indexed: i64 = conn
+ .query_row(
+ "SELECT COUNT(*) FROM workstream_events_fts \
+ WHERE workstream_events_fts MATCH '\"canaryevt\"'",
+ [],
+ |r| r.get(0),
+ )
+ .unwrap();
+ assert_eq!(indexed, 1, "the event is searchable before the purge");
+
+ purge_project(
+ &mut conn,
+ &ws,
+ &proj,
+ "default/scratch",
+ None,
+ true,
+ Compaction::Reclaim,
+ )
+ .unwrap();
+
+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();
+ let bytes = std::fs::read(&db_path).unwrap();
+ // Assert on the *token*, not the whole hyphenated body. The body only
+ // ever existed contiguously in the content table, which `VACUUM` clears
+ // on its own; what survives an ordinary delete is the tokenized form
+ // inside the FTS5 segments, and only a `rebuild` removes that. Asserting
+ // on the full string would pass with or without the rebuild and prove
+ // nothing.
+ assert!(
+ !bytes.windows(9).any(|w| w == b"canaryevt"),
+ "Reclaim must clear workstream event tokens; if this fails, the \
+ workstream_events_fts rebuild was dropped from reclaim_freed_pages \
+ and a managed transcript's text survives a compacted purge"
+ );
+ }
+
+ /// `delete_workspace` takes the same path and makes the same promise.
+ #[test]
+ fn delete_workspace_reclaim_removes_the_bytes_and_reports_it() {
+ let (tmp, mut conn, ws, proj) = fresh_db();
+ let db_path = tmp.path().join("test.sqlite");
+ seed_session(&mut conn, ws, proj, "canaryws");
+
+ let skipped = {
+ let other = get_or_create_workspace(&mut conn, "doomed").unwrap();
+ delete_workspace(&mut conn, &other, false, Compaction::Skip).unwrap()
+ };
+ assert!(!skipped.compacted, "the default does not VACUUM");
+
+ let summary = delete_workspace(&mut conn, &ws, true, Compaction::Reclaim).unwrap();
+ assert!(summary.compacted, "the summary reports that VACUUM ran");
+
+ conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE);").ok();
+ let bytes = std::fs::read(&db_path).unwrap();
+ assert!(
+ !bytes.windows(12).any(|w| w == b"obs-canaryws"),
+ "Reclaim must remove the deleted workspace's text from the file"
+ );
+ }
+
/// #387, the reporter's reproduction: after purging one session by UUID,
/// its rows are gone from every table that held them.
#[test]
@@ -8227,8 +8444,16 @@ pub(crate) mod tests {
let (_tmp, mut conn, ws, proj) = fresh_db();
// Simulate the post-purge state: project row gone.
// `purge_project` drives the cascading deletes we want here.
- let _ = purge_project(&mut conn, &ws, &proj, "default/scratch", None, false)
- .expect("purge of fresh project should succeed");
+ let _ = purge_project(
+ &mut conn,
+ &ws,
+ &proj,
+ "default/scratch",
+ None,
+ false,
+ Compaction::Skip,
+ )
+ .expect("purge of fresh project should succeed");
// Now try to rename the project that no longer exists. The
// pre-fix code returned `Ok(())` because `UPDATE` affected
// zero rows. The fix returns `NotFound` so admin handlers
@@ -8289,6 +8514,7 @@ pub(crate) mod tests {
"default/scratch",
Some(author),
false,
+ Compaction::Skip,
)
.expect("purge should succeed");
@@ -8336,8 +8562,16 @@ pub(crate) mod tests {
let (_tmp, mut conn, ws, proj) = fresh_db();
open_managed_run(&mut conn, &ws, &proj);
- let err = purge_project(&mut conn, &ws, &proj, "default/scratch", None, false)
- .expect_err("an active managed run must block the purge");
+ let err = purge_project(
+ &mut conn,
+ &ws,
+ &proj,
+ "default/scratch",
+ None,
+ false,
+ Compaction::Skip,
+ )
+ .expect_err("an active managed run must block the purge");
match err {
StoreError::ManagedRunActive { count, workstreams } => {
@@ -8369,8 +8603,16 @@ pub(crate) mod tests {
let (_tmp, mut conn, ws, proj) = fresh_db();
let prepared = open_managed_run(&mut conn, &ws, &proj);
- let summary = purge_project(&mut conn, &ws, &proj, "default/scratch", None, true)
- .expect("force purges regardless of the live lease");
+ let summary = purge_project(
+ &mut conn,
+ &ws,
+ &proj,
+ "default/scratch",
+ None,
+ true,
+ Compaction::Skip,
+ )
+ .expect("force purges regardless of the live lease");
assert_eq!(summary.workstreams_deleted, 1);
assert_eq!(summary.managed_runs_deleted, 1);
@@ -8397,8 +8639,16 @@ pub(crate) mod tests {
)
.unwrap();
- let summary = purge_project(&mut conn, &ws, &proj, "default/scratch", None, false)
- .expect("a lapsed lease is not a running agent");
+ let summary = purge_project(
+ &mut conn,
+ &ws,
+ &proj,
+ "default/scratch",
+ None,
+ false,
+ Compaction::Skip,
+ )
+ .expect("a lapsed lease is not a running agent");
assert_eq!(summary.workstreams_deleted, 1);
assert_eq!(summary.managed_runs_deleted, 1);
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index 7ea49a05..433659dc 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -295,9 +295,12 @@ pub(crate) enum WriteCmd {
dim: u32,
reply: oneshot::Sender<StoreResult<u64>>,
},
- /// Delete a project and all its data (pages, sessions, observations,
- /// handoffs, embeddings) in one transaction. Returns the paths of
- /// every page file that must be removed from disk by the caller.
+ /// Delete a project's rows (pages, sessions, observations, handoffs,
+ /// embeddings) in one transaction. Returns the paths of every page file
+ /// that must be removed from disk by the caller.
+ ///
+ /// A logical delete unless `compaction` is [`ops::Compaction::Reclaim`]:
+ /// freed bytes stay in the file until it is rewritten.
PurgeProject {
workspace_id: WorkspaceId,
project_id: ProjectId,
@@ -308,6 +311,8 @@ pub(crate) enum WriteCmd {
author_id: Option<ai_memory_core::UserId>,
/// Purge even when a managed workstream still holds a live run lease.
force: bool,
+ /// Whether to reclaim the freed bytes afterwards (`VACUUM`).
+ compaction: crate::ops::Compaction,
reply: oneshot::Sender<StoreResult<PurgeSummary>>,
},
/// Delete one session and everything derived from it, inside a single
@@ -327,6 +332,8 @@ pub(crate) enum WriteCmd {
DeleteWorkspace {
workspace_id: WorkspaceId,
force: bool,
+ /// Whether to reclaim the freed bytes afterwards (`VACUUM`).
+ compaction: crate::ops::Compaction,
reply: oneshot::Sender<StoreResult<DeleteWorkspaceSummary>>,
},
/// Rename a workspace's `name` column (UUID-keyed dir doesn't move).
@@ -1273,7 +1280,8 @@ impl WriterHandle {
rx.await.map_err(|_| StoreError::WriterClosed)?
}
- /// Delete a project and all its data in one atomic transaction.
+ /// Delete a project's rows in one atomic transaction. Logical unless
+ /// `compaction` is [`ops::Compaction::Reclaim`].
///
/// ON DELETE CASCADE propagates the delete through pages, sessions,
/// observations, handoffs, and page_embeddings automatically. The
@@ -1290,6 +1298,7 @@ impl WriterHandle {
label: impl Into<String>,
author_id: Option<ai_memory_core::UserId>,
force: bool,
+ compaction: crate::ops::Compaction,
) -> StoreResult<PurgeSummary> {
let (tx, rx) = oneshot::channel();
self.send(WriteCmd::PurgeProject {
@@ -1298,6 +1307,7 @@ impl WriterHandle {
label: label.into(),
author_id,
force,
+ compaction,
reply: tx,
})
.await?;
@@ -1347,11 +1357,13 @@ impl WriterHandle {
&self,
workspace_id: WorkspaceId,
force: bool,
+ compaction: crate::ops::Compaction,
) -> StoreResult<DeleteWorkspaceSummary> {
let (tx, rx) = oneshot::channel();
self.send(WriteCmd::DeleteWorkspace {
workspace_id,
force,
+ compaction,
reply: tx,
})
.await?;
@@ -2345,6 +2357,7 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
label,
author_id,
force,
+ compaction,
reply,
} => {
let result = ops::purge_project(
@@ -2354,6 +2367,7 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
&label,
author_id,
force,
+ compaction,
);
send_or_warn(reply, result, "purge_project");
}
@@ -2378,9 +2392,10 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
WriteCmd::DeleteWorkspace {
workspace_id,
force,
+ compaction,
reply,
} => {
- let result = ops::delete_workspace(&mut conn, &workspace_id, force);
+ let result = ops::delete_workspace(&mut conn, &workspace_id, force, compaction);
send_or_warn(reply, result, "delete_workspace");
}
WriteCmd::RenameWorkspace {
diff --git a/docs/lifecycle-ops.md b/docs/lifecycle-ops.md
index d3b0c73c..65161ed3 100644
--- a/docs/lifecycle-ops.md
+++ b/docs/lifecycle-ops.md
@@ -8,12 +8,12 @@ on a homelab box where mistakes are harder to undo.
| Command | Safe with server **running**? | Wipes data? | Reversible? | Notes |
|---|---|---|---|---|
-| `purge-project --confirm` | ✅ yes | the one project's data | no | Deletes the UUID-namespaced wiki root and raw workstream segments; sibling projects remain untouched. Refuses with `409` while a managed workstream under the project holds a live run lease — `--force` overrides. |
+| `purge-project --confirm` | ✅ yes | the one project's data | no | Deletes the UUID-namespaced wiki root and raw workstream segments; sibling projects remain untouched. Refuses with `409` while a managed workstream under the project holds a live run lease — `--force` overrides. Logical delete by default; `--compact` additionally rebuilds the FTS indexes and `VACUUM`s (see below). |
| `purge-session --session-id --confirm` | ✅ yes | the one session's data | no | Deletes one session by UUID: its row, its observations, the handoffs it **authored**, its `sessions/<id>.md` page and every superseded version, their embeddings, and its auto-improve runs. Strictly scoped — a session that does not belong to the named workspace/project is a `404` and nothing is deleted. Handoffs the session only *accepted* are kept: that text belongs to the session that wrote it. Logical delete by default; `--compact` additionally rebuilds the FTS indexes and `VACUUM`s (see below). |
| `handoffs --expire-all --confirm` | ✅ yes | no (state change only) | no (but nothing is destroyed) | Marks every **open** handoff in the scope `expired` so it stops being offered to an agent. Rows, summaries and provenance are kept and stay visible in the audit log. Unlike the automatic sweep it does **not** spare manual handoffs or ones from another directory — those exemptions are exactly what a leftover backlog is made of, so honouring them would clear nothing. `--older-than-days N` keeps recent batons. Owner-scoped: never touches another user's baton. |
| `rename-project --from --to` | ✅ yes | no | yes (rename back) | Column-only update on `projects.name`. The on-disk dir is keyed by `project_id` (UUID), so the rename never moves a file. |
| `/admin/rename-workspace` | ✅ yes | no | yes (rename back) | Column-only update on `workspaces.name`; refreshes `_meta.md` scope manifests and checkpoints the wiki tree. |
-| `/admin/delete-workspace` | ✅ yes | the workspace and every child project | no | Runs `purge_workspace` admission first, deletes SQLite rows in one cascade, removes the UUID-keyed workspace directory and managed-workstream raw segments, reports filesystem partial failures, and dispatches mirror notification after durable work. |
+| `/admin/delete-workspace` | ✅ yes | the workspace and every child project | no | Runs `purge_workspace` admission first, deletes SQLite rows in one cascade, removes the UUID-keyed workspace directory and managed-workstream raw segments, reports filesystem partial failures, and dispatches mirror notification after durable work. Logical delete by default; `"compact": true` additionally rebuilds the FTS indexes and `VACUUM`s (see below). |
| `move-project --confirm` | ✅ yes | source only in the merge case (a `Reject`-policy `purge_project` webhook can still abort the source teardown leaving everything intact) | no | Fresh destination → lossless **true move** (re-stamp `workspace_id`, keep `project_id`, rename the dir): sessions/observations/handoffs + history all survive. Destination with a same-named project → **copy+purge merge**: only latest pages migrate. |
| `move-session <id> --to --confirm` | ✅ yes | no | yes (move it back) | Re-stamps one session (or every session touching `--from-project`) into another project: `sessions`, `observations`, its `handoffs`, consolidation jobs, auto-improve runs/claims and its `sessions/<id>.md` page, one transaction per session; the page file moves with it (`--pages move`, default) or is retired for regeneration. Without `--confirm` it is a real dry run (rolled back). Refuses with `409` an open session or a pending consolidation job unless `--force`. |
| `backup --to` | ✅ yes | no | n/a | Streams a gzipped tarball from the server's online `sqlite3 .backup` plus the wiki tree. Safe alongside the live writer. |
@@ -29,7 +29,7 @@ fundamentally cannot run while another process holds the SQLite WAL writer. See
[CLAUDE.md §16](../CLAUDE.md) for the invariant.
-## `purge-session` and what "deleted" means
+## What "deleted" means (`purge-session`, `purge-project`, `delete-workspace`)
`purge-session` answers *"forget this conversation"*: after it runs, the
session is gone from the API, from `status` counts and from search.
@@ -65,6 +65,29 @@ the purge still contains everything. Removing the bytes from the live SQLite
file is worth doing on its own terms; do not describe it to a user as a
guarantee that the content is unrecoverable, because it is not.
+### The same is true of `purge-project` and `delete-workspace`
+
+Every row above applies unchanged to `ai-memory purge-project --compact` and to
+`POST /admin/delete-workspace` with `{"compact": true}`. The table is a
+property of *any* SQLite delete, not of one command: rows go, bytes stay in
+free pages until the file is rewritten.
+
+This is worth saying explicitly because the help text for those two commands
+used to promise "ALL its data", which read as byte-level removal they never
+performed (#540). Both now describe the same boundary and both accept the same
+opt-in.
+
+One extra detail applies at project and workspace scope. A managed
+workstream's `workstream_events` rows leave through the
+`projects → workstreams → workstream_events` cascade rather than through a
+`DELETE` against that table, and a cascade does not fire the `AFTER DELETE`
+trigger that would drop the row from `workstream_events_fts`. Compaction
+therefore rebuilds **all three** FTS indexes — `pages_fts`,
+`observations_fts` and `workstream_events_fts` — not just the two a session
+purge needs. Rebuilding only the indexes a given caller "should" have touched
+is what leaves a managed agent's transcript text in the file after an operator
+asked for it to be reclaimed.
+
### Scope containment
The session id is never authority on its own. Every statement is filtered on
@@ -123,8 +146,14 @@ so a clean SQLite DB can be rebuilt from the UUID-keyed wiki tree alone.
```bash
ai-memory purge-project --workspace default --project my-project --confirm
+
+# …and to reclaim the freed bytes as well (slow; rewrites the whole database):
+ai-memory purge-project --workspace default --project my-project --confirm --compact
```
+Like `purge-session`, this is a logical delete unless `--compact` is given —
+see [What "deleted" means](#what-deleted-means-purge-session-purge-project-delete-workspace).
+
What happens, in order:
1. Server looks up `(workspace_id, project_id)` by name. Returns 404
@@ -238,7 +267,10 @@ Failure modes:
Deletes a workspace row and all child projects/pages/sessions/managed
workstreams through the `workspace_id` cascade. The route is guarded by
`force: true` for non-empty workspaces and follows the destructive-operation
-ordering used by project purges:
+ordering used by project purges. It accepts the same opt-in reclaim as the
+other two destructive commands — `{"compact": true}` — and is otherwise a
+logical delete; see
+[What "deleted" means](#what-deleted-means-purge-session-purge-project-delete-workspace).
1. Look up the workspace without creating missing scopes.
2. Run blocking `op=purge_workspace` admission. A reject-policy webhook aborts