Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(cli): reclaim-ledger-versions drops the pre-#660 ledger residue

Yi-111-ad2b9896876b9
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8e1bdf65..a8374321 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- `ai-memory reclaim-ledger-versions` drops the superseded versions of the raw
+ hook event ledger that the pre-2.1.1 indexer left behind (#660), and nothing
+ else. It reports what it would remove and changes nothing unless
+ `--confirm`; `--drop-latest` also removes each ledger's live row, and
+ `--compact` rebuilds the FTS index and `VACUUM`s to return the freed bytes.
+ A path is only a candidate when its *content* is a hook ledger, so a real
+ page named `log-2026-09.md` keeps its whole version chain, and only
+ `is_latest=0 AND superseded_at IS NULL` rows are eligible, so decay-owned
+ rows stay with `forget-sweep`. Derived FTS/entity/vector/link rows go with
+ the page through the existing cascades, and the FTS delete trigger is stood
+ down for the bulk delete and restored from its own `sqlite_master` DDL
+ afterwards, so the cleanup does not re-tokenize tens of gigabytes of ledger
+ body one row at a time. (#914)
- The builtin web UI has a root-only `/web/pending` page to triage pending
auto-improvement proposals. It lists the proposals of all projects, with a
project filter and a sort, and shows the rationale and the proposed body.
diff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rs
index 00d0ce97..8e07a645 100644
--- a/crates/ai-memory-cli/src/cli.rs
+++ b/crates/ai-memory-cli/src/cli.rs
@@ -96,6 +96,23 @@ pub enum Command {
/// database, so every write blocks until it finishes and it needs free
/// disk space of roughly the database's own size.
Compact(CompactArgs),
+ /// Drop the superseded ledger versions the pre-2.1.1 indexer left behind.
+ ///
+ /// #660 stopped the indexer from rewriting the whole `log-YYYY-MM.md` row
+ /// on every hook append. The fix stopped new rows; it did not remove the
+ /// ones already written, and no other command reaches them — `compact`
+ /// deletes nothing, `forget-sweep` only hard-deletes decay tombstones, and
+ /// `reindex` loses the DB-only state. One reported store held 6,539
+ /// versions of 101 live pages.
+ ///
+ /// Only paths whose *content* is a hook event ledger are considered, so a
+ /// real page a human happened to name `log-2026-09.md` keeps its whole
+ /// version chain. Each dropped version is a byte prefix of the one after
+ /// it, so nothing the file on disk does not already hold is lost.
+ ///
+ /// Prints what it would remove and changes nothing unless `--confirm` is
+ /// passed. See also `ai-memory status` for the reclaimable figure.
+ ReclaimLedgerVersions(ReclaimLedgerVersionsArgs),
/// Snapshot wiki/, db/, and config.toml into a gzipped tarball.
Backup(BackupArgs),
/// Export one project's wiki as an OKF v0.2 bundle tarball.
@@ -1038,6 +1055,34 @@ pub struct CompactArgs {
pub confirm: bool,
}
+/// Arguments for `reclaim-ledger-versions`.
+#[derive(Debug, Args)]
+pub struct ReclaimLedgerVersionsArgs {
+ /// Actually delete. Without this the command reports what it would remove
+ /// — the ledger paths, the row count and the bytes — and changes nothing.
+ #[arg(long)]
+ pub confirm: bool,
+ /// Also drop each ledger's *live* row, not just its superseded versions.
+ ///
+ /// Since #660 the indexer skips ledgers, so each ledger's live row is also
+ /// left over from before the fix. It is kept by default: it is the
+ /// version the file on disk corresponds to, and dropping it is the
+ /// operator's call. Drop it only once the ledger file itself has been
+ /// removed, or the next hook append starts a fresh chain.
+ #[arg(long)]
+ pub drop_latest: bool,
+ /// Rebuild the FTS index and VACUUM afterwards, returning the freed bytes
+ /// to the filesystem.
+ ///
+ /// Without this the reclaim is a logical delete — the rows are gone and
+ /// nothing can reach them, but their bytes stay in free pages of the
+ /// database file until it is next rewritten. `VACUUM` rewrites the whole
+ /// file under an exclusive lock and needs free disk of roughly the
+ /// database's own size, so it is opt-in.
+ #[arg(long)]
+ pub compact: bool,
+}
+
/// Arguments for `purge-project`.
#[derive(Debug, Args)]
pub struct PurgeProjectArgs {
diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rs
index e9576392..86bbf726 100644
--- a/crates/ai-memory-cli/src/commands/mod.rs
+++ b/crates/ai-memory-cli/src/commands/mod.rs
@@ -63,6 +63,7 @@ pub mod project_registry;
pub mod purge_project;
pub mod purge_session;
pub mod read_page;
+pub mod reclaim_ledger_versions;
pub mod reindex;
pub mod rename_project;
pub mod rename_workstream;
diff --git a/crates/ai-memory-cli/src/commands/reclaim_ledger_versions.rs b/crates/ai-memory-cli/src/commands/reclaim_ledger_versions.rs
new file mode 100644
index 00000000..67811514
--- /dev/null
+++ b/crates/ai-memory-cli/src/commands/reclaim_ledger_versions.rs
@@ -0,0 +1,105 @@
+//! `ai-memory reclaim-ledger-versions` — thin HTTP client for dropping the
+//! superseded versions of the raw hook event ledger.
+
+use anyhow::Result;
+use serde::Serialize;
+
+use crate::cli::ReclaimLedgerVersionsArgs;
+use crate::config::Config;
+use crate::http_client::{ServerEndpoint, post_json};
+
+use super::compact::human_bytes;
+
+/// Request sent to `POST /admin/reclaim-ledger-versions`.
+#[derive(Serialize)]
+struct ReclaimLedgerVersionsRequest {
+ confirm: bool,
+ dry_run: bool,
+ drop_latest: bool,
+ compact: bool,
+}
+
+/// Run the `reclaim-ledger-versions` subcommand.
+///
+/// A dry run by default: the report says what would go, and `--confirm` is
+/// what makes it go. `--confirm` is required for the deleting run because,
+/// unlike `compact`, this one removes rows and nothing here can put them back.
+///
+/// # Errors
+/// Returns an error when the server is unreachable or returns a non-2xx
+/// response.
+pub async fn run(config: &Config, args: ReclaimLedgerVersionsArgs) -> Result<()> {
+ let dry_run = !args.confirm;
+
+ let endpoint = ServerEndpoint::from_config_resolving_auth(config).await;
+ let report: serde_json::Value = post_json(
+ &endpoint,
+ "/admin/reclaim-ledger-versions",
+ &ReclaimLedgerVersionsRequest {
+ confirm: args.confirm,
+ dry_run,
+ drop_latest: args.drop_latest,
+ compact: args.compact,
+ },
+ )
+ .await?;
+
+ let n = |key: &str| report[key].as_u64().unwrap_or(0);
+ let paths = n("ledger_paths");
+ let rows = n("pages_deleted");
+ let body_bytes = n("bytes_deleted");
+ let dropped_latest = report["dropped_latest"].as_bool().unwrap_or(false);
+ let compacted = report["compacted"].as_bool().unwrap_or(false);
+
+ if paths == 0 || rows == 0 {
+ println!(
+ "No superseded ledger versions to reclaim. Nothing written by the \
+ pre-2.1.1 indexer is left in this store."
+ );
+ return Ok(());
+ }
+
+ if dry_run {
+ println!(
+ "Would delete {rows} superseded version(s) of the hook event ledger \
+ across {paths} ledger path(s), carrying {} of page body.",
+ human_bytes(body_bytes),
+ );
+ if dropped_latest {
+ println!(" --drop-latest: each ledger's live row would go too.");
+ }
+ println!("Nothing was changed. To apply:");
+ let mut flags = String::from(" ai-memory reclaim-ledger-versions --confirm");
+ if args.drop_latest {
+ flags.push_str(" --drop-latest");
+ }
+ if args.compact {
+ flags.push_str(" --compact");
+ }
+ println!("{flags}");
+ return Ok(());
+ }
+
+ println!(
+ "Deleted {rows} superseded ledger version(s) across {paths} ledger \
+ path(s), carrying {} of page body.",
+ human_bytes(body_bytes),
+ );
+ if dropped_latest {
+ println!(" Each ledger's live row was dropped as well (--drop-latest).");
+ }
+ if compacted {
+ println!(
+ "Reclaimed {} ({} → {}).",
+ human_bytes(n("bytes_reclaimed")),
+ human_bytes(n("bytes_before")),
+ human_bytes(n("bytes_after")),
+ );
+ } else {
+ println!(
+ "The bytes are free pages now. Run `ai-memory compact --confirm` \
+ to return them to the filesystem."
+ );
+ }
+ Ok(())
+}
diff --git a/crates/ai-memory-cli/src/lib.rs b/crates/ai-memory-cli/src/lib.rs
index 1350a46c..70204583 100644
--- a/crates/ai-memory-cli/src/lib.rs
+++ b/crates/ai-memory-cli/src/lib.rs
@@ -130,6 +130,9 @@ pub async fn run() -> Result<()> {
Command::Serve(args) => commands::serve::run(&config, args).await,
Command::Reset(args) => commands::reset::run(&config, args),
Command::Compact(args) => commands::compact::run(&config, args).await,
+ Command::ReclaimLedgerVersions(args) => {
+ commands::reclaim_ledger_versions::run(&config, args).await
+ }
Command::Backup(args) => commands::backup::run(&config, args).await,
Command::ExportOkf(args) => commands::export_okf::run(&config, args).await,
Command::Restore(args) => commands::restore::run(&config, args),
diff --git a/crates/ai-memory-core/src/lib.rs b/crates/ai-memory-core/src/lib.rs
index 6d320bc5..29520e7a 100644
--- a/crates/ai-memory-core/src/lib.rs
+++ b/crates/ai-memory-core/src/lib.rs
@@ -12,6 +12,7 @@ pub mod handoff;
pub mod ingest_metrics;
pub use ingest_metrics::{IngestMetrics, IngestMetricsSnapshot};
pub mod ids;
+pub mod log_ledger;
pub mod message;
pub mod observation;
pub mod okf;
diff --git a/crates/ai-memory-core/src/log_ledger.rs b/crates/ai-memory-core/src/log_ledger.rs
new file mode 100644
index 00000000..4cb3a9c9
--- /dev/null
+++ b/crates/ai-memory-core/src/log_ledger.rs
@@ -0,0 +1,187 @@
+//! Recognizing the raw hook event ledger (`log.md` / `log-YYYY-MM.md`).
+//!
+//! A reserved-looking filename is only a ledger when its content opens with a
+//! hook log entry, never on filename alone (an ordinary page can be named
+//! `log-2026-09.md`). Every surface that has to tell the two apart must agree,
+//! or a hook append supersedes a real page's row — or, in the other
+//! direction, a cleanup aimed at ledgers reaches a real page's version chain.
+//!
+//! The predicates live here, in the pure-compute layer, because the surfaces
+//! that consume them do not all have the same reach:
+//!
+//! - the file watcher, the OKF conformance migration and the OKF bundle export
+//! hold a path on disk ([`ai_memory_wiki`](https://docs.rs/ai-memory-wiki))
+//! and open it;
+//! - the store's `reclaim-ledger-versions` holds a `pages.body` it already has
+//! in memory and must not re-read from the filesystem.
+//!
+//! One definition, two entry points, so "is this a ledger" cannot drift
+//! between the surface that indexes ledgers and the surface that deletes
+//! them.
+
+/// Upper bound on frontmatter lines scanned by [`body_opens_with_log_ledger`].
+/// Conformant OKF frontmatter is a handful of keys; this is slack, not a
+/// format limit.
+pub const MAX_FRONTMATTER_LINES: usize = 64;
+
+/// How much of a body a caller has to read to answer
+/// [`body_opens_with_log_ledger`].
+///
+/// The gate is decided by the first meaningful line, so a few kilobytes is
+/// already generous: it leaves room for [`MAX_FRONTMATTER_LINES`] lines of
+/// frontmatter many times over. The bound is what makes the gate affordable
+/// on the rows it has to classify — a hook ledger carries a whole event log,
+/// and a store can hold tens of gigabytes of them. A caller holding the body
+/// already (the store) and a caller reading it off disk (the indexer) both
+/// need it, so the bound lives with the predicate.
+pub const GATE_PREFIX_BYTES: usize = 8 * 1024;
+
+/// Opening line of a hook log entry. The ledger is `## [ts] ...` entries and
+/// never YAML frontmatter.
+const LEDGER_ENTRY_PREFIX: &str = "## [";
+
+/// True when `s` is the ledger filename shape: `log.md`, or a rotated
+/// `log-YYYY-MM.md`.
+///
+/// This is the *shape* test only. It is deliberately satisfiable by an
+/// ordinary page — pair it with [`body_opens_with_log_ledger`] before acting
+/// on the answer.
+#[must_use]
+pub fn is_log_ledger_path(s: &str) -> bool {
+ s == "log.md" || is_rotated_log_ledger_path(s)
+}
+
+/// True when `s` is the rotated ledger shape `log-YYYY-MM.md`.
+///
+/// Month and day are not range-checked, only digit-checked: a filename is a
+/// shape, and the content gate is what decides whether a file wearing it is
+/// really a ledger.
+#[must_use]
+pub fn is_rotated_log_ledger_path(s: &str) -> bool {
+ let Some(stem) = s.strip_prefix("log-").and_then(|v| v.strip_suffix(".md")) else {
+ return false;
+ };
+ let bytes = stem.as_bytes();
+ bytes.len() == "YYYY-MM".len()
+ && bytes[4] == b'-'
+ && bytes[..4].iter().all(u8::is_ascii_digit)
+ && bytes[5..].iter().all(u8::is_ascii_digit)
+}
+
+/// Cheap content check for the raw hook event ledger shape. The body's first
+/// meaningful line must be a hook log entry (`## [ts] ...`).
+///
+/// The body may sit under a YAML frontmatter block: the OKF v0.2 migration
+/// conforms every `.md` under `wiki/`, ledgers included, so a migrated store
+/// has `type: Note` stamped on top of each `log-YYYY-MM.md`. Stopping at the
+/// fence would classify those ledgers as ordinary pages, and each hook
+/// `append_event` would then supersede a multi-megabyte page row.
+///
+/// A body with an unterminated frontmatter block inside
+/// [`MAX_FRONTMATTER_LINES`] is **not** a ledger. So is a body that ends
+/// before the deciding line. Both answers say "no", which is the safe
+/// direction for a caller deciding what to delete.
+#[must_use]
+pub fn body_opens_with_log_ledger(body: &str) -> bool {
+ let mut lines = body.lines();
+
+ let Some(first) = lines.next() else {
+ return false;
+ };
+ if first.trim_end() != "---" {
+ return first.starts_with(LEDGER_ENTRY_PREFIX);
+ }
+
+ // Walk past the frontmatter block.
+ let mut closed = false;
+ for line in lines.by_ref().take(MAX_FRONTMATTER_LINES) {
+ if line.trim_end() == "---" {
+ closed = true;
+ break;
+ }
+ }
+ if !closed {
+ return false;
+ }
+
+ // First non-blank line after the fence decides.
+ lines
+ .find(|line| !line.trim().is_empty())
+ .is_some_and(|line| line.starts_with(LEDGER_ENTRY_PREFIX))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn log_md_and_rotated_shapes_are_recognized() {
+ assert!(is_log_ledger_path("log.md"));
+ assert!(is_log_ledger_path("log-2026-09.md"));
+ assert!(is_rotated_log_ledger_path("log-2026-09.md"));
+ }
+
+ #[test]
+ fn near_miss_filenames_are_not_the_ledger_shape() {
+ for s in [
+ "log.md.bak",
+ "log-2026-9.md",
+ "log-2026-09-01.md",
+ "log-2026.md",
+ "log-abcdefgh.md",
+ "logs/log-2026-09.md",
+ "notes/log.md",
+ "LOG-2026-09.md",
+ ] {
+ assert!(!is_log_ledger_path(s), "{s} should not be the ledger shape");
+ }
+ }
+
+ #[test]
+ fn a_bare_entry_opens_a_ledger() {
+ assert!(body_opens_with_log_ledger(
+ "## [2026-09-25T10:00:00Z] tool call"
+ ));
+ }
+
+ #[test]
+ fn frontmatter_does_not_hide_the_ledger() {
+ let body =
+ "---\ntype: Note\ntitle: Log 2026-09\n---\n\n## [2026-09-25T10:00:00Z] tool call\n";
+ assert!(body_opens_with_log_ledger(body));
+ }
+
+ #[test]
+ fn frontmatter_needs_a_closing_fence() {
+ let body = "---\ntype: Note\n\n## [2026-09-25T10:00:00Z] tool call\n";
+ assert!(!body_opens_with_log_ledger(body));
+ }
+
+ #[test]
+ fn an_unterminated_frontmatter_block_is_not_a_ledger() {
+ let mut body = String::from("---\n");
+ for _ in 0..MAX_FRONTMATTER_LINES + 5 {
+ body.push_str("type: Note\n");
+ }
+ body.push_str("## [2026-09-25T10:00:00Z] tool call\n");
+ assert!(!body_opens_with_log_ledger(&body));
+ }
+
+ #[test]
+ fn an_ordinary_page_wearing_the_ledger_name_is_not_a_ledger() {
+ assert!(!body_opens_with_log_ledger(
+ "# Log 2026-09\n\nA hand-written page.\n"
+ ));
+ assert!(!body_opens_with_log_ledger(""));
+ assert!(!body_opens_with_log_ledger("---"));
+ }
+
+ #[test]
+ fn a_blank_first_line_decides_against_the_ledger() {
+ // The shape test is exact: the *first* line is what is compared, and
+ // a leading blank line is not a fence and not an entry.
+ assert!(!body_opens_with_log_ledger(
+ "\n## [2026-09-25T10:00:00Z] tool call\n"
+ ));
+ }
+}
diff --git a/crates/ai-memory-mcp/src/admin.rs b/crates/ai-memory-mcp/src/admin.rs
index 54562feb..b4e66e81 100644
--- a/crates/ai-memory-mcp/src/admin.rs
+++ b/crates/ai-memory-mcp/src/admin.rs
@@ -28,6 +28,8 @@
//! - `POST /admin/rename-project` — rename a project (column-only; no files move).
//! - `POST /admin/rename-workspace` — rename a workspace and refresh scope manifests.
//! - `POST /admin/compact` — reclaim free pages; deletes nothing.
+//! - `POST /admin/reclaim-ledger-versions` — drop the superseded ledger
+//! versions the pre-#660 indexer left behind, and nothing else.
//! - `POST /admin/delete-workspace` — delete a workspace and its projects
//! (logical unless `compact` is set; see `ai_memory_store::Compaction`).
//! - `POST /admin/merge-workspace` — fold every project of one workspace into
@@ -683,6 +685,10 @@ pub fn admin_router_with_sweep_tuning(
.route("/admin/move-session", post(handle_move_session))
.route("/admin/delete-workspace", post(handle_delete_workspace))
.route("/admin/compact", post(handle_compact))
+ .route(
+ "/admin/reclaim-ledger-versions",
+ post(handle_reclaim_ledger_versions),
+ )
.route("/admin/rename-workspace", post(handle_rename_workspace))
.route("/admin/merge-workspace", post(handle_merge_workspace))
.route("/admin/write-page", post(handle_write_page))
@@ -4537,6 +4543,98 @@ async fn handle_compact(
}
}
+/// JSON request body for `POST /admin/reclaim-ledger-versions`.
+#[derive(Deserialize)]
+struct ReclaimLedgerVersionsRequest {
+ /// Mandatory for a run that deletes. This one removes rows, and unlike
+ /// `compact` it is not reversible by re-running anything: the bytes are
+ /// only in the markdown the ledger mirrors, and only if the file on disk
+ /// still holds them.
+ confirm: bool,
+ /// Report what would go without deleting it. A dry run changes nothing,
+ /// so it needs no `confirm` — that is the whole point of it.
+ #[serde(default)]
+ dry_run: bool,
+ /// Also drop each ledger's live row, not just its superseded versions.
+ #[serde(default)]
+ drop_latest: bool,
+ /// Rebuild the FTS index and `VACUUM` afterwards to return the bytes.
+ #[serde(default)]
+ compact: bool,
+}
+
+/// Wire-format summary returned by `POST /admin/reclaim-ledger-versions`.
+#[derive(Debug, Serialize)]
+pub struct ReclaimLedgerVersionsReport {
+ /// Ledger `(workspace, project, path)` coordinates holding residue.
+ pub ledger_paths: u64,
+ /// `pages` rows selected — removed, unless the run was a dry run.
+ pub pages_deleted: u64,
+ /// Bytes of page body those rows carried. Logical payload, not freed
+ /// bytes: the file only shrinks when `compact` was set.
+ pub bytes_deleted: u64,
+ /// Bytes returned to the filesystem. Zero unless `compact` was set.
+ pub bytes_reclaimed: u64,
+ /// Whether each ledger's live row went too.
+ pub dropped_latest: bool,
+ /// Database size in bytes before the delete.
+ pub bytes_before: u64,
+ /// Database size in bytes afterwards.
+ pub bytes_after: u64,
+ /// Whether the freed bytes were reclaimed (`VACUUM` ran).
+ pub compacted: bool,
+}
+
+/// `POST /admin/reclaim-ledger-versions` — delete the superseded versions of
+/// the raw hook event ledger, and nothing else. The online counterpart of the
+/// cleanup #660's fix left no command for.
+async fn handle_reclaim_ledger_versions(
+ State(state): State<Arc<AdminState>>,
+ Json(req): Json<ReclaimLedgerVersionsRequest>,
+) -> impl IntoResponse {
+ if !req.dry_run && !req.confirm {
+ return (
+ StatusCode::BAD_REQUEST,
+ Json(serde_json::json!({
+ "error": "reclaim-ledger-versions deletes page rows; \
+ run it without --confirm first to see what it would \
+ remove, then pass confirm: true"
+ })),
+ );
+ }
+ let compaction = if req.compact {
+ ai_memory_store::Compaction::Reclaim
+ } else {
+ ai_memory_store::Compaction::Skip
+ };
+ match state
+ .writer
+ .reclaim_ledger_versions(req.dry_run, req.drop_latest, compaction)
+ .await
+ {
+ Ok(summary) => {
+ let report = ReclaimLedgerVersionsReport {
+ ledger_paths: summary.ledger_paths,
+ pages_deleted: summary.pages_deleted,
+ bytes_deleted: summary.bytes_deleted,
+ bytes_reclaimed: summary.bytes_reclaimed(),
+ dropped_latest: summary.dropped_latest,
+ bytes_before: summary.bytes_before,
+ bytes_after: summary.bytes_after,
+ compacted: summary.compacted,
+ };
+ (
+ StatusCode::OK,
+ Json(serde_json::to_value(&report).unwrap_or_else(|_| serde_json::json!({}))),
+ )
+ }
+ Err(e) => (
+ StatusCode::INTERNAL_SERVER_ERROR,
+ Json(serde_json::json!({ "error": e.to_string() })),
+ ),
+ }
+}
+
async fn handle_delete_workspace(
State(state): State<Arc<AdminState>>,
actor_ext: Option<axum::Extension<ai_memory_core::ActorContext>>,
diff --git a/crates/ai-memory-mcp/tests/suite/admin_reclaim_ledger_versions.rs b/crates/ai-memory-mcp/tests/suite/admin_reclaim_ledger_versions.rs
new file mode 100644
index 00000000..887234d5
--- /dev/null
+++ b/crates/ai-memory-mcp/tests/suite/admin_reclaim_ledger_versions.rs
@@ -0,0 +1,205 @@
+//! `POST /admin/reclaim-ledger-versions` is a new destructive entry point, so
+//! it gets the adversarial treatment [`docs/security-boundaries.md`] requires:
+//! attempt the violation, assert the refusal, and include a legitimate
+//! control.
+//!
+//! The guard under test is the confirm gate. Unlike `compact`, this route
+//! removes page rows, so a request that deletes without saying so must be
+//! refused — and a request that only measures must be allowed through without
+//! `confirm`, because a dry run is precisely the step that lets an operator
+//! decide.
+
+use super::common::{get, post};
+use ai_memory_consolidate::{DEFAULT_CONTRADICTION_SIM_HIGH, DEFAULT_CONTRADICTION_SIM_LOW};
+use ai_memory_core::{NewPage, PagePath, Tier};
+use ai_memory_mcp::AdminState;
+use ai_memory_store::{DecayParams, Store};
+use ai_memory_wiki::Wiki;
+use axum::http::StatusCode;
+use serde_json::json;
+use tempfile::TempDir;
+
+async fn make_state(tmp: &TempDir) -> (AdminState, Store) {
+ let store = Store::open(tmp.path()).unwrap();
+ let wiki = Wiki::new(tmp.path(), store.writer.clone()).unwrap();
+ let db_path = store.db_path().to_path_buf();
+ let state = AdminState {
+ ingest_metrics: std::sync::Arc::new(ai_memory_core::IngestMetrics::default()),
+ writer: store.writer.clone(),
+ reader: store.reader.clone(),
+ wiki,
+ llm: None,
+ auto_improve_require_approval: false,
+ auto_improve_review_config: Default::default(),
+ embedder: None,
+ provider_health: ai_memory_llm::ProviderHealth::default(),
+ decay_params: DecayParams::default(),
+ contradiction_band_min: DEFAULT_CONTRADICTION_SIM_LOW,
+ contradiction_band_max: DEFAULT_CONTRADICTION_SIM_HIGH,
+ data_dir: tmp.path().to_path_buf(),
+ bind: "127.0.0.1:0".to_string(),
+ home_dir: None,
+ bootstrap_lock: std::sync::Arc::new(tokio::sync::Mutex::new(())),
+ token_pepper: None,
+ active_project: ai_memory_core::ActiveProject::new(),
+ scope_invalidator: None,
+ trusted_proxy_identity: false,
+ db_path,
+ };
+ (state, store)
+}
+
+/// Total `pages` rows, read back through the public reader rather than the
+/// handle the writer used, so a passing assertion is not the writer agreeing
+/// with itself.
+async fn page_rows(store: &Store) -> u64 {
+ store.reader.status_counts().await.unwrap().pages_all
+}
+
+async fn body_json(resp: axum::response::Response) -> serde_json::Value {
+ let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
+ .await
+ .unwrap();
+ serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null)
+}
+
+/// Seed a ledger with two versions, so there is residue to reclaim.
+async fn seed_ledger(store: &Store) -> ai_memory_core::ProjectId {
+ let ws = store
+ .writer
+ .get_or_create_workspace("default")
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app", None)
+ .await
+ .unwrap();
+ for stamp in ["2026-09-05T00:00:00Z", "2026-09-06T00:00:00Z"] {
+ store
+ .writer
+ .upsert_page(NewPage {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new("log-2026-09.md").unwrap(),
+ title: "t".into(),
+ body: format!("## [{stamp}] tool Bash: cargo test\n"),
+ tier: Tier::Episodic,
+ frontmatter_json: serde_json::json!({}),
+ pinned: false,
+ links: Vec::new(),
+ author_id: None,
+ expires_at: None,
+ entities: Vec::new(),
+ evidence: Vec::new(),
+ })
+ .await
+ .unwrap();
+ }
+ proj
+}
+
+#[tokio::test]
+async fn deleting_without_confirm_is_refused_and_changes_nothing() {
+ let tmp = TempDir::new().unwrap();
+ let (state, store) = make_state(&tmp).await;
+ seed_ledger(&store).await;
+ let before = page_rows(&store).await;
+ assert_eq!(before, 2);
+
+ let resp = post(
+ state,
+ "/admin/reclaim-ledger-versions",
+ json!({ "confirm": false }),
+ )
+ .await;
+
+ assert_eq!(
+ resp.status(),
+ StatusCode::BAD_REQUEST,
+ "a deleting run must say so"
+ );
+ assert_eq!(
+ page_rows(&store).await,
+ before,
+ "the refusal has to be a real refusal, not an error raised after the fact",
+ );
+}
+
+/// `confirm` has no serde default, so a body that omits it is rejected by the
+/// extractor before the handler runs — the same fail-closed shape
+/// `/admin/compact` uses. A request that never states its intent cannot
+/// delete.
+#[tokio::test]
+async fn omitting_confirm_entirely_is_refused() {
+ let tmp = TempDir::new().unwrap();
+ let (state, store) = make_state(&tmp).await;
+ seed_ledger(&store).await;
+
+ let resp = post(state, "/admin/reclaim-ledger-versions", json!({})).await;
+
+ assert!(
+ resp.status().is_client_error(),
+ "expected a client error, got {}",
+ resp.status(),
+ );
+ assert_eq!(page_rows(&store).await, 2);
+}
+
+/// Legitimate control: the measuring run needs no `confirm`, because it
+/// changes nothing — and that is the step the CLI puts in front of `--confirm`.
+#[tokio::test]
+async fn a_dry_run_is_allowed_without_confirm_and_reports_the_residue() {
+ let tmp = TempDir::new().unwrap();
+ let (state, store) = make_state(&tmp).await;
+ seed_ledger(&store).await;
+
+ let resp = post(
+ state,
+ "/admin/reclaim-ledger-versions",
+ json!({ "confirm": false, "dry_run": true }),
+ )
+ .await;
+
+ assert_eq!(resp.status(), StatusCode::OK);
+ let body = body_json(resp).await;
+ assert_eq!(body["ledger_paths"], 1);
+ assert_eq!(body["pages_deleted"], 1);
+ assert!(body["bytes_deleted"].as_u64().unwrap() > 0);
+ assert_eq!(page_rows(&store).await, 2, "a dry run deletes nothing");
+}
+
+/// Legitimate control: the confirmed run goes through, and takes the superseded
+/// version only.
+#[tokio::test]
+async fn the_confirmed_run_reclaims_the_superseded_version() {
+ let tmp = TempDir::new().unwrap();
+ let (state, store) = make_state(&tmp).await;
+ seed_ledger(&store).await;
+
+ let resp = post(
+ state,
+ "/admin/reclaim-ledger-versions",
+ json!({ "confirm": true }),
+ )
+ .await;
+
+ assert_eq!(resp.status(), StatusCode::OK);
+ let body = body_json(resp).await;
+ assert_eq!(body["pages_deleted"], 1);
+ assert_eq!(body["dropped_latest"], false);
+ assert_eq!(page_rows(&store).await, 1);
+}
+
+#[tokio::test]
+async fn get_is_not_a_valid_method_for_this_route() {
+ let tmp = TempDir::new().unwrap();
+ let (state, _store) = make_state(&tmp).await;
+
+ let resp = get(state, "/admin/reclaim-ledger-versions").await;
+ assert!(
+ resp.status().is_client_error(),
+ "the route is POST-only, got {}",
+ resp.status(),
+ );
+}
diff --git a/crates/ai-memory-mcp/tests/suite/mod.rs b/crates/ai-memory-mcp/tests/suite/mod.rs
index 434bcfdb..f3226d8c 100644
--- a/crates/ai-memory-mcp/tests/suite/mod.rs
+++ b/crates/ai-memory-mcp/tests/suite/mod.rs
@@ -15,6 +15,7 @@ mod admin_phase3;
mod admin_provider_error_logging;
mod admin_purge;
mod admin_read_page;
+mod admin_reclaim_ledger_versions;
mod admin_rename;
mod admin_status_search;
mod admin_write_page;
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index 4d57cfb8..91401c20 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -4177,6 +4177,282 @@ pub(crate) fn database_bytes(conn: &Connection) -> StoreResult<u64> {
Ok((page_count.max(0) as u64).saturating_mul(page_size.max(0) as u64))
}
+/// Rows per `DELETE` statement. Bounded so a 50k-row cleanup does not build
+/// one enormous statement, and so a `LIMIT`-less delete never holds a
+/// statement journal entry proportional to the whole residue.
+const LEDGER_DELETE_CHUNK: usize = 256;
+
+/// SQL selecting the residue #660 left behind: superseded versions of a
+/// ledger path that no decay tombstone owns.
+///
+/// - `is_latest = 0` — only superseded versions; the live row of a ledger is
+/// what the next hook append reads, and it is dropped only on request.
+/// - `superseded_at IS NULL` — only `decay` writes that column, so this is
+/// exactly the set `forget-sweep` cannot reach. Rows a decay tombstone owns
+/// are left to the sweep that already handles them.
+/// - the path predicate is the shared ledger filename gate, narrowed to the
+/// two shapes it accepts. `[0-9]` rather than `?` so a `log-abcd-ef.md`
+/// cannot widen the candidate set. The content gate runs in Rust.
+///
+/// - `length(body)` rides along so the report can state the bytes without
+/// reading them; `substr` bounds what the gate sees to
+/// [`ai_memory_core::log_ledger::GATE_PREFIX_BYTES`], because these rows
+/// carry whole ledger bodies. A prefix too short to decide leaves the row
+/// alone rather than deleting it.
+const LEDGER_RESIDUE_SQL: &str = "\
+SELECT id, workspace_id, project_id, path, length(body), substr(body, 1, ?1) \
+ FROM pages \
+ WHERE is_latest = 0 \
+ AND superseded_at IS NULL \
+ AND (path = 'log.md' OR path GLOB 'log-[0-9][0-9][0-9][0-9]-[0-9][0-9].md')";
+
+/// What one [`reclaim_ledger_versions`] run found, and removed unless
+/// `dry_run`. The counts are identical either way, so a dry run is a real
+/// measurement of what the confirmed run would do.
+#[derive(Debug, Default, Clone, PartialEq, Eq)]
+pub struct ReclaimLedgerVersionsSummary {
+ /// Ledger `(workspace, project, path)` coordinates holding residue.
+ pub ledger_paths: u64,
+ /// `pages` rows selected for deletion.
+ pub pages_deleted: u64,
+ /// Bytes of page body those rows carried. Logical payload, not freed
+ /// bytes: the file only shrinks when `compacted` ran.
+ pub bytes_deleted: u64,
+ /// Whether the live row of each ledger went too.
+ pub dropped_latest: bool,
+ /// Database size in bytes before the delete.
+ pub bytes_before: u64,
+ /// Database size in bytes afterwards.
+ pub bytes_after: u64,
+ /// Whether the freed bytes were reclaimed (`VACUUM` ran).
+ pub compacted: bool,
+ /// Whether the run was a dry run and changed nothing.
+ pub dry_run: bool,
+}
+
+impl ReclaimLedgerVersionsSummary {
+ /// Bytes returned to the filesystem. Zero unless `compacted`: a delete
+ /// without a `VACUUM` leaves its bytes in free pages, exactly as for any
+ /// other SQLite delete.
+ #[must_use]
+ pub fn bytes_reclaimed(&self) -> u64 {
+ if self.compacted {
+ self.bytes_before.saturating_sub(self.bytes_after)
+ } else {
+ 0
+ }
+ }
+}
+
+/// Delete the superseded ledger page versions the pre-#660 indexer wrote, and
+/// nothing else.
+///
+/// # Why this exists
+///
+/// #660 (2.1.1) stopped the indexer from superseding an OKF-conformed ledger
+/// on every hook append, because each append rewrote the whole row. The fix
+/// stops new rows; the rows already written stay, and nothing removed them:
+/// `compact` deletes nothing, `forget-sweep` only hard-deletes the ancestry of
+/// decay tombstones (only `decay` writes `superseded_at`), and `reindex` loses
+/// the DB-only state an operator is trying to keep. One reported store reached
+/// 6,539 versions and 14 GB of ledger rows for 101 live pages.
+///
+/// Every one of those rows is a byte prefix of the version after it, so
+/// deleting them loses nothing the file on disk does not already hold.
+///
+/// # Scope: a path is a ledger only if its content says so
+///
+/// The filename gate alone is not enough, and getting it wrong is the
+/// expensive direction: a prose page a human named `log-2026-09.md` wears the
+/// same shape as the ledger, and deleting its version chain destroys a real
+/// page's history (invariant #16 — a divergent write supersedes, it does not
+/// destroy). So the shared content gate from
+/// [`ai_memory_core::log_ledger`] decides, and it is the same gate the
+/// indexer, the OKF migration and the bundle export use. A real page's
+/// supersession chain is never a candidate.
+///
+/// # Derived rows
+///
+/// Nothing is deleted by hand. `page_embeddings`, `links`,
+/// `cross_project_links`, `entity_page_links` and `page_feedback` all carry
+/// `ON DELETE CASCADE` from `pages`, and `PRAGMA foreign_keys` is on, so one
+/// delete takes the vector, link, entity and feedback rows with it.
+///
+/// # Why the FTS delete trigger is dropped
+///
+/// `pages_fts` is an external-content FTS5 table, so dropping the trigger
+/// would leave the deleted rows' tokens in the index. Re-issuing them through
+/// the trigger is what makes an ordinary delete work — and on these rows it
+/// means re-tokenizing tens of gigabytes of ledger body, one row at a time,
+/// which is the cost the reporter hit. So the trigger's own SQL is read from
+/// `sqlite_master`, dropped for the delete, and re-executed afterwards, and
+/// `pages_fts` is rebuilt wholesale. Reading the DDL back rather than
+/// restating it keeps this honest if the index's shape ever changes again.
+///
+/// # `drop_latest`
+///
+/// Since #660 the indexer skips ledgers, so each ledger's live row is also
+/// left over from before the fix. It is kept by default — it is the version
+/// the file on disk corresponds to, and dropping it is the operator's call,
+/// not this command's.
+///
+/// # Cost
+///
+/// The FTS rebuild is a whole-index operation and `VACUUM` (under
+/// [`Compaction::Reclaim`]) rewrites the whole file and takes an exclusive
+/// lock, exactly as [`compact`] does. `VACUUM` needs free disk of roughly the
+/// database's own size.
+///
+/// # Errors
+/// Propagates the SQL error from the scan, the delete, the rebuild or the
+/// `VACUUM`. The delete runs in one transaction, so a failure leaves both the
+/// rows and the FTS trigger as they were.
+pub fn reclaim_ledger_versions(
+ conn: &mut Connection,
+ dry_run: bool,
+ drop_latest: bool,
+ compaction: Compaction,
+) -> StoreResult<ReclaimLedgerVersionsSummary> {
+ let bytes_before = database_bytes(conn)?;
+
+ // Collect first, delete second: the content gate is per-row, and deciding
+ // it while deleting would mean re-reading rows a rollback just restored.
+ let mut doomed: Vec<PageId> = Vec::new();
+ let mut ledger_paths: BTreeSet<(String, String, String)> = BTreeSet::new();
+ let mut bytes_deleted: u64 = 0;
+
+ {
+ let mut stmt = conn.prepare(LEDGER_RESIDUE_SQL)?;
+ let mut rows = stmt.query(params![
+ ai_memory_core::log_ledger::GATE_PREFIX_BYTES as i64
+ ])?;
+ while let Some(row) = rows.next()? {
+ let body_prefix: String = row.get(5)?;
+ if !ai_memory_core::log_ledger::body_opens_with_log_ledger(&body_prefix) {
+ continue;
+ }
+ let id: Vec<u8> = row.get(0)?;
+ let workspace: Vec<u8> = row.get(1)?;
+ let project: Vec<u8> = row.get(2)?;
+ let path: String = row.get(3)?;
+ let body_len: i64 = row.get(4)?;
+ ledger_paths.insert((
+ WorkspaceId::from_slice(&workspace)?.to_string(),
+ ProjectId::from_slice(&project)?.to_string(),
+ path,
+ ));
+ bytes_deleted = bytes_deleted.saturating_add(body_len.max(0) as u64);
+ doomed.push(PageId::from_slice(&id)?);
+ }
+ }
+
+ let mut summary = ReclaimLedgerVersionsSummary {
+ ledger_paths: ledger_paths.len() as u64,
+ pages_deleted: doomed.len() as u64,
+ bytes_deleted,
+ dropped_latest: false,
+ bytes_before,
+ bytes_after: bytes_before,
+ compacted: false,
+ dry_run,
+ };
+
+ if drop_latest {
+ // The live row of each ledger the gate already approved. Collected by
+ // the same gate, so a prose page wearing the ledger name is still
+ // untouched.
+ let mut stmt = conn.prepare(
+ "SELECT id, length(body), substr(body, 1, ?1), workspace_id, project_id, path \
+ FROM pages \
+ WHERE is_latest = 1 \
+ AND (path = 'log.md' OR path GLOB 'log-[0-9][0-9][0-9][0-9]-[0-9][0-9].md')",
+ )?;
+ let mut rows = stmt.query(params![
+ ai_memory_core::log_ledger::GATE_PREFIX_BYTES as i64
+ ])?;
+ let mut latest: Vec<PageId> = Vec::new();
+ while let Some(row) = rows.next()? {
+ let body_prefix: String = row.get(2)?;
+ if !ai_memory_core::log_ledger::body_opens_with_log_ledger(&body_prefix) {
+ continue;
+ }
+ let id: Vec<u8> = row.get(0)?;
+ latest.push(PageId::from_slice(&id)?);
+ bytes_deleted = bytes_deleted.saturating_add(row.get::<_, i64>(1)?.max(0) as u64);
+ }
+ summary.pages_deleted += latest.len() as u64;
+ summary.bytes_deleted = bytes_deleted;
+ summary.dropped_latest = !latest.is_empty();
+ doomed.extend(latest);
+ }
+
+ if dry_run {
+ return Ok(summary);
+ }
+
+ if !doomed.is_empty() {
+ delete_pages_without_fts_retokenizing(conn, &doomed)?;
+ }
+
+ summary.bytes_after = database_bytes(conn)?;
+ if compaction == Compaction::Reclaim && !doomed.is_empty() {
+ // `VACUUM` cannot run inside a transaction, and it is the whole-file
+ // rewrite `compact` already documents the cost of.
+ conn.execute_batch("VACUUM;")?;
+ summary.compacted = true;
+ summary.bytes_after = database_bytes(conn)?;
+ }
+ Ok(summary)
+}
+
+/// Delete `ids` from `pages` with the FTS delete trigger stood down, then put
+/// the trigger back and rebuild `pages_fts`.
+///
+/// The trigger's DDL is read from `sqlite_master` rather than restated, so a
+/// later change to the index's shape cannot leave this dropping a trigger that
+/// no longer matches the one it recreates. The drop, the delete and the
+/// re-create share one transaction: a failure rolls back to the original
+/// trigger with the rows intact.
+fn delete_pages_without_fts_retokenizing(conn: &mut Connection, ids: &[PageId]) -> StoreResult<()> {
+ let trigger_sql: Option<String> = conn
+ .query_row(
+ "SELECT sql FROM sqlite_master WHERE type = 'trigger' AND name = 'pages_fts_ad'",
+ [],
+ |r| r.get(0),
+ )
+ .ok();
+
+ let tx = conn.transaction()?;
+
+ if let Some(sql) = trigger_sql.as_deref() {
+ tx.execute_batch("DROP TRIGGER pages_fts_ad;")?;
+ for chunk in ids.chunks(LEDGER_DELETE_CHUNK) {
+ let mut del = tx.prepare("DELETE FROM pages WHERE id = ?1")?;
+ for id in chunk {
+ del.execute(params![id.as_bytes()])?;
+ }
+ }
+ tx.execute_batch(sql)?;
+ } else {
+ // No trigger to stand down: an ordinary delete already leaves the
+ // index consistent, so do not invent a different path for it.
+ for chunk in ids.chunks(LEDGER_DELETE_CHUNK) {
+ let mut del = tx.prepare("DELETE FROM pages WHERE id = ?1")?;
+ for id in chunk {
+ del.execute(params![id.as_bytes()])?;
+ }
+ }
+ }
+
+ tx.commit()?;
+
+ // The rows are gone but their tokens are not: an external-content FTS5
+ // table keeps them in its segments until a rebuild. One rebuild for the
+ // whole index instead of a delete command per row.
+ conn.execute_batch("INSERT INTO pages_fts(pages_fts) VALUES('rebuild');")?;
+ Ok(())
+}
+
/// What [`purge_session`] removed. All counts are rows actually deleted.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub struct PurgeSessionSummary {
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index 7aae7210..07901cd7 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -397,6 +397,17 @@ pub(crate) enum WriteCmd {
Compact {
reply: oneshot::Sender<StoreResult<crate::ops::CompactSummary>>,
},
+ /// Delete the superseded ledger page versions the pre-#660 indexer left
+ /// behind, and nothing else. See [`ops::reclaim_ledger_versions`].
+ ReclaimLedgerVersions {
+ /// Report what would go without deleting it.
+ dry_run: bool,
+ /// Also drop each ledger's live row, not just its superseded versions.
+ drop_latest: bool,
+ /// Whether to reclaim the freed bytes afterwards (`VACUUM`).
+ compaction: crate::ops::Compaction,
+ reply: oneshot::Sender<StoreResult<crate::ops::ReclaimLedgerVersionsSummary>>,
+ },
/// Delete a workspace row (its `workspace_id` FKs cascade projects/pages/
/// sessions/…). Refused when non-empty unless `force`.
DeleteWorkspace {
@@ -1845,6 +1856,29 @@ impl WriterHandle {
rx.await.map_err(|_| StoreError::WriterClosed)?
}
+ /// Delete the superseded ledger page versions the pre-#660 indexer left
+ /// behind. See [`ops::reclaim_ledger_versions`].
+ ///
+ /// # Errors
+ /// Propagates the store error from the scan, delete, FTS rebuild or
+ /// `VACUUM`, plus [`StoreError::WriterClosed`].
+ pub async fn reclaim_ledger_versions(
+ &self,
+ dry_run: bool,
+ drop_latest: bool,
+ compaction: crate::ops::Compaction,
+ ) -> StoreResult<crate::ops::ReclaimLedgerVersionsSummary> {
+ let (tx, rx) = oneshot::channel();
+ self.send(WriteCmd::ReclaimLedgerVersions {
+ dry_run,
+ drop_latest,
+ compaction,
+ reply: tx,
+ })
+ .await?;
+ rx.await.map_err(|_| StoreError::WriterClosed)?
+ }
+
/// Rename a workspace (column-only; the on-disk dir is UUID-keyed).
///
/// # Errors
@@ -3345,6 +3379,16 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
let result = ops::compact(&mut conn);
send_or_warn(reply, result, "compact");
}
+ WriteCmd::ReclaimLedgerVersions {
+ dry_run,
+ drop_latest,
+ compaction,
+ reply,
+ } => {
+ let result =
+ ops::reclaim_ledger_versions(&mut conn, dry_run, drop_latest, compaction);
+ send_or_warn(reply, result, "reclaim_ledger_versions");
+ }
WriteCmd::DeleteWorkspace {
workspace_id,
force,
diff --git a/crates/ai-memory-store/tests/suite/mod.rs b/crates/ai-memory-store/tests/suite/mod.rs
index e4a1745b..903fc0e1 100644
--- a/crates/ai-memory-store/tests/suite/mod.rs
+++ b/crates/ai-memory-store/tests/suite/mod.rs
@@ -15,6 +15,7 @@ mod most_recently_active_scope;
mod multi_session;
mod pinned_pages;
mod project_authz;
+mod reclaim_ledger_versions;
mod related_walk;
mod retrieval_superseded;
mod retrieval_tuning_streams;
diff --git a/crates/ai-memory-store/tests/suite/reclaim_ledger_versions.rs b/crates/ai-memory-store/tests/suite/reclaim_ledger_versions.rs
new file mode 100644
index 00000000..99a03d75
--- /dev/null
+++ b/crates/ai-memory-store/tests/suite/reclaim_ledger_versions.rs
@@ -0,0 +1,585 @@
+//! `reclaim_ledger_versions` must drop the superseded ledger versions the
+//! pre-#660 indexer wrote, and must not touch anything else.
+//!
+//! Every test here is a fail-before/pass-after case for a destructive
+//! operation, so each one states what must survive as loudly as what must go.
+//! The content gate is the load-bearing part: a real page a human named
+//! `log-2026-09.md` wears the ledger's filename shape, and deleting its version
+//! chain would destroy a real page's history (invariant #16).
+
+use ai_memory_core::{NewPage, PagePath, Tier};
+use ai_memory_store::{Compaction, Store};
+
+/// A hook event ledger body, exactly as the hooks append it.
+fn ledger_body(stamp: &str) -> String {
+ format!("## [{stamp}] tool Bash: cargo test\n")
+}
+
+/// The same ledger after the OKF v0.2 migration stamped frontmatter on it.
+fn conformed_ledger_body(stamp: &str) -> String {
+ format!("---\ntype: Note\ntitle: Log 2026-09\n---\n\n## [{stamp}] tool Bash: cargo test\n")
+}
+
+/// An ordinary page that merely wears the ledger's filename.
+fn prose_body() -> String {
+ "# Log 2026-09\n\nA hand-written page that happens to be named like the ledger.\n".to_string()
+}
+
+async fn write_page(
+ store: &Store,
+ ws: ai_memory_core::WorkspaceId,
+ proj: ai_memory_core::ProjectId,
+ path: &str,
+ body: &str,
+) {
+ store
+ .writer
+ .upsert_page(NewPage {
+ workspace_id: ws,
+ project_id: proj,
+ path: PagePath::new(path).unwrap(),
+ title: "t".into(),
+ body: body.into(),
+ tier: Tier::Episodic,
+ frontmatter_json: serde_json::json!({}),
+ pinned: false,
+ links: Vec::new(),
+ author_id: None,
+ expires_at: None,
+ entities: Vec::new(),
+ evidence: Vec::new(),
+ })
+ .await
+ .unwrap();
+}
+
+/// Rows in `pages` for `path`, newest first, as `(is_latest, body)`.
+fn versions(store: &Store, proj: ai_memory_core::ProjectId, path: &str) -> Vec<(i64, String)> {
+ let conn = rusqlite::Connection::open(store.db_path()).unwrap();
+ let mut stmt = conn
+ .prepare(
+ "SELECT is_latest, body FROM pages \
+ WHERE project_id = ?1 AND path = ?2 ORDER BY created_at DESC, rowid DESC",
+ )
+ .unwrap();
+ let rows = stmt
+ .query_map(rusqlite::params![proj.as_bytes(), path], |r| {
+ Ok((r.get(0)?, r.get(1)?))
+ })
+ .unwrap();
+ rows.map(std::result::Result::unwrap).collect()
+}
+
+#[tokio::test]
+async fn a_dry_run_reports_the_residue_and_deletes_nothing() {
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-05T00:00:00Z"),
+ )
+ .await;
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-06T00:00:00Z"),
+ )
+ .await;
+ assert_eq!(versions(&store, proj, "log-2026-09.md").len(), 2);
+
+ let report = store
+ .writer
+ .reclaim_ledger_versions(true, false, Compaction::Skip)
+ .await
+ .unwrap();
+
+ assert!(report.dry_run);
+ assert_eq!(report.ledger_paths, 1);
+ assert_eq!(report.pages_deleted, 1, "only the superseded version");
+ assert!(report.bytes_deleted > 0);
+ assert!(!report.dropped_latest);
+ assert!(!report.compacted);
+ assert_eq!(report.bytes_reclaimed(), 0, "a dry run reclaims nothing");
+ assert_eq!(
+ versions(&store, proj, "log-2026-09.md").len(),
+ 2,
+ "a dry run changes nothing",
+ );
+}
+
+#[tokio::test]
+async fn the_confirmed_run_drops_only_the_superseded_ledger_version() {
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-05T00:00:00Z"),
+ )
+ .await;
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-06T00:00:00Z"),
+ )
+ .await;
+
+ let report = store
+ .writer
+ .reclaim_ledger_versions(false, false, Compaction::Skip)
+ .await
+ .unwrap();
+
+ assert!(!report.dry_run);
+ assert_eq!(report.pages_deleted, 1);
+
+ let left = versions(&store, proj, "log-2026-09.md");
+ assert_eq!(left.len(), 1, "the live ledger version survives");
+ assert_eq!(left[0].0, 1, "and it is the latest one");
+ assert!(
+ left[0].1.contains("2026-09-06"),
+ "the surviving version is the one the file holds now",
+ );
+}
+
+#[tokio::test]
+async fn a_prose_page_wearing_the_ledger_name_keeps_its_whole_version_chain() {
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+
+ // A divergent write: two versions of a real page at the ledger's path.
+ write_page(&store, ws, proj, "log-2026-09.md", &prose_body()).await;
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ "# Log 2026-09\n\nSecond, hand-written revision.\n",
+ )
+ .await;
+
+ let report = store
+ .writer
+ .reclaim_ledger_versions(false, true, Compaction::Reclaim)
+ .await
+ .unwrap();
+
+ assert_eq!(
+ report.pages_deleted, 0,
+ "the filename alone is not a ledger"
+ );
+ assert_eq!(report.ledger_paths, 0);
+ assert_eq!(
+ versions(&store, proj, "log-2026-09.md").len(),
+ 2,
+ "a real page's supersession chain is never a candidate",
+ );
+}
+
+#[tokio::test]
+async fn an_ordinary_page_chain_in_another_project_is_untouched() {
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let other = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+ let sidecar = store
+ .writer
+ .get_or_create_project(ws, "sidecar".to_string(), None)
+ .await
+ .unwrap();
+
+ // The same ledger path, in two projects, one of them a real page. A
+ // cleanup that keyed on the path alone would take the wrong rows.
+ write_page(
+ &store,
+ ws,
+ other,
+ "log-2026-09.md",
+ &ledger_body("2026-09-05T00:00:00Z"),
+ )
+ .await;
+ write_page(
+ &store,
+ ws,
+ other,
+ "log-2026-09.md",
+ &ledger_body("2026-09-06T00:00:00Z"),
+ )
+ .await;
+ write_page(&store, ws, sidecar, "log-2026-09.md", &prose_body()).await;
+ write_page(
+ &store,
+ ws,
+ sidecar,
+ "log-2026-09.md",
+ "# Log 2026-09\n\nSecond revision.\n",
+ )
+ .await;
+ write_page(&store, ws, sidecar, "notes/keep.md", "a real page").await;
+ write_page(&store, ws, sidecar, "notes/keep.md", "a real page, revised").await;
+
+ store
+ .writer
+ .reclaim_ledger_versions(false, false, Compaction::Skip)
+ .await
+ .unwrap();
+
+ assert_eq!(
+ versions(&store, sidecar, "log-2026-09.md").len(),
+ 2,
+ "the other project's real page keeps its chain",
+ );
+ assert_eq!(
+ versions(&store, sidecar, "notes/keep.md").len(),
+ 2,
+ "an ordinary page is never a candidate at all",
+ );
+ assert_eq!(versions(&store, other, "log-2026-09.md").len(), 1);
+}
+
+#[tokio::test]
+async fn drop_latest_removes_the_ledger_entirely() {
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-05T00:00:00Z"),
+ )
+ .await;
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-06T00:00:00Z"),
+ )
+ .await;
+
+ let report = store
+ .writer
+ .reclaim_ledger_versions(false, true, Compaction::Skip)
+ .await
+ .unwrap();
+
+ assert!(report.dropped_latest);
+ assert_eq!(report.pages_deleted, 2, "superseded version and live row");
+ assert!(
+ versions(&store, proj, "log-2026-09.md").is_empty(),
+ "the ledger is gone, so the next hook append starts a fresh chain",
+ );
+}
+
+#[tokio::test]
+async fn a_migrated_ledger_is_still_a_ledger() {
+ // #669 stamps OKF frontmatter on every `.md`, ledgers included. Stopping
+ // at the fence would classify a migrated ledger as an ordinary page and
+ // leave the residue behind — or worse, mislabel a real page as one.
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-05T00:00:00Z"),
+ )
+ .await;
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &conformed_ledger_body("2026-09-06T00:00:00Z"),
+ )
+ .await;
+
+ let report = store
+ .writer
+ .reclaim_ledger_versions(false, false, Compaction::Skip)
+ .await
+ .unwrap();
+
+ assert_eq!(
+ report.pages_deleted, 1,
+ "frontmatter does not hide the ledger"
+ );
+ assert_eq!(versions(&store, proj, "log-2026-09.md").len(), 1);
+}
+
+#[tokio::test]
+async fn the_fts_delete_trigger_survives_a_reclaim_byte_for_byte() {
+ // The reclaim stands the trigger down for the bulk delete and puts the
+ // DDL it read back from `sqlite_master` afterwards. If that read ever
+ // stopped matching what is in the schema, the store would be left with no
+ // delete propagation at all, and that is silent until the next delete.
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+
+ let trigger_sql = |store: &Store| -> Option<String> {
+ let conn = rusqlite::Connection::open(store.db_path()).unwrap();
+ conn.query_row(
+ "SELECT sql FROM sqlite_master WHERE type = 'trigger' AND name = 'pages_fts_ad'",
+ [],
+ |r| r.get(0),
+ )
+ .ok()
+ };
+ let before = trigger_sql(&store);
+ assert!(before.is_some(), "the schema ships this trigger");
+
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-05T00:00:00Z"),
+ )
+ .await;
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-06T00:00:00Z"),
+ )
+ .await;
+
+ store
+ .writer
+ .reclaim_ledger_versions(false, false, Compaction::Skip)
+ .await
+ .unwrap();
+
+ assert_eq!(
+ trigger_sql(&store),
+ before,
+ "the trigger is restored exactly"
+ );
+}
+
+#[tokio::test]
+async fn a_reclaimed_ledger_row_leaves_no_searchable_text_behind() {
+ // The trigger was stood down, so nothing propagated the delete into the
+ // index. This is the check that the wholesale rebuild actually ran: the
+ // superseded body must be unfindable, and the live one must still be
+ // findable.
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ "## [2026-09-05T00:00:00Z] tool Bash: zzzsupersededtokenzzz\n",
+ )
+ .await;
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ "## [2026-09-06T00:00:00Z] tool Bash: yyycurrenttokenyyy\n",
+ )
+ .await;
+
+ store
+ .writer
+ .reclaim_ledger_versions(false, false, Compaction::Skip)
+ .await
+ .unwrap();
+
+ let drift = store.reader.derived_index_status().await.unwrap();
+ assert_eq!(
+ drift.pages_fts_rows, drift.pages_rows,
+ "the rebuilt index agrees with the content table again",
+ );
+
+ let conn = rusqlite::Connection::open(store.db_path()).unwrap();
+ let hits = |needle: &str| -> i64 {
+ conn.query_row(
+ "SELECT COUNT(*) FROM pages_fts WHERE pages_fts MATCH ?1",
+ rusqlite::params![needle],
+ |r| r.get(0),
+ )
+ .unwrap()
+ };
+ assert_eq!(hits("zzzsupersededtokenzzz"), 0, "the dropped body is gone");
+ assert_eq!(
+ hits("yyycurrenttokenyyy"),
+ 1,
+ "the live body is still indexed"
+ );
+}
+
+#[tokio::test]
+async fn a_decay_tombstone_stays_with_the_sweep_that_owns_it() {
+ // Only `decay` writes `superseded_at`, and `forget-sweep` hard-deletes
+ // that ancestry. Taking those rows here would rob the sweep of the state
+ // it reasons about, so the residue query excludes them.
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+ let proj = store
+ .writer
+ .get_or_create_project(ws, "app".to_string(), None)
+ .await
+ .unwrap();
+
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-05T00:00:00Z"),
+ )
+ .await;
+ write_page(
+ &store,
+ ws,
+ proj,
+ "log-2026-09.md",
+ &ledger_body("2026-09-06T00:00:00Z"),
+ )
+ .await;
+ {
+ let conn = rusqlite::Connection::open(store.db_path()).unwrap();
+ conn.execute(
+ "UPDATE pages SET superseded_at = created_at WHERE is_latest = 0",
+ [],
+ )
+ .unwrap();
+ }
+
+ let report = store
+ .writer
+ .reclaim_ledger_versions(false, false, Compaction::Skip)
+ .await
+ .unwrap();
+
+ assert_eq!(
+ report.pages_deleted, 0,
+ "decay's rows are not this command's"
+ );
+ assert_eq!(versions(&store, proj, "log-2026-09.md").len(), 2);
+}
+
+#[tokio::test]
+async fn an_empty_store_reports_nothing_and_touches_nothing() {
+ let tmp = tempfile::tempdir().unwrap();
+ let store = Store::open(tmp.path()).unwrap();
+ let _ws = store
+ .writer
+ .get_or_create_workspace("default".to_string())
+ .await
+ .unwrap();
+
+ let report = store
+ .writer
+ .reclaim_ledger_versions(false, true, Compaction::Reclaim)
+ .await
+ .unwrap();
+
+ assert_eq!(report.ledger_paths, 0);
+ assert_eq!(report.pages_deleted, 0);
+ assert!(!report.dropped_latest);
+ assert!(!report.compacted, "no rows went, so no VACUUM was needed");
+}
diff --git a/crates/ai-memory-wiki/src/ledger.rs b/crates/ai-memory-wiki/src/ledger.rs
index 3b09dec4..79e13f33 100644
--- a/crates/ai-memory-wiki/src/ledger.rs
+++ b/crates/ai-memory-wiki/src/ledger.rs
@@ -6,79 +6,60 @@
//! reserved-looking filename is only a ledger when its content opens with a
//! hook log entry, never on filename alone (an ordinary page can be named
//! `log-2026-09.md`).
+//!
+//! The predicates themselves live in [`ai_memory_core::log_ledger`], because
+//! the store's `reclaim-ledger-versions` has to answer the same question from
+//! a `pages.body` it already holds, with no filesystem access. This module is
+//! only the file-reading half of that contract.
use std::path::Path;
use ai_memory_core::PagePath;
/// `log.md` / `log-YYYY-MM.md` are the raw per-project event ledger the hooks
-/// append to (see `ai-memory-hooks::log::log_filename_for`): `## [ts] ...`
-/// entries, never YAML frontmatter.
+/// append to (see `ai-memory-hooks::log::log_filename_for`).
pub(crate) fn is_log_ledger_filename(page_path: &PagePath) -> bool {
- let s = page_path.as_str();
- s == "log.md" || is_rotated_log_filename(s)
+ ai_memory_core::log_ledger::is_log_ledger_path(page_path.as_str())
}
pub(crate) fn is_rotated_log_filename(s: &str) -> bool {
- let Some(stem) = s.strip_prefix("log-").and_then(|v| v.strip_suffix(".md")) else {
- return false;
- };
- let bytes = stem.as_bytes();
- bytes.len() == "YYYY-MM".len()
- && bytes[4] == b'-'
- && bytes[..4].iter().all(|b| b.is_ascii_digit())
- && bytes[5..].iter().all(|b| b.is_ascii_digit())
+ ai_memory_core::log_ledger::is_rotated_log_ledger_path(s)
}
/// Cheap check for the raw hook event ledger shape. A reserved-looking
/// filename is only a ledger when its first body line is a hook log entry
/// (`## [ts] ...`).
///
-/// The body may sit under a YAML frontmatter block: the OKF v0.2 migration
-/// conforms every `.md` under `wiki/`, ledgers included, so a migrated store
-/// has `type: Note` stamped on top of each `log-YYYY-MM.md`. Stopping at the
-/// fence would classify those ledgers as ordinary pages, and each hook
-/// `append_event` would then supersede a multi-megabyte page row.
+/// Reads only [`GATE_PREFIX_BYTES`] and defers the decision to
+/// [`ai_memory_core::log_ledger::body_opens_with_log_ledger`], so the file and
+/// the in-memory answer cannot drift apart. The bound is also what keeps a
+/// pathological file (a lone opening fence in a multi-gigabyte log) from a
+/// full scan.
pub(crate) fn opens_with_log_ledger(abs: &Path) -> bool {
- use std::io::{BufRead, BufReader};
- let Ok(file) = std::fs::File::open(abs) else {
+ use std::io::Read;
+ let Ok(mut file) = std::fs::File::open(abs) else {
return false;
};
- let mut reader = BufReader::new(file);
- let mut line = String::new();
- if reader.read_line(&mut line).is_err() {
- return false;
- }
- if line.trim_end() != "---" {
- return line.starts_with("## [");
- }
- // Walk past the frontmatter block. The bound keeps a pathological file
- // (a lone opening fence in a multi-gigabyte log) from a full scan.
- let mut closed = false;
- for _ in 0..MAX_FRONTMATTER_LINES {
- line.clear();
- match reader.read_line(&mut line) {
- Ok(0) | Err(_) => return false,
- Ok(_) => {}
- }
- if line.trim_end() == "---" {
- closed = true;
- break;
+ let mut prefix = vec![0u8; ai_memory_core::log_ledger::GATE_PREFIX_BYTES];
+ let read = match file.read(&mut prefix) {
+ Ok(n) => n,
+ Err(_) => return false,
+ };
+ // A file that filled the buffer may have been cut mid-line; a shorter read
+ // is the whole file, last line and all.
+ if read == prefix.len() {
+ match prefix.iter().rposition(|b| *b == b'\n') {
+ Some(last) => prefix.truncate(last + 1),
+ None => prefix.clear(),
}
+ } else {
+ prefix.truncate(read);
}
- if !closed {
- return false;
- }
- // First non-blank line after the fence decides.
- loop {
- line.clear();
- match reader.read_line(&mut line) {
- Ok(0) | Err(_) => return false,
- Ok(_) => {}
- }
- if !line.trim().is_empty() {
- return line.starts_with("## [");
- }
+ match std::str::from_utf8(&prefix) {
+ Ok(text) => ai_memory_core::log_ledger::body_opens_with_log_ledger(text),
+ // A ledger entry is ASCII, so invalid UTF-8 this early means the file
+ // is not one.
+ Err(_) => false,
}
}
@@ -97,7 +78,84 @@ pub fn is_rotated_event_ledger(abs: &Path) -> bool {
&& opens_with_log_ledger(abs)
}
-/// Upper bound on frontmatter lines scanned by [`opens_with_log_ledger`].
-/// Conformant OKF frontmatter is a handful of keys; this is slack, not a
-/// format limit.
-pub(crate) const MAX_FRONTMATTER_LINES: usize = 64;
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use std::io::Write;
+
+ /// Write `body` to a fresh file in a throwaway dir and hand back the path.
+ fn temp_file(dir: &tempfile::TempDir, name: &str, body: &str) -> std::path::PathBuf {
+ let path = dir.path().join(name);
+ let mut f = std::fs::File::create(&path).unwrap();
+ f.write_all(body.as_bytes()).unwrap();
+ path
+ }
+
+ /// The file reader and the in-memory predicate must answer identically:
+ /// the store's `reclaim-ledger-versions` decides from a `pages.body` and
+ /// the indexer decides from this file, and the two surfaces have to agree
+ /// about which paths are ledgers.
+ #[test]
+ fn the_file_and_the_body_answer_the_same_question() {
+ let dir = tempfile::tempdir().unwrap();
+ let cases: &[(&str, &str)] = &[
+ ("bare.md", "## [2026-09-25T10:00:00Z] tool call\n"),
+ (
+ "conformed.md",
+ "---\ntype: Note\ntitle: Log 2026-09\n---\n\n## [2026-09-25T10:00:00Z] tool call\n",
+ ),
+ ("prose.md", "# Log 2026-09\n\nA hand-written page.\n"),
+ ("frontmatter-only.md", "---\ntype: Note\ntitle: Log\n---\n"),
+ ("unterminated.md", "---\ntype: Note\n"),
+ ("empty.md", ""),
+ ("blank-first.md", "\n## [2026-09-25T10:00:00Z] tool call\n"),
+ (
+ "no-trailing-newline.md",
+ "## [2026-09-25T10:00:00Z] tool call",
+ ),
+ ];
+ for (name, body) in cases {
+ let path = temp_file(&dir, name, body);
+ assert_eq!(
+ opens_with_log_ledger(&path),
+ ai_memory_core::log_ledger::body_opens_with_log_ledger(body),
+ "file and body disagree for {name}",
+ );
+ }
+ }
+
+ /// A body longer than the gate prefix must still be classified from its
+ /// first meaningful line, not read whole.
+ #[test]
+ fn a_ledger_longer_than_the_gate_prefix_is_still_a_ledger() {
+ let dir = tempfile::tempdir().unwrap();
+ let filler = "## [2026-09-25T10:00:00Z] tool Bash: cargo test\n".repeat(4000);
+ assert!(filler.len() > ai_memory_core::log_ledger::GATE_PREFIX_BYTES);
+ let path = temp_file(&dir, "long.md", &filler);
+ assert!(opens_with_log_ledger(&path));
+ }
+
+ /// A prefix cut mid-line must not answer for a line it only half read.
+ #[test]
+ fn a_prefix_cut_mid_line_does_not_decide() {
+ let dir = tempfile::tempdir().unwrap();
+ // Push the deciding line past the prefix, with a huge frontmatter
+ // value so the cut lands inside it.
+ let mut body = String::from("---\ntitle: ");
+ body.push_str(&"x".repeat(ai_memory_core::log_ledger::GATE_PREFIX_BYTES));
+ body.push_str("\n---\n\n## [2026-09-25T10:00:00Z] tool call\n");
+ let path = temp_file(&dir, "cut.md", &body);
+ assert!(
+ !opens_with_log_ledger(&path),
+ "an unread deciding line is not a ledger"
+ );
+ }
+
+ /// A missing file is not a ledger, and neither reader may panic on it.
+ #[test]
+ fn a_missing_file_is_not_a_ledger() {
+ let dir = tempfile::tempdir().unwrap();
+ assert!(!opens_with_log_ledger(&dir.path().join("absent.md")));
+ assert!(!is_rotated_event_ledger(&dir.path().join("absent.md")));
+ }
+}
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 4daa8bbe..7d77ccd7 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -543,6 +543,7 @@ upgrade auth user
completions handoffs purge-session
compact api-key export-okf
message doctor backfill
+reclaim-ledger-versions compact
```
Run `ai-memory --help` for the full tree.
@@ -550,6 +551,22 @@ Run `ai-memory --help` for the full tree.
`auto-improve-report` is read-only by default; `--stage` creates one pending
telemetry report page for audit/approval without staging learning-memory edits.
+`reclaim-ledger-versions` drops the superseded versions of the raw hook event
+ledger that the pre-2.1.1 indexer left behind (#660). It is a dry run unless
+`--confirm` is passed. A path is only a candidate when its *content* opens
+with a hook log entry — the same
+`ai_memory_core::log_ledger::body_opens_with_log_ledger` gate the indexer
+(#660), the OKF conformance migration (#669) and the bundle export (#748) use
+— so a real page named `log-2026-09.md` keeps its whole version chain. Only
+`is_latest=0 AND superseded_at IS NULL` rows are eligible, so rows a decay
+tombstone owns stay with `forget-sweep`. Derived FTS/entity/vector/link rows
+go with the page through the existing `ON DELETE CASCADE`s, and the FTS delete
+trigger is stood down for the bulk delete (its DDL is read back from
+`sqlite_master` and re-executed) so the cleanup does not re-tokenize tens of
+gigabytes of ledger body row by row; `pages_fts` is then rebuilt wholesale.
+`--compact` additionally `VACUUM`s to return the bytes, at the cost `compact`
+documents.
+
## Cross-cutting invariants
Carved in M0/M1; every milestone has to respect them. Each comes from
diff --git a/docs/security-boundaries.md b/docs/security-boundaries.md
index 20652893..b0c4ce9f 100644
--- a/docs/security-boundaries.md
+++ b/docs/security-boundaries.md
@@ -43,6 +43,7 @@ boundary not yet built.
| 9 | Scope resolution fail-closed | `ai-memory-store/src/scope.rs` no-create `lookup_existing_*`; create only via `create_explicit_scope` | `scope.rs` no-auto-create + `unscoped_write_with_unresolvable_coordinate_errors` | STRONG |
| 10 | Destructive-op live-process refusal + confirm flags (invariant #9) | `ai-memory-cli/src/commands/process_guard.rs` `sibling_processes` + confirm flags in `reset`/`restore`/`reindex`/`uninstall --purge-data`/`purge_project` | `admin_purge.rs` confirm→400; `removal.rs` — injected live-sibling makes each destructive command bail before touching the data dir | STRONG |
| 10b | Session purge is scope+owner-bound (no cross-session/project over-delete) | `ai-memory-store/src/ops.rs` `purge_session` — selection scoped to `(workspace_id, project_id)` and keyed on this session's own `summary_page_id` **or** `path='sessions/<sid>.md'` + `json_extract(frontmatter_json,'$.session_id')=<sid>` (frontmatter owner, not the recursive latest-chain); `in_scope==0 → NotFound` fail-closed; whole op in one transaction | `ops.rs` `purge_session_leaves_a_sibling_session_in_the_same_project_intact`, `…refuses_a_session_from_another_project_and_deletes_nothing`, `…refuses_a_session_from_another_workspace`, `…does_not_delete_an_identically_pathed_page_in_another_project`, `…removes_older_summary_versions_without_deleting_prior_manual_page` (#862) | STRONG |
+| 10c | Ledger reclaim is content-gated, not filename-gated (invariant #16) | `ai-memory-store/src/ops.rs` `reclaim_ledger_versions` — a candidate row must pass `ai_memory_core::log_ledger::body_opens_with_log_ledger` on its own `body`, and only `is_latest=0 AND superseded_at IS NULL` rows are eligible, so decay-owned rows stay with `forget-sweep`; confirm gate in `admin.rs` `handle_reclaim_ledger_versions` | `reclaim_ledger_versions.rs` `a_prose_page_wearing_the_ledger_name_keeps_its_whole_version_chain`, `an_ordinary_page_chain_in_another_project_is_untouched`, `a_decay_tombstone_stays_with_the_sweep_that_owns_it`; `admin_reclaim_ledger_versions.rs` `deleting_without_confirm_is_refused_and_changes_nothing` (with dry-run and confirmed controls) | STRONG |
| 11a | Hook backpressure (202/429) + bounded fan-out (invariant #5) | `ai-memory-hooks/src/router.rs` semaphore→429, 202 immediately, `MAX_HOOK_BATCH_ITEMS`, bounded LRU limiter | `router.rs` `handle_hook_returns_429_when_ingest_saturated`, `ingest_rate_limiter_is_bounded` | STRONG |
| 11b | Capture exclusions drop before storage | `ai-memory-hooks` `capture_policy.rs` `inspect`→`Drop` (before semaphore/spawn) | `capture_policy.rs` per-agent `…honors_exclusions` tests | STRONG |
| 11c | Capture hook ≤200ms budget (invariant #5) | `hooks/_lib.sh` capture path `curl --max-time 0.2` (context-fetch 1.0s and background drain 2.0s are separate, larger-budget paths) | none (shell-script timeout; hard to unit-test) — watch on any capture-path change | WATCH |