Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(migration): take the safety archive before the DB schema migration (#633)
AkitaOnRails681b1a860ce1
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a80f1671..52defbe1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -35,14 +35,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`dest_free_bytes`, so the same number is visible right before a migration.### Fixed-- Corrected `docs/MIGRATION-2.0.md`, which wrongly promised the automatic- pre-migration archive could roll a store back to 1.x (#633). The archive is- written *after* `Store::open` migrates the SQLite schema forward, so its- `db/` is already at the 2.x schema and a 1.x binary refuses to open it. The- docs now state the archive is a 2.x recovery point and that a genuine 1.x- rollback requires a backup taken with the 1.x binary before upgrading.- (The underlying ordering — snapshotting before the DB migration runs — is- tracked separately.)+- The pre-migration safety archive is now taken **before** the SQLite schema+ is migrated, so it is a genuine pre-2.0 recovery point a 1.x binary can+ reopen — restoring the documented "reversible upgrade" (#633). Previously the+ archive was written inside the wiki migration, which runs *after*+ `Store::open` has already advanced the DB schema, so the archived `db/` was+ already at 2.x and a 1.x binary refused it — the documented rollback was+ impossible. The snapshot now runs in the boot path before `Store::open`,+ gated to the real 1.x→2.0 upgrade (a fresh install or already-migrated store+ takes nothing, unchanged) and idempotent; the OKF migration reuses that+ archive instead of taking a second one. An integration test asserts the+ archived DB reflects the pre-migration state.- Typed edges (`relations`) now actually work on OpenAI-family providers(#630). `ConsolidatedPage.relations` was an open map(`BTreeMap<String, Vec<String>>`), which the strict structured-outputdiff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex c04ceade..91e2122d 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -719,6 +719,20 @@ pub async fn run(config: &Config, args: ServeArgs) -> Result<()> {// second active-project pointer (#563). Held until `run` returns.let _serve_lock = acquire_serve_lock(&config.data_dir, args.force)?;+ // #633: take the pre-migration safety archive BEFORE `Store::open` advances+ // the SQLite schema, so the archive captures the true pre-2.0 state a 1.x+ // binary can reopen. `Store::open` below runs the DB migrations; if the+ // backup ran after it (as it used to, inside the wiki migration), the+ // archived `db/` would already be at the 2.x schema and the documented+ // rollback to 1.x would be impossible. A no-op for a fresh install or an+ // already-migrated store; the OKF wiki migration reuses this archive instead+ // of taking a second, post-DB-migration one.+ let backup_dest_override = std::env::var("AI_MEMORY_BACKUP_DIR")+ .ok()+ .map(std::path::PathBuf::from);+ migrations::snapshot_before_db_migration(&config.data_dir, backup_dest_override.as_deref())+ .with_context(|| "taking the pre-migration safety backup")?;+let store = Store::open(&config.data_dir).with_context(|| format!("opening store at {}", config.data_dir.display()))?;diff --git a/crates/ai-memory-wiki/src/migrations/m2026_09_okf_conformance.rs b/crates/ai-memory-wiki/src/migrations/m2026_09_okf_conformance.rsindex c477c5b0..eb252039 100644--- a/crates/ai-memory-wiki/src/migrations/m2026_09_okf_conformance.rs+++ b/crates/ai-memory-wiki/src/migrations/m2026_09_okf_conformance.rs@@ -48,6 +48,50 @@ impl OkfConformance {const BACKUP_LABEL: &str = "okf-v0.2";+/// Take the pre-migration safety archive **before** the store's DB schema is+/// migrated, so it captures the true pre-2.0 state a 1.x binary can reopen+/// (#633). Call this from the boot path *before* `Store::open`, which advances+/// the SQLite schema — otherwise the archive's `db/` is already at 2.x and the+/// documented 1.x rollback is impossible.+///+/// Gated to the real 1.x→2.0 upgrade and idempotent:+/// - if a usable backup already exists (`pre-migration-backup.json` pointing at+/// a present archive), reuse it — never overwrite a good pre-migration snapshot;+/// - if the wiki has no pre-OKF files, this is a fresh install or an already+/// conformant store — nothing to protect, take nothing (unchanged behaviour);+/// - otherwise archive the whole data dir now.+///+/// `dest_override` is the destination directory (the serve path passes+/// `AI_MEMORY_BACKUP_DIR`); `None` uses the default resolution. A `None`+/// *return* means "no backup was needed/taken here".+///+/// # Errors+/// [`WikiError`] if the wiki scan or the archive write fails.+pub fn snapshot_before_db_migration(+ data_dir: &Path,+ dest_override: Option<&Path>,+) -> WikiResult<Option<crate::backup::BackupReceipt>> {+ if crate::backup::BackupReceipt::load(data_dir).is_some_and(|r| r.archive_present()) {+ // A prior boot in this same upgrade already captured the pre-migration+ // state; do not overwrite it with a now-partially-migrated snapshot.+ return Ok(None);+ }+ let wiki_root = data_dir.join("wiki");+ if nonconformant_files(&wiki_root)?.is_empty() {+ // No pre-OKF wiki files → not a 1.x store being upgraded.+ return Ok(None);+ }+ let receipt =+ crate::backup::create_pre_migration_backup(data_dir, BACKUP_LABEL, dest_override)?;+ tracing::info!(+ archive = %receipt.archive_path.display(),+ size_bytes = receipt.size_bytes,+ dest_free_bytes = ?receipt.dest_free_bytes,+ "pre-migration backup taken before the DB schema migration (#633: 1.x-restorable)"+ );+ Ok(Some(receipt))+}+#[async_trait::async_trait]impl WikiMigration for OkfConformance {fn name(&self) -> &'static str {@@ -76,19 +120,36 @@ impl WikiMigration for OkfConformance {"OKF migration: taking the pre-migration backup first");- // 1. Backup gate.- let receipt = crate::backup::create_pre_migration_backup(- data_dir,- BACKUP_LABEL,- self.dest_override.as_deref(),- )?;- tracing::info!(- archive = %receipt.archive_path.display(),- size_bytes = receipt.size_bytes,- entries = receipt.entries,- dest_free_bytes = ?receipt.dest_free_bytes,- "pre-migration backup verified"- );+ // 1. Backup gate. The boot path takes this archive BEFORE `Store::open`+ // migrates the DB schema (#633), so a usable receipt normally already+ // exists here — reuse it rather than archive the now-DB-migrated tree+ // a second time. Only archive here for the rare path where the DB was+ // already at 2.x but the wiki was not yet conformant (no pre-open+ // snapshot was taken).+ match crate::backup::BackupReceipt::load(data_dir)+ .filter(crate::backup::BackupReceipt::archive_present)+ {+ Some(existing) => {+ tracing::info!(+ archive = %existing.archive_path.display(),+ "reusing the pre-open pre-migration backup (#633)"+ );+ }+ None => {+ let receipt = crate::backup::create_pre_migration_backup(+ data_dir,+ BACKUP_LABEL,+ self.dest_override.as_deref(),+ )?;+ tracing::info!(+ archive = %receipt.archive_path.display(),+ size_bytes = receipt.size_bytes,+ entries = receipt.entries,+ dest_free_bytes = ?receipt.dest_free_bytes,+ "pre-migration backup verified"+ );+ }+ }// 2. Checkpoint whatever the tree holds before touching it.let git = crate::git::GitAdapter::open_or_init(wiki_root)?;@@ -481,6 +542,140 @@ mod tests {assert!(crate::backup::BackupReceipt::load(tmp.path()).is_none());}+ // ---- #633: the safety archive must be taken BEFORE the DB migration ----++ /// The core assertion for #633: the pre-open snapshot captures the DB as it+ /// was at snapshot time, so a mutation applied afterwards — which is exactly+ /// what `Store::open`'s schema migration is — is NOT in the archive. Prove+ /// it end to end: write PRE, snapshot, overwrite to POST, and the extracted+ /// archive must still read PRE. If it read POST, the archive would be+ /// post-migration and a 1.x binary could not use it.+ #[test]+ fn snapshot_archive_captures_the_db_before_a_later_mutation() {+ let tmp = TempDir::new().unwrap();+ // A non-OKF wiki file makes this look like a 1.x store to the gate.+ let proj_dir = tmp.path().join("wiki").join("w").join("p");+ std::fs::create_dir_all(&proj_dir).unwrap();+ std::fs::write(proj_dir.join("note.md"), "# Note\n\nlegacy body\n").unwrap();+ // A DB carrying a sentinel row.+ let db_path = tmp.path().join("db").join("memory.sqlite");+ std::fs::create_dir_all(db_path.parent().unwrap()).unwrap();+ {+ let conn = rusqlite::Connection::open(&db_path).unwrap();+ conn.execute_batch(+ "CREATE TABLE snapshot_marker(v TEXT); INSERT INTO snapshot_marker VALUES('PRE');",+ )+ .unwrap();+ }++ let dest = TempDir::new().unwrap();+ let receipt = snapshot_before_db_migration(tmp.path(), Some(dest.path()))+ .unwrap()+ .expect("a legacy store must be snapshotted");++ // Mutate the on-disk DB AFTER the snapshot, as a schema migration would.+ {+ let conn = rusqlite::Connection::open(&db_path).unwrap();+ conn.execute("UPDATE snapshot_marker SET v = 'POST'", [])+ .unwrap();+ }++ // Extract the archive and read its DB: it must still say PRE.+ let extract = TempDir::new().unwrap();+ {+ let file = std::fs::File::open(&receipt.archive_path).unwrap();+ let dec = flate2::read::GzDecoder::new(file);+ tar::Archive::new(dec).unpack(extract.path()).unwrap();+ }+ let archived_db = extract.path().join("db").join("memory.sqlite");+ let conn = rusqlite::Connection::open(&archived_db).unwrap();+ let marker: String = conn+ .query_row("SELECT v FROM snapshot_marker", [], |r| r.get(0))+ .unwrap();+ assert_eq!(+ marker, "PRE",+ "the archive captured the DB after a later mutation — the backup is not pre-migration"+ );+ }++ #[tokio::test]+ async fn snapshot_backs_up_a_legacy_store() {+ let (tmp, _store, _ws, _proj) = legacy_store().await;+ let dest = TempDir::new().unwrap();+ let receipt = snapshot_before_db_migration(tmp.path(), Some(dest.path())).unwrap();+ assert!(+ receipt.is_some(),+ "a legacy store must be snapshotted pre-open"+ );+ assert_eq!(std::fs::read_dir(dest.path()).unwrap().count(), 1);+ assert!(crate::backup::BackupReceipt::load(tmp.path()).is_some());+ }++ #[tokio::test]+ async fn snapshot_skips_a_conformant_or_fresh_store() {+ let tmp = TempDir::new().unwrap();+ let _store = Store::open(tmp.path()).unwrap();+ std::fs::create_dir_all(tmp.path().join("wiki")).unwrap();+ let dest = TempDir::new().unwrap();+ assert!(+ snapshot_before_db_migration(tmp.path(), Some(dest.path()))+ .unwrap()+ .is_none()+ );+ assert_eq!(std::fs::read_dir(dest.path()).unwrap().count(), 0);+ }++ /// The pre-open snapshot and the OKF migration must not both archive: the+ /// migration reuses the receipt the boot path already wrote.+ #[tokio::test]+ async fn okf_migration_reuses_the_pre_open_snapshot() {+ let (tmp, store, _ws, _proj) = legacy_store().await;+ let pre_dest = TempDir::new().unwrap();+ let receipt = snapshot_before_db_migration(tmp.path(), Some(pre_dest.path()))+ .unwrap()+ .expect("legacy store snapshotted pre-open");+ assert_eq!(std::fs::read_dir(pre_dest.path()).unwrap().count(), 1);++ // A different dest: if the migration re-archived, an artifact would land+ // here. It must reuse the pre-open one instead.+ let okf_dest = TempDir::new().unwrap();+ let migration = OkfConformance {+ dest_override: Some(okf_dest.path().to_path_buf()),+ };+ migration+ .up(&store.writer, &tmp.path().join("wiki"))+ .await+ .unwrap();+ assert_eq!(+ std::fs::read_dir(okf_dest.path()).unwrap().count(),+ 0,+ "the OKF migration took a second backup instead of reusing the pre-open one"+ );+ assert!(+ receipt.archive_present(),+ "the reused archive must still exist"+ );+ }++ #[tokio::test]+ async fn snapshot_is_idempotent_and_never_overwrites() {+ let (tmp, _store, _ws, _proj) = legacy_store().await;+ let dest = TempDir::new().unwrap();+ assert!(+ snapshot_before_db_migration(tmp.path(), Some(dest.path()))+ .unwrap()+ .is_some()+ );+ let after_first = std::fs::read_dir(dest.path()).unwrap().count();+ assert!(+ snapshot_before_db_migration(tmp.path(), Some(dest.path()))+ .unwrap()+ .is_none(),+ "a second snapshot when a usable backup exists must be a no-op"+ );+ assert_eq!(std::fs::read_dir(dest.path()).unwrap().count(), after_first);+ }+/// The gate itself: when the backup cannot be written, the migration/// must fail and leave every row and file untouched.#[tokio::test]diff --git a/crates/ai-memory-wiki/src/migrations/mod.rs b/crates/ai-memory-wiki/src/migrations/mod.rsindex d713203a..7b8b4f40 100644--- a/crates/ai-memory-wiki/src/migrations/mod.rs+++ b/crates/ai-memory-wiki/src/migrations/mod.rs@@ -46,6 +46,7 @@ use std::path::Path;use ai_memory_store::WriterHandle;+pub use m2026_09_okf_conformance::snapshot_before_db_migration;pub use runner::run_pending;use crate::error::WikiResult;diff --git a/docs/MIGRATION-2.0.md b/docs/MIGRATION-2.0.mdindex 1e6fff08..2a28511e 100644--- a/docs/MIGRATION-2.0.md+++ b/docs/MIGRATION-2.0.md@@ -1,19 +1,14 @@# Upgrading to 2.02.0 changes the wiki's on-disk format to the [Open Knowledge Format-v0.2](okf.md). The upgrade is automatic and backup-gated. This page-describes exactly what happens and what the automatic archive can and-cannot restore.--> **Rollback to 1.x needs your own 1.x-era backup.** The automatic-> archive below is written **after** the SQLite schema has already been-> migrated forward, so its `db/` is at the 2.x schema and a 1.x binary-> will refuse to open it (`store schema … newer than this ai-memory-> build`). The archive is a full **2.x** recovery point (wiki content and-> business state intact); it is **not** a 1.x rollback point. If you want-> the option to return to 1.x, take a backup with the **1.x** binary (or-> `ai-memory backup` on 1.x, or a filesystem copy of the data dir) before-> the first 2.x start. See [#633](https://github.com/akitaonrails/ai-memory/issues/633).+v0.2](okf.md). The upgrade is automatic, backup-gated, and reversible.+This page describes exactly what happens and how to go back.++> **The safety archive is a true pre-migration recovery point.** It is+> written **before** the SQLite schema is migrated forward (#633), so its+> `db/` is still at the 1.x schema and restoring it lets you start the old+> 1.x binary again. (Earlier 2.0.x builds took this archive *after* the DB+> migration, which left it 2.x-only — fixed in 2.0.3.)## What happens on the first 2.0 start@@ -115,12 +110,11 @@ ai-memory status # server healthy, page counts unchangedgrep -L "^type:" <data_dir>/wiki/*/*/*/*.md # no output = all pages typed```-## Restoring the automatic archive+## Restoring the backup-Blunt and complete — returns the entire data directory to the state it-had **the moment the OKF wiki migration began**. Note this is *after* the-SQLite schema was migrated forward, so the restored `db/` is at the 2.x-schema: reopen it with a **2.x** binary, not 1.x.+Blunt and complete — returns the entire data directory to its exact+pre-migration state (the archive is taken before both the DB schema and+the wiki are migrated, #633), so you can start the old 1.x binary again:```bash# 1. stop the server (docker compose down / systemctl stop ai-memory)@@ -129,13 +123,9 @@ mv <data_dir> <data_dir>.post-migration# 3. unpack the archive as the new data dirmkdir <data_dir>tar -xzf ~/ai-memory-backup-okf-v0.2-<date>.tar.gz -C <data_dir>-# 4. start ai-memory 2.x against it (a 1.x binary will refuse the DB)+# 4. start the OLD (1.x) binary against it```-To actually roll **back to 1.x**, you need a backup taken with the 1.x-binary before the upgrade (see the note at the top of this page); the-automatic archive above cannot serve that purpose.-Surgical alternative (keeps post-migration work, reverts only the wikifiles): the `pre-okf-migration checkpoint` commit in the wiki's githistory, followed by `ai-memory reindex`.@@ -146,10 +136,11 @@ A 2.0-migrated data directory records the migration in the`wiki_migrations` table. A **newer** binary opening an **older** wikimigrates it (as above). An **older 2.0+** binary opening a **newer**wiki refuses to start with `NewerWikiFormat` instead of silently mixing-formats. (1.x binaries predate the *wiki* guard, but the **SQLite** schema guard-still stops them: a 1.x binary refuses a 2.x-migrated `db/` outright, so-you cannot stay on 1.x by restoring the automatic archive — only by-restoring a backup you took with the 1.x binary before upgrading.)+formats. (1.x binaries predate the guard: they can open a migrated+directory and will tolerate the extra frontmatter, but new writes from+1.x will not carry the OKF keys — avoid mixing. To stay on 1.x cleanly,+restore the pre-migration archive, which is captured before the DB+migration and so reopens under 1.x.)## Sharing bundles