Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(okf): MIGRATION-2.0.md upgrade guide + changelog + design deltas
AkitaOnRails576f67761e24
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ec47a354..095d853f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,17 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]-### Fixed-- `memory_query` no longer fails with `fts5: syntax error` when the query is- natural language containing parentheses, apostrophe-quoted phrases, or a- stray `OR`/`AND` (e.g. *"my visit to the Museum of Modern Art (MoMA) and- the 'Ancient Civilizations' exhibit"*). Prepared FTS5 queries are now- validated against the engine's own parser and degrade to an always-valid- quoted bag of words when the preserved operator form does not parse;- deliberate well-formed operator queries are preserved as before. Found by- the new LongMemEval retrieval harness on its first full run.-### Added+- `ai-memory export-okf --project <p> -o <bundle.tar.gz>` and+ `POST /admin/export-okf` — stream one project's wiki as a validated+ OKF v0.2 bundle with a freshly generated index; a non-conformant page+ fails the export. Importing needs no command: unpack a bundle's+ concept files into a project's wiki directory and the watcher (or+ `reindex`) ingests them.- Added a LongMemEval retrieval benchmark to the evaluation harness(`cargo run -p ai-memory-eval -- retrieval`). It drives a real`ai-memory serve` subprocess end to end: haystack chat histories replay@@ -27,6 +23,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0reported separately). The 278 MB dataset is fetched on demand with apinned sha256; baselines are published under `docs/benchmarks/`. Theexisting LLM A/B harness moved to the `ab` subcommand.+### Changed+- The wiki's on-disk format is now natively the Open Knowledge Format+ (OKF) v0.2: every page write fills the spec's `type`, `generated`,+ `sources` and `stale_after` keys (ai-memory's own fields ride along as+ spec-safe extensions), and each project directory is a conformant OKF+ bundle with an `okf_version` index. Existing stores are migrated+ automatically on the first 2.0 start — **backup-gated**: the entire+ data dir is archived to the user's home (or `AI_MEMORY_BACKUP_DIR`)+ and verified first, or the migration refuses to run; pages are+ rewritten in place (same ids, same version rows, timestamps+ untouched), and the wiki homepage shows where the archive is until it+ is deleted. A data dir migrated by a newer binary is refused by older+ 2.0+ binaries instead of silently mixing formats. See+ `docs/okf.md` and `docs/MIGRATION-2.0.md`.++### Fixed+- `memory_query` no longer fails with `fts5: syntax error` when the query is+ natural language containing parentheses, apostrophe-quoted phrases, or a+ stray `OR`/`AND` (e.g. *"my visit to the Museum of Modern Art (MoMA) and+ the 'Ancient Civilizations' exhibit"*). Prepared FTS5 queries are now+ validated against the engine's own parser and degrade to an always-valid+ quoted bag of words when the preserved operator form does not parse;+ deliberate well-formed operator queries are preserved as before. Found by+ the new LongMemEval retrieval harness on its first full run.## [1.39.0] - 2026-09-01diff --git a/README.md b/README.mdindex 8fbd3f20..1bab0702 100644--- a/README.md+++ b/README.md@@ -344,6 +344,8 @@ diagram, crate breakdown, schema notes, and invariants.- [`docs/security.md`](docs/security.md) - the full security model.- [`docs/research-2026-landscape.md`](docs/research-2026-landscape.md) - how the field looks and where we sit in it.- [`docs/ROADMAP-2.0.md`](docs/ROADMAP-2.0.md) - the plan for the 2.0 release, one item at a time.+- [`docs/okf.md`](docs/okf.md) - the wiki is natively an Open Knowledge Format (OKF v0.2) bundle; design and field mapping.+- [`docs/MIGRATION-2.0.md`](docs/MIGRATION-2.0.md) - upgrading an existing store to 2.0: the backup-gated automatic migration and how to restore.- [`docs/benchmarks/`](docs/benchmarks/README.md) - published retrieval-quality numbers with provenance, reproducible from the in-repo harness.- [`docs/ROADMAP-2.0.md`](docs/ROADMAP-2.0.md) - the 2.0 plan.diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex a3ee3c55..5d688c11 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -475,7 +475,7 @@ reorg purge-project rename-projectmove-project move-session uninstallauth user completionshandoffs purge-session compact-api-key+api-key export-okf```Run `ai-memory --help` for the full tree.diff --git a/docs/MIGRATION-2.0.md b/docs/MIGRATION-2.0.mdnew file mode 100644index 00000000..d1a87fa1--- /dev/null+++ b/docs/MIGRATION-2.0.md@@ -0,0 +1,100 @@+# Upgrading to 2.0++2.0 changes the wiki's on-disk format to the [Open Knowledge Format+v0.2](okf.md). The upgrade is automatic, backup-gated, and reversible.+This page describes exactly what happens and how to go back.++## What happens on the first 2.0 start++When `ai-memory serve` starts on a data directory created by 1.x, a+one-shot wiki migration runs before the server accepts any traffic:++1. **A full backup is taken first — or nothing happens at all.** The+ entire data directory (wiki files, SQLite database, config) is+ compressed to a timestamped archive in your home directory:++ ```+ ~/ai-memory-backup-okf-v0.2-<date>.tar.gz+ ```++ Set `AI_MEMORY_BACKUP_DIR=/somewhere/else` before starting if your+ home is small; the destination must be outside the data directory.+ The archive is re-opened and verified after writing. **If the backup+ cannot be written or verified, the migration aborts and the server+ refuses to start** — your data is untouched and the error says why.++2. The wiki git history gets a `pre-okf-migration checkpoint` commit.++3. Every page's frontmatter is rewritten **in place** — same page ids,+ same version rows, bodies untouched, timestamps untouched. No page+ is superseded, embeddings stay valid, nothing looks freshly edited.++4. Every project directory gains an `index.md` declaring+ `okf_version: "0.2"` (only if one does not already exist).++5. One `okf-migration` git commit records the whole rewrite.++The migration is idempotent: restarting the server re-runs nothing and+never takes a second backup. Fresh installs skip all of it, including+the backup.++## After the migration++The wiki homepage shows a notice with the archive's location, size and+date until you delete the archive file:++- **Everything looks right?** Delete the archive; the notice disappears+ on its own.+- **Something is missing?** Restore (below).++Verify the migration if you like:++```bash+ai-memory status # server healthy, page counts unchanged+grep -L "^type:" <data_dir>/wiki/*/*/*/*.md # no output = all pages typed+```++## Restoring the backup++Blunt and complete — returns the entire data directory to its exact+pre-migration state:++```bash+# 1. stop the server (docker compose down / systemctl stop ai-memory)+# 2. move the current data dir aside+mv <data_dir> <data_dir>.post-migration+# 3. unpack the archive as the new data dir+mkdir <data_dir>+tar -xzf ~/ai-memory-backup-okf-v0.2-<date>.tar.gz -C <data_dir>+# 4. start the OLD (1.x) binary against it+```++Surgical alternative (keeps post-migration work, reverts only the wiki+files): the `pre-okf-migration checkpoint` commit in the wiki's git+history, followed by `ai-memory reindex`.++## Downgrade guard++A 2.0-migrated data directory records the migration in the+`wiki_migrations` table. A **newer** binary opening an **older** wiki+migrates it (as above). An **older 2.0+** binary opening a **newer**+wiki refuses to start with `NewerWikiFormat` instead of silently mixing+formats. (1.x binaries predate the guard: they can open a migrated+directory and will tolerate the extra frontmatter, but new writes from+1.x will not carry the OKF keys — avoid mixing; restore the archive if+you need to stay on 1.x.)++## Sharing bundles++Post-migration, each project directory *is* an OKF v0.2 bundle. Hand a+copy to any OKF-aware tool, or export a validated tarball with a fresh+index:++```bash+ai-memory export-okf --project myproject -o myproject-bundle.tar.gz+```++Importing a foreign bundle needs no command: unpack its concept files+into a project's wiki directory and the watcher (or `ai-memory+reindex`) ingests them; anything missing from their frontmatter is+filled at write time.diff --git a/docs/okf.md b/docs/okf.mdindex 6b7ff599..3f92eea5 100644--- a/docs/okf.md+++ b/docs/okf.md@@ -78,9 +78,12 @@ Order is fixed; each step gates the next:embedding invalidation, no `updated_at` stampede. One git commit("okf-migration") on the wiki, after a pre-migration checkpointcommit. Reindex afterwards.-3. **`wiki_format` generation marker** in the wiki meta manifest;- `serve` refuses to open a newer-generation wiki with an older binary- (mirror of the DB schema-ahead guard).+3. **Generation marker**: the migration ships as a `WikiMigration`+ (tracked in the `wiki_migrations` table), and the runner now refuses+ to open a wiki whose table records a migration this binary does not+ know (`NewerWikiFormat`) — the downgrade guard mirroring the DB+ schema-ahead rule. Scope `_meta.md` manifests get their `type` only;+ they are identity records, not concept pages.4. **Idempotent**: a re-run migrates zero pages.5. **Homepage notice** until the archive is deleted: path, size, date,plus "everything looks right → delete the archive" and "something@@ -100,6 +103,10 @@ pre-migration git checkpoint + `reindex` (surgical).- Backup gate: archive step broken → migration refuses to run.- Homepage notice renders the recorded archive path and clears whenthe file is gone.-- Foreign OKF v0.2 bundle imports into a project; `export --okf`- emits a bundle a strict reader accepts.+- Foreign OKF v0.2 bundle imports into a project; `export-okf`+ emits a bundle a strict reader accepts (a non-conformant page fails+ the export). Import has no dedicated command by design: the format is+ native, so unpacking a bundle's concept files into a project's wiki+ directory and letting the watcher (or `reindex`) ingest them IS the+ import path.- Retrieval regression: LongMemEval baseline re-run; no material drop.