Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(okf): MIGRATION-2.0.md upgrade guide + changelog + design deltas

AkitaOnRails576f67761e24
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ec47a354..095d853f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,17 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
-### Fixed
-- `memory_query` no longer fails with `fts5: syntax error` when the query is
- natural language containing parentheses, apostrophe-quoted phrases, or a
- stray `OR`/`AND` (e.g. *"my visit to the Museum of Modern Art (MoMA) and
- the 'Ancient Civilizations' exhibit"*). Prepared FTS5 queries are now
- validated against the engine's own parser and degrade to an always-valid
- quoted bag of words when the preserved operator form does not parse;
- deliberate well-formed operator queries are preserved as before. Found by
- the new LongMemEval retrieval harness on its first full run.
-
### Added
+- `ai-memory export-okf --project <p> -o <bundle.tar.gz>` and
+ `POST /admin/export-okf` — stream one project's wiki as a validated
+ OKF v0.2 bundle with a freshly generated index; a non-conformant page
+ fails the export. Importing needs no command: unpack a bundle's
+ concept files into a project's wiki directory and the watcher (or
+ `reindex`) ingests them.
- Added a LongMemEval retrieval benchmark to the evaluation harness
(`cargo run -p ai-memory-eval -- retrieval`). It drives a real
`ai-memory serve` subprocess end to end: haystack chat histories replay
@@ -27,6 +23,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
reported separately). The 278 MB dataset is fetched on demand with a
pinned sha256; baselines are published under `docs/benchmarks/`. The
existing LLM A/B harness moved to the `ab` subcommand.
+### Changed
+- The wiki's on-disk format is now natively the Open Knowledge Format
+ (OKF) v0.2: every page write fills the spec's `type`, `generated`,
+ `sources` and `stale_after` keys (ai-memory's own fields ride along as
+ spec-safe extensions), and each project directory is a conformant OKF
+ bundle with an `okf_version` index. Existing stores are migrated
+ automatically on the first 2.0 start — **backup-gated**: the entire
+ data dir is archived to the user's home (or `AI_MEMORY_BACKUP_DIR`)
+ and verified first, or the migration refuses to run; pages are
+ rewritten in place (same ids, same version rows, timestamps
+ untouched), and the wiki homepage shows where the archive is until it
+ is deleted. A data dir migrated by a newer binary is refused by older
+ 2.0+ binaries instead of silently mixing formats. See
+ `docs/okf.md` and `docs/MIGRATION-2.0.md`.
+
+### Fixed
+- `memory_query` no longer fails with `fts5: syntax error` when the query is
+ natural language containing parentheses, apostrophe-quoted phrases, or a
+ stray `OR`/`AND` (e.g. *"my visit to the Museum of Modern Art (MoMA) and
+ the 'Ancient Civilizations' exhibit"*). Prepared FTS5 queries are now
+ validated against the engine's own parser and degrade to an always-valid
+ quoted bag of words when the preserved operator form does not parse;
+ deliberate well-formed operator queries are preserved as before. Found by
+ the new LongMemEval retrieval harness on its first full run.
## [1.39.0] - 2026-09-01
diff --git a/README.md b/README.md
index 8fbd3f20..1bab0702 100644
--- a/README.md
+++ b/README.md
@@ -344,6 +344,8 @@ diagram, crate breakdown, schema notes, and invariants.
- [`docs/security.md`](docs/security.md) - the full security model.
- [`docs/research-2026-landscape.md`](docs/research-2026-landscape.md) - how the field looks and where we sit in it.
- [`docs/ROADMAP-2.0.md`](docs/ROADMAP-2.0.md) - the plan for the 2.0 release, one item at a time.
+- [`docs/okf.md`](docs/okf.md) - the wiki is natively an Open Knowledge Format (OKF v0.2) bundle; design and field mapping.
+- [`docs/MIGRATION-2.0.md`](docs/MIGRATION-2.0.md) - upgrading an existing store to 2.0: the backup-gated automatic migration and how to restore.
- [`docs/benchmarks/`](docs/benchmarks/README.md) - published retrieval-quality numbers with provenance, reproducible from the in-repo harness.
- [`docs/ROADMAP-2.0.md`](docs/ROADMAP-2.0.md) - the 2.0 plan.
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index a3ee3c55..5d688c11 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -475,7 +475,7 @@ reorg purge-project rename-project
move-project move-session uninstall
auth user completions
handoffs purge-session compact
-api-key
+api-key export-okf
```
Run `ai-memory --help` for the full tree.
diff --git a/docs/MIGRATION-2.0.md b/docs/MIGRATION-2.0.md
new file mode 100644
index 00000000..d1a87fa1
--- /dev/null
+++ b/docs/MIGRATION-2.0.md
@@ -0,0 +1,100 @@
+# Upgrading to 2.0
+
+2.0 changes the wiki's on-disk format to the [Open Knowledge Format
+v0.2](okf.md). The upgrade is automatic, backup-gated, and reversible.
+This page describes exactly what happens and how to go back.
+
+## What happens on the first 2.0 start
+
+When `ai-memory serve` starts on a data directory created by 1.x, a
+one-shot wiki migration runs before the server accepts any traffic:
+
+1. **A full backup is taken first — or nothing happens at all.** The
+ entire data directory (wiki files, SQLite database, config) is
+ compressed to a timestamped archive in your home directory:
+
+ ```
+ ~/ai-memory-backup-okf-v0.2-<date>.tar.gz
+ ```
+
+ Set `AI_MEMORY_BACKUP_DIR=/somewhere/else` before starting if your
+ home is small; the destination must be outside the data directory.
+ The archive is re-opened and verified after writing. **If the backup
+ cannot be written or verified, the migration aborts and the server
+ refuses to start** — your data is untouched and the error says why.
+
+2. The wiki git history gets a `pre-okf-migration checkpoint` commit.
+
+3. Every page's frontmatter is rewritten **in place** — same page ids,
+ same version rows, bodies untouched, timestamps untouched. No page
+ is superseded, embeddings stay valid, nothing looks freshly edited.
+
+4. Every project directory gains an `index.md` declaring
+ `okf_version: "0.2"` (only if one does not already exist).
+
+5. One `okf-migration` git commit records the whole rewrite.
+
+The migration is idempotent: restarting the server re-runs nothing and
+never takes a second backup. Fresh installs skip all of it, including
+the backup.
+
+## After the migration
+
+The wiki homepage shows a notice with the archive's location, size and
+date until you delete the archive file:
+
+- **Everything looks right?** Delete the archive; the notice disappears
+ on its own.
+- **Something is missing?** Restore (below).
+
+Verify the migration if you like:
+
+```bash
+ai-memory status # server healthy, page counts unchanged
+grep -L "^type:" <data_dir>/wiki/*/*/*/*.md # no output = all pages typed
+```
+
+## Restoring the backup
+
+Blunt and complete — returns the entire data directory to its exact
+pre-migration state:
+
+```bash
+# 1. stop the server (docker compose down / systemctl stop ai-memory)
+# 2. move the current data dir aside
+mv <data_dir> <data_dir>.post-migration
+# 3. unpack the archive as the new data dir
+mkdir <data_dir>
+tar -xzf ~/ai-memory-backup-okf-v0.2-<date>.tar.gz -C <data_dir>
+# 4. start the OLD (1.x) binary against it
+```
+
+Surgical alternative (keeps post-migration work, reverts only the wiki
+files): the `pre-okf-migration checkpoint` commit in the wiki's git
+history, followed by `ai-memory reindex`.
+
+## Downgrade guard
+
+A 2.0-migrated data directory records the migration in the
+`wiki_migrations` table. A **newer** binary opening an **older** wiki
+migrates it (as above). An **older 2.0+** binary opening a **newer**
+wiki refuses to start with `NewerWikiFormat` instead of silently mixing
+formats. (1.x binaries predate the guard: they can open a migrated
+directory and will tolerate the extra frontmatter, but new writes from
+1.x will not carry the OKF keys — avoid mixing; restore the archive if
+you need to stay on 1.x.)
+
+## Sharing bundles
+
+Post-migration, each project directory *is* an OKF v0.2 bundle. Hand a
+copy to any OKF-aware tool, or export a validated tarball with a fresh
+index:
+
+```bash
+ai-memory export-okf --project myproject -o myproject-bundle.tar.gz
+```
+
+Importing a foreign bundle needs no command: unpack its concept files
+into a project's wiki directory and the watcher (or `ai-memory
+reindex`) ingests them; anything missing from their frontmatter is
+filled at write time.
diff --git a/docs/okf.md b/docs/okf.md
index 6b7ff599..3f92eea5 100644
--- a/docs/okf.md
+++ b/docs/okf.md
@@ -78,9 +78,12 @@ Order is fixed; each step gates the next:
embedding invalidation, no `updated_at` stampede. One git commit
("okf-migration") on the wiki, after a pre-migration checkpoint
commit. Reindex afterwards.
-3. **`wiki_format` generation marker** in the wiki meta manifest;
- `serve` refuses to open a newer-generation wiki with an older binary
- (mirror of the DB schema-ahead guard).
+3. **Generation marker**: the migration ships as a `WikiMigration`
+ (tracked in the `wiki_migrations` table), and the runner now refuses
+ to open a wiki whose table records a migration this binary does not
+ know (`NewerWikiFormat`) — the downgrade guard mirroring the DB
+ schema-ahead rule. Scope `_meta.md` manifests get their `type` only;
+ they are identity records, not concept pages.
4. **Idempotent**: a re-run migrates zero pages.
5. **Homepage notice** until the archive is deleted: path, size, date,
plus "everything looks right → delete the archive" and "something
@@ -100,6 +103,10 @@ pre-migration git checkpoint + `reindex` (surgical).
- Backup gate: archive step broken → migration refuses to run.
- Homepage notice renders the recorded archive path and clears when
the file is gone.
-- Foreign OKF v0.2 bundle imports into a project; `export --okf`
- emits a bundle a strict reader accepts.
+- Foreign OKF v0.2 bundle imports into a project; `export-okf`
+ emits a bundle a strict reader accepts (a non-conformant page fails
+ the export). Import has no dedicated command by design: the format is
+ native, so unpacking a bundle's concept files into a project's wiki
+ directory and letting the watcher (or `reindex`) ingest them IS the
+ import path.
- Retrieval regression: LongMemEval baseline re-run; no material drop.