Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(wiki): keep hook event ledgers out of the wiki's git history
Djalma Júniorbb6aaf474d83
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 53f77b46..1881a02a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -129,6 +129,15 @@ version in [`docs/changelog/`](docs/changelog/):were not re-stamped into the target project, which distorted cross-projectprofile convergence and caused a subsequent `purge-project` of the sourceproject to destroy them via cascade. They now re-stamp in the same transaction. (#1265)+- Fixed the hook event ledgers (`log.md`, `log-YYYY-MM.md`) being committed to+ the wiki's git repository. Every checkpoint after a hook append wrote a new+ whole-file blob of the ledger, and nothing packed them: one install's+ `wiki/.git` reached 3.49 GB, 229 MB per version of a single ledger.+ Checkpoints now leave ledgers out, recognized by name and by content, so an+ ordinary page named like a ledger stays versioned. A ledger an earlier+ version committed is dropped from the git index, never from disk, by the next+ checkpoint. Existing history keeps its old blobs; `docs/lifecycle-ops.md`+ explains how an operator can rewrite it. (#PRNUM)## [3.0.0] - 2026-10-13diff --git a/crates/ai-memory-hooks/src/log.rs b/crates/ai-memory-hooks/src/log.rsindex 7090238a..699e39be 100644--- a/crates/ai-memory-hooks/src/log.rs+++ b/crates/ai-memory-hooks/src/log.rs@@ -10,13 +10,13 @@//! Each calendar month gets its own file: `log-YYYY-MM.md` under the//! project root. The filename is derived from the event timestamp, so//! events written across a month boundary land in their respective-//! month's file. Old months never grow once closed — perfect for git's-//! per-blob storage (one terminal blob per month per project). The+//! month's file. Old months never grow once closed. Wiki checkpoints+//! leave ledgers out of git (every append would otherwise store a new+//! full-size blob), so the files on disk are their only copy. The//! watcher's reserved-name check (`is_reserved_filename`) covers the//! `log-*.md` pattern so rotated files are not indexed as wiki pages.//!-//! No explicit retention sweep yet — git history bounds growth-//! sub-linearly and old monthly blobs are tiny. A future+//! No explicit retention sweep yet. A future//! `forget-sweep --logs --older-than 12m` could delete cold files if//! it becomes worthwhile.diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rsindex 5f024f40..15f6dc04 100644--- a/crates/ai-memory-hooks/src/router.rs+++ b/crates/ai-memory-hooks/src/router.rs@@ -14810,6 +14810,18 @@ mod tests {).await.unwrap();+ // A page write nobody committed. The prompt above also appends to the+ // hook ledger, which checkpoints no longer commit, so the ledger alone+ // cannot be the pending write a recovery commit picks up.+ let pending = ai_memory_core::PagePath::new("notes/pending.md").unwrap();+ state+ .wiki+ .git()+ .write_atomic(+ &state.wiki.abs_path(state.workspace_id, target, &pending),+ b"pending\n",+ )+ .unwrap();state.ingest_metrics.record_persisted(123);let outcome = process_authorized(&state,diff --git a/crates/ai-memory-wiki/src/git.rs b/crates/ai-memory-wiki/src/git.rsindex b8032882..d0e60cea 100644--- a/crates/ai-memory-wiki/src/git.rs+++ b/crates/ai-memory-wiki/src/git.rs@@ -11,6 +11,11 @@//! safety net, decided in `take_staging`. The repository stays open//! between commits: reopening it dropped libgit2's index tree cache, so//! every commit rebuilt every directory tree.+//!+//! The raw hook event ledgers (`log.md`, `log-YYYY-MM.md`) are never staged.+//! They are append-only logs that reach hundreds of megabytes, git stores each+//! new version as a whole blob, and nothing reads them back from git; see+//! [`is_ledger_file`].use std::collections::BTreeSet;use std::path::{Path, PathBuf};@@ -538,7 +543,12 @@ impl GitAdapter {// and hash every file again, once.let touched = match staging {Staging::Everything => {- let walked = stage_working_tree(&mut index).map_err(CommitGit2Error::Other)?;+ let walked =+ stage_working_tree(&self.root, &mut index).map_err(CommitGit2Error::Other)?;+ // The walk only skips ledgers it would add or update; one a+ // past version already tracks, and that has not changed, is+ // not offered to it at all.+ untrack_ledgers(&self.root, &mut index, None).map_err(CommitGit2Error::Other)?;self.record_walk(&walked);walked.len()}@@ -559,7 +569,7 @@ impl GitAdapter {"index could not be written as a tree; re-hashing the working tree (#594)");index.clear().map_err(CommitGit2Error::Other)?;- stage_working_tree(&mut index).map_err(CommitGit2Error::Other)?;+ stage_working_tree(&self.root, &mut index).map_err(CommitGit2Error::Other)?;index.write().map_err(CommitGit2Error::Other)?;index.write_tree().map_err(CommitGit2Error::Other)?}@@ -847,18 +857,81 @@ pub(crate) fn slash_path(rel: &Path) -> String {/// entries, drop deleted ones, add untracked files. libgit2 does all three/// from one index-to-workdir diff. Returns the paths touched; does not/// write the index file.-fn stage_working_tree(index: &mut git2::Index) -> Result<Vec<PathBuf>, git2::Error> {+///+/// Ledgers are skipped: a positive return from the callback leaves that path+/// alone, neither added nor refreshed.+fn stage_working_tree(root: &Path, index: &mut git2::Index) -> Result<Vec<PathBuf>, git2::Error> {let mut touched = Vec::new();index.add_all(["*"].iter(),IndexAddOption::DEFAULT,Some(&mut |path: &Path, _: &[u8]| {+ if is_ledger_file(root, path) {+ return 1;+ }touched.push(path.to_path_buf());0}),)?;Ok(touched)}++/// True when `rel` (relative to the wiki root) is a raw hook event ledger:+/// a `log.md` / `log-YYYY-MM.md` file whose content also opens with a hook log+/// entry.+///+/// Both halves are needed, as everywhere else a ledger is recognized+/// ([`crate::ledger`]): an ordinary page may be named `log-2026-09.md`, and it+/// must stay versioned. The content half reads at most+/// `GATE_PREFIX_BYTES` from the file, so a ledger of any size costs one small+/// read per commit.+///+/// Why this is not a `.gitignore` or `info/exclude` entry: those match names,+/// so one entry would also hide an ordinary page that wears a ledger's name,+/// and the entry would outlive the file's being a ledger.+fn is_ledger_file(root: &Path, rel: &Path) -> bool {+ rel.file_name()+ .and_then(|name| name.to_str())+ .is_some_and(ai_memory_core::log_ledger::is_log_ledger_path)+ && crate::ledger::opens_with_log_ledger(&root.join(rel))+}++/// Drop ledgers that earlier versions committed from the index, under the+/// directory `under` (a `/`-separated path) or everywhere when it is `None`.+/// The files stay on disk: only the index entry goes, so the next commit's tree+/// no longer names them and they stop growing the repository. Their old blobs+/// remain in history. Returns how many entries were dropped.+fn untrack_ledgers(+ root: &Path,+ index: &mut git2::Index,+ under: Option<&str>,+) -> Result<usize, git2::Error> {+ let prefix = under.map(|dir| format!("{}/", dir.trim_end_matches('/')));+ let tracked: Vec<PathBuf> = index+ .iter()+ .filter_map(|entry| String::from_utf8(entry.path).ok())+ .filter(|path| prefix.as_ref().is_none_or(|p| path.starts_with(p)))+ .filter(|path| {+ // The shape test is a string compare; only a shape match pays for+ // the file read.+ path.rsplit('/')+ .next()+ .is_some_and(ai_memory_core::log_ledger::is_log_ledger_path)+ })+ .map(PathBuf::from)+ .filter(|path| is_ledger_file(root, path))+ .collect();+ for path in &tracked {+ index.remove_path(path)?;+ }+ if !tracked.is_empty() {+ debug!(+ count = tracked.len(),+ "hook event ledgers dropped from the index; the files stay on disk"+ );+ }+ Ok(tracked.len())+}/// Stage only `paths`: a file by path, a directory with its subtree, a/// gone path by removing its entries. Does not write the index file.fn stage_paths(@@ -880,10 +953,17 @@ fn stage_paths(index.add_all([spec.as_str()].iter(),IndexAddOption::DISABLE_PATHSPEC_MATCH,- None,+ Some(&mut |path: &Path, _: &[u8]| i32::from(is_ledger_file(root, path))),)?;+ untrack_ledgers(root, index, Some(&spec))?;} else if abs.is_file() {- index.add_path(rel)?;+ if is_ledger_file(root, rel) {+ if index.get_path(rel, 0).is_some() {+ index.remove_path(rel)?;+ }+ } else {+ index.add_path(rel)?;+ }} else if index.get_path(rel, 0).is_some() {index.remove_path(rel)?;} else {@@ -932,7 +1012,7 @@ fn commit_all_fallback() -> WikiResult<Option<git2::Oid>> {validate_cli_fallback_root(root)?;warn!(error = %original, root = %root.display(), "libgit2 commit failed; trying git CLI fallback");- run_git(root, ["add", "-A"])?;+ stage_all_except_ledgers_cli(root)?;let diff = git_command(root).args(["diff", "--cached", "--quiet", "--exit-code"]).status()@@ -1093,6 +1173,103 @@ fn run_git<const N: usize>(root: &Path, args: [&str; N]) -> WikiResult<()> {}}+/// `git add -A` for the CLI fallback, leaving the hook event ledgers out the+/// way [`stage_working_tree`] does for libgit2.+///+/// Ledgers a past version committed are dropped from the index first, then+/// everything else is added with the ledgers as `exclude` pathspecs, so git+/// never reads a ledger into a blob. Pathspecs go through stdin because a+/// project with a ledger per month can name more files than a Windows command+/// line holds.+#[cfg(windows)]+fn stage_all_except_ledgers_cli(root: &Path) -> WikiResult<()> {+ let mut ledgers = Vec::new();+ collect_ledger_files(root, root, &mut ledgers)?;+ if ledgers.is_empty() {+ return run_git(root, ["add", "-A"]);+ }+ let pathspecs = |prefix: &str, first: &str| -> Vec<u8> {+ let mut input = first.as_bytes().to_vec();+ for ledger in &ledgers {+ input.extend_from_slice(prefix.as_bytes());+ input.extend_from_slice(slash_path(ledger).as_bytes());+ input.push(0);+ }+ input+ };+ run_git_with_stdin(+ root,+ [+ "rm",+ "--cached",+ // Without `--force` git refuses an entry whose staged content+ // matches neither HEAD nor the file (an old version staged an+ // append and never committed it), and that refusal would abort+ // the whole checkpoint. Only paths already validated as ledgers+ // are listed, and only their index entries go.+ "--force",+ "-q",+ "--ignore-unmatch",+ "--pathspec-from-file=-",+ "--pathspec-file-nul",+ ],+ &pathspecs(":(literal)", ""),+ )?;+ run_git_with_stdin(+ root,+ ["add", "-A", "--pathspec-from-file=-", "--pathspec-file-nul"],+ &pathspecs(":(exclude,literal)", ".\0"),+ )+}++/// Every ledger file under `dir`, relative to `root`, skipping `.git` and+/// symlinks.+#[cfg(windows)]+fn collect_ledger_files(root: &Path, dir: &Path, found: &mut Vec<PathBuf>) -> WikiResult<()> {+ for entry in std::fs::read_dir(dir)? {+ let entry = entry?;+ let file_type = entry.file_type()?;+ let path = entry.path();+ if file_type.is_dir() {+ if entry.file_name() != ".git" {+ collect_ledger_files(root, &path, found)?;+ }+ } else if file_type.is_file()+ && let Ok(rel) = path.strip_prefix(root)+ && is_ledger_file(root, rel)+ {+ found.push(rel.to_path_buf());+ }+ }+ Ok(())+}++#[cfg(windows)]+fn run_git_with_stdin<const N: usize>(+ root: &Path,+ args: [&str; N],+ input: &[u8],+) -> WikiResult<()> {+ use std::io::Write;+ use std::process::Stdio;+ let mut child = git_command(root)+ .args(args)+ .stdin(Stdio::piped())+ .spawn()+ .map_err(|error| WikiError::Io(std::io::Error::other(error.to_string())))?;+ if let Some(mut stdin) = child.stdin.take() {+ stdin.write_all(input)?;+ }+ let status = child.wait()?;+ if status.success() {+ Ok(())+ } else {+ Err(WikiError::Io(std::io::Error::other(format!(+ "git fallback exited with status {status}"+ ))))+ }+}+#[cfg(windows)]fn git_output<const N: usize>(root: &Path, args: [&str; N]) -> WikiResult<std::process::Output> {let output = git_command(root)@@ -1633,7 +1810,7 @@ mod tests {let staged = |root: &Path| {let repo = Repository::open(root).unwrap();let mut index = repo.index().unwrap();- let touched = stage_working_tree(&mut index).unwrap().len();+ let touched = stage_working_tree(root, &mut index).unwrap().len();index.write().unwrap();touched};@@ -1727,6 +1904,221 @@ mod tests {assert!(adapter.written_paths().is_empty());}+ const LEDGER_BODY: &str = "## [2026-09-05T00:00:00Z] tool Bash: cargo test\n";+ const PROSE_BODY: &str = "# Log\n\nAn ordinary page named like a ledger.\n";++ /// Whether the index names `rel`, from a fresh read of the index file.+ fn indexed(root: &Path, rel: &str) -> bool {+ let repo = Repository::open(root).unwrap();+ let index = repo.index().unwrap();+ index.get_path(Path::new(rel), 0).is_some()+ }++ /// Path-scoped staging (the commit after the first walk): a ledger a past+ /// version tracked is dropped from the index when it is reported, by file+ /// or through its directory, and the file stays on disk. Ordinary files in+ /// the same call, including a page wearing a ledger's name, are staged.+ #[test]+ fn path_staging_untracks_a_ledger_and_keeps_a_page_wearing_its_name() {+ let tmp = tempdir();+ let root = tmp.path().join("wiki");+ let adapter = GitAdapter::open_or_init(&root).unwrap();+ write(&root, "p/log-2026-09.md", LEDGER_BODY);+ write(&root, "q/log-2026-09.md", LEDGER_BODY);+ write(&root, "r/log-2026-09.md", PROSE_BODY);+ let repo = Repository::open(&root).unwrap();+ let mut index = repo.index().unwrap();+ for rel in ["p/log-2026-09.md", "q/log-2026-09.md", "r/log-2026-09.md"] {+ index.add_path(Path::new(rel)).unwrap();+ }+ index.write().unwrap();+ drop(adapter);++ write(+ &root,+ "p/log-2026-09.md",+ &format!("{LEDGER_BODY}{LEDGER_BODY}"),+ );+ write(&root, "r/log-2026-09.md", &format!("{PROSE_BODY}edited\n"));+ write(&root, "q/other.md", "page\n");+ // Never tracked, reported through its directory: it must not even be+ // read into a blob.+ let fresh = format!("{LEDGER_BODY}## [2026-09-06T00:00:00Z] stop\n");+ write(&root, "s/log-2026-08.md", &fresh);+ let mut index = repo.index().unwrap();+ // `p` is reported by file, `q` and `s` through their directories.+ let paths: BTreeSet<PathBuf> = ["p/log-2026-09.md", "q", "r/log-2026-09.md", "s"]+ .iter()+ .map(PathBuf::from)+ .collect();+ stage_paths(&root, &mut index, &paths).unwrap();+ index.write().unwrap();++ assert!(!indexed(&root, "p/log-2026-09.md"), "file report");+ assert!(!indexed(&root, "q/log-2026-09.md"), "directory report");+ assert!(+ !indexed(&root, "s/log-2026-08.md"),+ "untracked, by directory"+ );+ assert!(+ !repo+ .odb()+ .unwrap()+ .exists(git2::Oid::hash_object(ObjectType::Blob, fresh.as_bytes()).unwrap()),+ "a ledger under a reported directory was hashed into git"+ );+ assert!(indexed(&root, "q/other.md"), "control: the page beside it");+ assert!(indexed(&root, "r/log-2026-09.md"), "control: a page");+ assert_eq!(+ std::fs::read_to_string(root.join("p/log-2026-09.md")).unwrap(),+ format!("{LEDGER_BODY}{LEDGER_BODY}"),+ "the file on disk is untouched"+ );+ assert_eq!(+ std::fs::read_to_string(root.join("q/log-2026-09.md")).unwrap(),+ LEDGER_BODY+ );+ }++ /// The CLI fallback (Windows only, when libgit2 cannot open the repo)+ /// must leave ledgers out as well, and must never read one into a blob.+ #[cfg(windows)]+ #[test]+ fn cli_fallback_commit_leaves_ledgers_out_and_untracks_a_tracked_one() {+ let tmp = tempdir();+ let root = tmp.path().join("wiki");+ let adapter = GitAdapter::open_or_init(&root).unwrap();+ // A repo-local setting beats a contributor's global `commit.gpgsign`.+ Repository::open(&root)+ .unwrap()+ .config()+ .unwrap()+ .set_bool("commit.gpgsign", false)+ .unwrap();+ write(&root, "p/log-2026-07.md", LEDGER_BODY);+ run_git(&root, ["add", "p/log-2026-07.md"]).unwrap();+ run_git(+ &root,+ [+ "-c",+ "user.name=fixture",+ "-c",+ "user.email=fixture@local",+ "commit",+ "-q",+ "--no-gpg-sign",+ "-m",+ "fixture: tracked by an earlier version",+ ],+ )+ .unwrap();+ // Content no other file or commit holds, so a blob for it can only come+ // from staging this file.+ let new_ledger = format!("{LEDGER_BODY}## [2026-09-06T00:00:00Z] stop\n");+ write(&root, "p/log-2026-09.md", &new_ledger);+ write(&root, "p/log.md", PROSE_BODY);+ write(&root, "p/page.md", "page\n");++ let oid = commit_all_fallback(&root, "fallback", git2::Error::from_str("forced"))+ .unwrap()+ .expect("the page is new");++ let repo = Repository::open(&root).unwrap();+ let tree = repo.find_commit(oid).unwrap().tree().unwrap();+ assert!(tree.get_path(Path::new("p/page.md")).is_ok());+ assert!(tree.get_path(Path::new("p/log.md")).is_ok(), "a page");+ assert!(tree.get_path(Path::new("p/log-2026-09.md")).is_err());+ assert!(+ tree.get_path(Path::new("p/log-2026-07.md")).is_err(),+ "the tracked ledger was untracked"+ );+ assert!(root.join("p/log-2026-07.md").is_file(), "and kept on disk");+ let unseen = git2::Oid::hash_object(ObjectType::Blob, new_ledger.as_bytes()).unwrap();+ assert!(+ !repo.odb().unwrap().exists(unseen),+ "git read the new ledger into a blob"+ );+ drop(adapter);+ }++ /// The state an older version could leave behind: it staged an append to a+ /// ledger and never committed it, and the hook appended again after that.+ /// HEAD, the index and the file then hold three different contents, which+ /// plain `git rm --cached` refuses; that refusal must not abort the+ /// checkpoint, which still has ordinary pages to save.+ #[cfg(windows)]+ #[test]+ fn cli_fallback_untracks_a_ledger_whose_index_matches_neither_head_nor_disk() {+ let tmp = tempdir();+ let root = tmp.path().join("wiki");+ let adapter = GitAdapter::open_or_init(&root).unwrap();+ Repository::open(&root)+ .unwrap()+ .config()+ .unwrap()+ .set_bool("commit.gpgsign", false)+ .unwrap();+ let ledger = "p/log-2026-09.md";+ let version_a = LEDGER_BODY.to_string();+ let version_b = format!(+ "{version_a}## [2026-09-06T00:00:00Z] stop+"+ );+ let version_c = format!(+ "{version_b}## [2026-09-07T00:00:00Z] start+"+ );++ write(&root, ledger, &version_a);+ run_git(&root, ["add", ledger]).unwrap();+ run_git(+ &root,+ [+ "-c",+ "user.name=fixture",+ "-c",+ "user.email=fixture@local",+ "commit",+ "-q",+ "--no-gpg-sign",+ "-m",+ "fixture: HEAD holds version A",+ ],+ )+ .unwrap();+ write(&root, ledger, &version_b);+ run_git(&root, ["add", ledger]).unwrap();+ write(&root, ledger, &version_c);+ write(+ &root,+ "p/page.md",+ "page+",+ );++ let oid = commit_all_fallback(&root, "fallback", git2::Error::from_str("forced"))+ .expect("the unforced removal aborted the whole checkpoint")+ .expect("the page is new");++ let repo = Repository::open(&root).unwrap();+ let tree = repo.find_commit(oid).unwrap().tree().unwrap();+ assert!(+ tree.get_path(Path::new("p/page.md")).is_ok(),+ "the ordinary page was not committed"+ );+ assert!(+ tree.get_path(Path::new(ledger)).is_err(),+ "the ledger is still in the commit"+ );+ assert!(!indexed(&root, ledger), "the ledger is still in the index");+ assert_eq!(+ std::fs::read_to_string(root.join(ledger)).unwrap(),+ version_c,+ "the file on disk was changed"+ );+ drop(adapter);+ }+#[test]fn the_adapters_writes_report_their_paths() {let (_tmp, root, adapter) = committed(&[("a.md", "a")]);diff --git a/crates/ai-memory-wiki/tests/suite/mod.rs b/crates/ai-memory-wiki/tests/suite/mod.rsindex 579699a2..2edf7da4 100644--- a/crates/ai-memory-wiki/tests/suite/mod.rs+++ b/crates/ai-memory-wiki/tests/suite/mod.rs@@ -3,3 +3,4 @@//! no extra binary; a new file must be declared below.mod admission;+mod wiki_git_ledgers;diff --git a/crates/ai-memory-wiki/tests/suite/wiki_git_ledgers.rs b/crates/ai-memory-wiki/tests/suite/wiki_git_ledgers.rsnew file mode 100644index 00000000..2ce8c345--- /dev/null+++ b/crates/ai-memory-wiki/tests/suite/wiki_git_ledgers.rs@@ -0,0 +1,338 @@+//! The raw hook event ledgers must stay out of the wiki's git history.+//!+//! Every checkpoint used to stage every changed file, so each hook append to a+//! `log-YYYY-MM.md` of hundreds of megabytes wrote a new whole-file blob, and+//! nothing ever packed them: one install's `wiki/.git` reached 3.49 GB. A+//! ledger is recognized by name *and* content (an ordinary page can be named+//! `log-2026-09.md` and must stay versioned), and the file itself is never+//! touched: only the git index entry goes.++use std::path::{Path, PathBuf};++use ai_memory_core::{PagePath, ProjectId, WorkspaceId};+use ai_memory_store::Store;+use ai_memory_wiki::Wiki;+use git2::{ObjectType, Oid, Repository, Signature, Time};+use tempfile::TempDir;++const LEDGER: &str = "log-2026-09.md";++/// A ledger line exactly as the hooks append it.+fn ledger_line(n: usize) -> String {+ format!(+ "## [2026-09-05T00:00:{:02}Z] tool Bash: cargo test\n",+ n % 60+ )+}++/// The same ledger after the OKF v0.2 migration stamped frontmatter on it.+fn conformed_ledger() -> String {+ format!(+ "---\ntype: Note\ntitle: Log 2026-09\n---\n\n{}",+ ledger_line(0)+ )+}++/// An ordinary page that merely wears the ledger's filename.+const PROSE: &str =+ "# Log 2026-09\n\nA hand-written page that happens to be named like the ledger.\n";++struct Fixture {+ _tmp: TempDir,+ _store: Store,+ wiki: Wiki,+ ws: WorkspaceId,+ proj: ProjectId,+}++async fn fixture() -> Fixture {+ let tmp = TempDir::new().unwrap();+ let store = Store::open(tmp.path()).unwrap();+ let ws = store+ .writer+ .get_or_create_workspace("default")+ .await+ .unwrap();+ let proj = store+ .writer+ .get_or_create_project(ws, "scratch", None)+ .await+ .unwrap();+ let wiki = Wiki::new(tmp.path(), store.writer.clone())+ .unwrap()+ .with_store_reader(store.reader.clone());+ Fixture {+ _tmp: tmp,+ _store: store,+ wiki,+ ws,+ proj,+ }+}++impl Fixture {+ fn root(&self) -> &Path {+ self.wiki.git().root()+ }++ /// Path of `name` inside the project, as git names it.+ fn rel(&self, name: &str) -> PathBuf {+ PathBuf::from(format!("{}/{}/{name}", self.ws, self.proj))+ }++ /// Append through the hook ledger's own entry point, which reports the+ /// write to the next checkpoint.+ fn append(&self, name: &str, text: &str) -> PathBuf {+ self.wiki+ .append_under_project(self.ws, self.proj, name, text.as_bytes())+ .unwrap()+ }++ /// Write a page the way the adapter's callers do, reporting the path.+ fn write_page(&self, name: &str, text: &str) -> PathBuf {+ let path = PagePath::new(name).unwrap();+ let abs = self.wiki.abs_path(self.ws, self.proj, &path);+ self.wiki.git().write_atomic(&abs, text.as_bytes()).unwrap();+ abs+ }++ fn commit(&self, message: &str) -> Oid {+ self.wiki+ .commit_all(message)+ .unwrap()+ .expect("the checkpoint had something to commit")+ }++ /// What an earlier version did: put the file in a commit of its own, behind+ /// the adapter's back. The signature is fixed and libgit2 never signs, so a+ /// contributor's global `commit.gpgsign` cannot reach the fixture.+ fn commit_tracking(&self, rel: &Path) {+ let repo = Repository::open(self.root()).unwrap();+ let mut index = repo.index().unwrap();+ index.add_path(rel).unwrap();+ index.write().unwrap();+ let tree = repo.find_tree(index.write_tree().unwrap()).unwrap();+ let sig = Signature::new("fixture", "fixture@local", &Time::new(0, 0)).unwrap();+ let parents: Vec<_> = repo+ .head()+ .ok()+ .and_then(|h| h.target())+ .map(|oid| repo.find_commit(oid).unwrap())+ .into_iter()+ .collect();+ let parent_refs: Vec<&git2::Commit<'_>> = parents.iter().collect();+ repo.commit(+ Some("HEAD"),+ &sig,+ &sig,+ "fixture: ledger tracked by an earlier version",+ &tree,+ &parent_refs,+ )+ .unwrap();+ }++ /// The file's content in HEAD's tree, `None` when HEAD does not name it.+ fn at_head(&self, rel: &Path) -> Option<Vec<u8>> {+ let repo = Repository::open(self.root()).unwrap();+ let tree = repo.head().unwrap().peel_to_tree().unwrap();+ let entry = tree.get_path(rel).ok()?;+ Some(repo.find_blob(entry.id()).unwrap().content().to_vec())+ }++ /// Whether a blob with exactly this content exists anywhere in the object+ /// database, loose or packed: the cost the fix is about.+ fn blob_stored(&self, bytes: &[u8]) -> bool {+ let repo = Repository::open(self.root()).unwrap();+ let oid = Oid::hash_object(ObjectType::Blob, bytes).unwrap();+ repo.odb().unwrap().exists(oid)+ }+}++#[tokio::test]+async fn a_checkpoint_after_a_ledger_append_does_not_commit_the_ledger() {+ let fx = fixture().await;+ fx.write_page("notes/a.md", "first\n");+ let abs = fx.append(LEDGER, &ledger_line(0));++ // The first checkpoint walks the whole tree.+ fx.commit("walk");+ assert!(+ fx.at_head(&fx.rel("notes/a.md")).is_some(),+ "control: the page is committed"+ );+ assert!(+ fx.at_head(&fx.rel(LEDGER)).is_none(),+ "the walk staged the ledger"+ );+ assert!(+ !fx.blob_stored(ledger_line(0).as_bytes()),+ "the ledger was read into a blob even though it was left out of the tree"+ );++ // The next one stages only the reported paths.+ let grown = format!("{}{}", ledger_line(0), ledger_line(1));+ fx.append(LEDGER, &ledger_line(1));+ fx.write_page("notes/a.md", "second\n");+ fx.commit("paths");+ assert_eq!(fx.at_head(&fx.rel("notes/a.md")).unwrap(), b"second\n");+ assert!(+ fx.at_head(&fx.rel(LEDGER)).is_none(),+ "a reported ledger append was staged"+ );+ assert!(!fx.blob_stored(grown.as_bytes()));+ assert_eq!(+ std::fs::read_to_string(abs).unwrap(),+ grown,+ "the ledger on disk is the hooks' file, untouched"+ );+}++#[tokio::test]+async fn a_ledger_alone_leaves_nothing_to_commit() {+ let fx = fixture().await;+ fx.write_page("notes/a.md", "first\n");+ fx.commit("walk");++ fx.append(LEDGER, &ledger_line(0));++ assert!(+ fx.wiki.commit_all("ledger only").unwrap().is_none(),+ "a hook append alone must not produce a checkpoint"+ );+}++#[tokio::test]+async fn every_shape_of_ledger_is_left_out() {+ let fx = fixture().await;+ fx.write_page("notes/a.md", "first\n");+ fx.append("log.md", &ledger_line(0));+ fx.append("log-2026-08.md", &conformed_ledger());+ fx.append(LEDGER, &ledger_line(0));++ fx.commit("walk");++ for name in ["log.md", "log-2026-08.md", LEDGER] {+ assert!(fx.at_head(&fx.rel(name)).is_none(), "{name} was committed");+ }+}++#[tokio::test]+async fn a_ledger_an_earlier_version_committed_leaves_the_index_but_not_the_disk() {+ let fx = fixture().await;+ let abs = fx.append(LEDGER, &ledger_line(0));+ fx.commit_tracking(&fx.rel(LEDGER));+ let old = ledger_line(0);+ assert_eq!(fx.at_head(&fx.rel(LEDGER)).unwrap(), old.as_bytes());++ // It keeps growing, as it does in production.+ fx.append(LEDGER, &ledger_line(1));+ let grown = format!("{old}{}", ledger_line(1));+ fx.write_page("notes/a.md", "first\n");+ fx.commit("after upgrade");++ assert!(+ fx.at_head(&fx.rel(LEDGER)).is_none(),+ "the tracked ledger is still in the tree"+ );+ assert!(fx.at_head(&fx.rel("notes/a.md")).is_some());+ assert_eq!(+ std::fs::read_to_string(abs).unwrap(),+ grown,+ "dropping it from the index must not delete or alter the file"+ );+ assert!(+ !fx.blob_stored(grown.as_bytes()),+ "the grown ledger was hashed into git"+ );+}++/// A closed month nobody appends to is not offered to the staging walk (its+/// stat data is unchanged), so only a scan of the index can untrack it.+#[tokio::test]+async fn an_unchanged_tracked_ledger_is_untracked_too() {+ let fx = fixture().await;+ let abs = fx.append("log-2026-07.md", &ledger_line(0));+ fx.commit_tracking(&fx.rel("log-2026-07.md"));++ fx.write_page("notes/a.md", "first\n");+ fx.commit("after upgrade");++ assert!(fx.at_head(&fx.rel("log-2026-07.md")).is_none());+ assert_eq!(std::fs::read_to_string(abs).unwrap(), ledger_line(0));+}++/// Adversarial control: the name alone must never keep a page out of git.+#[tokio::test]+async fn a_page_wearing_a_ledgers_name_is_still_committed() {+ let fx = fixture().await;+ fx.append(LEDGER, PROSE);+ fx.append("log.md", "# Log\n\nNot a hook ledger either.\n");+ fx.append(+ "log-2026-08.md",+ "---\ntitle: Notes\n---\n\nprose, not a log entry\n",+ );++ fx.commit("walk");+ assert_eq!(+ fx.at_head(&fx.rel(LEDGER)).unwrap(),+ PROSE.as_bytes(),+ "an ordinary page named like a ledger was dropped from git"+ );+ assert!(fx.at_head(&fx.rel("log.md")).is_some());+ assert!(fx.at_head(&fx.rel("log-2026-08.md")).is_some());++ // And its edits keep being versioned through the path-scoped checkpoint.+ let edited = format!("{PROSE}\nMore.\n");+ fx.append(LEDGER, "\nMore.\n");+ fx.commit("paths");+ assert_eq!(fx.at_head(&fx.rel(LEDGER)).unwrap(), edited.as_bytes());+}++/// The other half of the gate: a ledger body under a name that is not a ledger+/// filename is an ordinary file.+#[tokio::test]+async fn ledger_content_under_another_name_is_still_committed() {+ let fx = fixture().await;+ fx.append("log-2026-09-15.md", &ledger_line(0));+ fx.append("notes.md", &ledger_line(0));++ fx.commit("walk");++ assert!(fx.at_head(&fx.rel("log-2026-09-15.md")).is_some());+ assert!(fx.at_head(&fx.rel("notes.md")).is_some());+}++/// A directory named like a ledger is a directory: its files are pages like+/// any other.+#[tokio::test]+async fn a_directory_named_like_a_ledger_keeps_its_files_versioned() {+ let fx = fixture().await;+ fx.write_page("log.md/inner.md", "inside\n");++ fx.commit("walk");++ assert_eq!(fx.at_head(&fx.rel("log.md/inner.md")).unwrap(), b"inside\n");+}++#[tokio::test]+async fn restoring_a_page_from_a_checkpoint_still_works_beside_a_ledger() {+ let fx = fixture().await;+ let path = PagePath::new("sessions/restored.md").unwrap();+ fx.write_page(path.as_str(), "---\ntitle: Restored\n---\nold");+ fx.append(LEDGER, &ledger_line(0));+ let rev = fx.commit("checkpoint").to_string();++ fx.write_page(path.as_str(), "---\ntitle: Restored\n---\nnew");+ fx.append(LEDGER, &ledger_line(1));+ fx.commit("later");+ assert_eq!(fx.wiki.git().recent_checkpoints(10).unwrap().len(), 2);++ fx.wiki+ .restore_page_from_checkpoint(fx.ws, fx.proj, path.clone(), &rev)+ .await+ .unwrap();++ let md = fx.wiki.read_page(fx.ws, fx.proj, &path).unwrap();+ assert_eq!(md.body.trim(), "old");+}diff --git a/docs/lifecycle-ops.md b/docs/lifecycle-ops.mdindex 17a7bb45..cc582492 100644--- a/docs/lifecycle-ops.md+++ b/docs/lifecycle-ops.md@@ -944,6 +944,52 @@ What it does not recover:embeddings. Those live only in SQLite and require a full `backup` / `restore`if you need to roll them back.+### Hook event ledgers are not in the wiki's git history++The raw hook event ledgers (`log.md` and `log-YYYY-MM.md` under each project)+are append-only logs that can reach hundreds of megabytes. Git stores every new+version of a file as a whole blob, and the server never packs or prunes the wiki+repository, so each checkpoint that followed a hook append added another copy of+the ledger: one install's `wiki/.git` grew to 3.49 GB. Nothing reads ledger+history back from git: `restore-page` and `checkpoints` work on pages, and+`backup` copies the data directory from disk.++Since this change a checkpoint leaves ledgers out of the commit. A file counts as+a ledger only when its name has the ledger shape **and** its first 8 KiB open+with a hook log entry (`## [timestamp] ...`, under optional frontmatter), the+same test the indexer and `reclaim-ledger-versions` use. A page that merely is+named `log-2026-09.md` stays versioned. The ledger file itself is never touched.+A ledger that an earlier version already committed is dropped from the git index+(`git rm --cached`) by the first checkpoint that sees it, so existing installs+stop growing without any step from you. The check runs when a checkpoint stages+files and does a bounded prefix read (at most 8 KiB) per check, never the+whole file; a ledger appended between that read and the commit is simply not in+the commit.++What this does **not** do: history already written keeps its old blobs, so+`wiki/.git` does not shrink by itself, and checkpoints taken before the upgrade+still contain old ledger versions. The server never rewrites or garbage-collects+that history. Shrinking it is an operator task, and it is destructive: every+commit id changes, so OIDs you noted from `ai-memory checkpoints` stop resolving,+and `git gc --prune=now` makes the old blobs unrecoverable. Stop the server, take+a `backup` first, and only then, as an example:++```bash+cd <data-dir>/wiki+# Let the upgraded server take one checkpoint first, so HEAD no longer tracks the+# ledgers, and copy them aside anyway: a history rewrite resets the working tree.+# The globs match by name only. If you keep an ordinary page named like a+# ledger, they also delete that page's history, so narrow them first.+git filter-repo --force --invert-paths \+ --path-glob '*/log.md' \+ --path-glob '*/log-[0-9][0-9][0-9][0-9]-[0-9][0-9].md'+git reflog expire --expire=now --all+git gc --prune=now --aggressive+```++Start the server again afterwards. `git filter-repo` is a separate tool and is+not installed with ai-memory.+### `backup````bash