Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(wiki): fsync the wiki's git objects and refs, report an unreadable HEAD

AkitaOnRails8fdf1fd10898
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 91cde92d..c01e0d42 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -223,6 +223,16 @@ version in [`docs/changelog/`](docs/changelog/):
relay and wiki sync companions take the same `rustls-webpki` bump.
### Fixed
+- The wiki's git objects and refs are now synced to disk. libgit2 renamed them
+ into place without fsync, so a power loss could keep a ref naming a commit
+ whose object was never written, and every later checkpoint failed. Opening
+ the wiki now sets `core.fsyncObjectFiles` in the repository's own config (an
+ explicit `false` is kept), startup logs an `ERROR` when `HEAD` does not
+ resolve to a readable commit, and a failed bootstrap auto-commit is logged
+ instead of dropped. `docs/lifecycle-ops.md` describes the repair for a
+ history already damaged. The git CLI prints a deprecation warning for the
+ key, which libgit2 still needs.
+ ([#1511](https://github.com/akitaonrails/ai-memory/issues/1511))
- Reserved local-embedding capacity for interactive search queries instead of
allowing concurrent page writes and backfill work to occupy every inference
slot. Excess queries fall back immediately to non-vector retrieval, and a
diff --git a/crates/ai-memory-consolidate/src/bootstrap.rs b/crates/ai-memory-consolidate/src/bootstrap.rs
index 8308c430..aaed0e9c 100644
--- a/crates/ai-memory-consolidate/src/bootstrap.rs
+++ b/crates/ai-memory-consolidate/src/bootstrap.rs
@@ -680,10 +680,12 @@ impl Bootstrap {
}
let _ids = self.wiki.apply_batch(requests).await?;
- let _ = self.wiki.commit_all(&format!(
+ if let Err(e) = self.wiki.commit_all(&format!(
"bootstrap: {llm_chunks} LLM chunk(s), ingested {collected} sources, wrote {} pages",
written_paths.len() + 1,
- ));
+ )) {
+ warn!(error = %e, "bootstrap auto-commit failed");
+ }
// Best-effort: the run already succeeded (pages are written), so a
// failure here must not fail the whole run — it would just leave
diff --git a/crates/ai-memory-wiki/src/git.rs b/crates/ai-memory-wiki/src/git.rs
index ff1419af..90788ae7 100644
--- a/crates/ai-memory-wiki/src/git.rs
+++ b/crates/ai-memory-wiki/src/git.rs
@@ -16,6 +16,10 @@
//! They are append-only logs that reach hundreds of megabytes, git stores each
//! new version as a whole blob, and nothing reads them back from git; see
//! [`is_ledger_file`].
+//!
+//! Opening the repository turns on libgit2's fsync for `.git` through the
+//! repository's own config, so a crash cannot leave a ref naming an object
+//! whose bytes never reached disk; see [`enable_object_fsync`].
use std::collections::BTreeSet;
use std::path::{Path, PathBuf};
@@ -23,7 +27,7 @@ use std::sync::{Arc, Mutex, MutexGuard};
use std::time::{Duration, Instant};
use git2::{ErrorCode, IndexAddOption, ObjectType, Repository, Signature};
-use tracing::{debug, warn};
+use tracing::{debug, error, warn};
use crate::error::{WikiError, WikiResult};
@@ -185,18 +189,95 @@ fn map_open_error(error: OpenRepositoryError) -> WikiError {
}
}
-fn validate_repository_or_allow_cli_fallback(root: &Path) -> WikiResult<()> {
+/// The validated repository, or `None` when libgit2 cannot open it and the
+/// Windows git CLI fallback will serve it instead.
+fn validate_repository_or_allow_cli_fallback(root: &Path) -> WikiResult<Option<Repository>> {
match open_validated_repository(root) {
- Ok(_) => Ok(()),
+ Ok(repo) => Ok(Some(repo)),
Err(OpenRepositoryError::Git(error))
if cfg!(windows) && should_try_commit_cli_fallback(&error) =>
{
- validate_cli_fallback_root(root)
+ validate_cli_fallback_root(root).map(|()| None)
}
Err(error) => Err(map_open_error(error)),
}
}
+/// The repository-config key that makes libgit2 fsync each loose object,
+/// ref and reflog (and the directory it lands in) before renaming it into
+/// place.
+const FSYNC_OBJECT_FILES: &str = "core.fsyncObjectFiles";
+
+/// Turn on libgit2's fsync for the wiki repository in its local config,
+/// unless that config already names a value. Returns whether it is on.
+///
+/// libgit2 writes objects and refs to a temp file and renames it, without
+/// fsync by default. After a crash the rename, and the ref update that names
+/// the new commit, can survive while the object's bytes never reached disk:
+/// zero-length loose objects and a HEAD that no longer reads, so every later
+/// checkpoint fails (#1511). The process-wide `GIT_OPT_ENABLE_FSYNC_GITDIR`
+/// covers the same writes, but git2 has no safe wrapper for it and the
+/// workspace forbids `unsafe`. The config key is per repository, persists in
+/// `.git/config`, and the git CLI honours it too, with a warning that it is
+/// deprecated in favour of `core.fsync`. An explicit `false` is the
+/// operator's choice and is kept.
+fn enable_object_fsync(repo: &Repository) -> Result<bool, git2::Error> {
+ let mut local = repo.config()?.open_level(git2::ConfigLevel::Local)?;
+ match local.get_bool(FSYNC_OBJECT_FILES) {
+ Ok(enabled) => Ok(enabled),
+ Err(error) if error.code() == ErrorCode::NotFound => {
+ local.set_bool(FSYNC_OBJECT_FILES, true)?;
+ Ok(true)
+ }
+ Err(error) => Err(error),
+ }
+}
+
+/// `Ok` when HEAD is unborn (a fresh repository) or resolves to a commit
+/// whose tree reads back; the error otherwise, such as libgit2's
+/// "failed to parse loose object" for an object a crash left empty.
+fn readable_head(repo: &Repository) -> Result<(), git2::Error> {
+ let head = match repo.head() {
+ Ok(head) => head,
+ Err(error) if matches!(error.code(), ErrorCode::UnbornBranch | ErrorCode::NotFound) => {
+ return Ok(());
+ }
+ Err(error) => return Err(error),
+ };
+ head.peel_to_commit()?.tree()?;
+ Ok(())
+}
+
+/// Startup checks on the wiki repository. Neither fails the server: the
+/// markdown on disk is the source of truth and keeps being served, but a
+/// history that cannot take a commit is reported once, loudly, instead of
+/// only as a warning from each checkpoint that fails.
+fn check_repository_durability(root: &Path, repo: &Repository) {
+ match enable_object_fsync(repo) {
+ Ok(true) => {}
+ Ok(false) => warn!(
+ root = %root.display(),
+ "{FSYNC_OBJECT_FILES} is false in the wiki repository's config; a crash can leave \
+ its history with empty objects"
+ ),
+ Err(error) => warn!(
+ root = %root.display(),
+ %error,
+ "could not turn on fsync for the wiki repository; a crash can leave its history \
+ with empty objects"
+ ),
+ }
+ if let Err(error) = readable_head(repo) {
+ error!(
+ root = %root.display(),
+ %error,
+ "wiki git history is unreadable: HEAD does not resolve to a readable commit, so \
+ every checkpoint will fail until it is repaired. Pages on disk are not affected. \
+ See \"Wiki git history unreadable after a crash\" in docs/lifecycle-ops.md"
+ );
+ }
+}
+
fn validate_cli_fallback_root(root: &Path) -> WikiResult<()> {
let git_dir = crate::confinement::inspect_git_directory(root)?;
crate::confinement::inspect_tree_except(root, &[git_dir.as_path()])?;
@@ -286,17 +367,20 @@ impl GitAdapter {
pub fn open_or_init(root: &Path) -> WikiResult<Self> {
crate::confinement::initialize_root(root)?;
let git_dir = root.join(".git");
- match std::fs::symlink_metadata(&git_dir) {
+ let repo = match std::fs::symlink_metadata(&git_dir) {
Ok(_) => {
- validate_repository_or_allow_cli_fallback(root)?;
debug!(root = %root.display(), "wiki repo already initialised");
+ validate_repository_or_allow_cli_fallback(root)?
}
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
debug!(root = %root.display(), "initialising wiki repo");
init_repo(root)?;
- validate_repository_or_allow_cli_fallback(root)?;
+ validate_repository_or_allow_cli_fallback(root)?
}
Err(error) => return Err(error.into()),
+ };
+ if let Some(repo) = repo {
+ check_repository_durability(root, &repo);
}
Ok(Self {
root: root.to_path_buf(),
@@ -2695,4 +2779,79 @@ mod tests {
.unwrap();
assert_eq!(String::from_utf8(bytes).unwrap(), "one");
}
+
+ fn local_fsync_setting(root: &Path) -> Option<bool> {
+ Repository::open(root)
+ .unwrap()
+ .config()
+ .unwrap()
+ .open_level(git2::ConfigLevel::Local)
+ .unwrap()
+ .get_bool(FSYNC_OBJECT_FILES)
+ .ok()
+ }
+
+ /// #1511: without it libgit2 renames objects and refs into place unsynced,
+ /// and a crash left a ref naming an empty commit object.
+ #[test]
+ fn opening_the_wiki_turns_on_git_object_fsync() {
+ let tmp = tempdir();
+ let fresh = tmp.path().join("fresh");
+ GitAdapter::open_or_init(&fresh).unwrap();
+ assert_eq!(local_fsync_setting(&fresh), Some(true));
+
+ // A repository an older version created gets it on the next open.
+ let existing = tmp.path().join("existing");
+ Repository::init(&existing).unwrap();
+ assert_eq!(local_fsync_setting(&existing), None);
+ GitAdapter::open_or_init(&existing).unwrap();
+ assert_eq!(local_fsync_setting(&existing), Some(true));
+ }
+
+ #[test]
+ fn an_explicit_fsync_opt_out_is_kept() {
+ let tmp = tempdir();
+ let root = tmp.path().join("wiki");
+ Repository::init(&root)
+ .unwrap()
+ .config()
+ .unwrap()
+ .set_bool(FSYNC_OBJECT_FILES, false)
+ .unwrap();
+
+ GitAdapter::open_or_init(&root).unwrap();
+
+ assert_eq!(local_fsync_setting(&root), Some(false));
+ }
+
+ /// The crash shape from #1511: HEAD names a commit whose loose object is
+ /// empty. Startup still opens the wiki, the check reports it, and a
+ /// checkpoint returns the error instead of passing for a clean tree.
+ #[test]
+ fn an_empty_head_commit_object_is_reported_not_swallowed() {
+ let (_tmp, root, _adapter) = committed(&[("page.md", "page")]);
+ let repo = Repository::open(&root).unwrap();
+ readable_head(&repo).unwrap();
+ let head = repo.head().unwrap().target().unwrap().to_string();
+ let object = root.join(".git/objects").join(&head[..2]).join(&head[2..]);
+ let mut permissions = std::fs::metadata(&object).unwrap().permissions();
+ #[allow(clippy::permissions_set_readonly_false)]
+ permissions.set_readonly(false);
+ std::fs::set_permissions(&object, permissions).unwrap();
+ std::fs::write(&object, b"").unwrap();
+ drop(repo);
+
+ let reopened = GitAdapter::open_or_init(&root).unwrap();
+ assert!(readable_head(&Repository::open(&root).unwrap()).is_err());
+ write(&root, "page.md", "page, edited after the crash");
+ assert!(reopened.commit_all("after the crash").is_err());
+ }
+
+ #[test]
+ fn a_fresh_repository_has_a_readable_head() {
+ let tmp = tempdir();
+ let root = tmp.path().join("wiki");
+ GitAdapter::open_or_init(&root).unwrap();
+ readable_head(&Repository::open(&root).unwrap()).unwrap();
+ }
}
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 838b8675..a24fef82 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -1055,6 +1055,11 @@ that touch the relevant area.
[design-windows-ci.md](design-windows-ci.md#windows-transient-file-errors).
New file-I/O code with the same race follows the same pattern, with a
test that injects the error so it also runs off Windows.
+ The wiki's `.git` gets the same durability from libgit2: opening the
+ wiki sets `core.fsyncObjectFiles` in the repository's config, so objects
+ and refs are synced before their rename, and startup logs an `ERROR` when
+ `HEAD` does not resolve to a readable commit (the repair is in
+ [lifecycle-ops.md](lifecycle-ops.md#wiki-git-history-unreadable-after-a-crash)).
11. **Absolute canonical data dir** default; logged loudly on
startup. (agentmemory #303.)
12. **No global singletons / `lazy_static` configs.** All deps
diff --git a/docs/lifecycle-ops.md b/docs/lifecycle-ops.md
index bc8d1c73..949823b6 100644
--- a/docs/lifecycle-ops.md
+++ b/docs/lifecycle-ops.md
@@ -1368,6 +1368,40 @@ ai-memory move-session --from-project tmp --to my-project --confirm
ai-memory purge-project --project tmp --confirm
```
+### Wiki git history unreadable after a crash
+
+The server turns on `core.fsyncObjectFiles` in `wiki/.git/config` when it opens
+the wiki, so libgit2 syncs each object and ref before renaming it into place.
+Before that, a power loss or kernel crash could keep a ref update while the
+objects it names were never written: zero-length files under `.git/objects`,
+`HEAD` pointing at one of them, and every later checkpoint failing with
+`failed to parse loose object: invalid header`. The pages on disk are not
+affected; only their history stops. At startup the server logs an `ERROR`
+("wiki git history is unreadable") when `HEAD` does not resolve to a readable
+commit.
+
+The working tree is the source of truth, so repair points the branch back at
+the newest intact commit and lets the next checkpoint record everything since.
+Stop the server first, then from `<data_dir>/wiki`:
+
+```bash
+git fsck --full # names the empty objects and the bad ref
+mkdir -p ../wiki-empty-objects # move them out, don't delete them
+find .git/objects -type f -empty -exec mv {} ../wiki-empty-objects/ \;
+# Newest reflog entry whose commit and tree still read back:
+for oid in $(awk '{print $2}' .git/logs/HEAD | tac); do
+ git cat-file -e "$oid^{tree}" 2>/dev/null && echo "$oid" && break
+done
+git update-ref refs/heads/main <oid> # the branch `git symbolic-ref HEAD` names
+rm .git/index && git reset -q # rebuild the index; pages are untouched
+git reflog expire --stale-fix --expire=never --all
+git fsck --full # clean (a dangling blob is harmless)
+```
+
+Start the server again; the next checkpoint commits the pages written after
+that commit. Git prints a deprecation warning for `core.fsyncObjectFiles`
+(it prefers `core.fsync`); libgit2 reads only the older key, so leave it set.
+
## Why this matters: the flat-wiki incident
Before the per-project disk layout (commits up to `e7b9a17`), the