Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(wiki): keep the libgit2 fsync setting out of .git/config

AkitaOnRailscd88045f452e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c01e0d42..db24fcdb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -225,13 +225,14 @@ version in [`docs/changelog/`](docs/changelog/):
### Fixed
- The wiki's git objects and refs are now synced to disk. libgit2 renamed them
into place without fsync, so a power loss could keep a ref naming a commit
- whose object was never written, and every later checkpoint failed. Opening
- the wiki now sets `core.fsyncObjectFiles` in the repository's own config (an
- explicit `false` is kept), startup logs an `ERROR` when `HEAD` does not
+ whose object was never written, and every later checkpoint failed. Every
+ repository handle ai-memory opens now turns on `core.fsyncObjectFiles` from
+ `.git/ai-memory.gitconfig`, a file only libgit2 reads, so `.git/config` is
+ untouched and the git CLI shows no deprecation warning; an explicit `false`
+ in `.git/config` is kept. Startup logs an `ERROR` when `HEAD` does not
resolve to a readable commit, and a failed bootstrap auto-commit is logged
instead of dropped. `docs/lifecycle-ops.md` describes the repair for a
- history already damaged. The git CLI prints a deprecation warning for the
- key, which libgit2 still needs.
+ history already damaged.
([#1511](https://github.com/akitaonrails/ai-memory/issues/1511))
- Reserved local-embedding capacity for interactive search queries instead of
allowing concurrent page writes and backfill work to occupy every inference
diff --git a/crates/ai-memory-wiki/src/git.rs b/crates/ai-memory-wiki/src/git.rs
index 90788ae7..b2d9ca10 100644
--- a/crates/ai-memory-wiki/src/git.rs
+++ b/crates/ai-memory-wiki/src/git.rs
@@ -17,9 +17,9 @@
//! new version as a whole blob, and nothing reads them back from git; see
//! [`is_ledger_file`].
//!
-//! Opening the repository turns on libgit2's fsync for `.git` through the
-//! repository's own config, so a crash cannot leave a ref naming an object
-//! whose bytes never reached disk; see [`enable_object_fsync`].
+//! Every repository handle turns on libgit2's fsync for `.git` through a
+//! config file only libgit2 reads, so a crash cannot leave a ref naming an
+//! object whose bytes never reached disk; see [`attach_libgit2_config`].
use std::collections::BTreeSet;
use std::path::{Path, PathBuf};
@@ -179,6 +179,7 @@ fn open_validated_repository(root: &Path) -> Result<Repository, OpenRepositoryEr
reason: "wiki repository metadata escapes the in-root .git directory",
}));
}
+ attach_libgit2_config(&repo).map_err(OpenRepositoryError::Git)?;
Ok(repo)
}
@@ -203,13 +204,35 @@ fn validate_repository_or_allow_cli_fallback(root: &Path) -> WikiResult<Option<R
}
}
-/// The repository-config key that makes libgit2 fsync each loose object,
-/// ref and reflog (and the directory it lands in) before renaming it into
-/// place.
+/// The config key that makes libgit2 fsync each loose object, ref and reflog
+/// (and the directory it lands in) before renaming it into place.
const FSYNC_OBJECT_FILES: &str = "core.fsyncObjectFiles";
-/// Turn on libgit2's fsync for the wiki repository in its local config,
-/// unless that config already names a value. Returns whether it is on.
+/// Config file in the wiki's `.git` that only libgit2 reads: git looks for
+/// `config` (and `config.worktree`) there, never for this name.
+const LIBGIT2_CONFIG_FILE: &str = "ai-memory.gitconfig";
+
+const LIBGIT2_CONFIG: &str = "\
+# Written by ai-memory and read only by its libgit2 handles (#1511).
+[core]
+\tfsyncObjectFiles = true
+";
+
+/// Write [`LIBGIT2_CONFIG_FILE`] into the wiki's `.git` unless it already
+/// holds [`LIBGIT2_CONFIG`]. Runs once per adapter, at startup.
+fn write_libgit2_config(git_dir: &Path) -> WikiResult<()> {
+ let path = git_dir.join(LIBGIT2_CONFIG_FILE);
+ if std::fs::read(&path).is_ok_and(|bytes| bytes == LIBGIT2_CONFIG.as_bytes()) {
+ return Ok(());
+ }
+ // Inside `.git`, which a commit never stages, so there is nothing to report.
+ #[allow(clippy::disallowed_methods)]
+ crate::atomic::write_atomic(&path, LIBGIT2_CONFIG.as_bytes())?;
+ Ok(())
+}
+
+/// Give this handle libgit2's fsync for `.git`, unless the repository's own
+/// config turns it off.
///
/// libgit2 writes objects and refs to a temp file and renames it, without
/// fsync by default. After a crash the rename, and the ref update that names
@@ -217,20 +240,38 @@ const FSYNC_OBJECT_FILES: &str = "core.fsyncObjectFiles";
/// zero-length loose objects and a HEAD that no longer reads, so every later
/// checkpoint fails (#1511). The process-wide `GIT_OPT_ENABLE_FSYNC_GITDIR`
/// covers the same writes, but git2 has no safe wrapper for it and the
-/// workspace forbids `unsafe`. The config key is per repository, persists in
-/// `.git/config`, and the git CLI honours it too, with a warning that it is
-/// deprecated in favour of `core.fsync`. An explicit `false` is the
-/// operator's choice and is kept.
-fn enable_object_fsync(repo: &Repository) -> Result<bool, git2::Error> {
- let mut local = repo.config()?.open_level(git2::ConfigLevel::Local)?;
- match local.get_bool(FSYNC_OBJECT_FILES) {
- Ok(enabled) => Ok(enabled),
- Err(error) if error.code() == ErrorCode::NotFound => {
- local.set_bool(FSYNC_OBJECT_FILES, true)?;
- Ok(true)
- }
- Err(error) => Err(error),
+/// workspace forbids `unsafe`. Writing the key to `.git/config` would make the
+/// git CLI warn that it is deprecated on every command an operator runs in the
+/// wiki, so it goes in [`LIBGIT2_CONFIG_FILE`], added to this handle's live
+/// config at the app level.
+///
+/// Why this reaches the fsync decision (vendored libgit2 1.9.3):
+/// `Repository::config` returns the handle's own config object, not a
+/// snapshot, and libgit2 reads `core.fsyncObjectFiles` from that object
+/// through the configmap cache when it first loads the object database
+/// (`git_odb__set_caps`) and the refdb (`refdb_fs` backend). Opening a
+/// repository loads neither, so the cached value is still unset here; this
+/// runs before any caller touches the handle.
+///
+/// The app level outranks every file git itself reads, so an explicit
+/// `false` in `.git/config` is checked first and kept. The app backend also
+/// becomes the handle's preferred writer; ai-memory never writes config
+/// through a handle.
+fn attach_libgit2_config(repo: &Repository) -> Result<(), git2::Error> {
+ let mut config = repo.config()?;
+ let local = match config.open_level(git2::ConfigLevel::Local) {
+ Ok(local) => local.get_bool(FSYNC_OBJECT_FILES).ok(),
+ Err(error) if error.code() == ErrorCode::NotFound => None,
+ Err(error) => return Err(error),
+ };
+ if local == Some(false) {
+ return Ok(());
}
+ config.add_file(
+ &repo.path().join(LIBGIT2_CONFIG_FILE),
+ git2::ConfigLevel::App,
+ false,
+ )
}
/// `Ok` when HEAD is unborn (a fresh repository) or resolves to a commit
@@ -253,7 +294,10 @@ fn readable_head(repo: &Repository) -> Result<(), git2::Error> {
/// history that cannot take a commit is reported once, loudly, instead of
/// only as a warning from each checkpoint that fails.
fn check_repository_durability(root: &Path, repo: &Repository) {
- match enable_object_fsync(repo) {
+ match repo
+ .config()
+ .and_then(|config| config.get_bool(FSYNC_OBJECT_FILES))
+ {
Ok(true) => {}
Ok(false) => warn!(
root = %root.display(),
@@ -263,7 +307,7 @@ fn check_repository_durability(root: &Path, repo: &Repository) {
Err(error) => warn!(
root = %root.display(),
%error,
- "could not turn on fsync for the wiki repository; a crash can leave its history \
+ "could not read the wiki repository's fsync setting; a crash can leave its history \
with empty objects"
),
}
@@ -366,20 +410,16 @@ impl GitAdapter {
/// Propagates any underlying libgit2 error.
pub fn open_or_init(root: &Path) -> WikiResult<Self> {
crate::confinement::initialize_root(root)?;
- let git_dir = root.join(".git");
- let repo = match std::fs::symlink_metadata(&git_dir) {
- Ok(_) => {
- debug!(root = %root.display(), "wiki repo already initialised");
- validate_repository_or_allow_cli_fallback(root)?
- }
+ match std::fs::symlink_metadata(root.join(".git")) {
+ Ok(_) => debug!(root = %root.display(), "wiki repo already initialised"),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
debug!(root = %root.display(), "initialising wiki repo");
init_repo(root)?;
- validate_repository_or_allow_cli_fallback(root)?
}
Err(error) => return Err(error.into()),
- };
- if let Some(repo) = repo {
+ }
+ write_libgit2_config(&crate::confinement::inspect_git_directory(root)?)?;
+ if let Some(repo) = validate_repository_or_allow_cli_fallback(root)? {
check_repository_durability(root, &repo);
}
Ok(Self {
@@ -2791,25 +2831,56 @@ mod tests {
.ok()
}
+ fn fsync_setting(repo: &Repository) -> Option<bool> {
+ repo.config().unwrap().get_bool(FSYNC_OBJECT_FILES).ok()
+ }
+
+ /// What the git CLI prints for `status`, or `None` without git on PATH.
+ fn git_status_stderr(root: &Path) -> Option<String> {
+ let output = git_command(root).arg("status").output().ok()?;
+ assert!(output.status.success(), "git status failed: {output:?}");
+ Some(String::from_utf8_lossy(&output.stderr).into_owned())
+ }
+
/// #1511: without it libgit2 renames objects and refs into place unsynced,
- /// and a crash left a ref naming an empty commit object.
+ /// and a crash left a ref naming an empty commit object. The setting lives
+ /// on ai-memory's handles only, never in `.git/config`, where git warns
+ /// that the key is deprecated on every command.
#[test]
- fn opening_the_wiki_turns_on_git_object_fsync() {
+ fn every_wiki_handle_turns_on_git_object_fsync_without_touching_git_config() {
let tmp = tempdir();
let fresh = tmp.path().join("fresh");
- GitAdapter::open_or_init(&fresh).unwrap();
- assert_eq!(local_fsync_setting(&fresh), Some(true));
-
- // A repository an older version created gets it on the next open.
let existing = tmp.path().join("existing");
+ // A repository an older version created gets it on the next open.
Repository::init(&existing).unwrap();
- assert_eq!(local_fsync_setting(&existing), None);
- GitAdapter::open_or_init(&existing).unwrap();
- assert_eq!(local_fsync_setting(&existing), Some(true));
+
+ for root in [&fresh, &existing] {
+ let adapter = GitAdapter::open_or_init(root).unwrap();
+ write(root, "page.md", "page");
+ adapter.commit_all("checkpoint").unwrap().unwrap();
+
+ let slot = adapter.writing();
+ let kept = &slot.as_ref().expect("the commit keeps its handle").repo;
+ assert_eq!(fsync_setting(kept), Some(true));
+ drop(slot);
+ let validated = open_validated_repository(root).ok().unwrap();
+ assert_eq!(fsync_setting(&validated), Some(true));
+
+ assert_eq!(local_fsync_setting(root), None);
+ assert_eq!(fsync_setting(&Repository::open(root).unwrap()), None);
+ let git_config = std::fs::read_to_string(root.join(".git/config")).unwrap();
+ assert!(!git_config.to_ascii_lowercase().contains("fsyncobjectfiles"));
+ if let Some(stderr) = git_status_stderr(root) {
+ assert!(
+ !stderr.contains("deprecated"),
+ "git status warned: {stderr}"
+ );
+ }
+ }
}
#[test]
- fn an_explicit_fsync_opt_out_is_kept() {
+ fn an_explicit_fsync_opt_out_in_git_config_is_kept() {
let tmp = tempdir();
let root = tmp.path().join("wiki");
Repository::init(&root)
@@ -2821,6 +2892,8 @@ mod tests {
GitAdapter::open_or_init(&root).unwrap();
+ let validated = open_validated_repository(&root).ok().unwrap();
+ assert_eq!(fsync_setting(&validated), Some(false));
assert_eq!(local_fsync_setting(&root), Some(false));
}
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index a24fef82..4e4ed699 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -1055,8 +1055,9 @@ that touch the relevant area.
[design-windows-ci.md](design-windows-ci.md#windows-transient-file-errors).
New file-I/O code with the same race follows the same pattern, with a
test that injects the error so it also runs off Windows.
- The wiki's `.git` gets the same durability from libgit2: opening the
- wiki sets `core.fsyncObjectFiles` in the repository's config, so objects
+ The wiki's `.git` gets the same durability from libgit2: every handle
+ ai-memory opens carries `core.fsyncObjectFiles = true` from
+ `.git/ai-memory.gitconfig`, a file the git CLI never reads, so objects
and refs are synced before their rename, and startup logs an `ERROR` when
`HEAD` does not resolve to a readable commit (the repair is in
[lifecycle-ops.md](lifecycle-ops.md#wiki-git-history-unreadable-after-a-crash)).
diff --git a/docs/lifecycle-ops.md b/docs/lifecycle-ops.md
index 949823b6..894c3977 100644
--- a/docs/lifecycle-ops.md
+++ b/docs/lifecycle-ops.md
@@ -1370,8 +1370,11 @@ ai-memory purge-project --project tmp --confirm
### Wiki git history unreadable after a crash
-The server turns on `core.fsyncObjectFiles` in `wiki/.git/config` when it opens
-the wiki, so libgit2 syncs each object and ref before renaming it into place.
+Every repository handle the server opens carries `core.fsyncObjectFiles = true`
+from `wiki/.git/ai-memory.gitconfig`, a file the git CLI never reads, so libgit2
+syncs each object and ref before renaming it into place while `git status` in
+the wiki stays free of git's deprecation warning for that key. Setting
+`core.fsyncObjectFiles = false` in `wiki/.git/config` turns it off.
Before that, a power loss or kernel crash could keep a ref update while the
objects it names were never written: zero-length files under `.git/objects`,
`HEAD` pointing at one of them, and every later checkpoint failing with
@@ -1399,8 +1402,7 @@ git fsck --full # clean (a dangling blob is harmless)
```
Start the server again; the next checkpoint commits the pages written after
-that commit. Git prints a deprecation warning for `core.fsyncObjectFiles`
-(it prefers `core.fsync`); libgit2 reads only the older key, so leave it set.
+that commit.
## Why this matters: the flat-wiki incident