Git designed for AI agents.
WP76b follow-up 3: untrusted state survives restarts, recovery waits for git
Federico Elia32d0ac454220
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.mdindex 4d59379..683cca1 100644--- a/doc/graft/PLAN7.md+++ b/doc/graft/PLAN7.md@@ -610,9 +610,22 @@ repository-wide lock.* *Write failure.* Only a failed fsync means the kernel may have droppedwrites, and the kernel reports a failed writeback once: a later fsync of thesame file succeeds.- * So the first failed fsync makes the repository *untrusted* for good.+ * So the first failed fsync makes the repository *untrusted* for good. The+ marker is rewritten durably with that failure, so a restart of the+ process in the same boot keeps it untrusted too.* An untrusted repository's marker is never cleared; only the move aside isretried, until it succeeds. Then the hydrated repository starts clean.+ * If even rewriting the marker fails, the disk is failing; the state of the+ process still keeps the repository untrusted until it stops.+ * A repository moved aside (`Forget`) while a git process of the owner+ that writes without fsync still runs stays dirty with a plain marker+ until that process exited and a durable step settled it. git names the+ repository by its path, so the process's later writes land in what is+ there now, the hydrated repository say.+ * A copy that holds shadow-mode history not imported yet (the+ `checkReplaceable` guard of a sync's hydration) is moved into+ `<storage>/+gitaly/graft-untrusted/` instead of being deleted: it is+ never served, and the history is not lost.* The repository is moved aside at once, under the repository lock: themove (both directories and the new entry) is made durable, then themarker removed.@@ -638,13 +651,32 @@ repository-wide lock.whose local copy may be the only one (an import that is not complete).* A marker of the same boot is a process restart: the page cache keptevery write, so the repository stays dirty to be settled.- * It is settled no earlier than the recovery grace (twice- `reftable.lockTimeout`): a git process of the previous owner that- outlived it (killed with it, not reaped yet) may still be committing- for that long.- * The git children get no parent-death signal: the command package is- shared and not in m3's grant. The grace bounds their life instead,- since a killed parent closes their stdin and git then aborts.+ * A git process of the previous owner that outlived it (killed with it,+ not reaped yet) may still be committing. So the marker stays while the+ repository's reftable directory holds a git lock or a table being+ written, which a writing git process always has: `tables.list.lock`+ (prepare to commit), `<table>.lock` (the tables a compaction merges),+ `*.temp.*` (a table being written). On top of that there is a floor+ of 2 s (`RecoveryGrace`), for the instant between a commit's rename+ and the start of its auto-compaction.+ * A marker that records an untrusted repository is handled like a+ marker of another boot: the repository is moved aside if it is a+ cache, and its marker is never settled otherwise.+ * Why the grace bound holds: when the previous owner dies, the stdin+ of its `update-ref --stdin` children closes.+ * A child that has not been sent `commit` aborts at end of input and+ writes nothing.+ * A child that has been sent `commit` already holds the reference lock+ (taken in `prepare`). It only writes its table and list and+ compacts, which takes milliseconds.+ * A child still waiting for the lock writes nothing afterwards: the+ next thing it reads is the end of its input.+ * A child that is committing is not bounded in time. Hence the rule+ on git's lock and temporary files above, and only a short floor.+ * No parent-death signal (`Pdeathsig`) instead, deliberately (PM).+ Go's `Pdeathsig` fires when the OS thread that created the child exits,+ not the process, unless the spawn is pinned with `LockOSThread`. In the+ shared command package that is easy to get subtly wrong.* The ownership is taken in two steps. The locks come first; the recoveryruns; only then are the process's writes deferred (`Activate`). So nowriter of the new owner writes a marker that the recovery could take for@@ -706,6 +738,16 @@ repository-wide lock.symbolic split;* a failed fsync then a successful one; and the same with a failed moveaside.+* Third review (of 18e152a):+ * M-A: the untrusted state is in the marker and survives a process restart.+ * m-1: Forget keeps a repository with a live git process dirty.+ * m-2: recovered markers wait for git's lock and temporary files, with a 2 s+ floor.+ * m-3: the cache check applies the sync's shadow-history guard; such copies+ are kept, not deleted.+ * m-4: a test proves the end of a deferred process runs after it exited. The+ mutation that swaps the two defers fails it.+ * Clean states are dropped; MarkDirty takes its state from the map again.**Tests.*** `localrefs`:diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.mdindex 522c7a6..197fb36 100644--- a/doc/graft/STORAGE.md+++ b/doc/graft/STORAGE.md@@ -470,9 +470,13 @@ or re-import the repository from scratch (seefrom then on the repository's marker is never removed: if the move asidefails, it is retried every second. Other errors of the step (a file thatcannot be opened for a moment) are retried every second and counted.- - *Process restart*: the markers found are kept for 20 s (twice git's- reference lock timeout) before they can be removed, in case a git process of- the previous process is still finishing.+ - *Process restart*: a marker found is kept while the repository's reftable+ directory holds a git lock or a table being written (a git process of the+ previous process may still be finishing), and for 2 s at least. A+ repository the previous process had found untrusted is moved aside.+ - A repository moved aside that holds shadow-mode history not imported yet+ goes to `<storage>/+gitaly/graft-untrusted/` and stays there: import what+ it holds, or remove it.- *Clean stop*: after the servers stopped, `gitaly serve` makes every dirtyrepository durable and removes its marker, so a clean reboot hydratesnothing.diff --git a/internal/gitaly/hook/updateref/deferred_durability_test.go b/internal/gitaly/hook/updateref/deferred_durability_test.goindex 72efa27..a020189 100644--- a/internal/gitaly/hook/updateref/deferred_durability_test.go+++ b/internal/gitaly/hook/updateref/deferred_durability_test.go@@ -2,6 +2,11 @@ package updaterefimport ("context"+ "fmt"+ "os"+ "path/filepath"+ "strconv"+ "strings""testing""github.com/stretchr/testify/require"@@ -43,6 +48,89 @@ func (v durabilityVoter) DeferReferenceWrites(context.Context, *gitalypb.Repositreturn func() { *v.ended++ }, true}+// liveUpdateRefs returns the update-ref processes that are children of the+// test process and still running.+func liveUpdateRefs(t *testing.T) []int {+ t.Helper()++ entries, err := os.ReadDir("/proc")+ require.NoError(t, err)+ var pids []int+ for _, entry := range entries {+ pid, err := strconv.Atoi(entry.Name())+ if err != nil {+ continue+ }+ stat, err := os.ReadFile(filepath.Join("/proc", entry.Name(), "stat"))+ if err != nil {+ continue+ }+ // pid (comm) state ppid ...; comm may contain spaces, so the fields are+ // read after its closing parenthesis.+ fields := strings.Fields(string(stat[strings.LastIndexByte(string(stat), ')')+1:]))+ if len(fields) < 2 || fields[0] == "Z" || fields[1] != strconv.Itoa(os.Getpid()) {+ continue+ }+ cmdline, err := os.ReadFile(filepath.Join("/proc", entry.Name(), "cmdline"))+ if err != nil {+ continue+ }+ if strings.Contains(string(cmdline), "update-ref") {+ pids = append(pids, pid)+ }+ }+ return pids+}++// endingVoter is a voter that defers every reference write and checks, when+// the end of a deferred git process is reported, that the process has exited.+type endingVoter struct {+ durabilityVoter+ t *testing.T+ ends *int+ failed *string+}++func (v endingVoter) DeferReferenceWrites(context.Context, *gitalypb.Repository) (func(), bool) {+ return func() {+ *v.ends+++ if live := liveUpdateRefs(v.t); len(live) > 0 {+ *v.failed = fmt.Sprintf("the end ran while update-ref %v was still running", live)+ }+ }, true+}++// TestUpdateReferenceEndsTheDeferredProcessAfterItExited verifies that+// UpdateWithHooks reports the end of a deferred update-ref only once the+// process has exited (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE): until+// then it may still write a table or compact the stack. It does not run in+// parallel, so that no other test's update-ref is a child at the same time.+func TestUpdateReferenceEndsTheDeferredProcessAfterItExited(t *testing.T) {+ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ testcfg.BuildGitalyHooks(t, cfg)+ logger := testhelper.NewLogger(t)+ locator := config.NewLocator(cfg)+ gitCmdFactory := gittest.NewCommandFactory(t, cfg)++ repo, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})+ base := gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("main"))+ next := gittest.WriteCommit(t, cfg, repoPath, gittest.WithParents(base))++ var ends int+ var failed string+ manager := hook.NewManager(cfg, locator, logger, gitCmdFactory, nil, nil,+ gitlab.NewMockClient(t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive),+ hook.NewPostReceiveRegistry(),+ hook.WithStorageVoter(endingVoter{t: t, ends: &ends, failed: &failed}))+ updater := NewUpdaterWithHooks(cfg, logger, locator, manager, gitCmdFactory, nil)++ require.NoError(t, updater.UpdateReference(ctx, repo, gittest.TestUser, nil, "refs/heads/main", next, base))+ require.Equal(t, 1, ends)+ require.Empty(t, failed)+ require.Equal(t, next, gittest.ResolveRevision(t, cfg, repoPath, "refs/heads/main"))+}+// TestDeferReferenceWrites verifies that the updater writes references without fsync only when// the storage voter of its hook manager defers their durability: never without the storage// core, nor when the voter does not (shadow mode, authoritative mode where the kernel exposes nodiff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.goindex f96aeb3..35b7ad0 100644--- a/internal/graft/corestore/localrefs/durability.go+++ b/internal/graft/corestore/localrefs/durability.go@@ -21,7 +21,6 @@ import ("github.com/prometheus/client_golang/prometheus"- "github.com/feder-cr/git-ai/internal/git/gitcmd""github.com/feder-cr/git-ai/internal/gitaly/config""github.com/feder-cr/git-ai/internal/gitaly/storage""github.com/feder-cr/git-ai/proto/go/gitalypb"@@ -56,10 +55,11 @@ const (// write before the durable step removes its marker.DurableStepIdle = time.Second// RecoveryGrace is how long a repository found dirty when the process takes- // the ownership keeps its marker at least: longer than git waits for the- // reference lock (reftable.lockTimeout), so that a git process of the- // previous owner that is not reaped yet has finished.- RecoveryGrace = 2 * gitcmd.ReftableLockTimeoutMilliseconds * time.Millisecond+ // the ownership keeps its marker at least. The rule that decides is that no+ // git lock or temporary file is in its reftable directory (gitActive); the+ // grace only covers the instant between a commit's rename and the start of+ // its auto-compaction, when a git process has neither.+ RecoveryGrace = 2 * time.Second)// WithRecoveryGrace replaces RecoveryGrace. Tests shorten it.@@ -191,11 +191,16 @@ type refState struct {// may still write a table or compact the stack.nLive intlastWrite time.Time- // settleAfter keeps the marker of a repository recovered dirty until then.+ // recovered is set for a repository found dirty when the process took the+ // ownership: a git process of the previous owner that outlived it may still+ // be writing, so its marker stays while git's lock or temporary files are+ // there (gitActive), and not before settleAfter.+ recovered boolsettleAfter time.Time// untrusted is the first failed fsync of the repository. The kernel reports// a failed writeback once, so a later fsync can succeed although writes// were lost: the repository never becomes clean again, only moved aside.+ // The marker records it, so that a restart of the process keeps it.untrusted error// synced holds the tables made durable during the current dirty period that// the stack still lists. Tables are immutable, so one fsync each is enough.@@ -216,6 +221,9 @@ func repoOf(r RepoRef) *gitalypb.Repository {type marker struct {RepoRefBootID string `json:"boot_id"`+ // Untrusted is the failed fsync that made the repository untrusted, empty+ // while it is only dirty.+ Untrusted string `json:"untrusted,omitempty"`}// Reasons of graft_storage_ref_untrusted_total.@@ -297,7 +305,7 @@ func newDurability(locator storage.Locator, idle time.Duration) *Durability {}),inflight: prometheus.NewGauge(prometheus.GaugeOpts{Name: "graft_storage_ref_writes_inflight",- Help: "Number of hook reference transactions between their start and their end. One that never ends keeps its repository dirty.",+ Help: "Number of git processes of this process that write references without fsync and have not exited; their repositories keep their markers.",}),owning: prometheus.NewGauge(prometheus.GaugeOpts{Name: "graft_storage_ref_durability_owner",@@ -441,20 +449,32 @@ func (d *Durability) MarkDirty(ctx context.Context, repo RepoRef, path string) ereturn nil}- d.mu.Lock()- st := d.stateLocked(path)- st.lastWrite = time.Now()- d.mu.Unlock()-- st.file.Lock()- defer st.file.Unlock()-- d.mu.Lock()- dirty := st.dirty- d.mu.Unlock()- if dirty {- return nil+ // The state is the map's after its file lock is taken: a durable step may+ // have dropped a clean state meanwhile.+ var st *refState+ for {+ d.mu.Lock()+ st = d.stateLocked(path)+ d.mu.Unlock()+ st.file.Lock()+ d.mu.Lock()+ current := d.repos[path] == st+ if current {+ st.lastWrite = time.Now()+ }+ dirty := st.dirty+ d.mu.Unlock()+ if !current {+ st.file.Unlock()+ continue+ }+ if dirty {+ st.file.Unlock()+ return nil+ }+ break}+ defer st.file.Unlock()markerPath, err := d.markerPath(ctx, repo, path)if err != nil {@@ -583,13 +603,26 @@ func (d *Durability) Settle(path string, final bool) error {}now := time.Now()idle := st.nLive == 0 && now.After(st.settleAfter) && (final || now.Sub(st.lastWrite) >= d.idle)+ recovered := st.recoveredd.mu.Unlock()+ if idle && recovered {+ // A git process of the previous owner may still be writing.+ active, err := gitActive(path)+ if err != nil {+ return err+ }+ idle = !active+ }if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {st.file.Lock()defer st.file.Unlock()d.clear(st)- return d.removeMarker(st)+ if err := d.removeMarker(st); err != nil {+ return err+ }+ d.prune(path, st)+ return nil}if err := d.syncStack(path, st); err != nil {@@ -623,12 +656,31 @@ func (d *Durability) Settle(path string, final bool) error {if err := lock.release(); err != nil {return err}- return d.removeMarker(st)+ if err := d.removeMarker(st); err != nil {+ return err+ }+ d.prune(path, st)+ return nil+}++// prune drops the state of a repository that is clean and has no live git+// process, so that the states do not grow with every repository ever written.+// The caller holds the state's file lock: MarkDirty takes the state again from+// the map after its own.+func (d *Durability) prune(path string, st *refState) {+ d.mu.Lock()+ defer d.mu.Unlock()+ if d.repos[path] == st && !st.dirty && st.untrusted == nil && st.nLive == 0 {+ delete(d.repos, path)+ }}// stepError is the error of a durable step: an *UntrustedError for a failed// fsync, which makes the repository untrusted for good, the error itself-// otherwise.+// otherwise. The first failure is written into the marker, so that the+// repository stays untrusted across a restart of the process (the next fsync+// would succeed); when even that write fails, the disk is failing and the+// state of this process keeps it untrusted.func (d *Durability) stepError(st *refState, repo RepoRef, path string, err error) error {var writeback writebackErrorif !errors.As(err, &writeback) {@@ -636,13 +688,44 @@ func (d *Durability) stepError(st *refState, repo RepoRef, path string, err erro}d.mu.Lock()- if st.untrusted == nil {+ first := st.untrusted == nil+ if first {st.untrusted = err}+ markerPath := st.markerd.mu.Unlock()++ if first && markerPath != "" {+ st.file.Lock()+ content, encodeErr := json.Marshal(marker{RepoRef: repo, BootID: d.bootID, Untrusted: err.Error()})+ if encodeErr == nil {+ encodeErr = d.writeDurably(markerPath, content)+ }+ st.file.Unlock()+ if encodeErr != nil {+ return &UntrustedError{Repo: repo, Path: path, Err: fmt.Errorf("%w (recording it in the marker failed: %w)", err, encodeErr)}+ }+ }return &UntrustedError{Repo: repo, Path: path, Err: err}}+// gitActive reports whether a git process may be writing the reftable stack of+// the repository at path: git's lock of the list, a lock of a table being+// compacted, or a table being written is there.+func gitActive(path string) (bool, error) {+ entries, err := os.ReadDir(filepath.Join(path, "reftable"))+ if err != nil {+ return false, fmt.Errorf("read the reftable directory: %w", err)+ }+ for _, entry := range entries {+ name := entry.Name()+ if strings.HasSuffix(name, ".lock") || strings.Contains(name, ".temp.") {+ return true, nil+ }+ }+ return false, nil+}+// settleLocked makes the tables written since the first pass durable and, when// no transaction of the process is live, marks the state clean with its file// lock held (the caller releases it). The caller holds git's reference lock: no@@ -877,6 +960,35 @@ func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) errodefer st.file.Unlock()}+ d.mu.Lock()+ live := st != nil && d.repos[path] == st && st.nLive > 0+ d.mu.Unlock()+ if live {+ // A git process that writes without fsync is still running. It names the+ // repository by its path, so its later writes land in what is there now+ // (the hydrated repository, say): the repository stays dirty, with a plain+ // marker, until it exited and a durable step made the stack durable.+ content, err := json.Marshal(marker{RepoRef: repo, BootID: d.bootID})+ if err != nil {+ return fmt.Errorf("encode the dirty marker: %w", err)+ }+ if err := d.writeDurably(markerPath, content); err != nil {+ return fmt.Errorf("write the dirty marker: %w", err)+ }+ d.mu.Lock()+ defer d.mu.Unlock()+ if !st.dirty {+ st.dirty = true+ d.dirty.Inc()+ }+ st.repo = repo+ st.marker = markerPath+ st.untrusted = nil+ st.recovered = false+ st.synced = map[string]struct{}{}+ return nil+ }+if err := os.Remove(markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {return fmt.Errorf("remove the dirty marker: %w", err)}@@ -887,7 +999,6 @@ func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) erroif st.dirty {d.dirty.Dec()}- d.inflight.Sub(float64(st.nLive))delete(d.repos, path)}return nil@@ -900,14 +1011,20 @@ type Recovered struct {// Reboot reports that the marker is of another kernel boot: the repository// must be moved aside before anything reads it.Reboot bool+ // Untrusted reports that a durable step of the previous process found a+ // failed fsync: the repository must be moved aside too, and is never+ // settled.+ Untrusted bool}// Recover reads the dirty markers of the storages; only the owner calls it,// when it takes the ownership. A marker of the running boot makes its-// repository dirty again, to be settled, and what a durable step of the-// previous owner left (its reference lock, its temporary files) is removed. It-// returns every repository found; the caller moves aside the ones of another-// boot before serving. A marker that cannot be read is an error: its+// repository dirty again, to be settled once no git process of the previous+// owner can still write it (gitActive, and the recovery grace), or untrusted+// again when the marker says so; what a durable step of the previous owner+// left (its reference lock, its temporary files) is removed. It returns every+// repository found; the caller moves aside the ones of another boot and the+// untrusted ones before serving. A marker that cannot be read is an error: its// repository is unknown, and serving could read a torn stack.func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovered, error) {if d == nil || !d.ownsLocks() {@@ -975,10 +1092,14 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovere}st.repo = m.RepoRefst.marker = markerPath+ st.recovered = truest.settleAfter = time.Now().Add(d.recoveryGrace)+ if m.Untrusted != "" {+ st.untrusted = errors.New(m.Untrusted)+ }d.mu.Unlock()}- found = append(found, Recovered{Repo: m.RepoRef, Path: path, Reboot: reboot})+ found = append(found, Recovered{Repo: m.RepoRef, Path: path, Reboot: reboot, Untrusted: m.Untrusted != ""})}}return found, nildiff --git a/internal/graft/corestore/localrefs/durability_test.go b/internal/graft/corestore/localrefs/durability_test.goindex 33c35b1..974714c 100644--- a/internal/graft/corestore/localrefs/durability_test.go+++ b/internal/graft/corestore/localrefs/durability_test.go@@ -39,12 +39,14 @@ type syncRecorder struct {mu sync.Mutexpaths []stringfailWith error+ // failOnly, when set, limits the failures to the paths that contain it.+ failOnly string}func (s *syncRecorder) sync(f *os.File) error {s.mu.Lock()defer s.mu.Unlock()- if s.failWith != nil {+ if s.failWith != nil && strings.Contains(f.Name(), s.failOnly) {return s.failWith}s.paths = append(s.paths, f.Name())@@ -403,6 +405,115 @@ func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {require.Equal(t, 0.0, testutil.ToFloat64(d.dirty))}+// TestUntrustedSurvivesAProcessRestart verifies that the untrusted state is in+// the marker: a restart of the process in the same boot, whose fsyncs then+// succeed, finds the repository untrusted and never settles it.+func TestUntrustedSurvivesAProcessRestart(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)++ before := f.durability(t, "boot-a", 0)+ require.NoError(t, before.MarkDirty(f.ctx, f.repo, f.path))+ // The writes of the references were lost; the marker's directory is fine.+ f.synced.mu.Lock()+ f.synced.failOnly = "/reftable"+ f.synced.mu.Unlock()+ f.synced.fail(errors.New("input/output error"))+ require.ErrorIs(t, before.Settle(f.path, false), ErrUntrusted)+ f.synced.fail(nil)+ content, err := os.ReadFile(f.markerPath(t, before))+ require.NoError(t, err)+ require.Contains(t, string(content), `"untrusted":"fsync: input/output error"`)+ require.NoError(t, before.Close())++ after := f.durability(t, "boot-a", 0)+ found, err := after.Recover(f.ctx, []string{f.repo.Storage})+ require.NoError(t, err)+ require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path, Untrusted: true}}, found)+ err = after.Settle(f.path, true)+ require.ErrorIs(t, err, ErrUntrusted)+ require.FileExists(t, f.markerPath(t, after))+}++// TestForgetKeepsARepositoryWithALiveGitProcess verifies that moving a+// repository aside while a git process that writes without fsync is still+// running keeps the repository dirty: that process names the repository by its+// path, so its later writes land in what is there now.+func TestForgetKeepsARepositoryWithALiveGitProcess(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ end := d.Begin(f.path)+ f.synced.fail(errors.New("input/output error"))+ require.ErrorIs(t, d.Settle(f.path, false), ErrUntrusted)+ f.synced.fail(nil)++ require.NoError(t, d.Forget(f.ctx, f.repo, f.path))+ require.FileExists(t, f.markerPath(t, d))+ content, err := os.ReadFile(f.markerPath(t, d))+ require.NoError(t, err)+ require.NotContains(t, string(content), "untrusted", "what is at the path now is not the untrusted repository")+ require.Equal(t, []string{f.path}, d.Dirty())+ require.NoError(t, d.Settle(f.path, true))+ require.FileExists(t, f.markerPath(t, d), "the git process is still running")++ end()+ require.NoError(t, d.Settle(f.path, false))+ require.NoFileExists(t, f.markerPath(t, d))+}++// TestRecoveredMarkerWaitsForGit verifies that a marker found at the+// recovery stays while git's lock or a table being written is in the+// repository: a git process of the previous owner may still be writing.+func TestRecoveredMarkerWaitsForGit(t *testing.T) {+ t.Parallel()++ for _, leftover := range []string{"tables.list.lock", "0x000000000009-0x000000000009-abcdef01.ref.lock", "0x000000000009-0x000000000009-abcdef01.ref.temp.XyZ123"} {+ t.Run(leftover, func(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)++ old := f.durability(t, "boot-a", 0)+ require.NoError(t, old.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, old.Close())++ file := filepath.Join(f.path, "reftable", leftover)+ require.NoError(t, os.WriteFile(file, []byte("0x000000000001-0x000000000003-abcdef01.ref\n"), 0o666))++ next := f.durability(t, "boot-a", 0)+ _, err := next.Recover(f.ctx, []string{f.repo.Storage})+ require.NoError(t, err)+ require.NoError(t, next.Settle(f.path, true))+ require.FileExists(t, f.markerPath(t, next), "git may still be writing")++ require.NoError(t, os.Remove(file))+ require.NoError(t, next.Settle(f.path, true))+ require.NoFileExists(t, f.markerPath(t, next))+ })+ }+}++// TestSettleDropsTheStateOfACleanRepository verifies that the states do not+// grow with every repository ever written, and that a writer after the drop+// marks the repository again.+func TestSettleDropsTheStateOfACleanRepository(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, d.Settle(f.path, false))+ d.mu.Lock()+ require.Empty(t, d.repos)+ d.mu.Unlock()++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ require.FileExists(t, f.markerPath(t, d))+ require.Equal(t, []string{f.path}, d.Dirty())+}+// TestSettleTransientErrorIsNotUntrusted verifies that a failure to read the// stack (not a failed fsync) is retried: the repository is not distrusted.func TestSettleTransientErrorIsNotUntrusted(t *testing.T) {diff --git a/internal/graft/corestore/syncer/durability.go b/internal/graft/corestore/syncer/durability.goindex f5ab734..f428854 100644--- a/internal/graft/corestore/syncer/durability.go+++ b/internal/graft/corestore/syncer/durability.go@@ -9,6 +9,9 @@ import ("sync""time"+ "google.golang.org/grpc/codes"+ "google.golang.org/grpc/status"+"github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/localrefs""github.com/feder-cr/git-ai/internal/structerr"@@ -122,10 +125,14 @@ func (s *Syncer) runDurableSteps(ctx context.Context, storages []string, recover// durability. A repository dirty from a process restart of the same boot is// kept: the kernel kept every write, and the durable step settles it.//-// Only a cache of the storage core is moved: the database must hold the-// repository with changesets, and the local repository must have its-// watermark. Any other marker (the repository was removed, or replaced by one-// being imported, whose local copy may be the only one) is dropped.+// A repository whose marker says that a durable step of the previous process+// found a failed fsync is moved aside the same way: its writes may be lost,+// although the kernel would report nothing more.+//+// Only a cache of the storage core is moved (storageCoreCache). Any other+// marker of another boot (the repository was removed, or replaced by one being+// imported, whose local copy may be the only one) is dropped; an untrusted one+// is kept, with its repository untrusted, and logged.func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string) error {d := s.core.RefDurabilityfound, err := d.Recover(ctx, storages)@@ -134,24 +141,33 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)}for _, rec := range found {- if !rec.Reboot {+ if !rec.Reboot && !rec.Untrusted {continue}+ reason := localrefs.ReasonReboot+ if !rec.Reboot {+ reason = localrefs.ReasonSyncFailed+ }- cache, err := s.storageCoreCache(ctx, rec.Repo, rec.Path)+ cache, keep, err := s.storageCoreCache(ctx, rec.Repo, rec.Path)if err != nil {return fmt.Errorf("storage core: recover the local references of %s:%s: %w", rec.Repo.Storage, rec.Repo.RelativePath, err)}if !cache {- s.logger.WithField("repository_path", rec.Path).- Warn("storage core: dropped the dirty marker of a local repository that is not a cache of the storage core")- if err := d.Forget(ctx, rec.Repo, rec.Path); err != nil {- return err+ if rec.Reboot {+ s.logger.WithField("repository_path", rec.Path).+ Warn("storage core: dropped the dirty marker of a local repository that is not a cache of the storage core")+ if err := d.Forget(ctx, rec.Repo, rec.Path); err != nil {+ return err+ }+ } else {+ s.logger.WithField("repository_path", rec.Path).+ Error("storage core: the references of a repository that is not a cache of the storage core may have lost writes; it is kept in place, untrusted")}continue}- if err := s.distrust(ctx, rec.Repo, rec.Path, localrefs.ReasonReboot, nil); err != nil {+ if err := s.distrust(ctx, rec.Repo, rec.Path, reason, nil, keep); err != nil {return err}}@@ -161,27 +177,37 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)// storageCoreCache reports whether the local repository at path is a cache of// the storage core that a hydration can rebuild: the database holds it with// changesets (an import that is not complete has none, and its local copy may-// be the only one). A sync replaces such a local repository too when it is-// stale, whatever its watermark. An error (the database, or the repository's-// directory, cannot be read) is returned: the caller tries again.-func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, path string) (bool, error) {+// be the only one). keep reports that the copy holds history written in shadow+// mode and not imported yet (checkReplaceable, the guard of a sync's+// hydration): the copy is moved aside all the same, so that it is never served,+// but kept, so that the history is not lost. An error (the database, or the+// repository's files, cannot be read) is returned: the caller tries again.+func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, path string) (cache, keep bool, err error) {r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: repo.Storage, RelativePath: repo.RelativePath})if errors.Is(err, corestore.ErrRepositoryNotFound) {- return false, nil+ return false, false, nil}if err != nil {- return false, err+ return false, false, err}if r.HeadSeq == 0 {- return false, nil+ return false, false, nil}if _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) {- return false, nil+ return false, false, nil} else if err != nil {- return false, err+ return false, false, err+ }++ wm, ok := s.readWatermark(path)+ if err := s.checkReplaceable(ctx, r, path, wm, ok); err != nil {+ if status.Code(err) == codes.FailedPrecondition {+ return true, true, nil+ }+ return false, false, err}- return true, nil+ return true, false, nil}// durableStep makes the references of the repository at path durable under the@@ -213,13 +239,13 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {// A repository that is gone (moved aside as a stale cache meanwhile) has// nothing left to serve: its marker is dropped.if _, statErr := os.Lstat(path); errors.Is(statErr, os.ErrNotExist) {- if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted, false); err != nil {logger.WithField("forget_error", err.Error()).Error("storage core: could not drop the marker of a repository that is gone; retried at the next step")}return}- cache, cacheErr := s.storageCoreCache(ctx, untrusted.Repo, path)+ cache, keep, cacheErr := s.storageCoreCache(ctx, untrusted.Repo, path)if cacheErr != nil {d.CountStepError()logger.WithField("cache_check_error", cacheErr.Error()).@@ -232,7 +258,7 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {logger.Error("storage core: the references of a repository that is not a cache of the storage core may have lost writes; it is kept in place")return}- if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted, keep); err != nil {// The untrusted state is sticky: the next step moves it aside again, and// the marker is never cleared meanwhile.logger.WithField("move_error", err.Error()).@@ -241,21 +267,27 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {}// distrust is distrustLocked under the repository lock.-func (s *Syncer) distrust(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {+func (s *Syncer) distrust(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error, keep bool) error {release, err := s.lock(ctx, path)if err != nil {return err}defer release()- return s.distrustLocked(ctx, repo, path, reason, cause)+ return s.distrustLocked(ctx, repo, path, reason, cause, keep)}-// distrustLocked moves the local repository at path into the temporary-// directory of its storage and removes it there in the background, then drops-// its dirty marker: the next request hydrates it. The move is made durable-// (both directories, and the new directory's entry) before the marker goes, so-// a crash in between moves it again. The caller holds the repository lock.-func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {+// KeptDirName is the directory of a storage where the untrusted repositories+// that hold history not imported yet are moved: they are never served, and an+// operator imports what they hold or removes them.+const KeptDirName = "+gitaly/graft-untrusted"++// distrustLocked moves the local repository at path out of the way, then drops+// its dirty marker: the next request hydrates it. It goes into the temporary+// directory of its storage and is removed there in the background, or, with+// keep, into KeptDirName, where it stays. The move is made durable (both+// directories, and the new directory's entry) before the marker goes, so a+// crash in between moves it again. The caller holds the repository lock.+func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error, keep bool) error {d := s.core.RefDurabilityif _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) {@@ -264,19 +296,29 @@ func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, patreturn structerr.NewInternal("storage core: stat repository: %w", err)}- tmpDir, err := s.tmpDir(repo.Storage)+ parent, err := s.tmpDir(repo.Storage)if err != nil {return err}- stale, err := os.MkdirTemp(tmpDir, tmpStalePattern)+ if keep {+ root, err := s.locator.GetStorageByName(ctx, repo.Storage)+ if err != nil {+ return err+ }+ parent = filepath.Join(root, KeptDirName)+ if err := os.MkdirAll(parent, 0o700); err != nil {+ return structerr.NewInternal("storage core: create the directory of kept repositories: %w", err)+ }+ }+ aside, err := os.MkdirTemp(parent, tmpStalePattern)if err != nil {return structerr.NewInternal("storage core: create stale directory: %w", err)}- if err := os.Rename(path, filepath.Join(stale, "repo.git")); err != nil {- _ = os.Remove(stale)+ if err := os.Rename(path, filepath.Join(aside, "repo.git")); err != nil {+ _ = os.Remove(aside)return structerr.NewInternal("storage core: move aside an untrusted repository: %w", err)}- for _, dir := range []string{filepath.Dir(path), stale, tmpDir} {+ for _, dir := range []string{filepath.Dir(path), aside, parent, filepath.Dir(parent)} {if err := syncDir(dir); err != nil {return structerr.NewInternal("storage core: make the move of an untrusted repository durable: %w", err)}@@ -288,13 +330,17 @@ func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, pat}d.CountUntrusted(reason)- logger := s.logger.WithField("repository_path", path).WithField("stale_path", stale).WithField("reason", reason)+ logger := s.logger.WithField("repository_path", path).WithField("moved_path", aside).WithField("reason", reason)if cause != nil {logger = logger.WithError(cause)}+ if keep {+ logger.Error("storage core: moved aside a local repository whose references may have lost writes; it holds shadow-mode history not imported yet and is kept for an operator; the next request hydrates the repository")+ return nil+ }logger.Error("storage core: moved aside a local repository whose references may have lost writes, the next request hydrates it")- go func() { _ = os.RemoveAll(stale) }()+ go func() { _ = os.RemoveAll(aside) }()return nil}diff --git a/internal/graft/corestore/syncer/syncer_localrefs_test.go b/internal/graft/corestore/syncer/syncer_localrefs_test.goindex 1dd81d5..a2e2a3b 100644--- a/internal/graft/corestore/syncer/syncer_localrefs_test.go+++ b/internal/graft/corestore/syncer/syncer_localrefs_test.go@@ -14,6 +14,7 @@ import ("github.com/feder-cr/git-ai/internal/git""github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/graft/changelog""github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/coretest""github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"@@ -247,6 +248,78 @@ func TestSyncerUntrustedRepositoryIsNeverSettled(t *testing.T) {e.verifySynced(t, repo)}+// failReferences returns a sync function that fails the fsyncs of reference+// files while failing is set: the writes of the references were lost, the rest+// of the disk is fine.+func failReferences(failing *atomic.Bool) func(f *os.File) error {+ return func(f *os.File) error {+ if failing.Load() && strings.Contains(f.Name(), "/reftable") {+ return errors.New("input/output error")+ }+ return f.Sync()+ }+}++// TestSyncerUntrustedRepositoryAcrossAProcessRestart is a failed fsync whose+// move aside fails, followed by a restart of the process in the same boot:+// the restarted process's fsyncs succeed, and the repository is moved aside+// all the same, never settled.+func TestSyncerUntrustedRepositoryAcrossAProcessRestart(t *testing.T) {+ t.Parallel()++ var failing atomic.Bool+ e, repo, _ := caughtUpDirty(t, localrefs.WithSync(failReferences(&failing)))+ tmpDir, err := e.locator.TempDir(e.storageName)+ require.NoError(t, err)+ require.NoError(t, os.MkdirAll(tmpDir, 0o700))++ failing.Store(true)+ require.NoError(t, os.Chmod(tmpDir, 0o500))+ e.sy.DurableStep(e.ctx, e.localPath(), false)+ require.NoError(t, os.Chmod(tmpDir, 0o700))+ failing.Store(false)+ require.DirExists(t, e.localPath(), "the move aside failed")++ // The process restarts in the same boot.+ coretest.EnableLocalRefs(t, e.core, e.cfg)+ e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)+ require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))++ require.NoDirExists(t, e.localPath(), "an untrusted repository is moved aside at the restart")+ require.Empty(t, e.markers(t))+ require.Equal(t, float64(1), e.untrusted(t, localrefs.ReasonSyncFailed))+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ e.verifySynced(t, repo)+}++// TestSyncerUntrustedRepositoryWithShadowHistoryIsKept verifies that an+// untrusted repository whose local copy holds history written in shadow mode+// and not imported yet (the guard of a sync's hydration) is moved aside but+// not deleted: it is never served, and the history is not lost.+func TestSyncerUntrustedRepositoryWithShadowHistoryIsKept(t *testing.T) {+ t.Parallel()++ var failing atomic.Bool+ e, _, c2 := caughtUpDirty(t, localrefs.WithSync(failReferences(&failing)))+ _, err := changelog.Append(e.localPath(), changelog.Meta{Operation: "push", Actor: "alice"}, []changelog.Update{+ {Reference: "refs/heads/shadow", OldOID: gittest.DefaultObjectHash.ZeroOID.String(), NewOID: c2.String()},+ })+ require.NoError(t, err)++ failing.Store(true)+ e.sy.DurableStep(e.ctx, e.localPath(), false)+ failing.Store(false)++ require.NoDirExists(t, e.localPath(), "never served")+ kept, err := filepath.Glob(filepath.Join(e.storagePath, syncer.KeptDirName, "*", "repo.git"))+ require.NoError(t, err)+ require.Len(t, kept, 1, "the copy with the shadow history is kept")+ last, err := changelog.LastSequence(kept[0])+ require.NoError(t, err)+ require.Positive(t, last)+ require.Empty(t, e.markers(t))+}+// TestSyncerUntrustedRepositoryThatIsGoneLosesItsMarker verifies that an// untrusted repository that was moved away meanwhile (a stale cache) does not// keep its marker forever.