Skip to content

Git designed for AI agents.

WP76b follow-up 3: untrusted state survives restarts, recovery waits for git

Federico Elia32d0ac454220
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.md
index 4d59379..683cca1 100644
--- a/doc/graft/PLAN7.md
+++ b/doc/graft/PLAN7.md
@@ -610,9 +610,22 @@ repository-wide lock.
* *Write failure.* Only a failed fsync means the kernel may have dropped
writes, and the kernel reports a failed writeback once: a later fsync of the
same file succeeds.
- * So the first failed fsync makes the repository *untrusted* for good.
+ * So the first failed fsync makes the repository *untrusted* for good. The
+ marker is rewritten durably with that failure, so a restart of the
+ process in the same boot keeps it untrusted too.
* An untrusted repository's marker is never cleared; only the move aside is
retried, until it succeeds. Then the hydrated repository starts clean.
+ * If even rewriting the marker fails, the disk is failing; the state of the
+ process still keeps the repository untrusted until it stops.
+ * A repository moved aside (`Forget`) while a git process of the owner
+ that writes without fsync still runs stays dirty with a plain marker
+ until that process exited and a durable step settled it. git names the
+ repository by its path, so the process's later writes land in what is
+ there now, the hydrated repository say.
+ * A copy that holds shadow-mode history not imported yet (the
+ `checkReplaceable` guard of a sync's hydration) is moved into
+ `<storage>/+gitaly/graft-untrusted/` instead of being deleted: it is
+ never served, and the history is not lost.
* The repository is moved aside at once, under the repository lock: the
move (both directories and the new entry) is made durable, then the
marker removed.
@@ -638,13 +651,32 @@ repository-wide lock.
whose local copy may be the only one (an import that is not complete).
* A marker of the same boot is a process restart: the page cache kept
every write, so the repository stays dirty to be settled.
- * It is settled no earlier than the recovery grace (twice
- `reftable.lockTimeout`): a git process of the previous owner that
- outlived it (killed with it, not reaped yet) may still be committing
- for that long.
- * The git children get no parent-death signal: the command package is
- shared and not in m3's grant. The grace bounds their life instead,
- since a killed parent closes their stdin and git then aborts.
+ * A git process of the previous owner that outlived it (killed with it,
+ not reaped yet) may still be committing. So the marker stays while the
+ repository's reftable directory holds a git lock or a table being
+ written, which a writing git process always has: `tables.list.lock`
+ (prepare to commit), `<table>.lock` (the tables a compaction merges),
+ `*.temp.*` (a table being written). On top of that there is a floor
+ of 2 s (`RecoveryGrace`), for the instant between a commit's rename
+ and the start of its auto-compaction.
+ * A marker that records an untrusted repository is handled like a
+ marker of another boot: the repository is moved aside if it is a
+ cache, and its marker is never settled otherwise.
+ * Why the grace bound holds: when the previous owner dies, the stdin
+ of its `update-ref --stdin` children closes.
+ * A child that has not been sent `commit` aborts at end of input and
+ writes nothing.
+ * A child that has been sent `commit` already holds the reference lock
+ (taken in `prepare`). It only writes its table and list and
+ compacts, which takes milliseconds.
+ * A child still waiting for the lock writes nothing afterwards: the
+ next thing it reads is the end of its input.
+ * A child that is committing is not bounded in time. Hence the rule
+ on git's lock and temporary files above, and only a short floor.
+ * No parent-death signal (`Pdeathsig`) instead, deliberately (PM).
+ Go's `Pdeathsig` fires when the OS thread that created the child exits,
+ not the process, unless the spawn is pinned with `LockOSThread`. In the
+ shared command package that is easy to get subtly wrong.
* The ownership is taken in two steps. The locks come first; the recovery
runs; only then are the process's writes deferred (`Activate`). So no
writer of the new owner writes a marker that the recovery could take for
@@ -706,6 +738,16 @@ repository-wide lock.
symbolic split;
* a failed fsync then a successful one; and the same with a failed move
aside.
+* Third review (of 18e152a):
+ * M-A: the untrusted state is in the marker and survives a process restart.
+ * m-1: Forget keeps a repository with a live git process dirty.
+ * m-2: recovered markers wait for git's lock and temporary files, with a 2 s
+ floor.
+ * m-3: the cache check applies the sync's shadow-history guard; such copies
+ are kept, not deleted.
+ * m-4: a test proves the end of a deferred process runs after it exited. The
+ mutation that swaps the two defers fails it.
+ * Clean states are dropped; MarkDirty takes its state from the map again.
**Tests.**
* `localrefs`:
diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.md
index 522c7a6..197fb36 100644
--- a/doc/graft/STORAGE.md
+++ b/doc/graft/STORAGE.md
@@ -470,9 +470,13 @@ or re-import the repository from scratch (see
from then on the repository's marker is never removed: if the move aside
fails, it is retried every second. Other errors of the step (a file that
cannot be opened for a moment) are retried every second and counted.
- - *Process restart*: the markers found are kept for 20 s (twice git's
- reference lock timeout) before they can be removed, in case a git process of
- the previous process is still finishing.
+ - *Process restart*: a marker found is kept while the repository's reftable
+ directory holds a git lock or a table being written (a git process of the
+ previous process may still be finishing), and for 2 s at least. A
+ repository the previous process had found untrusted is moved aside.
+ - A repository moved aside that holds shadow-mode history not imported yet
+ goes to `<storage>/+gitaly/graft-untrusted/` and stays there: import what
+ it holds, or remove it.
- *Clean stop*: after the servers stopped, `gitaly serve` makes every dirty
repository durable and removes its marker, so a clean reboot hydrates
nothing.
diff --git a/internal/gitaly/hook/updateref/deferred_durability_test.go b/internal/gitaly/hook/updateref/deferred_durability_test.go
index 72efa27..a020189 100644
--- a/internal/gitaly/hook/updateref/deferred_durability_test.go
+++ b/internal/gitaly/hook/updateref/deferred_durability_test.go
@@ -2,6 +2,11 @@ package updateref
import (
"context"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strconv"
+ "strings"
"testing"
"github.com/stretchr/testify/require"
@@ -43,6 +48,89 @@ func (v durabilityVoter) DeferReferenceWrites(context.Context, *gitalypb.Reposit
return func() { *v.ended++ }, true
}
+// liveUpdateRefs returns the update-ref processes that are children of the
+// test process and still running.
+func liveUpdateRefs(t *testing.T) []int {
+ t.Helper()
+
+ entries, err := os.ReadDir("/proc")
+ require.NoError(t, err)
+ var pids []int
+ for _, entry := range entries {
+ pid, err := strconv.Atoi(entry.Name())
+ if err != nil {
+ continue
+ }
+ stat, err := os.ReadFile(filepath.Join("/proc", entry.Name(), "stat"))
+ if err != nil {
+ continue
+ }
+ // pid (comm) state ppid ...; comm may contain spaces, so the fields are
+ // read after its closing parenthesis.
+ fields := strings.Fields(string(stat[strings.LastIndexByte(string(stat), ')')+1:]))
+ if len(fields) < 2 || fields[0] == "Z" || fields[1] != strconv.Itoa(os.Getpid()) {
+ continue
+ }
+ cmdline, err := os.ReadFile(filepath.Join("/proc", entry.Name(), "cmdline"))
+ if err != nil {
+ continue
+ }
+ if strings.Contains(string(cmdline), "update-ref") {
+ pids = append(pids, pid)
+ }
+ }
+ return pids
+}
+
+// endingVoter is a voter that defers every reference write and checks, when
+// the end of a deferred git process is reported, that the process has exited.
+type endingVoter struct {
+ durabilityVoter
+ t *testing.T
+ ends *int
+ failed *string
+}
+
+func (v endingVoter) DeferReferenceWrites(context.Context, *gitalypb.Repository) (func(), bool) {
+ return func() {
+ *v.ends++
+ if live := liveUpdateRefs(v.t); len(live) > 0 {
+ *v.failed = fmt.Sprintf("the end ran while update-ref %v was still running", live)
+ }
+ }, true
+}
+
+// TestUpdateReferenceEndsTheDeferredProcessAfterItExited verifies that
+// UpdateWithHooks reports the end of a deferred update-ref only once the
+// process has exited (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE): until
+// then it may still write a table or compact the stack. It does not run in
+// parallel, so that no other test's update-ref is a child at the same time.
+func TestUpdateReferenceEndsTheDeferredProcessAfterItExited(t *testing.T) {
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+ testcfg.BuildGitalyHooks(t, cfg)
+ logger := testhelper.NewLogger(t)
+ locator := config.NewLocator(cfg)
+ gitCmdFactory := gittest.NewCommandFactory(t, cfg)
+
+ repo, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})
+ base := gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("main"))
+ next := gittest.WriteCommit(t, cfg, repoPath, gittest.WithParents(base))
+
+ var ends int
+ var failed string
+ manager := hook.NewManager(cfg, locator, logger, gitCmdFactory, nil, nil,
+ gitlab.NewMockClient(t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive),
+ hook.NewPostReceiveRegistry(),
+ hook.WithStorageVoter(endingVoter{t: t, ends: &ends, failed: &failed}))
+ updater := NewUpdaterWithHooks(cfg, logger, locator, manager, gitCmdFactory, nil)
+
+ require.NoError(t, updater.UpdateReference(ctx, repo, gittest.TestUser, nil, "refs/heads/main", next, base))
+ require.Equal(t, 1, ends)
+ require.Empty(t, failed)
+ require.Equal(t, next, gittest.ResolveRevision(t, cfg, repoPath, "refs/heads/main"))
+}
+
// TestDeferReferenceWrites verifies that the updater writes references without fsync only when
// the storage voter of its hook manager defers their durability: never without the storage
// core, nor when the voter does not (shadow mode, authoritative mode where the kernel exposes no
diff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.go
index f96aeb3..35b7ad0 100644
--- a/internal/graft/corestore/localrefs/durability.go
+++ b/internal/graft/corestore/localrefs/durability.go
@@ -21,7 +21,6 @@ import (
"github.com/prometheus/client_golang/prometheus"
- "github.com/feder-cr/git-ai/internal/git/gitcmd"
"github.com/feder-cr/git-ai/internal/gitaly/config"
"github.com/feder-cr/git-ai/internal/gitaly/storage"
"github.com/feder-cr/git-ai/proto/go/gitalypb"
@@ -56,10 +55,11 @@ const (
// write before the durable step removes its marker.
DurableStepIdle = time.Second
// RecoveryGrace is how long a repository found dirty when the process takes
- // the ownership keeps its marker at least: longer than git waits for the
- // reference lock (reftable.lockTimeout), so that a git process of the
- // previous owner that is not reaped yet has finished.
- RecoveryGrace = 2 * gitcmd.ReftableLockTimeoutMilliseconds * time.Millisecond
+ // the ownership keeps its marker at least. The rule that decides is that no
+ // git lock or temporary file is in its reftable directory (gitActive); the
+ // grace only covers the instant between a commit's rename and the start of
+ // its auto-compaction, when a git process has neither.
+ RecoveryGrace = 2 * time.Second
)
// WithRecoveryGrace replaces RecoveryGrace. Tests shorten it.
@@ -191,11 +191,16 @@ type refState struct {
// may still write a table or compact the stack.
nLive int
lastWrite time.Time
- // settleAfter keeps the marker of a repository recovered dirty until then.
+ // recovered is set for a repository found dirty when the process took the
+ // ownership: a git process of the previous owner that outlived it may still
+ // be writing, so its marker stays while git's lock or temporary files are
+ // there (gitActive), and not before settleAfter.
+ recovered bool
settleAfter time.Time
// untrusted is the first failed fsync of the repository. The kernel reports
// a failed writeback once, so a later fsync can succeed although writes
// were lost: the repository never becomes clean again, only moved aside.
+ // The marker records it, so that a restart of the process keeps it.
untrusted error
// synced holds the tables made durable during the current dirty period that
// the stack still lists. Tables are immutable, so one fsync each is enough.
@@ -216,6 +221,9 @@ func repoOf(r RepoRef) *gitalypb.Repository {
type marker struct {
RepoRef
BootID string `json:"boot_id"`
+ // Untrusted is the failed fsync that made the repository untrusted, empty
+ // while it is only dirty.
+ Untrusted string `json:"untrusted,omitempty"`
}
// Reasons of graft_storage_ref_untrusted_total.
@@ -297,7 +305,7 @@ func newDurability(locator storage.Locator, idle time.Duration) *Durability {
}),
inflight: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "graft_storage_ref_writes_inflight",
- Help: "Number of hook reference transactions between their start and their end. One that never ends keeps its repository dirty.",
+ Help: "Number of git processes of this process that write references without fsync and have not exited; their repositories keep their markers.",
}),
owning: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "graft_storage_ref_durability_owner",
@@ -441,20 +449,32 @@ func (d *Durability) MarkDirty(ctx context.Context, repo RepoRef, path string) e
return nil
}
- d.mu.Lock()
- st := d.stateLocked(path)
- st.lastWrite = time.Now()
- d.mu.Unlock()
-
- st.file.Lock()
- defer st.file.Unlock()
-
- d.mu.Lock()
- dirty := st.dirty
- d.mu.Unlock()
- if dirty {
- return nil
+ // The state is the map's after its file lock is taken: a durable step may
+ // have dropped a clean state meanwhile.
+ var st *refState
+ for {
+ d.mu.Lock()
+ st = d.stateLocked(path)
+ d.mu.Unlock()
+ st.file.Lock()
+ d.mu.Lock()
+ current := d.repos[path] == st
+ if current {
+ st.lastWrite = time.Now()
+ }
+ dirty := st.dirty
+ d.mu.Unlock()
+ if !current {
+ st.file.Unlock()
+ continue
+ }
+ if dirty {
+ st.file.Unlock()
+ return nil
+ }
+ break
}
+ defer st.file.Unlock()
markerPath, err := d.markerPath(ctx, repo, path)
if err != nil {
@@ -583,13 +603,26 @@ func (d *Durability) Settle(path string, final bool) error {
}
now := time.Now()
idle := st.nLive == 0 && now.After(st.settleAfter) && (final || now.Sub(st.lastWrite) >= d.idle)
+ recovered := st.recovered
d.mu.Unlock()
+ if idle && recovered {
+ // A git process of the previous owner may still be writing.
+ active, err := gitActive(path)
+ if err != nil {
+ return err
+ }
+ idle = !active
+ }
if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {
st.file.Lock()
defer st.file.Unlock()
d.clear(st)
- return d.removeMarker(st)
+ if err := d.removeMarker(st); err != nil {
+ return err
+ }
+ d.prune(path, st)
+ return nil
}
if err := d.syncStack(path, st); err != nil {
@@ -623,12 +656,31 @@ func (d *Durability) Settle(path string, final bool) error {
if err := lock.release(); err != nil {
return err
}
- return d.removeMarker(st)
+ if err := d.removeMarker(st); err != nil {
+ return err
+ }
+ d.prune(path, st)
+ return nil
+}
+
+// prune drops the state of a repository that is clean and has no live git
+// process, so that the states do not grow with every repository ever written.
+// The caller holds the state's file lock: MarkDirty takes the state again from
+// the map after its own.
+func (d *Durability) prune(path string, st *refState) {
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ if d.repos[path] == st && !st.dirty && st.untrusted == nil && st.nLive == 0 {
+ delete(d.repos, path)
+ }
}
// stepError is the error of a durable step: an *UntrustedError for a failed
// fsync, which makes the repository untrusted for good, the error itself
-// otherwise.
+// otherwise. The first failure is written into the marker, so that the
+// repository stays untrusted across a restart of the process (the next fsync
+// would succeed); when even that write fails, the disk is failing and the
+// state of this process keeps it untrusted.
func (d *Durability) stepError(st *refState, repo RepoRef, path string, err error) error {
var writeback writebackError
if !errors.As(err, &writeback) {
@@ -636,13 +688,44 @@ func (d *Durability) stepError(st *refState, repo RepoRef, path string, err erro
}
d.mu.Lock()
- if st.untrusted == nil {
+ first := st.untrusted == nil
+ if first {
st.untrusted = err
}
+ markerPath := st.marker
d.mu.Unlock()
+
+ if first && markerPath != "" {
+ st.file.Lock()
+ content, encodeErr := json.Marshal(marker{RepoRef: repo, BootID: d.bootID, Untrusted: err.Error()})
+ if encodeErr == nil {
+ encodeErr = d.writeDurably(markerPath, content)
+ }
+ st.file.Unlock()
+ if encodeErr != nil {
+ return &UntrustedError{Repo: repo, Path: path, Err: fmt.Errorf("%w (recording it in the marker failed: %w)", err, encodeErr)}
+ }
+ }
return &UntrustedError{Repo: repo, Path: path, Err: err}
}
+// gitActive reports whether a git process may be writing the reftable stack of
+// the repository at path: git's lock of the list, a lock of a table being
+// compacted, or a table being written is there.
+func gitActive(path string) (bool, error) {
+ entries, err := os.ReadDir(filepath.Join(path, "reftable"))
+ if err != nil {
+ return false, fmt.Errorf("read the reftable directory: %w", err)
+ }
+ for _, entry := range entries {
+ name := entry.Name()
+ if strings.HasSuffix(name, ".lock") || strings.Contains(name, ".temp.") {
+ return true, nil
+ }
+ }
+ return false, nil
+}
+
// settleLocked makes the tables written since the first pass durable and, when
// no transaction of the process is live, marks the state clean with its file
// lock held (the caller releases it). The caller holds git's reference lock: no
@@ -877,6 +960,35 @@ func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) erro
defer st.file.Unlock()
}
+ d.mu.Lock()
+ live := st != nil && d.repos[path] == st && st.nLive > 0
+ d.mu.Unlock()
+ if live {
+ // A git process that writes without fsync is still running. It names the
+ // repository by its path, so its later writes land in what is there now
+ // (the hydrated repository, say): the repository stays dirty, with a plain
+ // marker, until it exited and a durable step made the stack durable.
+ content, err := json.Marshal(marker{RepoRef: repo, BootID: d.bootID})
+ if err != nil {
+ return fmt.Errorf("encode the dirty marker: %w", err)
+ }
+ if err := d.writeDurably(markerPath, content); err != nil {
+ return fmt.Errorf("write the dirty marker: %w", err)
+ }
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ if !st.dirty {
+ st.dirty = true
+ d.dirty.Inc()
+ }
+ st.repo = repo
+ st.marker = markerPath
+ st.untrusted = nil
+ st.recovered = false
+ st.synced = map[string]struct{}{}
+ return nil
+ }
+
if err := os.Remove(markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("remove the dirty marker: %w", err)
}
@@ -887,7 +999,6 @@ func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) erro
if st.dirty {
d.dirty.Dec()
}
- d.inflight.Sub(float64(st.nLive))
delete(d.repos, path)
}
return nil
@@ -900,14 +1011,20 @@ type Recovered struct {
// Reboot reports that the marker is of another kernel boot: the repository
// must be moved aside before anything reads it.
Reboot bool
+ // Untrusted reports that a durable step of the previous process found a
+ // failed fsync: the repository must be moved aside too, and is never
+ // settled.
+ Untrusted bool
}
// Recover reads the dirty markers of the storages; only the owner calls it,
// when it takes the ownership. A marker of the running boot makes its
-// repository dirty again, to be settled, and what a durable step of the
-// previous owner left (its reference lock, its temporary files) is removed. It
-// returns every repository found; the caller moves aside the ones of another
-// boot before serving. A marker that cannot be read is an error: its
+// repository dirty again, to be settled once no git process of the previous
+// owner can still write it (gitActive, and the recovery grace), or untrusted
+// again when the marker says so; what a durable step of the previous owner
+// left (its reference lock, its temporary files) is removed. It returns every
+// repository found; the caller moves aside the ones of another boot and the
+// untrusted ones before serving. A marker that cannot be read is an error: its
// repository is unknown, and serving could read a torn stack.
func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovered, error) {
if d == nil || !d.ownsLocks() {
@@ -975,10 +1092,14 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovere
}
st.repo = m.RepoRef
st.marker = markerPath
+ st.recovered = true
st.settleAfter = time.Now().Add(d.recoveryGrace)
+ if m.Untrusted != "" {
+ st.untrusted = errors.New(m.Untrusted)
+ }
d.mu.Unlock()
}
- found = append(found, Recovered{Repo: m.RepoRef, Path: path, Reboot: reboot})
+ found = append(found, Recovered{Repo: m.RepoRef, Path: path, Reboot: reboot, Untrusted: m.Untrusted != ""})
}
}
return found, nil
diff --git a/internal/graft/corestore/localrefs/durability_test.go b/internal/graft/corestore/localrefs/durability_test.go
index 33c35b1..974714c 100644
--- a/internal/graft/corestore/localrefs/durability_test.go
+++ b/internal/graft/corestore/localrefs/durability_test.go
@@ -39,12 +39,14 @@ type syncRecorder struct {
mu sync.Mutex
paths []string
failWith error
+ // failOnly, when set, limits the failures to the paths that contain it.
+ failOnly string
}
func (s *syncRecorder) sync(f *os.File) error {
s.mu.Lock()
defer s.mu.Unlock()
- if s.failWith != nil {
+ if s.failWith != nil && strings.Contains(f.Name(), s.failOnly) {
return s.failWith
}
s.paths = append(s.paths, f.Name())
@@ -403,6 +405,115 @@ func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {
require.Equal(t, 0.0, testutil.ToFloat64(d.dirty))
}
+// TestUntrustedSurvivesAProcessRestart verifies that the untrusted state is in
+// the marker: a restart of the process in the same boot, whose fsyncs then
+// succeed, finds the repository untrusted and never settles it.
+func TestUntrustedSurvivesAProcessRestart(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+
+ before := f.durability(t, "boot-a", 0)
+ require.NoError(t, before.MarkDirty(f.ctx, f.repo, f.path))
+ // The writes of the references were lost; the marker's directory is fine.
+ f.synced.mu.Lock()
+ f.synced.failOnly = "/reftable"
+ f.synced.mu.Unlock()
+ f.synced.fail(errors.New("input/output error"))
+ require.ErrorIs(t, before.Settle(f.path, false), ErrUntrusted)
+ f.synced.fail(nil)
+ content, err := os.ReadFile(f.markerPath(t, before))
+ require.NoError(t, err)
+ require.Contains(t, string(content), `"untrusted":"fsync: input/output error"`)
+ require.NoError(t, before.Close())
+
+ after := f.durability(t, "boot-a", 0)
+ found, err := after.Recover(f.ctx, []string{f.repo.Storage})
+ require.NoError(t, err)
+ require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path, Untrusted: true}}, found)
+ err = after.Settle(f.path, true)
+ require.ErrorIs(t, err, ErrUntrusted)
+ require.FileExists(t, f.markerPath(t, after))
+}
+
+// TestForgetKeepsARepositoryWithALiveGitProcess verifies that moving a
+// repository aside while a git process that writes without fsync is still
+// running keeps the repository dirty: that process names the repository by its
+// path, so its later writes land in what is there now.
+func TestForgetKeepsARepositoryWithALiveGitProcess(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ end := d.Begin(f.path)
+ f.synced.fail(errors.New("input/output error"))
+ require.ErrorIs(t, d.Settle(f.path, false), ErrUntrusted)
+ f.synced.fail(nil)
+
+ require.NoError(t, d.Forget(f.ctx, f.repo, f.path))
+ require.FileExists(t, f.markerPath(t, d))
+ content, err := os.ReadFile(f.markerPath(t, d))
+ require.NoError(t, err)
+ require.NotContains(t, string(content), "untrusted", "what is at the path now is not the untrusted repository")
+ require.Equal(t, []string{f.path}, d.Dirty())
+ require.NoError(t, d.Settle(f.path, true))
+ require.FileExists(t, f.markerPath(t, d), "the git process is still running")
+
+ end()
+ require.NoError(t, d.Settle(f.path, false))
+ require.NoFileExists(t, f.markerPath(t, d))
+}
+
+// TestRecoveredMarkerWaitsForGit verifies that a marker found at the
+// recovery stays while git's lock or a table being written is in the
+// repository: a git process of the previous owner may still be writing.
+func TestRecoveredMarkerWaitsForGit(t *testing.T) {
+ t.Parallel()
+
+ for _, leftover := range []string{"tables.list.lock", "0x000000000009-0x000000000009-abcdef01.ref.lock", "0x000000000009-0x000000000009-abcdef01.ref.temp.XyZ123"} {
+ t.Run(leftover, func(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+
+ old := f.durability(t, "boot-a", 0)
+ require.NoError(t, old.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, old.Close())
+
+ file := filepath.Join(f.path, "reftable", leftover)
+ require.NoError(t, os.WriteFile(file, []byte("0x000000000001-0x000000000003-abcdef01.ref\n"), 0o666))
+
+ next := f.durability(t, "boot-a", 0)
+ _, err := next.Recover(f.ctx, []string{f.repo.Storage})
+ require.NoError(t, err)
+ require.NoError(t, next.Settle(f.path, true))
+ require.FileExists(t, f.markerPath(t, next), "git may still be writing")
+
+ require.NoError(t, os.Remove(file))
+ require.NoError(t, next.Settle(f.path, true))
+ require.NoFileExists(t, f.markerPath(t, next))
+ })
+ }
+}
+
+// TestSettleDropsTheStateOfACleanRepository verifies that the states do not
+// grow with every repository ever written, and that a writer after the drop
+// marks the repository again.
+func TestSettleDropsTheStateOfACleanRepository(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, d.Settle(f.path, false))
+ d.mu.Lock()
+ require.Empty(t, d.repos)
+ d.mu.Unlock()
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ require.FileExists(t, f.markerPath(t, d))
+ require.Equal(t, []string{f.path}, d.Dirty())
+}
+
// TestSettleTransientErrorIsNotUntrusted verifies that a failure to read the
// stack (not a failed fsync) is retried: the repository is not distrusted.
func TestSettleTransientErrorIsNotUntrusted(t *testing.T) {
diff --git a/internal/graft/corestore/syncer/durability.go b/internal/graft/corestore/syncer/durability.go
index f5ab734..f428854 100644
--- a/internal/graft/corestore/syncer/durability.go
+++ b/internal/graft/corestore/syncer/durability.go
@@ -9,6 +9,9 @@ import (
"sync"
"time"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
"github.com/feder-cr/git-ai/internal/structerr"
@@ -122,10 +125,14 @@ func (s *Syncer) runDurableSteps(ctx context.Context, storages []string, recover
// durability. A repository dirty from a process restart of the same boot is
// kept: the kernel kept every write, and the durable step settles it.
//
-// Only a cache of the storage core is moved: the database must hold the
-// repository with changesets, and the local repository must have its
-// watermark. Any other marker (the repository was removed, or replaced by one
-// being imported, whose local copy may be the only one) is dropped.
+// A repository whose marker says that a durable step of the previous process
+// found a failed fsync is moved aside the same way: its writes may be lost,
+// although the kernel would report nothing more.
+//
+// Only a cache of the storage core is moved (storageCoreCache). Any other
+// marker of another boot (the repository was removed, or replaced by one being
+// imported, whose local copy may be the only one) is dropped; an untrusted one
+// is kept, with its repository untrusted, and logged.
func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string) error {
d := s.core.RefDurability
found, err := d.Recover(ctx, storages)
@@ -134,24 +141,33 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)
}
for _, rec := range found {
- if !rec.Reboot {
+ if !rec.Reboot && !rec.Untrusted {
continue
}
+ reason := localrefs.ReasonReboot
+ if !rec.Reboot {
+ reason = localrefs.ReasonSyncFailed
+ }
- cache, err := s.storageCoreCache(ctx, rec.Repo, rec.Path)
+ cache, keep, err := s.storageCoreCache(ctx, rec.Repo, rec.Path)
if err != nil {
return fmt.Errorf("storage core: recover the local references of %s:%s: %w", rec.Repo.Storage, rec.Repo.RelativePath, err)
}
if !cache {
- s.logger.WithField("repository_path", rec.Path).
- Warn("storage core: dropped the dirty marker of a local repository that is not a cache of the storage core")
- if err := d.Forget(ctx, rec.Repo, rec.Path); err != nil {
- return err
+ if rec.Reboot {
+ s.logger.WithField("repository_path", rec.Path).
+ Warn("storage core: dropped the dirty marker of a local repository that is not a cache of the storage core")
+ if err := d.Forget(ctx, rec.Repo, rec.Path); err != nil {
+ return err
+ }
+ } else {
+ s.logger.WithField("repository_path", rec.Path).
+ Error("storage core: the references of a repository that is not a cache of the storage core may have lost writes; it is kept in place, untrusted")
}
continue
}
- if err := s.distrust(ctx, rec.Repo, rec.Path, localrefs.ReasonReboot, nil); err != nil {
+ if err := s.distrust(ctx, rec.Repo, rec.Path, reason, nil, keep); err != nil {
return err
}
}
@@ -161,27 +177,37 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)
// storageCoreCache reports whether the local repository at path is a cache of
// the storage core that a hydration can rebuild: the database holds it with
// changesets (an import that is not complete has none, and its local copy may
-// be the only one). A sync replaces such a local repository too when it is
-// stale, whatever its watermark. An error (the database, or the repository's
-// directory, cannot be read) is returned: the caller tries again.
-func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, path string) (bool, error) {
+// be the only one). keep reports that the copy holds history written in shadow
+// mode and not imported yet (checkReplaceable, the guard of a sync's
+// hydration): the copy is moved aside all the same, so that it is never served,
+// but kept, so that the history is not lost. An error (the database, or the
+// repository's files, cannot be read) is returned: the caller tries again.
+func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, path string) (cache, keep bool, err error) {
r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: repo.Storage, RelativePath: repo.RelativePath})
if errors.Is(err, corestore.ErrRepositoryNotFound) {
- return false, nil
+ return false, false, nil
}
if err != nil {
- return false, err
+ return false, false, err
}
if r.HeadSeq == 0 {
- return false, nil
+ return false, false, nil
}
if _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) {
- return false, nil
+ return false, false, nil
} else if err != nil {
- return false, err
+ return false, false, err
+ }
+
+ wm, ok := s.readWatermark(path)
+ if err := s.checkReplaceable(ctx, r, path, wm, ok); err != nil {
+ if status.Code(err) == codes.FailedPrecondition {
+ return true, true, nil
+ }
+ return false, false, err
}
- return true, nil
+ return true, false, nil
}
// durableStep makes the references of the repository at path durable under the
@@ -213,13 +239,13 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {
// A repository that is gone (moved aside as a stale cache meanwhile) has
// nothing left to serve: its marker is dropped.
if _, statErr := os.Lstat(path); errors.Is(statErr, os.ErrNotExist) {
- if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {
+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted, false); err != nil {
logger.WithField("forget_error", err.Error()).Error("storage core: could not drop the marker of a repository that is gone; retried at the next step")
}
return
}
- cache, cacheErr := s.storageCoreCache(ctx, untrusted.Repo, path)
+ cache, keep, cacheErr := s.storageCoreCache(ctx, untrusted.Repo, path)
if cacheErr != nil {
d.CountStepError()
logger.WithField("cache_check_error", cacheErr.Error()).
@@ -232,7 +258,7 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {
logger.Error("storage core: the references of a repository that is not a cache of the storage core may have lost writes; it is kept in place")
return
}
- if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {
+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted, keep); err != nil {
// The untrusted state is sticky: the next step moves it aside again, and
// the marker is never cleared meanwhile.
logger.WithField("move_error", err.Error()).
@@ -241,21 +267,27 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {
}
// distrust is distrustLocked under the repository lock.
-func (s *Syncer) distrust(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {
+func (s *Syncer) distrust(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error, keep bool) error {
release, err := s.lock(ctx, path)
if err != nil {
return err
}
defer release()
- return s.distrustLocked(ctx, repo, path, reason, cause)
+ return s.distrustLocked(ctx, repo, path, reason, cause, keep)
}
-// distrustLocked moves the local repository at path into the temporary
-// directory of its storage and removes it there in the background, then drops
-// its dirty marker: the next request hydrates it. The move is made durable
-// (both directories, and the new directory's entry) before the marker goes, so
-// a crash in between moves it again. The caller holds the repository lock.
-func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {
+// KeptDirName is the directory of a storage where the untrusted repositories
+// that hold history not imported yet are moved: they are never served, and an
+// operator imports what they hold or removes them.
+const KeptDirName = "+gitaly/graft-untrusted"
+
+// distrustLocked moves the local repository at path out of the way, then drops
+// its dirty marker: the next request hydrates it. It goes into the temporary
+// directory of its storage and is removed there in the background, or, with
+// keep, into KeptDirName, where it stays. The move is made durable (both
+// directories, and the new directory's entry) before the marker goes, so a
+// crash in between moves it again. The caller holds the repository lock.
+func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error, keep bool) error {
d := s.core.RefDurability
if _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) {
@@ -264,19 +296,29 @@ func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, pat
return structerr.NewInternal("storage core: stat repository: %w", err)
}
- tmpDir, err := s.tmpDir(repo.Storage)
+ parent, err := s.tmpDir(repo.Storage)
if err != nil {
return err
}
- stale, err := os.MkdirTemp(tmpDir, tmpStalePattern)
+ if keep {
+ root, err := s.locator.GetStorageByName(ctx, repo.Storage)
+ if err != nil {
+ return err
+ }
+ parent = filepath.Join(root, KeptDirName)
+ if err := os.MkdirAll(parent, 0o700); err != nil {
+ return structerr.NewInternal("storage core: create the directory of kept repositories: %w", err)
+ }
+ }
+ aside, err := os.MkdirTemp(parent, tmpStalePattern)
if err != nil {
return structerr.NewInternal("storage core: create stale directory: %w", err)
}
- if err := os.Rename(path, filepath.Join(stale, "repo.git")); err != nil {
- _ = os.Remove(stale)
+ if err := os.Rename(path, filepath.Join(aside, "repo.git")); err != nil {
+ _ = os.Remove(aside)
return structerr.NewInternal("storage core: move aside an untrusted repository: %w", err)
}
- for _, dir := range []string{filepath.Dir(path), stale, tmpDir} {
+ for _, dir := range []string{filepath.Dir(path), aside, parent, filepath.Dir(parent)} {
if err := syncDir(dir); err != nil {
return structerr.NewInternal("storage core: make the move of an untrusted repository durable: %w", err)
}
@@ -288,13 +330,17 @@ func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, pat
}
d.CountUntrusted(reason)
- logger := s.logger.WithField("repository_path", path).WithField("stale_path", stale).WithField("reason", reason)
+ logger := s.logger.WithField("repository_path", path).WithField("moved_path", aside).WithField("reason", reason)
if cause != nil {
logger = logger.WithError(cause)
}
+ if keep {
+ logger.Error("storage core: moved aside a local repository whose references may have lost writes; it holds shadow-mode history not imported yet and is kept for an operator; the next request hydrates the repository")
+ return nil
+ }
logger.Error("storage core: moved aside a local repository whose references may have lost writes, the next request hydrates it")
- go func() { _ = os.RemoveAll(stale) }()
+ go func() { _ = os.RemoveAll(aside) }()
return nil
}
diff --git a/internal/graft/corestore/syncer/syncer_localrefs_test.go b/internal/graft/corestore/syncer/syncer_localrefs_test.go
index 1dd81d5..a2e2a3b 100644
--- a/internal/graft/corestore/syncer/syncer_localrefs_test.go
+++ b/internal/graft/corestore/syncer/syncer_localrefs_test.go
@@ -14,6 +14,7 @@ import (
"github.com/feder-cr/git-ai/internal/git"
"github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/graft/changelog"
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/coretest"
"github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
@@ -247,6 +248,78 @@ func TestSyncerUntrustedRepositoryIsNeverSettled(t *testing.T) {
e.verifySynced(t, repo)
}
+// failReferences returns a sync function that fails the fsyncs of reference
+// files while failing is set: the writes of the references were lost, the rest
+// of the disk is fine.
+func failReferences(failing *atomic.Bool) func(f *os.File) error {
+ return func(f *os.File) error {
+ if failing.Load() && strings.Contains(f.Name(), "/reftable") {
+ return errors.New("input/output error")
+ }
+ return f.Sync()
+ }
+}
+
+// TestSyncerUntrustedRepositoryAcrossAProcessRestart is a failed fsync whose
+// move aside fails, followed by a restart of the process in the same boot:
+// the restarted process's fsyncs succeed, and the repository is moved aside
+// all the same, never settled.
+func TestSyncerUntrustedRepositoryAcrossAProcessRestart(t *testing.T) {
+ t.Parallel()
+
+ var failing atomic.Bool
+ e, repo, _ := caughtUpDirty(t, localrefs.WithSync(failReferences(&failing)))
+ tmpDir, err := e.locator.TempDir(e.storageName)
+ require.NoError(t, err)
+ require.NoError(t, os.MkdirAll(tmpDir, 0o700))
+
+ failing.Store(true)
+ require.NoError(t, os.Chmod(tmpDir, 0o500))
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
+ require.NoError(t, os.Chmod(tmpDir, 0o700))
+ failing.Store(false)
+ require.DirExists(t, e.localPath(), "the move aside failed")
+
+ // The process restarts in the same boot.
+ coretest.EnableLocalRefs(t, e.core, e.cfg)
+ e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)
+ require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))
+
+ require.NoDirExists(t, e.localPath(), "an untrusted repository is moved aside at the restart")
+ require.Empty(t, e.markers(t))
+ require.Equal(t, float64(1), e.untrusted(t, localrefs.ReasonSyncFailed))
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ e.verifySynced(t, repo)
+}
+
+// TestSyncerUntrustedRepositoryWithShadowHistoryIsKept verifies that an
+// untrusted repository whose local copy holds history written in shadow mode
+// and not imported yet (the guard of a sync's hydration) is moved aside but
+// not deleted: it is never served, and the history is not lost.
+func TestSyncerUntrustedRepositoryWithShadowHistoryIsKept(t *testing.T) {
+ t.Parallel()
+
+ var failing atomic.Bool
+ e, _, c2 := caughtUpDirty(t, localrefs.WithSync(failReferences(&failing)))
+ _, err := changelog.Append(e.localPath(), changelog.Meta{Operation: "push", Actor: "alice"}, []changelog.Update{
+ {Reference: "refs/heads/shadow", OldOID: gittest.DefaultObjectHash.ZeroOID.String(), NewOID: c2.String()},
+ })
+ require.NoError(t, err)
+
+ failing.Store(true)
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
+ failing.Store(false)
+
+ require.NoDirExists(t, e.localPath(), "never served")
+ kept, err := filepath.Glob(filepath.Join(e.storagePath, syncer.KeptDirName, "*", "repo.git"))
+ require.NoError(t, err)
+ require.Len(t, kept, 1, "the copy with the shadow history is kept")
+ last, err := changelog.LastSequence(kept[0])
+ require.NoError(t, err)
+ require.Positive(t, last)
+ require.Empty(t, e.markers(t))
+}
+
// TestSyncerUntrustedRepositoryThatIsGoneLosesItsMarker verifies that an
// untrusted repository that was moved away meanwhile (a stale cache) does not
// keep its marker forever.