Git designed for AI agents.
WP76b follow-up 2: live count per git process, sticky untrusted state
Federico Elia18e152ad52a0
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.mdindex cb98074..4d59379 100644--- a/doc/graft/PLAN7.md+++ b/doc/graft/PLAN7.md@@ -575,19 +575,21 @@ repository-wide lock.the marker cannot be written, Prepared fails and git aborts; catch-uprepairs the database's write, like any local failure.* The syncer writes it under its repository lock.- * A hook transaction is live from its start to its end, and the durable- step never clears the marker while any is live.- * Start: the voter's Preparing, or `Denied` for a transaction the policy- denies.- * End: Committed or Aborted.- * Every transaction starts once and ends once, so the count per input is- the number of live transactions, whichever of two with the same input- ends first.+ * The durable step never clears the marker while a git process of the owner+ that writes references without fsync is alive.+ * Only `UpdateWithHooks`' update-ref writes that way. It is counted from+ just before it starts (`DeferReferenceWrites`, which also decides the+ deferral) until after it exited (`updater.Close` waits for it).+ * The syncer's update-ref runs under the syncer lock, which the durable+ step holds too.+ * The count follows the process, not the hooks. git skips the Aborted and+ Committed hooks in several cases: a failed prepare (a stale old value,+ a busy lock, a name conflict), a failed finish, initial transactions, a+ kill. Its Committed hook names a symbolic reference's target instead of+ the reference. Neither can leak a count or end the wrong one.* Reasons from git's source: the new table is written in`transaction_finish` after "prepared", and auto-compaction runs inside`reftable_addition_commit`, before "committed".- * A transaction whose end is never reported keeps its repository dirty- (conservative).* *Durable step* (`Settle`, every second for every dirty repository, under thesyncer's repository lock).* It fsyncs the tables not yet fsynced, `tables.list` and the directory.@@ -606,10 +608,16 @@ repository-wide lock.* A graceful stop runs a last step for every dirty repository after theservers stopped, so a clean reboot does not rehydrate them.* *Write failure.* Only a failed fsync means the kernel may have dropped- writes. The repository is then moved aside at once, under the repository- lock: the move (both directories and the new entry) is made durable, then- the marker removed. This happens only when the repository is a cache of the- storage core, the same check as the recovery. The failure is logged and+ writes, and the kernel reports a failed writeback once: a later fsync of the+ same file succeeds.+ * So the first failed fsync makes the repository *untrusted* for good.+ * An untrusted repository's marker is never cleared; only the move aside is+ retried, until it succeeds. Then the hydrated repository starts clean.+ * The repository is moved aside at once, under the repository lock: the+ move (both directories and the new entry) is made durable, then the+ marker removed.+ * This happens only when the repository is a cache of the storage core:+ the database holds it with changesets, as for the recovery. The failure is logged andcounted (`sync_failed`), and the next request hydrates the repository. Anyother error (opening a file, reading the list) is transient: counted(`graft_storage_ref_durable_step_errors_total`) and retried at the next@@ -625,10 +633,22 @@ repository-wide lock.moved aside before anything reads it (`reboot`), and the next requesthydrates it.* The move only happens when the database holds the repository with- changesets and the local copy has a watermark; any other marker is- dropped, keeping a repository whose local copy may be the only one.+ changesets. A sync would replace such a local copy too when it is stale,+ whatever its watermark. Any other marker is dropped, keeping a repository+ whose local copy may be the only one (an import that is not complete).* A marker of the same boot is a process restart: the page cache keptevery write, so the repository stays dirty to be settled.+ * It is settled no earlier than the recovery grace (twice+ `reftable.lockTimeout`): a git process of the previous owner that+ outlived it (killed with it, not reaped yet) may still be committing+ for that long.+ * The git children get no parent-death signal: the command package is+ shared and not in m3's grant. The grace bounds their life instead,+ since a killed parent closes their stdin and git then aborts.+ * The ownership is taken in two steps. The locks come first; the recovery+ runs; only then are the process's writes deferred (`Activate`). So no+ writer of the new owner writes a marker that the recovery could take for+ a leftover.* A lock with the durable step's tag and the step's temporary files, leftby a process that died, are removed.* A corrupt marker fails the startup: its repository is unknown.@@ -655,7 +675,7 @@ repository-wide lock.* PM grant: `updateref.WithDeferredReferenceDurability`; the `gitcmd` constants`FsyncComponents`, `FsyncComponentsWithoutReferences` and`ReftableLockTimeoutMilliseconds`; `hook/storage_voter.go`- (`DeferredReferenceDurability`, `Denied`); the optional interface asserted in+ (`DeferReferenceWrites`); the optional interface asserted in`UpdateWithHooks`.* Follow-up after the independent review of 310facd:* MAJOR: process-local state with overlapping processes. Fixed by one owner@@ -668,6 +688,24 @@ repository-wide lock.* The directory fsyncs (the marker directory's parent; the stale and temporarydirectories of a move) are added.* The step's temporary files are cleaned at recovery.+* Second review (of 8953e8e):+ * M1: the live count is tied to the deferred git process itself; it no+ longer depends on hooks git may skip. `StorageVoter.Denied` is gone.+ * M2: untrusted is sticky.+ * The recovery runs before the writes are deferred (`TryOwn`, `Recover`,+ `Activate`).+ * The recovery grace for recovered markers.+ * The dirty gauge counts a repository once.+ * Tests:+ * the reused inode is simulated deterministically;+ * a non-owner neither recovers nor removes the live owner's lock;+ * the e2e upgrade test reads the owner and marker counters instead of the+ racing marker file;+ * a non-owner's write marks nothing, at unit, syncer and e2e level;+ * git skipping its hooks, with the race, the failed prepare and the+ symbolic split;+ * a failed fsync then a successful one; and the same with a failed move+ aside.**Tests.*** `localrefs`:diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.mdindex 1276228..522c7a6 100644--- a/doc/graft/STORAGE.md+++ b/doc/graft/STORAGE.md@@ -463,12 +463,16 @@ or re-import the repository from scratch (seeloss: one hydration per repository that had a write in the second before theloss, or that was still being written. On a busy node that can be everyactive repository. A repository is only moved when the database holds it- with changesets and the local copy has a watermark; other markers are- dropped.+ with changesets; other markers are dropped.- *A write the kernel reports as failed* (a writeback error, same boot): thedurable step's fsync fails, the repository is moved aside at once and logged,- and the next request hydrates it. Other errors of the step (a file that+ and the next request hydrates it. The kernel reports such a failure once, so+ from then on the repository's marker is never removed: if the move aside+ fails, it is retried every second. Other errors of the step (a file thatcannot be opened for a moment) are retried every second and counted.+ - *Process restart*: the markers found are kept for 20 s (twice git's+ reference lock timeout) before they can be removed, in case a git process of+ the previous process is still finishing.- *Clean stop*: after the servers stopped, `gitaly serve` makes every dirtyrepository durable and removes its marker, so a clean reboot hydratesnothing.@@ -523,7 +527,7 @@ without a single divergence.| `graft_storage_ref_admission_expired_total` | Turns ended by the 10 s bound instead of their release (a git process that died between its phases). || `graft_storage_ref_dirty_repositories` | Repositories with reference writes not yet made durable. || `graft_storage_ref_dirty_marked_total`, `graft_storage_ref_dirty_cleared_total` | Dirty markers written and removed by the durable step. |-| `graft_storage_ref_writes_inflight` | Hook transactions between their start and their end (Committed or Aborted); one whose end is never reported keeps its repository dirty. |+| `graft_storage_ref_writes_inflight` | git processes of this node that write references without fsync and have not exited; their repositories keep their markers. || `graft_storage_ref_durability_owner` | 1 when this process owns the deferred durability of its storages, 0 while another process (the old one of a graceful upgrade) does. || `graft_storage_ref_durable_step_errors_total` | Durable steps that failed with a transient error (not a failed fsync); retried every second. || `graft_storage_ref_untrusted_total{reason}` | Repositories moved aside to be hydrated again: `reboot` (dirty at a kernel restart), `sync_failed` (the durable step's fsync failed). |diff --git a/internal/gitaly/hook/storage_voter.go b/internal/gitaly/hook/storage_voter.goindex a0fe77a..4d56528 100644--- a/internal/gitaly/hook/storage_voter.go+++ b/internal/gitaly/hook/storage_voter.go@@ -5,6 +5,7 @@ import ("github.com/feder-cr/git-ai/internal/git/gitcmd""github.com/feder-cr/git-ai/internal/graft/changelog"+ "github.com/feder-cr/git-ai/proto/go/gitalypb")// StorageVoter is the write path of the Graft storage core@@ -15,11 +16,6 @@ type StorageVoter interface {// its pack) before git takes the reference locks. It never fails the// transaction: when it did not complete, Prepared does that work itself.Preparing(ctx context.Context, payload gitcmd.HooksPayload, changes []byte)- // Denied is called instead of Preparing for a transaction that the- // standalone access policy denies: it prepares nothing, and the transaction- // is aborted in its Prepared phase. Every transaction gets exactly one of- // Preparing and Denied before its Aborted or Committed phase.- Denied(ctx context.Context, payload gitcmd.HooksPayload, changes []byte)// Prepared commits the transaction to the storage core while git holds// the reference locks. An error aborts the transaction.Prepared(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, meta changelog.Meta) error@@ -30,19 +26,24 @@ type StorageVoter interface {// Authoritative reports whether the storage core's database replaces the// file changelog.Authoritative() bool- // DeferredReferenceDurability reports whether reference transactions that- // the voter commits may write their references without fsync: the database+ // DeferReferenceWrites decides whether the git process the caller is about+ // to start for repo writes its references without fsync: the database// commit is their durable point, and the storage core makes the local// references durable later (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).- DeferredReferenceDurability() bool+ // When ok, the caller calls end once that process has exited.+ DeferReferenceWrites(ctx context.Context, repo *gitalypb.Repository) (end func(), ok bool)}-// DeferredReferenceDurability reports whether the reference transactions that-// the manager drives itself (UpdateWithHooks) may write their references-// without fsync. It is false without a storage voter, so only the storage-// core's authoritative mode ever turns the fsync of references off.-func (m *GitLabHookManager) DeferredReferenceDurability() bool {- return m.storageVoter != nil && m.storageVoter.DeferredReferenceDurability()+// DeferReferenceWrites decides whether the git process that the manager's+// caller (UpdateWithHooks) is about to start writes its references without+// fsync; when ok, end is called once that process has exited. It is false+// without a storage voter, so only the storage core's authoritative mode ever+// turns the fsync of references off.+func (m *GitLabHookManager) DeferReferenceWrites(ctx context.Context, repo *gitalypb.Repository) (end func(), ok bool) {+ if m.storageVoter == nil {+ return nil, false+ }+ return m.storageVoter.DeferReferenceWrites(ctx, repo)}// WithStorageVoter makes the manager commit every reference transaction to@@ -70,7 +71,6 @@ func (m *GitLabHookManager) storagePreparing(ctx context.Context, payload gitcmd}if message, err := m.graftPolicyDecision(env, changes); err != nil || message != "" {- m.storageVoter.Denied(ctx, payload, changes)return}diff --git a/internal/gitaly/hook/storage_voter_test.go b/internal/gitaly/hook/storage_voter_test.goindex abb3694..f125613 100644--- a/internal/gitaly/hook/storage_voter_test.go+++ b/internal/gitaly/hook/storage_voter_test.go@@ -17,6 +17,7 @@ import ("github.com/feder-cr/git-ai/internal/grpc/backchannel""github.com/feder-cr/git-ai/internal/testhelper""github.com/feder-cr/git-ai/internal/testhelper/testcfg"+ "github.com/feder-cr/git-ai/proto/go/gitalypb""github.com/stretchr/testify/require""google.golang.org/grpc/codes""google.golang.org/grpc/status"@@ -44,10 +45,6 @@ func (v *recordingVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte)v.record("preparing")}-func (v *recordingVoter) Denied(context.Context, gitcmd.HooksPayload, []byte) {- v.record("denied")-}-func (v *recordingVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {v.record("prepared")return nil@@ -63,7 +60,9 @@ func (v *recordingVoter) Aborted(context.Context, gitcmd.HooksPayload, []byte) {func (v *recordingVoter) Authoritative() bool { return true }-func (v *recordingVoter) DeferredReferenceDurability() bool { return false }+func (v *recordingVoter) DeferReferenceWrites(context.Context, *gitalypb.Repository) (func(), bool) {+ return nil, false+}// TestReferenceTransactionHook_storageVoterPhases checks that the hook manager// hands every phase of a reference transaction to the storage core, the@@ -86,11 +85,9 @@ func TestReferenceTransactionHook_storageVoterPhases(t *testing.T) {want: []string{"preparing", "prepared", "committed"},},{- // The voter learns of its start (Denied), so that it counts the- // transaction until its abort, but prepares nothing.desc: "a denied transaction is neither prepared nor committed",policy: []config.GraftPolicyRule{{Refs: []string{"refs/heads/main"}, Allow: false}},- want: []string{"denied"},+ want: nil,denied: true,},} {diff --git a/internal/gitaly/hook/updateref/deferred_durability_test.go b/internal/gitaly/hook/updateref/deferred_durability_test.goindex a381daf..72efa27 100644--- a/internal/gitaly/hook/updateref/deferred_durability_test.go+++ b/internal/gitaly/hook/updateref/deferred_durability_test.go@@ -14,18 +14,18 @@ import ("github.com/feder-cr/git-ai/internal/graft/changelog""github.com/feder-cr/git-ai/internal/testhelper""github.com/feder-cr/git-ai/internal/testhelper/testcfg"+ "github.com/feder-cr/git-ai/proto/go/gitalypb")// durabilityVoter is a storage voter that only answers whether it defers the durability of the-// references.+// references, and counts the ends of the processes it deferred.type durabilityVoter struct {deferred bool+ ended *int}func (durabilityVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte) {}-func (durabilityVoter) Denied(context.Context, gitcmd.HooksPayload, []byte) {}-func (durabilityVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {return nil}@@ -36,21 +36,29 @@ func (durabilityVoter) Aborted(context.Context, gitcmd.HooksPayload, []byte) {}func (durabilityVoter) Authoritative() bool { return true }-func (v durabilityVoter) DeferredReferenceDurability() bool { return v.deferred }+func (v durabilityVoter) DeferReferenceWrites(context.Context, *gitalypb.Repository) (func(), bool) {+ if !v.deferred {+ return nil, false+ }+ return func() { *v.ended++ }, true+}-// TestDeferReferenceDurability verifies that the updater writes references without fsync only-// when the storage voter of its hook manager defers their durability: never without the storage-// core, nor when the voter does not (shadow mode, or authoritative mode where the kernel exposes-// no boot identity; the voter's own answer is tested in the voter package).-func TestDeferReferenceDurability(t *testing.T) {+// TestDeferReferenceWrites verifies that the updater writes references without fsync only when+// the storage voter of its hook manager defers their durability: never without the storage+// core, nor when the voter does not (shadow mode, authoritative mode where the kernel exposes no+// boot identity, or a process that does not own the storage; the voter's own answer is tested in+// the voter package).+func TestDeferReferenceWrites(t *testing.T) {t.Parallel()+ ctx := testhelper.Context(t)cfg := testcfg.Build(t)newManager := func(opts ...hook.ManagerOption) hook.Manager {return hook.NewManager(cfg, config.NewLocator(cfg), testhelper.NewLogger(t), gittest.NewCommandFactory(t, cfg), nil, nil,gitlab.NewMockClient(t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive),hook.NewPostReceiveRegistry(), opts...)}+ var ended intfor _, tc := range []struct {desc string@@ -58,12 +66,17 @@ func TestDeferReferenceDurability(t *testing.T) {expected bool}{{desc: "storage core off", manager: newManager()},- {desc: "voter keeps the fsync", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: false}))},- {desc: "voter defers the durability", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: true})), expected: true},+ {desc: "voter keeps the fsync", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: false, ended: &ended}))},+ {desc: "voter defers the durability", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: true, ended: &ended})), expected: true},{desc: "manager without a storage voter", manager: hook.NewMockManager(t, hook.NopPreReceive, hook.NopPostReceive, hook.NopUpdate, hook.NopReferenceTransaction)},} {t.Run(tc.desc, func(t *testing.T) {- require.Equal(t, tc.expected, deferReferenceDurability(tc.manager))+ end, ok := deferReferenceWrites(ctx, tc.manager, &gitalypb.Repository{StorageName: "default", RelativePath: "repo.git"})+ require.Equal(t, tc.expected, ok)+ if ok {+ end()+ require.Equal(t, 1, ended)+ }})}}diff --git a/internal/gitaly/hook/updateref/update_with_hooks.go b/internal/gitaly/hook/updateref/update_with_hooks.goindex acf21e7..917d6b6 100644--- a/internal/gitaly/hook/updateref/update_with_hooks.go+++ b/internal/gitaly/hook/updateref/update_with_hooks.go@@ -143,14 +143,18 @@ func (e Error) Proto() *gitalypb.ReferenceUpdateError {// deferredDurability is implemented by a hook manager whose storage core writes the references// of the transactions it commits without fsync (hook.GitLabHookManager).type deferredDurability interface {- DeferredReferenceDurability() bool+ DeferReferenceWrites(ctx context.Context, repo *gitalypb.Repository) (end func(), ok bool)}-// deferReferenceDurability reports whether the reference transactions driven with hookManager-// may write their references without fsync: only when its storage voter says so.-func deferReferenceDurability(hookManager hook.Manager) bool {- d, ok := hookManager.(deferredDurability)- return ok && d.DeferredReferenceDurability()+// deferReferenceWrites decides whether the update-ref process about to start for repo with+// hookManager writes its references without fsync: only when its storage voter says so. When+// ok, end is called once the process has exited.+func deferReferenceWrites(ctx context.Context, hookManager hook.Manager, repo *gitalypb.Repository) (end func(), ok bool) {+ d, isDeferring := hookManager.(deferredDurability)+ if !isDeferring {+ return nil, false+ }+ return d.DeferReferenceWrites(ctx, repo)}// NewUpdaterWithHooks creates a new instance of a struct that will update a Git reference.@@ -272,8 +276,11 @@ func (u *UpdaterWithHooks) UpdateReference(// this problem.updaterOpts := []updateref.UpdaterOpt{updateref.WithDisabledTransactions()}// The storage core's authoritative mode makes the references durable itself: its database- // commit in the prepared phase is the durable point of the transaction.- if deferReferenceDurability(u.hookManager) {+ // commit in the prepared phase is the durable point of the transaction. It counts the+ // update-ref process as writing without fsync until the process has exited (end runs after+ // updater.Close, which waits for it).+ if end, ok := deferReferenceWrites(ctx, u.hookManager, repoProto); ok {+ defer end()updaterOpts = append(updaterOpts, updateref.WithDeferredReferenceDurability())}updater, err := updateref.New(ctx, repo, updaterOpts...)diff --git a/internal/graft/corestore/coretest/coretest.go b/internal/graft/corestore/coretest/coretest.goindex f7e1da8..e2dc835 100644--- a/internal/graft/corestore/coretest/coretest.go+++ b/internal/graft/corestore/coretest/coretest.go@@ -202,7 +202,8 @@ func Open(tb testing.TB, opts Options) *corestore.Core {// EnableLocalRefs gives the core the admission and the deferred durability of// the local reference writes, owning the storages of cfg, as gitaly serve does// in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). The-// boot identity is a fixed test value unless opts set another one. A deferred+// boot identity is a fixed test value and the recovery grace 0 unless opts set+// others. A deferred// durability the core had is closed first, as when its process exits.func EnableLocalRefs(tb testing.TB, core *corestore.Core, cfg config.Cfg, opts ...localrefs.DurabilityOption) {tb.Helper()@@ -211,11 +212,17 @@ func EnableLocalRefs(tb testing.TB, core *corestore.Core, cfg config.Cfg, opts .require.NoError(tb, core.RefDurability.Close())}core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)- durability, err := localrefs.NewDurability(config.NewLocator(cfg), 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID)}, opts...)...)+ durability, err := localrefs.NewDurability(config.NewLocator(cfg), 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID), localrefs.WithRecoveryGrace(0)}, opts...)...)require.NoError(tb, err)- owned, err := durability.TryOwn(context.Background(), StorageNames(cfg))- require.NoError(tb, err)- require.True(tb, owned, "another durability of the test owns the storages")+ // A git process forked at the moment a previous durability closed its lock+ // holds a copy of it until it execs (close-on-exec): the lock can be busy for+ // that short while.+ require.Eventually(tb, func() bool {+ owned, err := durability.TryOwn(context.Background(), StorageNames(cfg))+ require.NoError(tb, err)+ return owned+ }, 10*time.Second, time.Millisecond, "another durability of the test owns the storages")+ durability.Activate()tb.Cleanup(func() { _ = durability.Close() })core.RefDurability = durability}diff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.goindex 66147b1..f96aeb3 100644--- a/internal/graft/corestore/localrefs/durability.go+++ b/internal/graft/corestore/localrefs/durability.go@@ -21,6 +21,7 @@ import ("github.com/prometheus/client_golang/prometheus"+ "github.com/feder-cr/git-ai/internal/git/gitcmd""github.com/feder-cr/git-ai/internal/gitaly/config""github.com/feder-cr/git-ai/internal/gitaly/storage""github.com/feder-cr/git-ai/proto/go/gitalypb"@@ -54,8 +55,18 @@ const (// DurableStepIdle is how long a dirty repository must be without a reference// write before the durable step removes its marker.DurableStepIdle = time.Second+ // RecoveryGrace is how long a repository found dirty when the process takes+ // the ownership keeps its marker at least: longer than git waits for the+ // reference lock (reftable.lockTimeout), so that a git process of the+ // previous owner that is not reaped yet has finished.+ RecoveryGrace = 2 * gitcmd.ReftableLockTimeoutMilliseconds * time.Millisecond)+// WithRecoveryGrace replaces RecoveryGrace. Tests shorten it.+func WithRecoveryGrace(grace time.Duration) DurabilityOption {+ return func(d *Durability) { d.recoveryGrace = grace }+}+// CrashEnv names the test-only environment variable that makes the process// exit (exit code 75) in the durable step, while it holds git's reference lock// after making the stack durable and before it removes the marker: "durable_step".@@ -138,11 +149,17 @@ type Durability struct {// crashAt is the crash point selected with CrashEnv.crashAt string- // owner is set once the process owns the deferred durability of its- // storages; it never goes back. ownerFiles hold the owner locks.+ // ownerFiles hold the owner locks, taken by TryOwn. owner is set by+ // Activate once the process has recovered the storages: from then on its+ // writes are deferred. Neither goes back.owner atomic.BoolownerMu sync.MutexownerFiles []*os.File+ locked bool+ // recoveryGrace is how long a repository recovered dirty keeps its marker+ // at least: a git process of the previous owner that outlived it (killed+ // with it, but not reaped yet) may still be committing for that long.+ recoveryGrace time.Durationmu sync.Mutex// repos holds the state of every repository written since the process@@ -169,13 +186,17 @@ type refState struct {repo RepoRefmarker stringdirty bool- // live counts, by key, the hook transactions of the process from their- // start (Preparing, or a policy denial) to their end (Committed or- // Aborted): any of them may still write a table or compact the stack- // without fsync.- live map[Key]int+ // nLive counts the git processes of this process that write the+ // repository's references without fsync and have not exited: any of them+ // may still write a table or compact the stack.nLive intlastWrite time.Time+ // settleAfter keeps the marker of a repository recovered dirty until then.+ settleAfter time.Time+ // untrusted is the first failed fsync of the repository. The kernel reports+ // a failed writeback once, so a later fsync can succeed although writes+ // were lost: the repository never becomes clean again, only moved aside.+ untrusted error// synced holds the tables made durable during the current dirty period that// the stack still lists. Tables are immutable, so one fsync each is enough.synced map[string]struct{}@@ -248,11 +269,12 @@ func NewDurability(locator storage.Locator, idle time.Duration, opts ...Durabilifunc newDurability(locator storage.Locator, idle time.Duration) *Durability {return &Durability{- locator: locator,- idle: idle,- sync: (*os.File).Sync,- crashAt: os.Getenv(CrashEnv),- repos: map[string]*refState{},+ locator: locator,+ idle: idle,+ sync: (*os.File).Sync,+ crashAt: os.Getenv(CrashEnv),+ recoveryGrace: RecoveryGrace,+ repos: map[string]*refState{},marked: prometheus.NewCounter(prometheus.CounterOpts{Name: "graft_storage_ref_dirty_marked_total",Help: "Number of dirty markers written: a repository got local reference writes that are not durable yet.",@@ -293,19 +315,22 @@ func (d *Durability) Enabled() bool { return d != nil }// once.func (d *Durability) Deferring() bool { return d != nil && d.owner.Load() }-// TryOwn takes the ownership of the deferred durability of the storages-// without waiting. It returns false when another process holds it for one of-// them; the locks taken are kept, and a later call takes the rest.+// TryOwn takes the owner locks of the storages without waiting. It returns+// false when another process holds one of them; the locks taken are kept, and+// a later call takes the rest. Owning the locks lets the process recover the+// storages (Recover); its writes are deferred only after Activate, so that no+// writer of this process writes a marker that the recovery could take for one+// left behind.func (d *Durability) TryOwn(ctx context.Context, storages []string) (bool, error) {if d == nil {return false, nil}- if d.owner.Load() {- return true, nil- }d.ownerMu.Lock()defer d.ownerMu.Unlock()+ if d.locked {+ return true, nil+ }for _, storageName := range storages[len(d.ownerFiles):] {dir, err := d.markerDir(ctx, storageName)@@ -329,9 +354,31 @@ func (d *Durability) TryOwn(ctx context.Context, storages []string) (bool, errord.ownerFiles = append(d.ownerFiles, f)}+ d.locked = true+ return true, nil+}++// Activate makes the writes of the process deferred; it holds the owner locks+// (TryOwn) and has recovered the storages.+func (d *Durability) Activate() {+ if d == nil {+ return+ }++ d.ownerMu.Lock()+ defer d.ownerMu.Unlock()+ if !d.locked {+ panic("localrefs: Activate without the owner locks")+ }d.owner.Store(true)d.owning.Set(1)- return true, nil+}++// ownsLocks reports whether the process holds the owner locks.+func (d *Durability) ownsLocks() bool {+ d.ownerMu.Lock()+ defer d.ownerMu.Unlock()+ return d.locked}// Close releases the ownership. The process must not write references@@ -348,6 +395,7 @@ func (d *Durability) Close() error {errs = append(errs, f.Close())}d.ownerFiles = nil+ d.locked = falsereturn errors.Join(errs...)}@@ -374,7 +422,7 @@ func (d *Durability) markerDir(ctx context.Context, storageName string) (string,func (d *Durability) stateLocked(path string) *refState {st := d.repos[path]if st == nil {- st = &refState{live: map[Key]int{}, synced: map[string]struct{}{}}+ st = &refState{synced: map[string]struct{}{}}d.repos[path] = st}return st@@ -422,53 +470,45 @@ func (d *Durability) MarkDirty(ctx context.Context, repo RepoRef, path string) ed.mu.Lock()defer d.mu.Unlock()- st.dirty = truest.repo = repost.marker = markerPathd.marked.Inc()- d.dirty.Inc()+ if !st.dirty {+ st.dirty = true+ d.dirty.Inc()+ }return nil}-// Begin records that the hook transaction key started on the repository at-// path: from its Preparing phase (or its policy denial, which aborts it), before-// git takes the reference lock, to its end (End), it may write a table and-// compact the stack. Every transaction begins exactly once and ends exactly-// once (Committed or Aborted), so the count of a key is the number of its live-// transactions, whichever of two transactions with the same key ends first.-func (d *Durability) Begin(path string, key Key) {+// Begin records that a git process that writes the references of the+// repository at path without fsync is about to start; the caller calls end+// once the process has exited. Until then the process may write a table or+// compact the stack, whatever hooks git runs or skips (a failed prepare, a+// failed commit, a kill), so the durable step keeps the marker. end may be+// called more than once.+func (d *Durability) Begin(path string) (end func()) {if d == nil {- return+ return func() {}}d.mu.Lock()- defer d.mu.Unlock()st := d.stateLocked(path)- st.live[key]++st.nLive++d.inflight.Inc()-}--// End records that a hook transaction with key ended (Committed or Aborted).-// An end without a begin (the process started after the transaction began)-// changes nothing.-func (d *Durability) End(path string, key Key) {- if d == nil {- return- }+ d.mu.Unlock()- d.mu.Lock()- defer d.mu.Unlock()- st := d.repos[path]- if st == nil || st.live[key] == 0 {- return- }- st.live[key]--- if st.live[key] == 0 {- delete(st.live, key)+ var once sync.Once+ return func() {+ once.Do(func() {+ d.mu.Lock()+ defer d.mu.Unlock()+ // Forget may have dropped the state; the count belongs to it.+ if d.repos[path] == st {+ st.nLive--+ d.inflight.Dec()+ }+ })}- st.nLive--- d.inflight.Dec()}// Dirty returns the paths of the dirty repositories.@@ -535,7 +575,14 @@ func (d *Durability) Settle(path string, final bool) error {return nil}repo := st.repo- idle := st.nLive == 0 && (final || time.Since(st.lastWrite) >= d.idle)+ if st.untrusted != nil {+ // Sticky: the move aside is retried, the marker never cleared.+ untrusted := st.untrusted+ d.mu.Unlock()+ return &UntrustedError{Repo: repo, Path: path, Err: untrusted}+ }+ now := time.Now()+ idle := st.nLive == 0 && now.After(st.settleAfter) && (final || now.Sub(st.lastWrite) >= d.idle)d.mu.Unlock()if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {@@ -546,7 +593,7 @@ func (d *Durability) Settle(path string, final bool) error {}if err := d.syncStack(path, st); err != nil {- return d.stepError(repo, path, err)+ return d.stepError(st, repo, path, err)}if !idle {return nil@@ -563,7 +610,7 @@ func (d *Durability) Settle(path string, final bool) error {cleared, err := d.settleLocked(path, st)if err != nil {_ = lock.release()- return d.stepError(repo, path, err)+ return d.stepError(st, repo, path, err)}if !cleared {return lock.release()@@ -580,13 +627,20 @@ func (d *Durability) Settle(path string, final bool) error {}// stepError is the error of a durable step: an *UntrustedError for a failed-// fsync, the error itself otherwise.-func (d *Durability) stepError(repo RepoRef, path string, err error) error {+// fsync, which makes the repository untrusted for good, the error itself+// otherwise.+func (d *Durability) stepError(st *refState, repo RepoRef, path string, err error) error {var writeback writebackError- if errors.As(err, &writeback) {- return &UntrustedError{Repo: repo, Path: path, Err: err}+ if !errors.As(err, &writeback) {+ return err}- return err++ d.mu.Lock()+ if st.untrusted == nil {+ st.untrusted = err+ }+ d.mu.Unlock()+ return &UntrustedError{Repo: repo, Path: path, Err: err}}// settleLocked makes the tables written since the first pass durable and, when@@ -596,7 +650,7 @@ func (d *Durability) stepError(repo RepoRef, path string, err error) error {// before they take it until their end.func (d *Durability) settleLocked(path string, st *refState) (bool, error) {d.mu.Lock()- busy := st.nLive > 0+ busy := st.nLive > 0 || st.untrusted != nild.mu.Unlock()if busy {return false, nil@@ -856,7 +910,7 @@ type Recovered struct {// boot before serving. A marker that cannot be read is an error: its// repository is unknown, and serving could read a torn stack.func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovered, error) {- if !d.Deferring() {+ if d == nil || !d.ownsLocks() {return nil, nil}@@ -917,10 +971,11 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recoverest := d.stateLocked(path)if !st.dirty {st.dirty = true- st.repo = m.RepoRef- st.marker = markerPathd.dirty.Inc()}+ st.repo = m.RepoRef+ st.marker = markerPath+ st.settleAfter = time.Now().Add(d.recoveryGrace)d.mu.Unlock()}found = append(found, Recovered{Repo: m.RepoRef, Path: path, Reboot: reboot})diff --git a/internal/graft/corestore/localrefs/durability_test.go b/internal/graft/corestore/localrefs/durability_test.goindex 74434c1..33c35b1 100644--- a/internal/graft/corestore/localrefs/durability_test.go+++ b/internal/graft/corestore/localrefs/durability_test.go@@ -87,25 +87,39 @@ func setupDurability(t *testing.T) durabilityFixture {}// unowned returns a durability of the fixture's storage that does not own it.-func (f durabilityFixture) unowned(t *testing.T, bootID string, idle time.Duration) *Durability {+// The recovery grace is 0 unless opts set it.+func (f durabilityFixture) unowned(t *testing.T, bootID string, idle time.Duration, opts ...DurabilityOption) *Durability {t.Helper()- d, err := NewDurability(f.locator, idle, WithBootID(bootID), WithSync(f.synced.sync))+ d, err := NewDurability(f.locator, idle, append([]DurabilityOption{WithBootID(bootID), WithSync(f.synced.sync), WithRecoveryGrace(0)}, opts...)...)require.NoError(t, err)t.Cleanup(func() { require.NoError(t, d.Close()) })return d}-// durability returns a durability that owns the fixture's storage, as the-// process that runs the durable steps. Close it before another one owns.-func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration) *Durability {+// durability returns a durability that owns the fixture's storage and defers,+// as the process that runs the durable steps (the recovery, when a test runs+// it, comes after: no writer of these tests runs before it). Close it before+// another one owns.+func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration, opts ...DurabilityOption) *Durability {t.Helper()- d := f.unowned(t, bootID, idle)- owned, err := d.TryOwn(f.ctx, []string{f.repo.Storage})- require.NoError(t, err)- require.True(t, owned)+ d := f.unowned(t, bootID, idle, opts...)+ f.own(t, d)+ d.Activate()return d}+// own takes the owner locks for d. A git process forked by another test at the+// moment the previous owner closed its lock holds a copy of the lock until it+// execs (close-on-exec), so the lock can be busy for that short while.+func (f durabilityFixture) own(t *testing.T, d *Durability) {+ t.Helper()+ require.Eventually(t, func() bool {+ owned, err := d.TryOwn(f.ctx, []string{f.repo.Storage})+ require.NoError(t, err)+ return owned+ }, 10*time.Second, time.Millisecond)+}+func (f durabilityFixture) markerPath(t *testing.T, d *Durability) string {t.Helper()p, err := d.markerPath(f.ctx, f.repo, f.path)@@ -165,7 +179,11 @@ func TestOneOwnerPerStorage(t *testing.T) {old := f.durability(t, "boot-a", 0)require.True(t, old.Deferring())require.NoError(t, old.MarkDirty(f.ctx, f.repo, f.path))- old.Begin(f.path, key(1))+ _ = old.Begin(f.path)+ // The old process's durable step holds git's lock right now.+ liveLock, ok, err := lockStack(f.path)+ require.NoError(t, err)+ require.True(t, ok)next := f.unowned(t, "boot-a", 0)owned, err := next.TryOwn(f.ctx, storages)@@ -174,27 +192,49 @@ func TestOneOwnerPerStorage(t *testing.T) {require.False(t, next.Deferring())require.Equal(t, 0.0, testutil.ToFloat64(next.owning))- // The next process neither recovers nor settles: the old process's- // transaction is live, and only the old process counts it.+ // The next process neither recovers, nor removes the old process's lock,+ // nor settles: the old process's writer is live, and only the old process+ // counts it.found, err := next.Recover(f.ctx, storages)require.NoError(t, err)require.Empty(t, found)+ require.FileExists(t, filepath.Join(f.path, "reftable", "tables.list.lock"), "the live owner's lock is kept")require.NoError(t, next.Settle(f.path, true))require.FileExists(t, f.markerPath(t, old))+ require.NoError(t, liveLock.release())- // The old process exits; the next one owns and recovers the marker.+ // The old process exits; the next one owns, recovers the marker before it+ // defers, and settles it.require.NoError(t, old.Close())- owned, err = next.TryOwn(f.ctx, storages)- require.NoError(t, err)- require.True(t, owned)- require.True(t, next.Deferring())+ f.own(t, next)+ require.False(t, next.Deferring(), "no write is deferred before the recovery")found, err = next.Recover(f.ctx, storages)require.NoError(t, err)require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path}}, found)+ next.Activate()+ require.True(t, next.Deferring())require.NoError(t, next.Settle(f.path, false))require.NoFileExists(t, f.markerPath(t, next))}+// TestRecoveredMarkerWaitsForTheGrace verifies that a marker found by the+// recovery stays for the recovery grace: a git process of the previous owner+// that is not reaped yet may still be committing.+func TestRecoveredMarkerWaitsForTheGrace(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)++ old := f.durability(t, "boot-a", 0)+ require.NoError(t, old.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, old.Close())++ next := f.durability(t, "boot-a", 0, WithRecoveryGrace(time.Hour))+ _, err := next.Recover(f.ctx, []string{f.repo.Storage})+ require.NoError(t, err)+ require.NoError(t, next.Settle(f.path, true))+ require.FileExists(t, f.markerPath(t, next), "not even the last step removes it within the grace")+}+func TestSettleMakesTheStackDurableAndRemovesTheMarker(t *testing.T) {t.Parallel()f := setupDurability(t)@@ -240,43 +280,40 @@ func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {require.NoFileExists(t, f.markerPath(t, d))})- t.Run("a live transaction", func(t *testing.T) {+ t.Run("a live git process", func(t *testing.T) {t.Parallel()f := setupDurability(t)d := f.durability(t, "boot-a", 0)require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))- d.Begin(f.path, key(1))+ end := d.Begin(f.path)require.NoError(t, d.Settle(f.path, true))require.FileExists(t, f.markerPath(t, d), "not even the last step removes it")- // An end of another key does not count; the end of the transaction does.- d.End(f.path, key(2))- require.NoError(t, d.Settle(f.path, false))- require.FileExists(t, f.markerPath(t, d))-- d.End(f.path, key(1))+ end()require.NoError(t, d.Settle(f.path, false))require.NoFileExists(t, f.markerPath(t, d))require.Equal(t, 0.0, testutil.ToFloat64(d.inflight))})- t.Run("two live transactions with the same key", func(t *testing.T) {+ t.Run("two live git processes", func(t *testing.T) {t.Parallel()f := setupDurability(t)d := f.durability(t, "boot-a", 0)- // Whichever of the two ends first (an abort before Prepared, say), the- // other one is still counted.+ // Whatever hooks they ran or skipped, each counts until its own end, and+ // an end called twice counts once.require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))- d.Begin(f.path, key(1))- d.Begin(f.path, key(1))- d.End(f.path, key(1))+ first := d.Begin(f.path)+ second := d.Begin(f.path)+ first()+ first()require.NoError(t, d.Settle(f.path, false))require.FileExists(t, f.markerPath(t, d))- d.End(f.path, key(1))+ second()require.NoError(t, d.Settle(f.path, false))require.NoFileExists(t, f.markerPath(t, d))+ require.Equal(t, 0.0, testutil.ToFloat64(d.inflight))})t.Run("git holds the reference lock", func(t *testing.T) {@@ -309,14 +346,20 @@ func TestStackLockReleaseRemovesOnlyItsOwnLock(t *testing.T) {require.NoError(t, err)require.False(t, ok, "the lock is exclusive")- // Housekeeping removed the lock as stale, and another durable step took it- // (the freed inode may well be reused for it).- require.NoError(t, os.Remove(lockPath))+ // The same inode with another content: what a reused inode looks like to+ // the file check. The nonce tells it is not the step's lock.require.NoError(t, os.WriteFile(lockPath, []byte(stackLockTag+"0123456789abcdef\n"), 0o666))+ current, err := os.Lstat(lockPath)+ require.NoError(t, err)+ require.True(t, os.SameFile(current, lock.info), "the file check alone would take it for the step's lock")require.NoError(t, lock.release())- require.FileExists(t, lockPath, "a lock of another holder, even with the same tag, is not removed")+ require.FileExists(t, lockPath, "a lock with another nonce is not removed, even in the same inode")+ // Housekeeping removed the lock as stale: the release has nothing to do.require.NoError(t, os.Remove(lockPath))+ require.NoError(t, lock.release())+ require.NoFileExists(t, lockPath)+lock, ok, err = lockStack(f.path)require.NoError(t, err)require.True(t, ok)@@ -345,7 +388,15 @@ func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {require.FileExists(t, f.markerPath(t, d))require.Equal(t, []string{f.path}, d.Dirty())+ // The kernel reports a failed writeback once: the next fsync succeeds, and+ // the repository stays untrusted all the same.f.synced.fail(nil)+ err = d.Settle(f.path, true)+ require.ErrorIs(t, err, ErrUntrusted, "a later successful fsync does not make the repository clean")+ require.ErrorIs(t, err, writeback)+ require.FileExists(t, f.markerPath(t, d))++ // Only moving it aside (Forget) ends it.require.NoError(t, d.Forget(f.ctx, f.repo, f.path))require.NoFileExists(t, f.markerPath(t, d))require.Empty(t, d.Dirty())@@ -503,8 +554,8 @@ func TestNilDurabilityIsOff(t *testing.T) {require.False(t, owned)require.NoError(t, d.MarkDirty(ctx, RepoRef{}, "path"))require.NoError(t, d.Settle("path", true))- d.Begin("path", key(1))- d.End("path", key(1))+ d.Begin("path")()+ d.Activate()found, err := d.Recover(ctx, []string{"default"})require.NoError(t, err)require.Empty(t, found)diff --git a/internal/graft/corestore/syncer/durability.go b/internal/graft/corestore/syncer/durability.goindex 4b468c6..f5ab734 100644--- a/internal/graft/corestore/syncer/durability.go+++ b/internal/graft/corestore/syncer/durability.go@@ -10,7 +10,6 @@ import ("time""github.com/feder-cr/git-ai/internal/graft/corestore"- "github.com/feder-cr/git-ai/internal/graft/corestore/gitops""github.com/feder-cr/git-ai/internal/graft/corestore/localrefs""github.com/feder-cr/git-ai/internal/structerr")@@ -49,6 +48,7 @@ func (s *Syncer) StartLocalReferences(ctx context.Context, storages []string) (sif err := s.RecoverLocalReferences(ctx, storages); err != nil {return nil, err}+ d.Activate()recovered = true} else {s.logger.Warn("storage core: another gitaly process owns the deferred durability of the storages; references are written with fsync until it exits")@@ -103,6 +103,7 @@ func (s *Syncer) runDurableSteps(ctx context.Context, storages []string, recovers.logger.WithError(err).Error("storage core: recover the local references after taking the deferred durability; retried")continue}+ d.Activate()recovered = trues.logger.Info("storage core: this process owns the deferred durability of the storages now")}@@ -159,7 +160,10 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)// storageCoreCache reports whether the local repository at path is a cache of// the storage core that a hydration can rebuild: the database holds it with-// changesets, and it has a watermark.+// changesets (an import that is not complete has none, and its local copy may+// be the only one). A sync replaces such a local repository too when it is+// stale, whatever its watermark. An error (the database, or the repository's+// directory, cannot be read) is returned: the caller tries again.func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, path string) (bool, error) {r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: repo.Storage, RelativePath: repo.RelativePath})if errors.Is(err, corestore.ErrRepositoryNotFound) {@@ -177,13 +181,7 @@ func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, p} else if err != nil {return false, err}- _, ok, err := gitops.ReadWatermark(path)- if err != nil {- // A watermark that cannot be read is no proof that the storage core wrote- // the repository: it is kept, as untracked() keeps it.- return false, nil- }- return ok, nil+ return true, nil}// durableStep makes the references of the repository at path durable under the@@ -212,6 +210,15 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {return}+ // A repository that is gone (moved aside as a stale cache meanwhile) has+ // nothing left to serve: its marker is dropped.+ if _, statErr := os.Lstat(path); errors.Is(statErr, os.ErrNotExist) {+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {+ logger.WithField("forget_error", err.Error()).Error("storage core: could not drop the marker of a repository that is gone; retried at the next step")+ }+ return+ }+cache, cacheErr := s.storageCoreCache(ctx, untrusted.Repo, path)if cacheErr != nil {d.CountStepError()@@ -220,12 +227,16 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {return}if !cache {+ // Its marker stays (the untrusted state is sticky): after a reboot the+ // recovery decides again.logger.Error("storage core: the references of a repository that is not a cache of the storage core may have lost writes; it is kept in place")return}if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {+ // The untrusted state is sticky: the next step moves it aside again, and+ // the marker is never cleared meanwhile.logger.WithField("move_error", err.Error()).- Error("storage core: could not move aside a repository whose references may have lost writes")+ Error("storage core: could not move aside a repository whose references may have lost writes; retried at the next step")}}diff --git a/internal/graft/corestore/syncer/syncer_localrefs_test.go b/internal/graft/corestore/syncer/syncer_localrefs_test.goindex 38d33f5..1dd81d5 100644--- a/internal/graft/corestore/syncer/syncer_localrefs_test.go+++ b/internal/graft/corestore/syncer/syncer_localrefs_test.go@@ -211,6 +211,96 @@ func TestSyncerDurableStepMovesAsideOnAFailedWrite(t *testing.T) {e.verifySynced(t, repo)}+// TestSyncerUntrustedRepositoryIsNeverSettled is a failed fsync whose move+// aside fails too (the temporary directory cannot be written for a moment).+// The kernel reports the failed writeback once, so the next step's fsyncs+// succeed: the repository must still be moved aside, never settled.+func TestSyncerUntrustedRepositoryIsNeverSettled(t *testing.T) {+ t.Parallel()++ var failing atomic.Bool+ e, repo, _ := caughtUpDirty(t, localrefs.WithSync(func(f *os.File) error {+ if failing.Load() {+ return errors.New("input/output error")+ }+ return f.Sync()+ }))+ tmpDir, err := e.locator.TempDir(e.storageName)+ require.NoError(t, err)+ require.NoError(t, os.MkdirAll(tmpDir, 0o700))++ failing.Store(true)+ require.NoError(t, os.Chmod(tmpDir, 0o500))+ e.sy.DurableStep(e.ctx, e.localPath(), false)+ failing.Store(false)+ require.NoError(t, os.Chmod(tmpDir, 0o700))++ require.DirExists(t, e.localPath(), "the move aside failed")+ require.Len(t, e.markers(t), 1, "the marker stays")++ e.sy.DurableStep(e.ctx, e.localPath(), true)+ require.NoDirExists(t, e.localPath(), "moved aside, although its fsyncs succeed now")+ require.Empty(t, e.markers(t))+ require.Equal(t, float64(1), e.untrusted(t, localrefs.ReasonSyncFailed))++ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ e.verifySynced(t, repo)+}++// TestSyncerUntrustedRepositoryThatIsGoneLosesItsMarker verifies that an+// untrusted repository that was moved away meanwhile (a stale cache) does not+// keep its marker forever.+func TestSyncerUntrustedRepositoryThatIsGoneLosesItsMarker(t *testing.T) {+ t.Parallel()++ var failing atomic.Bool+ e, _, _ := caughtUpDirty(t, localrefs.WithSync(func(f *os.File) error {+ if failing.Load() {+ return errors.New("input/output error")+ }+ return f.Sync()+ }))++ tmpDir, err := e.locator.TempDir(e.storageName)+ require.NoError(t, err)+ require.NoError(t, os.MkdirAll(tmpDir, 0o700))+ failing.Store(true)+ require.NoError(t, os.Chmod(tmpDir, 0o500))+ e.sy.DurableStep(e.ctx, e.localPath(), false)+ require.NoError(t, os.Chmod(tmpDir, 0o700))+ failing.Store(false)+ require.Len(t, e.markers(t), 1)++ require.NoError(t, os.Rename(e.localPath(), filepath.Join(tmpDir, "moved.git")))+ e.sy.DurableStep(e.ctx, e.localPath(), false)+ require.Empty(t, e.markers(t))+ require.Empty(t, e.core.RefDurability.Dirty())+}++// TestSyncerCatchUpOfANonOwnerWritesWithFsync verifies that a process that+// does not own the deferred durability catches up with fsync: it writes no+// marker.+func TestSyncerCatchUpOfANonOwnerWritesWithFsync(t *testing.T) {+ t.Parallel()++ e := newEnv(t, envOptions{})+ durability, err := localrefs.NewDurability(e.locator, 0, localrefs.WithBootID(coretest.TestBootID))+ require.NoError(t, err)+ e.core.RefDurability = durability+ require.False(t, durability.Deferring())++ src := e.newSource(t)+ c1 := e.writeCommit(t, src)+ repo := e.register(t, e.key(localRelPath), src, c1)+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ e.advanceMain(t, repo.ID, src, c1)+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ require.Equal(t, float64(1), syncCount(t, e.sy, "caught_up"))++ require.Empty(t, e.markers(t))+ require.Empty(t, durability.Dirty())+}+// TestSyncerDurableStepRetriesATransientError verifies that a durable step that// fails without a failed fsync (the stack cannot be read for a moment) keeps// the repository and its marker and is retried.@@ -243,7 +333,7 @@ func TestSyncerSecondProcessWaitsForTheOwner(t *testing.T) {storages := []string{e.storageName}next := *e.core- durability, err := localrefs.NewDurability(e.locator, 0, localrefs.WithBootID(coretest.TestBootID))+ durability, err := localrefs.NewDurability(e.locator, 0, localrefs.WithBootID(coretest.TestBootID), localrefs.WithRecoveryGrace(0))require.NoError(t, err)next.RefDurability = durabilityt.Cleanup(func() { _ = durability.Close() })@@ -270,7 +360,7 @@ func TestSyncerStopSettlesEveryRepository(t *testing.T) {e := newEnv(t, envOptions{})e.core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)- durability, err := localrefs.NewDurability(e.locator, time.Hour, localrefs.WithBootID(coretest.TestBootID))+ durability, err := localrefs.NewDurability(e.locator, time.Hour, localrefs.WithBootID(coretest.TestBootID), localrefs.WithRecoveryGrace(0))require.NoError(t, err)t.Cleanup(func() { _ = durability.Close() })e.core.RefDurability = durabilitydiff --git a/internal/graft/corestore/voter/voter.go b/internal/graft/corestore/voter/voter.goindex e3d67bd..6d3853c 100644--- a/internal/graft/corestore/voter/voter.go+++ b/internal/graft/corestore/voter/voter.go@@ -176,8 +176,6 @@ func (v *Voter) Preparing(ctx context.Context, payload gitcmd.HooksPayload, chanreturn}- // The transaction is live from now on (it ends in Committed or Aborted).- v.begin(ctx, payload, changes)// Deferred first, so that it runs last: the admission is waited for after// the pack work, when git is about to take the reference lock.defer v.admit(ctx, payload, changes)@@ -774,7 +772,6 @@ func (v *Voter) Committed(ctx context.Context, payload gitcmd.HooksPayload, chan// git finished the transaction, its auto-compaction included, and released// the reference lock.v.core.RefAdmission.Release(repoPath, localrefs.Key(key))- v.core.RefDurability.End(repoPath, localrefs.Key(key))if !ok || v.applier == nil {return@@ -801,36 +798,31 @@ func (v *Voter) localRefs() bool {// DeferredReferenceDurability reports whether the reference writes of the// transactions this voter commits may run without fsync: authoritative mode,// and this process owns the deferred durability of the node (doc/graft/PLAN7.md,-// LOCAL REFERENCE WRITES RULE). The hook manager asks it for UpdateWithHooks.-// It goes from false to true at most once in a process.+// LOCAL REFERENCE WRITES RULE). It goes from false to true at most once in a+// process.func (v *Voter) DeferredReferenceDurability() bool {return v != nil && v.core.Authoritative() && v.core.RefDurability.Deferring()}-// Denied records the start of a transaction that the standalone policy denies:-// it is live until its Aborted phase, like any other (see begin).-func (v *Voter) Denied(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {- if v == nil || !v.core.Enabled() || payload.Repo == nil {- return+// DeferReferenceWrites decides whether the git process that the caller is about+// to start for repo (UpdateWithHooks' update-ref) writes its references without+// fsync. When ok, the process counts as live for the repository's deferred+// durability, and the caller calls end once it has exited: until then it may+// write a table or compact the stack, whichever hooks git runs or skips.+func (v *Voter) DeferReferenceWrites(ctx context.Context, repo *gitalypb.Repository) (end func(), ok bool) {+ if !v.DeferredReferenceDurability() {+ return nil, false}- v.begin(ctx, payload, changes)-}-// begin counts the transaction as live for the deferred durability: from-// before git takes the reference lock until Committed or Aborted, it may write-// a table or compact the stack, so the durable step keeps the repository's-// marker meanwhile.-func (v *Voter) begin(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {- if !v.core.Authoritative() || !v.core.RefDurability.Enabled() {- return- }-- repoPath, err := v.repoPath(ctx, payload)+ repoPath, err := v.locator.GetRepoPath(ctx, &gitalypb.Repository{+ StorageName: repo.GetStorageName(),+ RelativePath: repo.GetRelativePath(),+ }, storage.WithRepositoryVerificationSkipped())if err != nil {- v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path of a starting transaction")- return+ v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path of a reference write; it is made durable by git")+ return nil, false}- v.core.RefDurability.Begin(repoPath, localrefs.Key(pendingKey(payload.Repo, changes)))+ return v.core.RefDurability.Begin(repoPath), true}// admit waits for the admission of the transaction to the reference lock of@@ -853,9 +845,11 @@ func (v *Voter) admit(ctx context.Context, payload gitcmd.HooksPayload, changes// deferDurability lets git write the references of a transaction that the// database committed without fsync (authoritative mode, and this process owns// the deferred durability): the repository gets its durable dirty marker-// first. The transaction is live since its start (begin), so the durable step-// keeps the marker until it ends. It runs in the Prepared phase, under git's-// reference lock and before git writes anything.+// first. A git process that writes without fsync counts as live from before it+// started until it exited (DeferReferenceWrites), so the durable step keeps the+// marker meanwhile. It runs in the Prepared phase, under git's reference lock+// and before git writes anything. A transaction whose git process writes with+// fsync (a push) gets a marker too, which only delays its removal.func (v *Voter) deferDurability(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, shadow bool) error {if shadow || !v.DeferredReferenceDurability() {return nil@@ -888,13 +882,11 @@ func (v *Voter) Aborted(ctx context.Context, payload gitcmd.HooksPayload, changekey := pendingKey(payload.Repo, changes)- // git wrote nothing more of the transaction once it reports the abort.if v.localRefs() {if repoPath, err := v.repoPath(ctx, payload); err != nil {v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path after an abort, the admission ends at its bound")} else {v.core.RefAdmission.Release(repoPath, localrefs.Key(key))- v.core.RefDurability.End(repoPath, localrefs.Key(key))}}diff --git a/internal/graft/corestore/voter/voter_localrefs_test.go b/internal/graft/corestore/voter/voter_localrefs_test.goindex 35a9109..9bffd15 100644--- a/internal/graft/corestore/voter/voter_localrefs_test.go+++ b/internal/graft/corestore/voter/voter_localrefs_test.go@@ -14,6 +14,7 @@ import ("github.com/feder-cr/git-ai/internal/git""github.com/feder-cr/git-ai/internal/git/gittest""github.com/feder-cr/git-ai/internal/git/localrepo"+ "github.com/feder-cr/git-ai/internal/gitaly/config""github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/localrefs""github.com/feder-cr/git-ai/internal/graft/corestore/voter"@@ -54,10 +55,20 @@ func (f *fixture) markers(t *testing.T) []string {return names}+// deferred starts a deferred git process for the fixture's repository, as+// UpdateWithHooks does before its update-ref.+func (f *fixture) deferred(t *testing.T) (end func()) {+ t.Helper()+ end, ok := f.voter.DeferReferenceWrites(f.ctx, f.repo)+ require.True(t, ok)+ return end+}+// TestLocalReferenceWritesOfACommittedTransaction follows one authoritative// transaction through the LOCAL REFERENCE WRITES RULE (doc/graft/PLAN7.md):-// admitted at the end of Preparing, marked dirty and in flight after the-// database commit in Prepared, released and ended in Committed.+// its git process is live from before it starts until it exits, it is+// admitted at the end of Preparing and released in Committed, and its+// repository is marked dirty after the database commit in Prepared.func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {t.Parallel()@@ -68,6 +79,7 @@ func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {require.True(t, f.voter.DeferredReferenceDurability())require.False(t, f.admissionHeld(t))+ end := f.deferred(t)f.voter.Preparing(f.ctx, f.payload, input)require.True(t, f.admissionHeld(t), "the transaction is admitted at the end of Preparing")require.Empty(t, f.markers(t), "nothing is dirty before the database commit")@@ -75,58 +87,72 @@ func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {require.NoError(t, f.prepared(input))require.Len(t, f.markers(t), 1)- // git may still write and compact until Committed: the durable step keeps- // the marker.- require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))- require.Len(t, f.markers(t), 1)-gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName(ref), f.a)f.voter.Committed(f.ctx, f.payload, input)require.False(t, f.admissionHeld(t), "Committed releases the admission")require.Len(t, f.applier.calls, 1, "the local commit is still noted")+ // The git process may still compact until it exits.+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, true))+ require.Len(t, f.markers(t), 1)++ end()require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))require.Empty(t, f.markers(t))}// TestLocalReferenceWritesOfAnAbortedTransaction verifies that an abort after-// Prepared releases the admission and ends the transaction: the repository is-// settled once nothing else is live.+// Prepared releases the admission, and that the repository is settled once its+// git process exited.func TestLocalReferenceWritesOfAnAbortedTransaction(t *testing.T) {t.Parallel()f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})input := line(f.zero, f.a.String(), "refs/heads/main")+ end := f.deferred(t)f.voter.Preparing(f.ctx, f.payload, input)require.NoError(t, f.prepared(input))f.voter.Aborted(f.ctx, f.payload, input)require.False(t, f.admissionHeld(t))+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))+ require.Len(t, f.markers(t), 1, "the git process has not exited yet")+ end()require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))require.Empty(t, f.markers(t))}-// TestLocalReferenceWritesOfTransactionsWithTheSameInput verifies that two-// live transactions with the same input are both counted: the one denied by-// the policy and aborted does not end the other one, which may still write.-func TestLocalReferenceWritesOfTransactionsWithTheSameInput(t *testing.T) {+// TestLocalReferenceWritesWhenGitSkipsItsHooks covers the transactions whose+// later phases git never reports: a prepare that fails (a stale old value of a+// writer that lost a race, a lock that stayed busy, a name conflict), a commit+// whose finish fails, and the Committed phase of a symbolic reference that+// names its target instead of the reference of Preparing. The live count+// follows the git processes, so none of them leaks: the repository settles once+// both processes exited.+func TestLocalReferenceWritesWhenGitSkipsItsHooks(t *testing.T) {t.Parallel()f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})- input := line(f.zero, f.a.String(), "refs/heads/main")-- f.voter.Preparing(f.ctx, f.payload, input)- require.NoError(t, f.prepared(input))-- // The same input from an agent the policy denies.- f.voter.Denied(f.ctx, f.payload, input)- f.voter.Aborted(f.ctx, f.payload, input)- require.NoError(t, f.core.RefDurability.Settle(f.repoPath, true))- require.Len(t, f.markers(t), 1, "the first transaction may still write")-- gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName("refs/heads/main"), f.a)- f.voter.Committed(f.ctx, f.payload, input)+ winner := line(f.zero, f.a.String(), "refs/heads/main")+ loser := line(f.zero, f.b.String(), "refs/heads/main")++ endWinner := f.deferred(t)+ endLoser := f.deferred(t)+ f.voter.Preparing(f.ctx, f.payload, winner)+ f.voter.Preparing(f.ctx, f.payload, loser)++ require.NoError(t, f.prepared(winner))+ gittest.WriteRef(t, f.cfg, f.repoPath, "refs/heads/main", f.a)+ // The Committed phase names another reference than Preparing (a symbolic+ // reference split into its target).+ f.voter.Committed(f.ctx, f.payload, line(f.zero, f.a.String(), "HEAD"))+ endWinner()++ // The loser's prepare failed in git: no Prepared, no Aborted, no Committed.+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))+ require.Len(t, f.markers(t), 1, "the loser's git process is still live")+ endLoser()require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))require.Empty(t, f.markers(t))}@@ -141,9 +167,11 @@ func TestLocalReferenceWritesOfARejectedTransaction(t *testing.T) {f.seed(t, ref, corestore.Target(f.b.String()))input := line(f.a.String(), f.c.String(), ref)+ end := f.deferred(t)f.voter.Preparing(f.ctx, f.payload, input)requireMismatch(t, f.prepared(input), ref, f.a.String(), f.b.String())f.voter.Aborted(f.ctx, f.payload, input)+ end()require.False(t, f.admissionHeld(t))require.Empty(t, f.markers(t))@@ -171,11 +199,13 @@ func TestLocalReferenceWritesWhenTheMarkerCannotBeWritten(t *testing.T) {}// TestDeferredReferenceDurabilityByMode verifies that references are written-// without fsync in authoritative mode with deferred durability only: off and-// shadow modes, and authoritative mode without the boot identity, keep git's-// fsync of references.+// without fsync in authoritative mode by the process that owns the deferred+// durability only: off and shadow modes, authoritative mode without the boot+// identity, and a process that does not own the storage (the new process of a+// graceful upgrade) keep git's fsync of references.func TestDeferredReferenceDurabilityByMode(t *testing.T) {t.Parallel()+ ctx := testhelper.Context(t)var nilVoter *voter.Voterrequire.False(t, nilVoter.DeferredReferenceDurability())@@ -185,12 +215,28 @@ func TestDeferredReferenceDurabilityByMode(t *testing.T) {shadow := newFixture(t, fixtureOptions{mode: corestore.ModeShadow, localRefs: true})require.False(t, shadow.voter.DeferredReferenceDurability())+ _, ok := shadow.voter.DeferReferenceWrites(ctx, shadow.repo)+ require.False(t, ok)authoritative := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative})require.False(t, authoritative.voter.DeferredReferenceDurability())+ _, ok = authoritative.voter.DeferReferenceWrites(ctx, authoritative.repo)+ require.False(t, ok)deferred := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})require.True(t, deferred.voter.DeferredReferenceDurability())++ // Another process owns the storage: this one's durability holds no lock.+ notOwner := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative})+ durability, err := localrefs.NewDurability(config.NewLocator(notOwner.cfg), 0, localrefs.WithBootID("boot"))+ require.NoError(t, err)+ notOwner.core.RefDurability = durability+ require.False(t, notOwner.voter.DeferredReferenceDurability())+ _, ok = notOwner.voter.DeferReferenceWrites(ctx, notOwner.repo)+ require.False(t, ok)+ // Its Prepared phase marks nothing: its writes are made with fsync.+ require.NoError(t, notOwner.prepared(line(notOwner.zero, notOwner.a.String(), "refs/heads/main")))+ require.Empty(t, notOwner.markers(t))}// TestLocalReferenceWritesNotInShadowMode verifies that shadow mode, where gitdiff --git a/internal/graft/e2e/storage_localrefs_test.go b/internal/graft/e2e/storage_localrefs_test.goindex 8b5af52..f682252 100644--- a/internal/graft/e2e/storage_localrefs_test.go+++ b/internal/graft/e2e/storage_localrefs_test.go@@ -5,10 +5,13 @@ import ("encoding/json""errors""fmt"+ "io""math"+ "net/http""os""os/exec""path/filepath"+ "strconv""strings""sync""syscall"@@ -71,6 +74,7 @@ func newRealNode(t *testing.T, configure ...func(*config.GraftStorage)) *realNodc(&cfg.Graft.Storage)}cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)+ cfg.PrometheusListenAddr = freeAddr(t)testcfg.BuildGitalySSH(t, cfg)testcfg.BuildGitalyHooks(t, cfg)@@ -103,6 +107,7 @@ func (n *realNode) sibling(t *testing.T) *realNode {s := *ns.cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)+ s.cfg.PrometheusListenAddr = freeAddr(t)s.clientCfg = s.cfgs.clientCfg.SocketPath = "unix://" + s.cfg.SocketPaths.cfgPath = testcfg.WriteTemporaryGitalyConfigFile(t, s.cfg)@@ -181,6 +186,34 @@ func (n *realNode) log(t *testing.T) string {func (n *realNode) storagePath() string { return n.cfg.Storages[0].Path }+// freeAddr returns a local TCP address that was free a moment ago.+func freeAddr(t *testing.T) string {+ t.Helper()+ l, addr := testhelper.GetLocalhostListener(t)+ require.NoError(t, l.Close())+ return addr+}++// counter returns the value of the counter name of the server's metrics.+func (n *realNode) counter(t *testing.T, name string) float64 {+ t.Helper()++ resp, err := http.Get("http://" + n.cfg.PrometheusListenAddr + "/metrics")+ require.NoError(t, err)+ defer resp.Body.Close()+ body, err := io.ReadAll(resp.Body)+ require.NoError(t, err)+ for _, line := range strings.Split(string(body), "\n") {+ if value, ok := strings.CutPrefix(line, name+" "); ok {+ v, err := strconv.ParseFloat(strings.TrimSpace(value), 64)+ require.NoError(t, err)+ return v+ }+ }+ t.Fatalf("metric %s not found", name)+ return 0+}+// createRepository creates an empty repository through the server and returns// its relative path. In authoritative mode the gate registers it.func (n *realNode) createRepository(t *testing.T) string {@@ -300,7 +333,7 @@ func TestStorageLocalReferencesAfterAProcessKill(t *testing.T) {// The durable step runs every second and removes the marker of an idle// repository.- require.Eventually(t, func() bool { return len(n.markers(t)) == 0 }, 30*time.Second, 50*time.Millisecond)+ require.Eventually(t, func() bool { return len(n.markers(t)) == 0 }, 90*time.Second, 50*time.Millisecond)n.requireServed(t, rel, written)}@@ -408,6 +441,8 @@ func TestStorageLocalReferencesDuringAGracefulUpgrade(t *testing.T) {written[branch] = commit}next.requireServed(t, rel, written)+ require.Equal(t, float64(0), next.counter(t, "graft_storage_ref_durability_owner"))+ require.Equal(t, float64(0), next.counter(t, "graft_storage_ref_dirty_marked_total"), "the new process writes with fsync")// The old process owns the marker and settles it once the repository is// idle; the new process's writes were made with fsync and need none.@@ -421,7 +456,8 @@ func TestStorageLocalReferencesDuringAGracefulUpgrade(t *testing.T) {for branch, commit := range writeBranches(t, next.agent(t, rel), next.ctx, base, "after", 2) {written[branch] = commit}- require.Len(t, next.markers(t), 1, "the new owner writes without fsync")+ require.Equal(t, float64(1), next.counter(t, "graft_storage_ref_durability_owner"))+ require.GreaterOrEqual(t, next.counter(t, "graft_storage_ref_dirty_marked_total"), float64(1), "the new owner writes without fsync")next.requireServed(t, rel, written)}