Skip to content

Git designed for AI agents.

WP76b follow-up 2: live count per git process, sticky untrusted state

Federico Elia18e152ad52a0
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.md
index cb98074..4d59379 100644
--- a/doc/graft/PLAN7.md
+++ b/doc/graft/PLAN7.md
@@ -575,19 +575,21 @@ repository-wide lock.
the marker cannot be written, Prepared fails and git aborts; catch-up
repairs the database's write, like any local failure.
* The syncer writes it under its repository lock.
- * A hook transaction is live from its start to its end, and the durable
- step never clears the marker while any is live.
- * Start: the voter's Preparing, or `Denied` for a transaction the policy
- denies.
- * End: Committed or Aborted.
- * Every transaction starts once and ends once, so the count per input is
- the number of live transactions, whichever of two with the same input
- ends first.
+ * The durable step never clears the marker while a git process of the owner
+ that writes references without fsync is alive.
+ * Only `UpdateWithHooks`' update-ref writes that way. It is counted from
+ just before it starts (`DeferReferenceWrites`, which also decides the
+ deferral) until after it exited (`updater.Close` waits for it).
+ * The syncer's update-ref runs under the syncer lock, which the durable
+ step holds too.
+ * The count follows the process, not the hooks. git skips the Aborted and
+ Committed hooks in several cases: a failed prepare (a stale old value,
+ a busy lock, a name conflict), a failed finish, initial transactions, a
+ kill. Its Committed hook names a symbolic reference's target instead of
+ the reference. Neither can leak a count or end the wrong one.
* Reasons from git's source: the new table is written in
`transaction_finish` after "prepared", and auto-compaction runs inside
`reftable_addition_commit`, before "committed".
- * A transaction whose end is never reported keeps its repository dirty
- (conservative).
* *Durable step* (`Settle`, every second for every dirty repository, under the
syncer's repository lock).
* It fsyncs the tables not yet fsynced, `tables.list` and the directory.
@@ -606,10 +608,16 @@ repository-wide lock.
* A graceful stop runs a last step for every dirty repository after the
servers stopped, so a clean reboot does not rehydrate them.
* *Write failure.* Only a failed fsync means the kernel may have dropped
- writes. The repository is then moved aside at once, under the repository
- lock: the move (both directories and the new entry) is made durable, then
- the marker removed. This happens only when the repository is a cache of the
- storage core, the same check as the recovery. The failure is logged and
+ writes, and the kernel reports a failed writeback once: a later fsync of the
+ same file succeeds.
+ * So the first failed fsync makes the repository *untrusted* for good.
+ * An untrusted repository's marker is never cleared; only the move aside is
+ retried, until it succeeds. Then the hydrated repository starts clean.
+ * The repository is moved aside at once, under the repository lock: the
+ move (both directories and the new entry) is made durable, then the
+ marker removed.
+ * This happens only when the repository is a cache of the storage core:
+ the database holds it with changesets, as for the recovery. The failure is logged and
counted (`sync_failed`), and the next request hydrates the repository. Any
other error (opening a file, reading the list) is transient: counted
(`graft_storage_ref_durable_step_errors_total`) and retried at the next
@@ -625,10 +633,22 @@ repository-wide lock.
moved aside before anything reads it (`reboot`), and the next request
hydrates it.
* The move only happens when the database holds the repository with
- changesets and the local copy has a watermark; any other marker is
- dropped, keeping a repository whose local copy may be the only one.
+ changesets. A sync would replace such a local copy too when it is stale,
+ whatever its watermark. Any other marker is dropped, keeping a repository
+ whose local copy may be the only one (an import that is not complete).
* A marker of the same boot is a process restart: the page cache kept
every write, so the repository stays dirty to be settled.
+ * It is settled no earlier than the recovery grace (twice
+ `reftable.lockTimeout`): a git process of the previous owner that
+ outlived it (killed with it, not reaped yet) may still be committing
+ for that long.
+ * The git children get no parent-death signal: the command package is
+ shared and not in m3's grant. The grace bounds their life instead,
+ since a killed parent closes their stdin and git then aborts.
+ * The ownership is taken in two steps. The locks come first; the recovery
+ runs; only then are the process's writes deferred (`Activate`). So no
+ writer of the new owner writes a marker that the recovery could take for
+ a leftover.
* A lock with the durable step's tag and the step's temporary files, left
by a process that died, are removed.
* A corrupt marker fails the startup: its repository is unknown.
@@ -655,7 +675,7 @@ repository-wide lock.
* PM grant: `updateref.WithDeferredReferenceDurability`; the `gitcmd` constants
`FsyncComponents`, `FsyncComponentsWithoutReferences` and
`ReftableLockTimeoutMilliseconds`; `hook/storage_voter.go`
- (`DeferredReferenceDurability`, `Denied`); the optional interface asserted in
+ (`DeferReferenceWrites`); the optional interface asserted in
`UpdateWithHooks`.
* Follow-up after the independent review of 310facd:
* MAJOR: process-local state with overlapping processes. Fixed by one owner
@@ -668,6 +688,24 @@ repository-wide lock.
* The directory fsyncs (the marker directory's parent; the stale and temporary
directories of a move) are added.
* The step's temporary files are cleaned at recovery.
+* Second review (of 8953e8e):
+ * M1: the live count is tied to the deferred git process itself; it no
+ longer depends on hooks git may skip. `StorageVoter.Denied` is gone.
+ * M2: untrusted is sticky.
+ * The recovery runs before the writes are deferred (`TryOwn`, `Recover`,
+ `Activate`).
+ * The recovery grace for recovered markers.
+ * The dirty gauge counts a repository once.
+ * Tests:
+ * the reused inode is simulated deterministically;
+ * a non-owner neither recovers nor removes the live owner's lock;
+ * the e2e upgrade test reads the owner and marker counters instead of the
+ racing marker file;
+ * a non-owner's write marks nothing, at unit, syncer and e2e level;
+ * git skipping its hooks, with the race, the failed prepare and the
+ symbolic split;
+ * a failed fsync then a successful one; and the same with a failed move
+ aside.
**Tests.**
* `localrefs`:
diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.md
index 1276228..522c7a6 100644
--- a/doc/graft/STORAGE.md
+++ b/doc/graft/STORAGE.md
@@ -463,12 +463,16 @@ or re-import the repository from scratch (see
loss: one hydration per repository that had a write in the second before the
loss, or that was still being written. On a busy node that can be every
active repository. A repository is only moved when the database holds it
- with changesets and the local copy has a watermark; other markers are
- dropped.
+ with changesets; other markers are dropped.
- *A write the kernel reports as failed* (a writeback error, same boot): the
durable step's fsync fails, the repository is moved aside at once and logged,
- and the next request hydrates it. Other errors of the step (a file that
+ and the next request hydrates it. The kernel reports such a failure once, so
+ from then on the repository's marker is never removed: if the move aside
+ fails, it is retried every second. Other errors of the step (a file that
cannot be opened for a moment) are retried every second and counted.
+ - *Process restart*: the markers found are kept for 20 s (twice git's
+ reference lock timeout) before they can be removed, in case a git process of
+ the previous process is still finishing.
- *Clean stop*: after the servers stopped, `gitaly serve` makes every dirty
repository durable and removes its marker, so a clean reboot hydrates
nothing.
@@ -523,7 +527,7 @@ without a single divergence.
| `graft_storage_ref_admission_expired_total` | Turns ended by the 10 s bound instead of their release (a git process that died between its phases). |
| `graft_storage_ref_dirty_repositories` | Repositories with reference writes not yet made durable. |
| `graft_storage_ref_dirty_marked_total`, `graft_storage_ref_dirty_cleared_total` | Dirty markers written and removed by the durable step. |
-| `graft_storage_ref_writes_inflight` | Hook transactions between their start and their end (Committed or Aborted); one whose end is never reported keeps its repository dirty. |
+| `graft_storage_ref_writes_inflight` | git processes of this node that write references without fsync and have not exited; their repositories keep their markers. |
| `graft_storage_ref_durability_owner` | 1 when this process owns the deferred durability of its storages, 0 while another process (the old one of a graceful upgrade) does. |
| `graft_storage_ref_durable_step_errors_total` | Durable steps that failed with a transient error (not a failed fsync); retried every second. |
| `graft_storage_ref_untrusted_total{reason}` | Repositories moved aside to be hydrated again: `reboot` (dirty at a kernel restart), `sync_failed` (the durable step's fsync failed). |
diff --git a/internal/gitaly/hook/storage_voter.go b/internal/gitaly/hook/storage_voter.go
index a0fe77a..4d56528 100644
--- a/internal/gitaly/hook/storage_voter.go
+++ b/internal/gitaly/hook/storage_voter.go
@@ -5,6 +5,7 @@ import (
"github.com/feder-cr/git-ai/internal/git/gitcmd"
"github.com/feder-cr/git-ai/internal/graft/changelog"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
)
// StorageVoter is the write path of the Graft storage core
@@ -15,11 +16,6 @@ type StorageVoter interface {
// its pack) before git takes the reference locks. It never fails the
// transaction: when it did not complete, Prepared does that work itself.
Preparing(ctx context.Context, payload gitcmd.HooksPayload, changes []byte)
- // Denied is called instead of Preparing for a transaction that the
- // standalone access policy denies: it prepares nothing, and the transaction
- // is aborted in its Prepared phase. Every transaction gets exactly one of
- // Preparing and Denied before its Aborted or Committed phase.
- Denied(ctx context.Context, payload gitcmd.HooksPayload, changes []byte)
// Prepared commits the transaction to the storage core while git holds
// the reference locks. An error aborts the transaction.
Prepared(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, meta changelog.Meta) error
@@ -30,19 +26,24 @@ type StorageVoter interface {
// Authoritative reports whether the storage core's database replaces the
// file changelog.
Authoritative() bool
- // DeferredReferenceDurability reports whether reference transactions that
- // the voter commits may write their references without fsync: the database
+ // DeferReferenceWrites decides whether the git process the caller is about
+ // to start for repo writes its references without fsync: the database
// commit is their durable point, and the storage core makes the local
// references durable later (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).
- DeferredReferenceDurability() bool
+ // When ok, the caller calls end once that process has exited.
+ DeferReferenceWrites(ctx context.Context, repo *gitalypb.Repository) (end func(), ok bool)
}
-// DeferredReferenceDurability reports whether the reference transactions that
-// the manager drives itself (UpdateWithHooks) may write their references
-// without fsync. It is false without a storage voter, so only the storage
-// core's authoritative mode ever turns the fsync of references off.
-func (m *GitLabHookManager) DeferredReferenceDurability() bool {
- return m.storageVoter != nil && m.storageVoter.DeferredReferenceDurability()
+// DeferReferenceWrites decides whether the git process that the manager's
+// caller (UpdateWithHooks) is about to start writes its references without
+// fsync; when ok, end is called once that process has exited. It is false
+// without a storage voter, so only the storage core's authoritative mode ever
+// turns the fsync of references off.
+func (m *GitLabHookManager) DeferReferenceWrites(ctx context.Context, repo *gitalypb.Repository) (end func(), ok bool) {
+ if m.storageVoter == nil {
+ return nil, false
+ }
+ return m.storageVoter.DeferReferenceWrites(ctx, repo)
}
// WithStorageVoter makes the manager commit every reference transaction to
@@ -70,7 +71,6 @@ func (m *GitLabHookManager) storagePreparing(ctx context.Context, payload gitcmd
}
if message, err := m.graftPolicyDecision(env, changes); err != nil || message != "" {
- m.storageVoter.Denied(ctx, payload, changes)
return
}
diff --git a/internal/gitaly/hook/storage_voter_test.go b/internal/gitaly/hook/storage_voter_test.go
index abb3694..f125613 100644
--- a/internal/gitaly/hook/storage_voter_test.go
+++ b/internal/gitaly/hook/storage_voter_test.go
@@ -17,6 +17,7 @@ import (
"github.com/feder-cr/git-ai/internal/grpc/backchannel"
"github.com/feder-cr/git-ai/internal/testhelper"
"github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
"github.com/stretchr/testify/require"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
@@ -44,10 +45,6 @@ func (v *recordingVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte)
v.record("preparing")
}
-func (v *recordingVoter) Denied(context.Context, gitcmd.HooksPayload, []byte) {
- v.record("denied")
-}
-
func (v *recordingVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {
v.record("prepared")
return nil
@@ -63,7 +60,9 @@ func (v *recordingVoter) Aborted(context.Context, gitcmd.HooksPayload, []byte) {
func (v *recordingVoter) Authoritative() bool { return true }
-func (v *recordingVoter) DeferredReferenceDurability() bool { return false }
+func (v *recordingVoter) DeferReferenceWrites(context.Context, *gitalypb.Repository) (func(), bool) {
+ return nil, false
+}
// TestReferenceTransactionHook_storageVoterPhases checks that the hook manager
// hands every phase of a reference transaction to the storage core, the
@@ -86,11 +85,9 @@ func TestReferenceTransactionHook_storageVoterPhases(t *testing.T) {
want: []string{"preparing", "prepared", "committed"},
},
{
- // The voter learns of its start (Denied), so that it counts the
- // transaction until its abort, but prepares nothing.
desc: "a denied transaction is neither prepared nor committed",
policy: []config.GraftPolicyRule{{Refs: []string{"refs/heads/main"}, Allow: false}},
- want: []string{"denied"},
+ want: nil,
denied: true,
},
} {
diff --git a/internal/gitaly/hook/updateref/deferred_durability_test.go b/internal/gitaly/hook/updateref/deferred_durability_test.go
index a381daf..72efa27 100644
--- a/internal/gitaly/hook/updateref/deferred_durability_test.go
+++ b/internal/gitaly/hook/updateref/deferred_durability_test.go
@@ -14,18 +14,18 @@ import (
"github.com/feder-cr/git-ai/internal/graft/changelog"
"github.com/feder-cr/git-ai/internal/testhelper"
"github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
)
// durabilityVoter is a storage voter that only answers whether it defers the durability of the
-// references.
+// references, and counts the ends of the processes it deferred.
type durabilityVoter struct {
deferred bool
+ ended *int
}
func (durabilityVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte) {}
-func (durabilityVoter) Denied(context.Context, gitcmd.HooksPayload, []byte) {}
-
func (durabilityVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {
return nil
}
@@ -36,21 +36,29 @@ func (durabilityVoter) Aborted(context.Context, gitcmd.HooksPayload, []byte) {}
func (durabilityVoter) Authoritative() bool { return true }
-func (v durabilityVoter) DeferredReferenceDurability() bool { return v.deferred }
+func (v durabilityVoter) DeferReferenceWrites(context.Context, *gitalypb.Repository) (func(), bool) {
+ if !v.deferred {
+ return nil, false
+ }
+ return func() { *v.ended++ }, true
+}
-// TestDeferReferenceDurability verifies that the updater writes references without fsync only
-// when the storage voter of its hook manager defers their durability: never without the storage
-// core, nor when the voter does not (shadow mode, or authoritative mode where the kernel exposes
-// no boot identity; the voter's own answer is tested in the voter package).
-func TestDeferReferenceDurability(t *testing.T) {
+// TestDeferReferenceWrites verifies that the updater writes references without fsync only when
+// the storage voter of its hook manager defers their durability: never without the storage
+// core, nor when the voter does not (shadow mode, authoritative mode where the kernel exposes no
+// boot identity, or a process that does not own the storage; the voter's own answer is tested in
+// the voter package).
+func TestDeferReferenceWrites(t *testing.T) {
t.Parallel()
+ ctx := testhelper.Context(t)
cfg := testcfg.Build(t)
newManager := func(opts ...hook.ManagerOption) hook.Manager {
return hook.NewManager(cfg, config.NewLocator(cfg), testhelper.NewLogger(t), gittest.NewCommandFactory(t, cfg), nil, nil,
gitlab.NewMockClient(t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive),
hook.NewPostReceiveRegistry(), opts...)
}
+ var ended int
for _, tc := range []struct {
desc string
@@ -58,12 +66,17 @@ func TestDeferReferenceDurability(t *testing.T) {
expected bool
}{
{desc: "storage core off", manager: newManager()},
- {desc: "voter keeps the fsync", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: false}))},
- {desc: "voter defers the durability", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: true})), expected: true},
+ {desc: "voter keeps the fsync", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: false, ended: &ended}))},
+ {desc: "voter defers the durability", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: true, ended: &ended})), expected: true},
{desc: "manager without a storage voter", manager: hook.NewMockManager(t, hook.NopPreReceive, hook.NopPostReceive, hook.NopUpdate, hook.NopReferenceTransaction)},
} {
t.Run(tc.desc, func(t *testing.T) {
- require.Equal(t, tc.expected, deferReferenceDurability(tc.manager))
+ end, ok := deferReferenceWrites(ctx, tc.manager, &gitalypb.Repository{StorageName: "default", RelativePath: "repo.git"})
+ require.Equal(t, tc.expected, ok)
+ if ok {
+ end()
+ require.Equal(t, 1, ended)
+ }
})
}
}
diff --git a/internal/gitaly/hook/updateref/update_with_hooks.go b/internal/gitaly/hook/updateref/update_with_hooks.go
index acf21e7..917d6b6 100644
--- a/internal/gitaly/hook/updateref/update_with_hooks.go
+++ b/internal/gitaly/hook/updateref/update_with_hooks.go
@@ -143,14 +143,18 @@ func (e Error) Proto() *gitalypb.ReferenceUpdateError {
// deferredDurability is implemented by a hook manager whose storage core writes the references
// of the transactions it commits without fsync (hook.GitLabHookManager).
type deferredDurability interface {
- DeferredReferenceDurability() bool
+ DeferReferenceWrites(ctx context.Context, repo *gitalypb.Repository) (end func(), ok bool)
}
-// deferReferenceDurability reports whether the reference transactions driven with hookManager
-// may write their references without fsync: only when its storage voter says so.
-func deferReferenceDurability(hookManager hook.Manager) bool {
- d, ok := hookManager.(deferredDurability)
- return ok && d.DeferredReferenceDurability()
+// deferReferenceWrites decides whether the update-ref process about to start for repo with
+// hookManager writes its references without fsync: only when its storage voter says so. When
+// ok, end is called once the process has exited.
+func deferReferenceWrites(ctx context.Context, hookManager hook.Manager, repo *gitalypb.Repository) (end func(), ok bool) {
+ d, isDeferring := hookManager.(deferredDurability)
+ if !isDeferring {
+ return nil, false
+ }
+ return d.DeferReferenceWrites(ctx, repo)
}
// NewUpdaterWithHooks creates a new instance of a struct that will update a Git reference.
@@ -272,8 +276,11 @@ func (u *UpdaterWithHooks) UpdateReference(
// this problem.
updaterOpts := []updateref.UpdaterOpt{updateref.WithDisabledTransactions()}
// The storage core's authoritative mode makes the references durable itself: its database
- // commit in the prepared phase is the durable point of the transaction.
- if deferReferenceDurability(u.hookManager) {
+ // commit in the prepared phase is the durable point of the transaction. It counts the
+ // update-ref process as writing without fsync until the process has exited (end runs after
+ // updater.Close, which waits for it).
+ if end, ok := deferReferenceWrites(ctx, u.hookManager, repoProto); ok {
+ defer end()
updaterOpts = append(updaterOpts, updateref.WithDeferredReferenceDurability())
}
updater, err := updateref.New(ctx, repo, updaterOpts...)
diff --git a/internal/graft/corestore/coretest/coretest.go b/internal/graft/corestore/coretest/coretest.go
index f7e1da8..e2dc835 100644
--- a/internal/graft/corestore/coretest/coretest.go
+++ b/internal/graft/corestore/coretest/coretest.go
@@ -202,7 +202,8 @@ func Open(tb testing.TB, opts Options) *corestore.Core {
// EnableLocalRefs gives the core the admission and the deferred durability of
// the local reference writes, owning the storages of cfg, as gitaly serve does
// in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). The
-// boot identity is a fixed test value unless opts set another one. A deferred
+// boot identity is a fixed test value and the recovery grace 0 unless opts set
+// others. A deferred
// durability the core had is closed first, as when its process exits.
func EnableLocalRefs(tb testing.TB, core *corestore.Core, cfg config.Cfg, opts ...localrefs.DurabilityOption) {
tb.Helper()
@@ -211,11 +212,17 @@ func EnableLocalRefs(tb testing.TB, core *corestore.Core, cfg config.Cfg, opts .
require.NoError(tb, core.RefDurability.Close())
}
core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)
- durability, err := localrefs.NewDurability(config.NewLocator(cfg), 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID)}, opts...)...)
+ durability, err := localrefs.NewDurability(config.NewLocator(cfg), 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID), localrefs.WithRecoveryGrace(0)}, opts...)...)
require.NoError(tb, err)
- owned, err := durability.TryOwn(context.Background(), StorageNames(cfg))
- require.NoError(tb, err)
- require.True(tb, owned, "another durability of the test owns the storages")
+ // A git process forked at the moment a previous durability closed its lock
+ // holds a copy of it until it execs (close-on-exec): the lock can be busy for
+ // that short while.
+ require.Eventually(tb, func() bool {
+ owned, err := durability.TryOwn(context.Background(), StorageNames(cfg))
+ require.NoError(tb, err)
+ return owned
+ }, 10*time.Second, time.Millisecond, "another durability of the test owns the storages")
+ durability.Activate()
tb.Cleanup(func() { _ = durability.Close() })
core.RefDurability = durability
}
diff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.go
index 66147b1..f96aeb3 100644
--- a/internal/graft/corestore/localrefs/durability.go
+++ b/internal/graft/corestore/localrefs/durability.go
@@ -21,6 +21,7 @@ import (
"github.com/prometheus/client_golang/prometheus"
+ "github.com/feder-cr/git-ai/internal/git/gitcmd"
"github.com/feder-cr/git-ai/internal/gitaly/config"
"github.com/feder-cr/git-ai/internal/gitaly/storage"
"github.com/feder-cr/git-ai/proto/go/gitalypb"
@@ -54,8 +55,18 @@ const (
// DurableStepIdle is how long a dirty repository must be without a reference
// write before the durable step removes its marker.
DurableStepIdle = time.Second
+ // RecoveryGrace is how long a repository found dirty when the process takes
+ // the ownership keeps its marker at least: longer than git waits for the
+ // reference lock (reftable.lockTimeout), so that a git process of the
+ // previous owner that is not reaped yet has finished.
+ RecoveryGrace = 2 * gitcmd.ReftableLockTimeoutMilliseconds * time.Millisecond
)
+// WithRecoveryGrace replaces RecoveryGrace. Tests shorten it.
+func WithRecoveryGrace(grace time.Duration) DurabilityOption {
+ return func(d *Durability) { d.recoveryGrace = grace }
+}
+
// CrashEnv names the test-only environment variable that makes the process
// exit (exit code 75) in the durable step, while it holds git's reference lock
// after making the stack durable and before it removes the marker: "durable_step".
@@ -138,11 +149,17 @@ type Durability struct {
// crashAt is the crash point selected with CrashEnv.
crashAt string
- // owner is set once the process owns the deferred durability of its
- // storages; it never goes back. ownerFiles hold the owner locks.
+ // ownerFiles hold the owner locks, taken by TryOwn. owner is set by
+ // Activate once the process has recovered the storages: from then on its
+ // writes are deferred. Neither goes back.
owner atomic.Bool
ownerMu sync.Mutex
ownerFiles []*os.File
+ locked bool
+ // recoveryGrace is how long a repository recovered dirty keeps its marker
+ // at least: a git process of the previous owner that outlived it (killed
+ // with it, but not reaped yet) may still be committing for that long.
+ recoveryGrace time.Duration
mu sync.Mutex
// repos holds the state of every repository written since the process
@@ -169,13 +186,17 @@ type refState struct {
repo RepoRef
marker string
dirty bool
- // live counts, by key, the hook transactions of the process from their
- // start (Preparing, or a policy denial) to their end (Committed or
- // Aborted): any of them may still write a table or compact the stack
- // without fsync.
- live map[Key]int
+ // nLive counts the git processes of this process that write the
+ // repository's references without fsync and have not exited: any of them
+ // may still write a table or compact the stack.
nLive int
lastWrite time.Time
+ // settleAfter keeps the marker of a repository recovered dirty until then.
+ settleAfter time.Time
+ // untrusted is the first failed fsync of the repository. The kernel reports
+ // a failed writeback once, so a later fsync can succeed although writes
+ // were lost: the repository never becomes clean again, only moved aside.
+ untrusted error
// synced holds the tables made durable during the current dirty period that
// the stack still lists. Tables are immutable, so one fsync each is enough.
synced map[string]struct{}
@@ -248,11 +269,12 @@ func NewDurability(locator storage.Locator, idle time.Duration, opts ...Durabili
func newDurability(locator storage.Locator, idle time.Duration) *Durability {
return &Durability{
- locator: locator,
- idle: idle,
- sync: (*os.File).Sync,
- crashAt: os.Getenv(CrashEnv),
- repos: map[string]*refState{},
+ locator: locator,
+ idle: idle,
+ sync: (*os.File).Sync,
+ crashAt: os.Getenv(CrashEnv),
+ recoveryGrace: RecoveryGrace,
+ repos: map[string]*refState{},
marked: prometheus.NewCounter(prometheus.CounterOpts{
Name: "graft_storage_ref_dirty_marked_total",
Help: "Number of dirty markers written: a repository got local reference writes that are not durable yet.",
@@ -293,19 +315,22 @@ func (d *Durability) Enabled() bool { return d != nil }
// once.
func (d *Durability) Deferring() bool { return d != nil && d.owner.Load() }
-// TryOwn takes the ownership of the deferred durability of the storages
-// without waiting. It returns false when another process holds it for one of
-// them; the locks taken are kept, and a later call takes the rest.
+// TryOwn takes the owner locks of the storages without waiting. It returns
+// false when another process holds one of them; the locks taken are kept, and
+// a later call takes the rest. Owning the locks lets the process recover the
+// storages (Recover); its writes are deferred only after Activate, so that no
+// writer of this process writes a marker that the recovery could take for one
+// left behind.
func (d *Durability) TryOwn(ctx context.Context, storages []string) (bool, error) {
if d == nil {
return false, nil
}
- if d.owner.Load() {
- return true, nil
- }
d.ownerMu.Lock()
defer d.ownerMu.Unlock()
+ if d.locked {
+ return true, nil
+ }
for _, storageName := range storages[len(d.ownerFiles):] {
dir, err := d.markerDir(ctx, storageName)
@@ -329,9 +354,31 @@ func (d *Durability) TryOwn(ctx context.Context, storages []string) (bool, error
d.ownerFiles = append(d.ownerFiles, f)
}
+ d.locked = true
+ return true, nil
+}
+
+// Activate makes the writes of the process deferred; it holds the owner locks
+// (TryOwn) and has recovered the storages.
+func (d *Durability) Activate() {
+ if d == nil {
+ return
+ }
+
+ d.ownerMu.Lock()
+ defer d.ownerMu.Unlock()
+ if !d.locked {
+ panic("localrefs: Activate without the owner locks")
+ }
d.owner.Store(true)
d.owning.Set(1)
- return true, nil
+}
+
+// ownsLocks reports whether the process holds the owner locks.
+func (d *Durability) ownsLocks() bool {
+ d.ownerMu.Lock()
+ defer d.ownerMu.Unlock()
+ return d.locked
}
// Close releases the ownership. The process must not write references
@@ -348,6 +395,7 @@ func (d *Durability) Close() error {
errs = append(errs, f.Close())
}
d.ownerFiles = nil
+ d.locked = false
return errors.Join(errs...)
}
@@ -374,7 +422,7 @@ func (d *Durability) markerDir(ctx context.Context, storageName string) (string,
func (d *Durability) stateLocked(path string) *refState {
st := d.repos[path]
if st == nil {
- st = &refState{live: map[Key]int{}, synced: map[string]struct{}{}}
+ st = &refState{synced: map[string]struct{}{}}
d.repos[path] = st
}
return st
@@ -422,53 +470,45 @@ func (d *Durability) MarkDirty(ctx context.Context, repo RepoRef, path string) e
d.mu.Lock()
defer d.mu.Unlock()
- st.dirty = true
st.repo = repo
st.marker = markerPath
d.marked.Inc()
- d.dirty.Inc()
+ if !st.dirty {
+ st.dirty = true
+ d.dirty.Inc()
+ }
return nil
}
-// Begin records that the hook transaction key started on the repository at
-// path: from its Preparing phase (or its policy denial, which aborts it), before
-// git takes the reference lock, to its end (End), it may write a table and
-// compact the stack. Every transaction begins exactly once and ends exactly
-// once (Committed or Aborted), so the count of a key is the number of its live
-// transactions, whichever of two transactions with the same key ends first.
-func (d *Durability) Begin(path string, key Key) {
+// Begin records that a git process that writes the references of the
+// repository at path without fsync is about to start; the caller calls end
+// once the process has exited. Until then the process may write a table or
+// compact the stack, whatever hooks git runs or skips (a failed prepare, a
+// failed commit, a kill), so the durable step keeps the marker. end may be
+// called more than once.
+func (d *Durability) Begin(path string) (end func()) {
if d == nil {
- return
+ return func() {}
}
d.mu.Lock()
- defer d.mu.Unlock()
st := d.stateLocked(path)
- st.live[key]++
st.nLive++
d.inflight.Inc()
-}
-
-// End records that a hook transaction with key ended (Committed or Aborted).
-// An end without a begin (the process started after the transaction began)
-// changes nothing.
-func (d *Durability) End(path string, key Key) {
- if d == nil {
- return
- }
+ d.mu.Unlock()
- d.mu.Lock()
- defer d.mu.Unlock()
- st := d.repos[path]
- if st == nil || st.live[key] == 0 {
- return
- }
- st.live[key]--
- if st.live[key] == 0 {
- delete(st.live, key)
+ var once sync.Once
+ return func() {
+ once.Do(func() {
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ // Forget may have dropped the state; the count belongs to it.
+ if d.repos[path] == st {
+ st.nLive--
+ d.inflight.Dec()
+ }
+ })
}
- st.nLive--
- d.inflight.Dec()
}
// Dirty returns the paths of the dirty repositories.
@@ -535,7 +575,14 @@ func (d *Durability) Settle(path string, final bool) error {
return nil
}
repo := st.repo
- idle := st.nLive == 0 && (final || time.Since(st.lastWrite) >= d.idle)
+ if st.untrusted != nil {
+ // Sticky: the move aside is retried, the marker never cleared.
+ untrusted := st.untrusted
+ d.mu.Unlock()
+ return &UntrustedError{Repo: repo, Path: path, Err: untrusted}
+ }
+ now := time.Now()
+ idle := st.nLive == 0 && now.After(st.settleAfter) && (final || now.Sub(st.lastWrite) >= d.idle)
d.mu.Unlock()
if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {
@@ -546,7 +593,7 @@ func (d *Durability) Settle(path string, final bool) error {
}
if err := d.syncStack(path, st); err != nil {
- return d.stepError(repo, path, err)
+ return d.stepError(st, repo, path, err)
}
if !idle {
return nil
@@ -563,7 +610,7 @@ func (d *Durability) Settle(path string, final bool) error {
cleared, err := d.settleLocked(path, st)
if err != nil {
_ = lock.release()
- return d.stepError(repo, path, err)
+ return d.stepError(st, repo, path, err)
}
if !cleared {
return lock.release()
@@ -580,13 +627,20 @@ func (d *Durability) Settle(path string, final bool) error {
}
// stepError is the error of a durable step: an *UntrustedError for a failed
-// fsync, the error itself otherwise.
-func (d *Durability) stepError(repo RepoRef, path string, err error) error {
+// fsync, which makes the repository untrusted for good, the error itself
+// otherwise.
+func (d *Durability) stepError(st *refState, repo RepoRef, path string, err error) error {
var writeback writebackError
- if errors.As(err, &writeback) {
- return &UntrustedError{Repo: repo, Path: path, Err: err}
+ if !errors.As(err, &writeback) {
+ return err
}
- return err
+
+ d.mu.Lock()
+ if st.untrusted == nil {
+ st.untrusted = err
+ }
+ d.mu.Unlock()
+ return &UntrustedError{Repo: repo, Path: path, Err: err}
}
// settleLocked makes the tables written since the first pass durable and, when
@@ -596,7 +650,7 @@ func (d *Durability) stepError(repo RepoRef, path string, err error) error {
// before they take it until their end.
func (d *Durability) settleLocked(path string, st *refState) (bool, error) {
d.mu.Lock()
- busy := st.nLive > 0
+ busy := st.nLive > 0 || st.untrusted != nil
d.mu.Unlock()
if busy {
return false, nil
@@ -856,7 +910,7 @@ type Recovered struct {
// boot before serving. A marker that cannot be read is an error: its
// repository is unknown, and serving could read a torn stack.
func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovered, error) {
- if !d.Deferring() {
+ if d == nil || !d.ownsLocks() {
return nil, nil
}
@@ -917,10 +971,11 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovere
st := d.stateLocked(path)
if !st.dirty {
st.dirty = true
- st.repo = m.RepoRef
- st.marker = markerPath
d.dirty.Inc()
}
+ st.repo = m.RepoRef
+ st.marker = markerPath
+ st.settleAfter = time.Now().Add(d.recoveryGrace)
d.mu.Unlock()
}
found = append(found, Recovered{Repo: m.RepoRef, Path: path, Reboot: reboot})
diff --git a/internal/graft/corestore/localrefs/durability_test.go b/internal/graft/corestore/localrefs/durability_test.go
index 74434c1..33c35b1 100644
--- a/internal/graft/corestore/localrefs/durability_test.go
+++ b/internal/graft/corestore/localrefs/durability_test.go
@@ -87,25 +87,39 @@ func setupDurability(t *testing.T) durabilityFixture {
}
// unowned returns a durability of the fixture's storage that does not own it.
-func (f durabilityFixture) unowned(t *testing.T, bootID string, idle time.Duration) *Durability {
+// The recovery grace is 0 unless opts set it.
+func (f durabilityFixture) unowned(t *testing.T, bootID string, idle time.Duration, opts ...DurabilityOption) *Durability {
t.Helper()
- d, err := NewDurability(f.locator, idle, WithBootID(bootID), WithSync(f.synced.sync))
+ d, err := NewDurability(f.locator, idle, append([]DurabilityOption{WithBootID(bootID), WithSync(f.synced.sync), WithRecoveryGrace(0)}, opts...)...)
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, d.Close()) })
return d
}
-// durability returns a durability that owns the fixture's storage, as the
-// process that runs the durable steps. Close it before another one owns.
-func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration) *Durability {
+// durability returns a durability that owns the fixture's storage and defers,
+// as the process that runs the durable steps (the recovery, when a test runs
+// it, comes after: no writer of these tests runs before it). Close it before
+// another one owns.
+func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration, opts ...DurabilityOption) *Durability {
t.Helper()
- d := f.unowned(t, bootID, idle)
- owned, err := d.TryOwn(f.ctx, []string{f.repo.Storage})
- require.NoError(t, err)
- require.True(t, owned)
+ d := f.unowned(t, bootID, idle, opts...)
+ f.own(t, d)
+ d.Activate()
return d
}
+// own takes the owner locks for d. A git process forked by another test at the
+// moment the previous owner closed its lock holds a copy of the lock until it
+// execs (close-on-exec), so the lock can be busy for that short while.
+func (f durabilityFixture) own(t *testing.T, d *Durability) {
+ t.Helper()
+ require.Eventually(t, func() bool {
+ owned, err := d.TryOwn(f.ctx, []string{f.repo.Storage})
+ require.NoError(t, err)
+ return owned
+ }, 10*time.Second, time.Millisecond)
+}
+
func (f durabilityFixture) markerPath(t *testing.T, d *Durability) string {
t.Helper()
p, err := d.markerPath(f.ctx, f.repo, f.path)
@@ -165,7 +179,11 @@ func TestOneOwnerPerStorage(t *testing.T) {
old := f.durability(t, "boot-a", 0)
require.True(t, old.Deferring())
require.NoError(t, old.MarkDirty(f.ctx, f.repo, f.path))
- old.Begin(f.path, key(1))
+ _ = old.Begin(f.path)
+ // The old process's durable step holds git's lock right now.
+ liveLock, ok, err := lockStack(f.path)
+ require.NoError(t, err)
+ require.True(t, ok)
next := f.unowned(t, "boot-a", 0)
owned, err := next.TryOwn(f.ctx, storages)
@@ -174,27 +192,49 @@ func TestOneOwnerPerStorage(t *testing.T) {
require.False(t, next.Deferring())
require.Equal(t, 0.0, testutil.ToFloat64(next.owning))
- // The next process neither recovers nor settles: the old process's
- // transaction is live, and only the old process counts it.
+ // The next process neither recovers, nor removes the old process's lock,
+ // nor settles: the old process's writer is live, and only the old process
+ // counts it.
found, err := next.Recover(f.ctx, storages)
require.NoError(t, err)
require.Empty(t, found)
+ require.FileExists(t, filepath.Join(f.path, "reftable", "tables.list.lock"), "the live owner's lock is kept")
require.NoError(t, next.Settle(f.path, true))
require.FileExists(t, f.markerPath(t, old))
+ require.NoError(t, liveLock.release())
- // The old process exits; the next one owns and recovers the marker.
+ // The old process exits; the next one owns, recovers the marker before it
+ // defers, and settles it.
require.NoError(t, old.Close())
- owned, err = next.TryOwn(f.ctx, storages)
- require.NoError(t, err)
- require.True(t, owned)
- require.True(t, next.Deferring())
+ f.own(t, next)
+ require.False(t, next.Deferring(), "no write is deferred before the recovery")
found, err = next.Recover(f.ctx, storages)
require.NoError(t, err)
require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path}}, found)
+ next.Activate()
+ require.True(t, next.Deferring())
require.NoError(t, next.Settle(f.path, false))
require.NoFileExists(t, f.markerPath(t, next))
}
+// TestRecoveredMarkerWaitsForTheGrace verifies that a marker found by the
+// recovery stays for the recovery grace: a git process of the previous owner
+// that is not reaped yet may still be committing.
+func TestRecoveredMarkerWaitsForTheGrace(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+
+ old := f.durability(t, "boot-a", 0)
+ require.NoError(t, old.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, old.Close())
+
+ next := f.durability(t, "boot-a", 0, WithRecoveryGrace(time.Hour))
+ _, err := next.Recover(f.ctx, []string{f.repo.Storage})
+ require.NoError(t, err)
+ require.NoError(t, next.Settle(f.path, true))
+ require.FileExists(t, f.markerPath(t, next), "not even the last step removes it within the grace")
+}
+
func TestSettleMakesTheStackDurableAndRemovesTheMarker(t *testing.T) {
t.Parallel()
f := setupDurability(t)
@@ -240,43 +280,40 @@ func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {
require.NoFileExists(t, f.markerPath(t, d))
})
- t.Run("a live transaction", func(t *testing.T) {
+ t.Run("a live git process", func(t *testing.T) {
t.Parallel()
f := setupDurability(t)
d := f.durability(t, "boot-a", 0)
require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
- d.Begin(f.path, key(1))
+ end := d.Begin(f.path)
require.NoError(t, d.Settle(f.path, true))
require.FileExists(t, f.markerPath(t, d), "not even the last step removes it")
- // An end of another key does not count; the end of the transaction does.
- d.End(f.path, key(2))
- require.NoError(t, d.Settle(f.path, false))
- require.FileExists(t, f.markerPath(t, d))
-
- d.End(f.path, key(1))
+ end()
require.NoError(t, d.Settle(f.path, false))
require.NoFileExists(t, f.markerPath(t, d))
require.Equal(t, 0.0, testutil.ToFloat64(d.inflight))
})
- t.Run("two live transactions with the same key", func(t *testing.T) {
+ t.Run("two live git processes", func(t *testing.T) {
t.Parallel()
f := setupDurability(t)
d := f.durability(t, "boot-a", 0)
- // Whichever of the two ends first (an abort before Prepared, say), the
- // other one is still counted.
+ // Whatever hooks they ran or skipped, each counts until its own end, and
+ // an end called twice counts once.
require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
- d.Begin(f.path, key(1))
- d.Begin(f.path, key(1))
- d.End(f.path, key(1))
+ first := d.Begin(f.path)
+ second := d.Begin(f.path)
+ first()
+ first()
require.NoError(t, d.Settle(f.path, false))
require.FileExists(t, f.markerPath(t, d))
- d.End(f.path, key(1))
+ second()
require.NoError(t, d.Settle(f.path, false))
require.NoFileExists(t, f.markerPath(t, d))
+ require.Equal(t, 0.0, testutil.ToFloat64(d.inflight))
})
t.Run("git holds the reference lock", func(t *testing.T) {
@@ -309,14 +346,20 @@ func TestStackLockReleaseRemovesOnlyItsOwnLock(t *testing.T) {
require.NoError(t, err)
require.False(t, ok, "the lock is exclusive")
- // Housekeeping removed the lock as stale, and another durable step took it
- // (the freed inode may well be reused for it).
- require.NoError(t, os.Remove(lockPath))
+ // The same inode with another content: what a reused inode looks like to
+ // the file check. The nonce tells it is not the step's lock.
require.NoError(t, os.WriteFile(lockPath, []byte(stackLockTag+"0123456789abcdef\n"), 0o666))
+ current, err := os.Lstat(lockPath)
+ require.NoError(t, err)
+ require.True(t, os.SameFile(current, lock.info), "the file check alone would take it for the step's lock")
require.NoError(t, lock.release())
- require.FileExists(t, lockPath, "a lock of another holder, even with the same tag, is not removed")
+ require.FileExists(t, lockPath, "a lock with another nonce is not removed, even in the same inode")
+ // Housekeeping removed the lock as stale: the release has nothing to do.
require.NoError(t, os.Remove(lockPath))
+ require.NoError(t, lock.release())
+ require.NoFileExists(t, lockPath)
+
lock, ok, err = lockStack(f.path)
require.NoError(t, err)
require.True(t, ok)
@@ -345,7 +388,15 @@ func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {
require.FileExists(t, f.markerPath(t, d))
require.Equal(t, []string{f.path}, d.Dirty())
+ // The kernel reports a failed writeback once: the next fsync succeeds, and
+ // the repository stays untrusted all the same.
f.synced.fail(nil)
+ err = d.Settle(f.path, true)
+ require.ErrorIs(t, err, ErrUntrusted, "a later successful fsync does not make the repository clean")
+ require.ErrorIs(t, err, writeback)
+ require.FileExists(t, f.markerPath(t, d))
+
+ // Only moving it aside (Forget) ends it.
require.NoError(t, d.Forget(f.ctx, f.repo, f.path))
require.NoFileExists(t, f.markerPath(t, d))
require.Empty(t, d.Dirty())
@@ -503,8 +554,8 @@ func TestNilDurabilityIsOff(t *testing.T) {
require.False(t, owned)
require.NoError(t, d.MarkDirty(ctx, RepoRef{}, "path"))
require.NoError(t, d.Settle("path", true))
- d.Begin("path", key(1))
- d.End("path", key(1))
+ d.Begin("path")()
+ d.Activate()
found, err := d.Recover(ctx, []string{"default"})
require.NoError(t, err)
require.Empty(t, found)
diff --git a/internal/graft/corestore/syncer/durability.go b/internal/graft/corestore/syncer/durability.go
index 4b468c6..f5ab734 100644
--- a/internal/graft/corestore/syncer/durability.go
+++ b/internal/graft/corestore/syncer/durability.go
@@ -10,7 +10,6 @@ import (
"time"
"github.com/feder-cr/git-ai/internal/graft/corestore"
- "github.com/feder-cr/git-ai/internal/graft/corestore/gitops"
"github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
"github.com/feder-cr/git-ai/internal/structerr"
)
@@ -49,6 +48,7 @@ func (s *Syncer) StartLocalReferences(ctx context.Context, storages []string) (s
if err := s.RecoverLocalReferences(ctx, storages); err != nil {
return nil, err
}
+ d.Activate()
recovered = true
} else {
s.logger.Warn("storage core: another gitaly process owns the deferred durability of the storages; references are written with fsync until it exits")
@@ -103,6 +103,7 @@ func (s *Syncer) runDurableSteps(ctx context.Context, storages []string, recover
s.logger.WithError(err).Error("storage core: recover the local references after taking the deferred durability; retried")
continue
}
+ d.Activate()
recovered = true
s.logger.Info("storage core: this process owns the deferred durability of the storages now")
}
@@ -159,7 +160,10 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)
// storageCoreCache reports whether the local repository at path is a cache of
// the storage core that a hydration can rebuild: the database holds it with
-// changesets, and it has a watermark.
+// changesets (an import that is not complete has none, and its local copy may
+// be the only one). A sync replaces such a local repository too when it is
+// stale, whatever its watermark. An error (the database, or the repository's
+// directory, cannot be read) is returned: the caller tries again.
func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, path string) (bool, error) {
r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: repo.Storage, RelativePath: repo.RelativePath})
if errors.Is(err, corestore.ErrRepositoryNotFound) {
@@ -177,13 +181,7 @@ func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, p
} else if err != nil {
return false, err
}
- _, ok, err := gitops.ReadWatermark(path)
- if err != nil {
- // A watermark that cannot be read is no proof that the storage core wrote
- // the repository: it is kept, as untracked() keeps it.
- return false, nil
- }
- return ok, nil
+ return true, nil
}
// durableStep makes the references of the repository at path durable under the
@@ -212,6 +210,15 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {
return
}
+ // A repository that is gone (moved aside as a stale cache meanwhile) has
+ // nothing left to serve: its marker is dropped.
+ if _, statErr := os.Lstat(path); errors.Is(statErr, os.ErrNotExist) {
+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {
+ logger.WithField("forget_error", err.Error()).Error("storage core: could not drop the marker of a repository that is gone; retried at the next step")
+ }
+ return
+ }
+
cache, cacheErr := s.storageCoreCache(ctx, untrusted.Repo, path)
if cacheErr != nil {
d.CountStepError()
@@ -220,12 +227,16 @@ func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {
return
}
if !cache {
+ // Its marker stays (the untrusted state is sticky): after a reboot the
+ // recovery decides again.
logger.Error("storage core: the references of a repository that is not a cache of the storage core may have lost writes; it is kept in place")
return
}
if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {
+ // The untrusted state is sticky: the next step moves it aside again, and
+ // the marker is never cleared meanwhile.
logger.WithField("move_error", err.Error()).
- Error("storage core: could not move aside a repository whose references may have lost writes")
+ Error("storage core: could not move aside a repository whose references may have lost writes; retried at the next step")
}
}
diff --git a/internal/graft/corestore/syncer/syncer_localrefs_test.go b/internal/graft/corestore/syncer/syncer_localrefs_test.go
index 38d33f5..1dd81d5 100644
--- a/internal/graft/corestore/syncer/syncer_localrefs_test.go
+++ b/internal/graft/corestore/syncer/syncer_localrefs_test.go
@@ -211,6 +211,96 @@ func TestSyncerDurableStepMovesAsideOnAFailedWrite(t *testing.T) {
e.verifySynced(t, repo)
}
+// TestSyncerUntrustedRepositoryIsNeverSettled is a failed fsync whose move
+// aside fails too (the temporary directory cannot be written for a moment).
+// The kernel reports the failed writeback once, so the next step's fsyncs
+// succeed: the repository must still be moved aside, never settled.
+func TestSyncerUntrustedRepositoryIsNeverSettled(t *testing.T) {
+ t.Parallel()
+
+ var failing atomic.Bool
+ e, repo, _ := caughtUpDirty(t, localrefs.WithSync(func(f *os.File) error {
+ if failing.Load() {
+ return errors.New("input/output error")
+ }
+ return f.Sync()
+ }))
+ tmpDir, err := e.locator.TempDir(e.storageName)
+ require.NoError(t, err)
+ require.NoError(t, os.MkdirAll(tmpDir, 0o700))
+
+ failing.Store(true)
+ require.NoError(t, os.Chmod(tmpDir, 0o500))
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
+ failing.Store(false)
+ require.NoError(t, os.Chmod(tmpDir, 0o700))
+
+ require.DirExists(t, e.localPath(), "the move aside failed")
+ require.Len(t, e.markers(t), 1, "the marker stays")
+
+ e.sy.DurableStep(e.ctx, e.localPath(), true)
+ require.NoDirExists(t, e.localPath(), "moved aside, although its fsyncs succeed now")
+ require.Empty(t, e.markers(t))
+ require.Equal(t, float64(1), e.untrusted(t, localrefs.ReasonSyncFailed))
+
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ e.verifySynced(t, repo)
+}
+
+// TestSyncerUntrustedRepositoryThatIsGoneLosesItsMarker verifies that an
+// untrusted repository that was moved away meanwhile (a stale cache) does not
+// keep its marker forever.
+func TestSyncerUntrustedRepositoryThatIsGoneLosesItsMarker(t *testing.T) {
+ t.Parallel()
+
+ var failing atomic.Bool
+ e, _, _ := caughtUpDirty(t, localrefs.WithSync(func(f *os.File) error {
+ if failing.Load() {
+ return errors.New("input/output error")
+ }
+ return f.Sync()
+ }))
+
+ tmpDir, err := e.locator.TempDir(e.storageName)
+ require.NoError(t, err)
+ require.NoError(t, os.MkdirAll(tmpDir, 0o700))
+ failing.Store(true)
+ require.NoError(t, os.Chmod(tmpDir, 0o500))
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
+ require.NoError(t, os.Chmod(tmpDir, 0o700))
+ failing.Store(false)
+ require.Len(t, e.markers(t), 1)
+
+ require.NoError(t, os.Rename(e.localPath(), filepath.Join(tmpDir, "moved.git")))
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
+ require.Empty(t, e.markers(t))
+ require.Empty(t, e.core.RefDurability.Dirty())
+}
+
+// TestSyncerCatchUpOfANonOwnerWritesWithFsync verifies that a process that
+// does not own the deferred durability catches up with fsync: it writes no
+// marker.
+func TestSyncerCatchUpOfANonOwnerWritesWithFsync(t *testing.T) {
+ t.Parallel()
+
+ e := newEnv(t, envOptions{})
+ durability, err := localrefs.NewDurability(e.locator, 0, localrefs.WithBootID(coretest.TestBootID))
+ require.NoError(t, err)
+ e.core.RefDurability = durability
+ require.False(t, durability.Deferring())
+
+ src := e.newSource(t)
+ c1 := e.writeCommit(t, src)
+ repo := e.register(t, e.key(localRelPath), src, c1)
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ e.advanceMain(t, repo.ID, src, c1)
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ require.Equal(t, float64(1), syncCount(t, e.sy, "caught_up"))
+
+ require.Empty(t, e.markers(t))
+ require.Empty(t, durability.Dirty())
+}
+
// TestSyncerDurableStepRetriesATransientError verifies that a durable step that
// fails without a failed fsync (the stack cannot be read for a moment) keeps
// the repository and its marker and is retried.
@@ -243,7 +333,7 @@ func TestSyncerSecondProcessWaitsForTheOwner(t *testing.T) {
storages := []string{e.storageName}
next := *e.core
- durability, err := localrefs.NewDurability(e.locator, 0, localrefs.WithBootID(coretest.TestBootID))
+ durability, err := localrefs.NewDurability(e.locator, 0, localrefs.WithBootID(coretest.TestBootID), localrefs.WithRecoveryGrace(0))
require.NoError(t, err)
next.RefDurability = durability
t.Cleanup(func() { _ = durability.Close() })
@@ -270,7 +360,7 @@ func TestSyncerStopSettlesEveryRepository(t *testing.T) {
e := newEnv(t, envOptions{})
e.core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)
- durability, err := localrefs.NewDurability(e.locator, time.Hour, localrefs.WithBootID(coretest.TestBootID))
+ durability, err := localrefs.NewDurability(e.locator, time.Hour, localrefs.WithBootID(coretest.TestBootID), localrefs.WithRecoveryGrace(0))
require.NoError(t, err)
t.Cleanup(func() { _ = durability.Close() })
e.core.RefDurability = durability
diff --git a/internal/graft/corestore/voter/voter.go b/internal/graft/corestore/voter/voter.go
index e3d67bd..6d3853c 100644
--- a/internal/graft/corestore/voter/voter.go
+++ b/internal/graft/corestore/voter/voter.go
@@ -176,8 +176,6 @@ func (v *Voter) Preparing(ctx context.Context, payload gitcmd.HooksPayload, chan
return
}
- // The transaction is live from now on (it ends in Committed or Aborted).
- v.begin(ctx, payload, changes)
// Deferred first, so that it runs last: the admission is waited for after
// the pack work, when git is about to take the reference lock.
defer v.admit(ctx, payload, changes)
@@ -774,7 +772,6 @@ func (v *Voter) Committed(ctx context.Context, payload gitcmd.HooksPayload, chan
// git finished the transaction, its auto-compaction included, and released
// the reference lock.
v.core.RefAdmission.Release(repoPath, localrefs.Key(key))
- v.core.RefDurability.End(repoPath, localrefs.Key(key))
if !ok || v.applier == nil {
return
@@ -801,36 +798,31 @@ func (v *Voter) localRefs() bool {
// DeferredReferenceDurability reports whether the reference writes of the
// transactions this voter commits may run without fsync: authoritative mode,
// and this process owns the deferred durability of the node (doc/graft/PLAN7.md,
-// LOCAL REFERENCE WRITES RULE). The hook manager asks it for UpdateWithHooks.
-// It goes from false to true at most once in a process.
+// LOCAL REFERENCE WRITES RULE). It goes from false to true at most once in a
+// process.
func (v *Voter) DeferredReferenceDurability() bool {
return v != nil && v.core.Authoritative() && v.core.RefDurability.Deferring()
}
-// Denied records the start of a transaction that the standalone policy denies:
-// it is live until its Aborted phase, like any other (see begin).
-func (v *Voter) Denied(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {
- if v == nil || !v.core.Enabled() || payload.Repo == nil {
- return
+// DeferReferenceWrites decides whether the git process that the caller is about
+// to start for repo (UpdateWithHooks' update-ref) writes its references without
+// fsync. When ok, the process counts as live for the repository's deferred
+// durability, and the caller calls end once it has exited: until then it may
+// write a table or compact the stack, whichever hooks git runs or skips.
+func (v *Voter) DeferReferenceWrites(ctx context.Context, repo *gitalypb.Repository) (end func(), ok bool) {
+ if !v.DeferredReferenceDurability() {
+ return nil, false
}
- v.begin(ctx, payload, changes)
-}
-// begin counts the transaction as live for the deferred durability: from
-// before git takes the reference lock until Committed or Aborted, it may write
-// a table or compact the stack, so the durable step keeps the repository's
-// marker meanwhile.
-func (v *Voter) begin(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {
- if !v.core.Authoritative() || !v.core.RefDurability.Enabled() {
- return
- }
-
- repoPath, err := v.repoPath(ctx, payload)
+ repoPath, err := v.locator.GetRepoPath(ctx, &gitalypb.Repository{
+ StorageName: repo.GetStorageName(),
+ RelativePath: repo.GetRelativePath(),
+ }, storage.WithRepositoryVerificationSkipped())
if err != nil {
- v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path of a starting transaction")
- return
+ v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path of a reference write; it is made durable by git")
+ return nil, false
}
- v.core.RefDurability.Begin(repoPath, localrefs.Key(pendingKey(payload.Repo, changes)))
+ return v.core.RefDurability.Begin(repoPath), true
}
// admit waits for the admission of the transaction to the reference lock of
@@ -853,9 +845,11 @@ func (v *Voter) admit(ctx context.Context, payload gitcmd.HooksPayload, changes
// deferDurability lets git write the references of a transaction that the
// database committed without fsync (authoritative mode, and this process owns
// the deferred durability): the repository gets its durable dirty marker
-// first. The transaction is live since its start (begin), so the durable step
-// keeps the marker until it ends. It runs in the Prepared phase, under git's
-// reference lock and before git writes anything.
+// first. A git process that writes without fsync counts as live from before it
+// started until it exited (DeferReferenceWrites), so the durable step keeps the
+// marker meanwhile. It runs in the Prepared phase, under git's reference lock
+// and before git writes anything. A transaction whose git process writes with
+// fsync (a push) gets a marker too, which only delays its removal.
func (v *Voter) deferDurability(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, shadow bool) error {
if shadow || !v.DeferredReferenceDurability() {
return nil
@@ -888,13 +882,11 @@ func (v *Voter) Aborted(ctx context.Context, payload gitcmd.HooksPayload, change
key := pendingKey(payload.Repo, changes)
- // git wrote nothing more of the transaction once it reports the abort.
if v.localRefs() {
if repoPath, err := v.repoPath(ctx, payload); err != nil {
v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path after an abort, the admission ends at its bound")
} else {
v.core.RefAdmission.Release(repoPath, localrefs.Key(key))
- v.core.RefDurability.End(repoPath, localrefs.Key(key))
}
}
diff --git a/internal/graft/corestore/voter/voter_localrefs_test.go b/internal/graft/corestore/voter/voter_localrefs_test.go
index 35a9109..9bffd15 100644
--- a/internal/graft/corestore/voter/voter_localrefs_test.go
+++ b/internal/graft/corestore/voter/voter_localrefs_test.go
@@ -14,6 +14,7 @@ import (
"github.com/feder-cr/git-ai/internal/git"
"github.com/feder-cr/git-ai/internal/git/gittest"
"github.com/feder-cr/git-ai/internal/git/localrepo"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
"github.com/feder-cr/git-ai/internal/graft/corestore/voter"
@@ -54,10 +55,20 @@ func (f *fixture) markers(t *testing.T) []string {
return names
}
+// deferred starts a deferred git process for the fixture's repository, as
+// UpdateWithHooks does before its update-ref.
+func (f *fixture) deferred(t *testing.T) (end func()) {
+ t.Helper()
+ end, ok := f.voter.DeferReferenceWrites(f.ctx, f.repo)
+ require.True(t, ok)
+ return end
+}
+
// TestLocalReferenceWritesOfACommittedTransaction follows one authoritative
// transaction through the LOCAL REFERENCE WRITES RULE (doc/graft/PLAN7.md):
-// admitted at the end of Preparing, marked dirty and in flight after the
-// database commit in Prepared, released and ended in Committed.
+// its git process is live from before it starts until it exits, it is
+// admitted at the end of Preparing and released in Committed, and its
+// repository is marked dirty after the database commit in Prepared.
func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {
t.Parallel()
@@ -68,6 +79,7 @@ func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {
require.True(t, f.voter.DeferredReferenceDurability())
require.False(t, f.admissionHeld(t))
+ end := f.deferred(t)
f.voter.Preparing(f.ctx, f.payload, input)
require.True(t, f.admissionHeld(t), "the transaction is admitted at the end of Preparing")
require.Empty(t, f.markers(t), "nothing is dirty before the database commit")
@@ -75,58 +87,72 @@ func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {
require.NoError(t, f.prepared(input))
require.Len(t, f.markers(t), 1)
- // git may still write and compact until Committed: the durable step keeps
- // the marker.
- require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
- require.Len(t, f.markers(t), 1)
-
gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName(ref), f.a)
f.voter.Committed(f.ctx, f.payload, input)
require.False(t, f.admissionHeld(t), "Committed releases the admission")
require.Len(t, f.applier.calls, 1, "the local commit is still noted")
+ // The git process may still compact until it exits.
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, true))
+ require.Len(t, f.markers(t), 1)
+
+ end()
require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
require.Empty(t, f.markers(t))
}
// TestLocalReferenceWritesOfAnAbortedTransaction verifies that an abort after
-// Prepared releases the admission and ends the transaction: the repository is
-// settled once nothing else is live.
+// Prepared releases the admission, and that the repository is settled once its
+// git process exited.
func TestLocalReferenceWritesOfAnAbortedTransaction(t *testing.T) {
t.Parallel()
f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
input := line(f.zero, f.a.String(), "refs/heads/main")
+ end := f.deferred(t)
f.voter.Preparing(f.ctx, f.payload, input)
require.NoError(t, f.prepared(input))
f.voter.Aborted(f.ctx, f.payload, input)
require.False(t, f.admissionHeld(t))
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
+ require.Len(t, f.markers(t), 1, "the git process has not exited yet")
+ end()
require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
require.Empty(t, f.markers(t))
}
-// TestLocalReferenceWritesOfTransactionsWithTheSameInput verifies that two
-// live transactions with the same input are both counted: the one denied by
-// the policy and aborted does not end the other one, which may still write.
-func TestLocalReferenceWritesOfTransactionsWithTheSameInput(t *testing.T) {
+// TestLocalReferenceWritesWhenGitSkipsItsHooks covers the transactions whose
+// later phases git never reports: a prepare that fails (a stale old value of a
+// writer that lost a race, a lock that stayed busy, a name conflict), a commit
+// whose finish fails, and the Committed phase of a symbolic reference that
+// names its target instead of the reference of Preparing. The live count
+// follows the git processes, so none of them leaks: the repository settles once
+// both processes exited.
+func TestLocalReferenceWritesWhenGitSkipsItsHooks(t *testing.T) {
t.Parallel()
f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
- input := line(f.zero, f.a.String(), "refs/heads/main")
-
- f.voter.Preparing(f.ctx, f.payload, input)
- require.NoError(t, f.prepared(input))
-
- // The same input from an agent the policy denies.
- f.voter.Denied(f.ctx, f.payload, input)
- f.voter.Aborted(f.ctx, f.payload, input)
- require.NoError(t, f.core.RefDurability.Settle(f.repoPath, true))
- require.Len(t, f.markers(t), 1, "the first transaction may still write")
-
- gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName("refs/heads/main"), f.a)
- f.voter.Committed(f.ctx, f.payload, input)
+ winner := line(f.zero, f.a.String(), "refs/heads/main")
+ loser := line(f.zero, f.b.String(), "refs/heads/main")
+
+ endWinner := f.deferred(t)
+ endLoser := f.deferred(t)
+ f.voter.Preparing(f.ctx, f.payload, winner)
+ f.voter.Preparing(f.ctx, f.payload, loser)
+
+ require.NoError(t, f.prepared(winner))
+ gittest.WriteRef(t, f.cfg, f.repoPath, "refs/heads/main", f.a)
+ // The Committed phase names another reference than Preparing (a symbolic
+ // reference split into its target).
+ f.voter.Committed(f.ctx, f.payload, line(f.zero, f.a.String(), "HEAD"))
+ endWinner()
+
+ // The loser's prepare failed in git: no Prepared, no Aborted, no Committed.
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
+ require.Len(t, f.markers(t), 1, "the loser's git process is still live")
+ endLoser()
require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
require.Empty(t, f.markers(t))
}
@@ -141,9 +167,11 @@ func TestLocalReferenceWritesOfARejectedTransaction(t *testing.T) {
f.seed(t, ref, corestore.Target(f.b.String()))
input := line(f.a.String(), f.c.String(), ref)
+ end := f.deferred(t)
f.voter.Preparing(f.ctx, f.payload, input)
requireMismatch(t, f.prepared(input), ref, f.a.String(), f.b.String())
f.voter.Aborted(f.ctx, f.payload, input)
+ end()
require.False(t, f.admissionHeld(t))
require.Empty(t, f.markers(t))
@@ -171,11 +199,13 @@ func TestLocalReferenceWritesWhenTheMarkerCannotBeWritten(t *testing.T) {
}
// TestDeferredReferenceDurabilityByMode verifies that references are written
-// without fsync in authoritative mode with deferred durability only: off and
-// shadow modes, and authoritative mode without the boot identity, keep git's
-// fsync of references.
+// without fsync in authoritative mode by the process that owns the deferred
+// durability only: off and shadow modes, authoritative mode without the boot
+// identity, and a process that does not own the storage (the new process of a
+// graceful upgrade) keep git's fsync of references.
func TestDeferredReferenceDurabilityByMode(t *testing.T) {
t.Parallel()
+ ctx := testhelper.Context(t)
var nilVoter *voter.Voter
require.False(t, nilVoter.DeferredReferenceDurability())
@@ -185,12 +215,28 @@ func TestDeferredReferenceDurabilityByMode(t *testing.T) {
shadow := newFixture(t, fixtureOptions{mode: corestore.ModeShadow, localRefs: true})
require.False(t, shadow.voter.DeferredReferenceDurability())
+ _, ok := shadow.voter.DeferReferenceWrites(ctx, shadow.repo)
+ require.False(t, ok)
authoritative := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative})
require.False(t, authoritative.voter.DeferredReferenceDurability())
+ _, ok = authoritative.voter.DeferReferenceWrites(ctx, authoritative.repo)
+ require.False(t, ok)
deferred := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
require.True(t, deferred.voter.DeferredReferenceDurability())
+
+ // Another process owns the storage: this one's durability holds no lock.
+ notOwner := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative})
+ durability, err := localrefs.NewDurability(config.NewLocator(notOwner.cfg), 0, localrefs.WithBootID("boot"))
+ require.NoError(t, err)
+ notOwner.core.RefDurability = durability
+ require.False(t, notOwner.voter.DeferredReferenceDurability())
+ _, ok = notOwner.voter.DeferReferenceWrites(ctx, notOwner.repo)
+ require.False(t, ok)
+ // Its Prepared phase marks nothing: its writes are made with fsync.
+ require.NoError(t, notOwner.prepared(line(notOwner.zero, notOwner.a.String(), "refs/heads/main")))
+ require.Empty(t, notOwner.markers(t))
}
// TestLocalReferenceWritesNotInShadowMode verifies that shadow mode, where git
diff --git a/internal/graft/e2e/storage_localrefs_test.go b/internal/graft/e2e/storage_localrefs_test.go
index 8b5af52..f682252 100644
--- a/internal/graft/e2e/storage_localrefs_test.go
+++ b/internal/graft/e2e/storage_localrefs_test.go
@@ -5,10 +5,13 @@ import (
"encoding/json"
"errors"
"fmt"
+ "io"
"math"
+ "net/http"
"os"
"os/exec"
"path/filepath"
+ "strconv"
"strings"
"sync"
"syscall"
@@ -71,6 +74,7 @@ func newRealNode(t *testing.T, configure ...func(*config.GraftStorage)) *realNod
c(&cfg.Graft.Storage)
}
cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)
+ cfg.PrometheusListenAddr = freeAddr(t)
testcfg.BuildGitalySSH(t, cfg)
testcfg.BuildGitalyHooks(t, cfg)
@@ -103,6 +107,7 @@ func (n *realNode) sibling(t *testing.T) *realNode {
s := *n
s.cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)
+ s.cfg.PrometheusListenAddr = freeAddr(t)
s.clientCfg = s.cfg
s.clientCfg.SocketPath = "unix://" + s.cfg.SocketPath
s.cfgPath = testcfg.WriteTemporaryGitalyConfigFile(t, s.cfg)
@@ -181,6 +186,34 @@ func (n *realNode) log(t *testing.T) string {
func (n *realNode) storagePath() string { return n.cfg.Storages[0].Path }
+// freeAddr returns a local TCP address that was free a moment ago.
+func freeAddr(t *testing.T) string {
+ t.Helper()
+ l, addr := testhelper.GetLocalhostListener(t)
+ require.NoError(t, l.Close())
+ return addr
+}
+
+// counter returns the value of the counter name of the server's metrics.
+func (n *realNode) counter(t *testing.T, name string) float64 {
+ t.Helper()
+
+ resp, err := http.Get("http://" + n.cfg.PrometheusListenAddr + "/metrics")
+ require.NoError(t, err)
+ defer resp.Body.Close()
+ body, err := io.ReadAll(resp.Body)
+ require.NoError(t, err)
+ for _, line := range strings.Split(string(body), "\n") {
+ if value, ok := strings.CutPrefix(line, name+" "); ok {
+ v, err := strconv.ParseFloat(strings.TrimSpace(value), 64)
+ require.NoError(t, err)
+ return v
+ }
+ }
+ t.Fatalf("metric %s not found", name)
+ return 0
+}
+
// createRepository creates an empty repository through the server and returns
// its relative path. In authoritative mode the gate registers it.
func (n *realNode) createRepository(t *testing.T) string {
@@ -300,7 +333,7 @@ func TestStorageLocalReferencesAfterAProcessKill(t *testing.T) {
// The durable step runs every second and removes the marker of an idle
// repository.
- require.Eventually(t, func() bool { return len(n.markers(t)) == 0 }, 30*time.Second, 50*time.Millisecond)
+ require.Eventually(t, func() bool { return len(n.markers(t)) == 0 }, 90*time.Second, 50*time.Millisecond)
n.requireServed(t, rel, written)
}
@@ -408,6 +441,8 @@ func TestStorageLocalReferencesDuringAGracefulUpgrade(t *testing.T) {
written[branch] = commit
}
next.requireServed(t, rel, written)
+ require.Equal(t, float64(0), next.counter(t, "graft_storage_ref_durability_owner"))
+ require.Equal(t, float64(0), next.counter(t, "graft_storage_ref_dirty_marked_total"), "the new process writes with fsync")
// The old process owns the marker and settles it once the repository is
// idle; the new process's writes were made with fsync and need none.
@@ -421,7 +456,8 @@ func TestStorageLocalReferencesDuringAGracefulUpgrade(t *testing.T) {
for branch, commit := range writeBranches(t, next.agent(t, rel), next.ctx, base, "after", 2) {
written[branch] = commit
}
- require.Len(t, next.markers(t), 1, "the new owner writes without fsync")
+ require.Equal(t, float64(1), next.counter(t, "graft_storage_ref_durability_owner"))
+ require.GreaterOrEqual(t, next.counter(t, "graft_storage_ref_dirty_marked_total"), float64(1), "the new owner writes without fsync")
next.requireServed(t, rel, written)
}