Git designed for AI agents.
WP76b follow-up: one owner per storage, exact live count, review fixes
Federico Elia8953e8e0b973
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.mdindex d3e15d3..89c2705 100644--- a/doc/graft/PLAN7.md+++ b/doc/graft/PLAN7.md@@ -541,8 +541,31 @@ repository-wide lock.2. *Deferred durability* (`corestore/localrefs.Durability`). The hook's`UpdateWithHooks` transactions and the syncer's catch-up write references with`core.fsync` without `reference`. Only the voter can turn this on- (`DeferredReferenceDurability`: authoritative mode and the kernel's boot- identity readable). Objects stay durable as before.+ (`DeferredReferenceDurability`: authoritative mode, the kernel's boot identity+ readable, and this process owns the storage's deferred durability). Objects+ stay durable as before.+ * *One owner per storage.* The deferred durability of a storage belongs to the+ process that holds an exclusive lock (flock) on+ `<storage>/+gitaly/graft-ref-durability/owner.lock` for its whole life.+ * Only the owner writes references without fsync, runs the durable steps,+ recovers, and removes a durable step's leftovers. A second process on the+ storage (a graceful upgrade overlaps the old and the new process) writes+ references with fsync and touches no marker or lock. It tries every second+ to take the ownership, and recovers when it does.+ * Ownership only goes from false to true within a process. So a write+ decided as deferred always runs in the owner.+ * Why this is safe:+ * The unfsynced writes, the live transactions and the syncer lock that+ the durable step reasons about all belong to one process.+ * A non-owner's writes are durable by themselves. Its compactions fsync+ the new table and list before they delete the tables they replace.+ * A new owner takes over only after the old process exited (the flock is+ released at exit), so no durable step of the old process is still+ running.+ * While an old process lives, no foreign-boot marker can exist: that+ process recovered at its own start in this boot.+ * Measured with real binaries: a second `gitaly serve` during the first's+ life, then the first killed.* *Dirty marker.* Before the first such write after a durable step, therepository gets a marker, written durably, with the kernel's boot identity(`<storage>/+gitaly/graft-ref-durability/<sha256(path)>.json`, not the state@@ -552,28 +575,49 @@ repository-wide lock.the marker cannot be written, Prepared fails and git aborts; catch-uprepairs the database's write, like any local failure.* The syncer writes it under its repository lock.- * A hook transaction counts as in flight from Prepared to Committed.- Reasons from git's source: the new table is written in- `transaction_finish` after "prepared", and auto-compaction runs inside- `reftable_addition_commit`, before "committed". An abort after Prepared- is not counted as an end (two transactions with the same input cannot be- told apart), so the repository stays dirty.+ * A hook transaction is live from its start to its end, and the durable+ step never clears the marker while any is live.+ * Start: the voter's Preparing, or `Denied` for a transaction the policy+ denies.+ * End: Committed or Aborted.+ * Every transaction starts once and ends once, so the count per input is+ the number of live transactions, whichever of two with the same input+ ends first.+ * Reasons from git's source: the new table is written in+ `transaction_finish` after "prepared", and auto-compaction runs inside+ `reftable_addition_commit`, before "committed".+ * A transaction whose end is never reported keeps its repository dirty+ (conservative).* *Durable step* (`Settle`, every second for every dirty repository, under thesyncer's repository lock).* It fsyncs the tables not yet fsynced, `tables.list` and the directory.- * When the repository has been idle for a second and nothing is in flight,- it takes git's lock without waiting (the lock is created with a tag, by- linking a temporary file). It then fsyncs what was written since, marks- the state clean, and releases the lock. Last, it removes the marker,- under a per-repository file lock that `MarkDirty` waits for.+ * When the repository has been idle for a second, or the process stops,+ and nothing is live, the step clears the marker:+ * It takes git's lock without waiting. The lock is created by linking a+ temporary file that holds a tag and a random nonce.+ * It fsyncs what was written since, marks the state clean, and releases+ the lock. The release removes the lock only if it is still the same+ file with the same nonce: an inode is reused as soon as it is freed, so+ a lock that someone else replaced is never removed.+ * Last, it removes the marker, under a per-repository file lock that+ `MarkDirty` waits for.* A busy lock leaves the marker for the next step.- * *Write failure.* A failed fsync in the durable step means the kernel may- have dropped writes. The repository is moved aside at once, under the- repository lock: the move is made durable, then the marker removed. The- failure is logged and counted (`sync_failed`), and the next request- hydrates the repository.+ * The set of fsynced tables keeps only the tables the stack still lists.+ * A graceful stop runs a last step for every dirty repository after the+ servers stopped, so a clean reboot does not rehydrate them.+ * *Write failure.* Only a failed fsync means the kernel may have dropped+ writes. The repository is then moved aside at once, under the repository+ lock: the move (both directories and the new entry) is made durable, then+ the marker removed. This happens only when the repository is a cache of the+ storage core, the same check as the recovery. The failure is logged and+ counted (`sync_failed`), and the next request hydrates the repository. Any+ other error (opening a file, reading the list) is transient: counted+ (`graft_storage_ref_durable_step_errors_total`) and retried at the next+ step.* *Recovery* (`Syncer.RecoverLocalReferences`). `gitaly serve` runs it before- any listener opens and before any background worker.+ any listener opens and before any background worker, when it owns the+ storage (`Syncer.StartLocalReferences`); otherwise it runs when the process+ takes the ownership.* A marker of another boot means the kernel restarted with writes that maynot have reached the disk. git reads a torn stack as an empty repositorywith exit code 0 (experiment 1, kept as@@ -585,8 +629,8 @@ repository-wide lock.dropped, keeping a repository whose local copy may be the only one.* A marker of the same boot is a process restart: the page cache keptevery write, so the repository stays dirty to be settled.- * A lock with the durable step's tag, left by a process that died, is- removed.+ * A lock with the durable step's tag and the step's temporary files, left+ by a process that died, are removed.* A corrupt marker fails the startup: its repository is unknown.* Not Linux (no boot identity): references stay durable per transaction.3. *Read paths.* After the recovery, no read path in authoritative mode can see@@ -605,32 +649,62 @@ repository-wide lock.* `localrefs` (`admission.go`, `durability.go`); `Core.RefAdmission`/`RefDurability`.* `voter` (`admit`, `deferDurability`, Committed/Aborted).* `syncer` (`refWriter`, `durability.go`).-* `serve.go`: `openLocalReferenceWrites`, the recovery before the listeners, the- durable-step loop.+* `serve.go`: `openLocalReferenceWrites`; `StartLocalReferences` before the+ listeners (ownership, recovery, durable-step loop), and its stop, deferred so+ that it runs after the servers stopped.* PM grant: `updateref.WithDeferredReferenceDurability`; the `gitcmd` constants`FsyncComponents`, `FsyncComponentsWithoutReferences` and`ReftableLockTimeoutMilliseconds`; `hook/storage_voter.go`- (`DeferredReferenceDurability`); the optional interface asserted in+ (`DeferredReferenceDurability`, `Denied`); the optional interface asserted in`UpdateWithHooks`.+* Follow-up after the independent review of 310facd:+ * MAJOR: process-local state with overlapping processes. Fixed by one owner+ per storage.+ * The unlock is ownership-checked (nonce and file).+ * An abort ends its transaction (the live count).+ * The fsynced-table set is pruned.+ * A final step runs on a graceful stop.+ * Only a failed fsync distrusts a repository, and only after the cache check.+ * The directory fsyncs (the marker directory's parent; the stale and temporary+ directories of a move) are added.+ * The step's temporary files are cleaned at recovery.**Tests.**-* `localrefs`: FIFO order, a cancelled waiter, the wait bound, the hold bound,- unique keys; the marker, the durable step (idle, in flight, git's lock held,- failed fsync, repository gone), recovery (process restart, kernel restart,- stale lock, corrupt marker), the torn-stack experiment.-* `voter`: committed, aborted and rejected transactions; a marker that cannot be+* `localrefs`:+ * admission: FIFO order, a cancelled waiter, the wait bound, the hold bound,+ unique keys;+ * ownership: one owner per storage; a non-owner neither defers, settles nor+ recovers; ownership taken after the owner closes; the marker directory+ created durably;+ * durable step: the marker; idle, live transactions (including two with the+ same key), git's lock held, the final step; the checked release (a+ replaced lock with a reused inode is kept); failed fsync versus a transient+ error; pruning; repository gone;+ * recovery: process restart, kernel restart, stale lock and temporary files,+ corrupt marker;+ * the torn-stack experiment.+* `voter`: committed, aborted (now settled) and rejected transactions; a denied+ and an allowed transaction with the same input; a marker that cannot bewritten; durability per mode (off, shadow and authoritative without bootidentity keep the fsync).* `updateref`: the effective `core.fsync` of `update-ref`.* `hook/updateref`: only a voter that defers turns it off.* `syncer`: catch-up marks; reboot recovery hydrates (a read before it listsnothing); process-restart recovery keeps; non-caches are kept; a failed fsync- moves aside and hydrates.-* e2e with the real `gitaly` binary: SIGKILL after writes (same boot, kept,- settled); SIGKILL in the durable step (the lock is removed at restart, writes- go on); a simulated power loss (torn table, other boot: moved aside, every- write served); stack depth under 4×30 concurrent writes (git's geometric- compaction, at most 2⌈log₂ n⌉+2 tables; measured 4 for 122 updates).+ moves aside and hydrates; a transient error is retried; a second process waits+ for the owner, then owns and settles; the stop settles a repository that is+ not idle.+* e2e with the real `gitaly` binary:+ * SIGKILL after writes: same boot, kept, settled;+ * SIGKILL in the durable step: the lock is removed at restart and writes go+ on;+ * a simulated power loss (torn table, other boot): moved aside, every write+ served;+ * a graceful-upgrade overlap: a second `gitaly serve` on the same storage+ writes with fsync while the first owns; after a SIGKILL of the first, the+ second owns and defers;+ * stack depth under 4×30 concurrent writes: git's geometric compaction, at+ most 2⌈log₂ n⌉+2 tables; measured 4 for 122 updates.**Measured** (2026-10-19, same method as the problem's numbers, alternatingbefore = b665f56 and after = WP76b, two runs each, load 3–7; mean ms, 50 / 100diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.mdindex b616dec..29cb166 100644--- a/doc/graft/STORAGE.md+++ b/doc/graft/STORAGE.md@@ -456,7 +456,18 @@ or re-import the repository from scratch (seedropped.- *A write the kernel reports as failed* (a writeback error, same boot): thedurable step's fsync fails, the repository is moved aside at once and logged,- and the next request hydrates it.+ and the next request hydrates it. Other errors of the step (a file that+ cannot be opened for a moment) are retried every second and counted.+ - *Clean stop*: after the servers stopped, `gitaly serve` makes every dirty+ repository durable and removes its marker, so a clean reboot hydrates+ nothing.+ - *Graceful upgrade* (`GITALY_UPGRADES_ENABLED`): the new process serves while+ the old one drains. One process per storage owns the deferred durability, by+ an exclusive lock on `<storage>/+gitaly/graft-ref-durability/owner.lock`.+ Until the old process exits, the new one writes references with fsync, as+ in the other modes; then it takes over (`graft_storage_ref_durability_owner`).+ During the overlap two processes also open an SQLite database, and their+ writes meet in SQLite's own lock wait, see Limits; Postgres is not affected.- Offline tools that read a repository from disk (`gitaly git -C <repo>`,`gitaly-debug simulate-http-clone`) must not be pointed at a repository ofa node after an unclean shutdown before `gitaly serve` has started once:@@ -501,7 +512,9 @@ without a single divergence.| `graft_storage_ref_admission_expired_total` | Turns ended by the 10 s bound instead of their release (a git process that died between its phases). || `graft_storage_ref_dirty_repositories` | Repositories with reference writes not yet made durable. || `graft_storage_ref_dirty_marked_total`, `graft_storage_ref_dirty_cleared_total` | Dirty markers written and removed by the durable step. |-| `graft_storage_ref_writes_inflight` | Hook transactions between their Prepared and Committed phases; one that never commits keeps its repository dirty. |+| `graft_storage_ref_writes_inflight` | Hook transactions between their start and their end (Committed or Aborted); one whose end is never reported keeps its repository dirty. |+| `graft_storage_ref_durability_owner` | 1 when this process owns the deferred durability of its storages, 0 while another process (the old one of a graceful upgrade) does. |+| `graft_storage_ref_durable_step_errors_total` | Durable steps that failed with a transient error (not a failed fsync); retried every second. || `graft_storage_ref_untrusted_total{reason}` | Repositories moved aside to be hydrated again: `reboot` (dirty at a kernel restart), `sync_failed` (the durable step's fsync failed). |## Benchmark@@ -614,9 +627,12 @@ show:after a power loss it is always hydrated again, even though its referenceswere made durable less than a second earlier: the marker does not record whichwrites are already durable.-- **SQLite serves one node.** SQLite has a single write lock. The writes of- one gitaly process take it in the order they arrive, but several processes on- one SQLite file (several nodes on one host) meet in SQLite's own lock wait,+- **SQLite serves one node.** Only `gitaly serve` opens the database: the+ `graft` CLI and the other tools go through the server. SQLite has a single+ write lock. The writes of one gitaly process take it in the order they+ arrive, but several processes on one SQLite file (several nodes on one host,+ or the old and the new process during a graceful upgrade) meet in SQLite's+ own lock wait,which keeps no queue: under steady contention an unlucky write can wait pastthe 10 s busy timeout and fail. A deployment with more than one node usesPostgres; SQLite is for one node, or for tests.diff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.goindex 61993e3..116a080 100644--- a/internal/cli/gitaly/serve.go+++ b/internal/cli/gitaly/serve.go@@ -638,12 +638,13 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {for _, s := range cfg.Storages {storageNames = append(storageNames, s.Name)}- if err := syncer.RecoverLocalReferences(ctx, storageNames); err != nil {+ stopLocalReferences, err := syncer.StartLocalReferences(ctx, storageNames)+ if err != nil {return err}- durableCtx, stopDurableSteps := context.WithCancel(ctx)- defer stopDurableSteps()- go syncer.RunDurableSteps(durableCtx, localrefs.DurableStepInterval)+ // Deferred here, so it runs after the servers stopped: a last durable step+ // settles every repository whose writes ended.+ defer stopLocalReferences()storageGate = gate.New(storageCore, syncer, logger, locator, localrepoFactory, protoregistry.GitalyProtoPreregistered)}diff --git a/internal/gitaly/hook/storage_voter.go b/internal/gitaly/hook/storage_voter.goindex eb297fd..a0fe77a 100644--- a/internal/gitaly/hook/storage_voter.go+++ b/internal/gitaly/hook/storage_voter.go@@ -15,6 +15,11 @@ type StorageVoter interface {// its pack) before git takes the reference locks. It never fails the// transaction: when it did not complete, Prepared does that work itself.Preparing(ctx context.Context, payload gitcmd.HooksPayload, changes []byte)+ // Denied is called instead of Preparing for a transaction that the+ // standalone access policy denies: it prepares nothing, and the transaction+ // is aborted in its Prepared phase. Every transaction gets exactly one of+ // Preparing and Denied before its Aborted or Committed phase.+ Denied(ctx context.Context, payload gitcmd.HooksPayload, changes []byte)// Prepared commits the transaction to the storage core while git holds// the reference locks. An error aborts the transaction.Prepared(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, meta changelog.Meta) error@@ -65,6 +70,7 @@ func (m *GitLabHookManager) storagePreparing(ctx context.Context, payload gitcmd}if message, err := m.graftPolicyDecision(env, changes); err != nil || message != "" {+ m.storageVoter.Denied(ctx, payload, changes)return}diff --git a/internal/gitaly/hook/storage_voter_test.go b/internal/gitaly/hook/storage_voter_test.goindex 7c411ad..abb3694 100644--- a/internal/gitaly/hook/storage_voter_test.go+++ b/internal/gitaly/hook/storage_voter_test.go@@ -44,6 +44,10 @@ func (v *recordingVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte)v.record("preparing")}+func (v *recordingVoter) Denied(context.Context, gitcmd.HooksPayload, []byte) {+ v.record("denied")+}+func (v *recordingVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {v.record("prepared")return nil@@ -82,9 +86,11 @@ func TestReferenceTransactionHook_storageVoterPhases(t *testing.T) {want: []string{"preparing", "prepared", "committed"},},{+ // The voter learns of its start (Denied), so that it counts the+ // transaction until its abort, but prepares nothing.desc: "a denied transaction is neither prepared nor committed",policy: []config.GraftPolicyRule{{Refs: []string{"refs/heads/main"}, Allow: false}},- want: nil,+ want: []string{"denied"},denied: true,},} {diff --git a/internal/gitaly/hook/updateref/deferred_durability_test.go b/internal/gitaly/hook/updateref/deferred_durability_test.goindex 1b9d414..a381daf 100644--- a/internal/gitaly/hook/updateref/deferred_durability_test.go+++ b/internal/gitaly/hook/updateref/deferred_durability_test.go@@ -24,6 +24,8 @@ type durabilityVoter struct {func (durabilityVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte) {}+func (durabilityVoter) Denied(context.Context, gitcmd.HooksPayload, []byte) {}+func (durabilityVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {return nil}diff --git a/internal/graft/corestore/coretest/coretest.go b/internal/graft/corestore/coretest/coretest.goindex 829d77f..f7e1da8 100644--- a/internal/graft/corestore/coretest/coretest.go+++ b/internal/graft/corestore/coretest/coretest.go@@ -20,7 +20,7 @@ import ("testing""time"- "github.com/feder-cr/git-ai/internal/gitaly/storage"+ "github.com/feder-cr/git-ai/internal/gitaly/config""github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/blobstore""github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"@@ -200,18 +200,35 @@ func Open(tb testing.TB, opts Options) *corestore.Core {}// EnableLocalRefs gives the core the admission and the deferred durability of-// the local reference writes, as gitaly serve does in authoritative mode-// (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). The boot identity is a-// fixed test value unless opts set another one.-func EnableLocalRefs(tb testing.TB, core *corestore.Core, locator storage.Locator, opts ...localrefs.DurabilityOption) {+// the local reference writes, owning the storages of cfg, as gitaly serve does+// in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). The+// boot identity is a fixed test value unless opts set another one. A deferred+// durability the core had is closed first, as when its process exits.+func EnableLocalRefs(tb testing.TB, core *corestore.Core, cfg config.Cfg, opts ...localrefs.DurabilityOption) {tb.Helper()+ if core.RefDurability != nil {+ require.NoError(tb, core.RefDurability.Close())+ }core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)- durability, err := localrefs.NewDurability(locator, 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID)}, opts...)...)+ durability, err := localrefs.NewDurability(config.NewLocator(cfg), 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID)}, opts...)...)+ require.NoError(tb, err)+ owned, err := durability.TryOwn(context.Background(), StorageNames(cfg))require.NoError(tb, err)+ require.True(tb, owned, "another durability of the test owns the storages")+ tb.Cleanup(func() { _ = durability.Close() })core.RefDurability = durability}+// StorageNames returns the names of the storages of cfg.+func StorageNames(cfg config.Cfg) []string {+ names := make([]string, 0, len(cfg.Storages))+ for _, s := range cfg.Storages {+ names = append(names, s.Name)+ }+ return names+}+// TestBootID is the boot identity EnableLocalRefs gives the deferred// durability by default.const TestBootID = "test-boot"diff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.goindex ca0a905..66147b1 100644--- a/internal/graft/corestore/localrefs/durability.go+++ b/internal/graft/corestore/localrefs/durability.go@@ -4,6 +4,7 @@ import ("bufio""bytes""context"+ "crypto/rand""crypto/sha256""encoding/hex""encoding/json"@@ -14,6 +15,8 @@ import ("path/filepath""strings""sync"+ "sync/atomic"+ "syscall""time""github.com/prometheus/client_golang/prometheus"@@ -28,13 +31,25 @@ import (// cache walker removes after an hour.const MarkerDirName = config.GitalyDataPrefix + "/graft-ref-durability"+// ownerLockName is the file in MarkerDirName whose exclusive lock makes a+// process the owner of the storage's deferred durability.+const ownerLockName = "owner.lock"++// markerTempPrefix prefixes the temporary files of marker writes.+const markerTempPrefix = ".marker-"++// stepTempPrefix prefixes the temporary files the durable step links its+// reference lock from, in the repository directory.+const stepTempPrefix = ".graft-durable-step-"+// bootIDPath is where Linux exposes the identity of the running kernel boot.const bootIDPath = "/proc/sys/kernel/random/boot_id"const (// DurableStepInterval is how often the durable step runs for every dirty// repository: it bounds how long a write failure the kernel reports goes- // unnoticed.+ // unnoticed. A process that does not own the deferred durability tries to+ // take it as often.DurableStepInterval = time.Second// DurableStepIdle is how long a dirty repository must be without a reference// write before the durable step removes its marker.@@ -52,14 +67,16 @@ const (crashExitCode = 75)-// stackLockContent is what the durable step writes into the reference lock-// it takes. git never writes it (a lock of git holds table names), so a lock-// with this content found at startup was left by a durable step that died.-const stackLockContent = "graft durable step\n"+// stackLockTag starts the content of the reference lock the durable step+// takes, followed by a random nonce of that lock. git never writes it (a lock+// of git holds table names), so a lock with this tag found by a new owner was+// left by a durable step that died, and a step removes a lock only when it+// still holds its own nonce.+const stackLockTag = "graft durable step "-// ErrUntrusted is returned by Settle when making the references of a-// repository durable failed: the kernel may have dropped some of their writes,-// so the local repository must not be served before it is rebuilt.+// ErrUntrusted is returned by Settle when the kernel reported that making the+// references of a repository durable failed: it may have dropped some of their+// writes, so the local repository must not be served before it is rebuilt.var ErrUntrusted = errors.New("storage core: the local references may have lost writes")// UntrustedError is ErrUntrusted for one repository.@@ -76,23 +93,37 @@ func (e *UntrustedError) Error() string {// Unwrap returns ErrUntrusted and the cause.func (e *UntrustedError) Unwrap() []error { return []error{ErrUntrusted, e.Err} }+// writebackError is an fsync that failed: the kernel reports a write it could+// not make durable. Every other error of the durable step (opening a file,+// reading the table list) is transient and retried at the next step.+type writebackError struct{ err error }++func (e writebackError) Error() string { return "fsync: " + e.err.Error() }+func (e writebackError) Unwrap() error { return e.err }+// Durability defers the durability of the local reference writes of the// storage core in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE// WRITES RULE). The database commit of a transaction is its durable commit// point and the local repository is a cache of it, so git writes the references// without fsync and Durability keeps the cache safe to serve://+// - one process per storage owns the deferred durability (an exclusive lock+// held for the life of the process); only the owner writes references+// without fsync, runs the durable steps and touches the durable step's+// locks. Another process on the same storage (a graceful upgrade overlaps+// the old and the new process) writes them with fsync until it owns;// - before the first reference write that is not durable, the repository// gets a dirty marker, written durably, that holds the identity of the// kernel boot (MarkDirty);// - a durable step makes the references durable and removes the marker,-// under git's reference lock and only when no writer is between its-// Prepared and Committed phases (Settle);-// - at startup, a marker of another boot means the kernel restarted while-// the repository had writes that may never have reached the disk: its-// reftable stack may be torn, which git reads as an empty repository, so-// the repository is moved aside and hydrated again (Recover). A marker of-// the running boot is a process restart: the kernel kept every write.+// under git's reference lock and only when no transaction of the process is+// live (Settle);+// - when it takes the ownership, a marker of another boot means the kernel+// restarted while the repository had writes that may never have reached the+// disk: its reftable stack may be torn, which git reads as an empty+// repository, so the repository is moved aside and hydrated again+// (Recover). A marker of the running boot is a process restart: the kernel+// kept every write.//// A nil *Durability is off: every reference write is durable by itself// (core.fsync with references), as outside authoritative mode.@@ -107,16 +138,24 @@ type Durability struct {// crashAt is the crash point selected with CrashEnv.crashAt string+ // owner is set once the process owns the deferred durability of its+ // storages; it never goes back. ownerFiles hold the owner locks.+ owner atomic.Bool+ ownerMu sync.Mutex+ ownerFiles []*os.File+mu sync.Mutex// repos holds the state of every repository written since the process// started (one entry per repository; Forget drops it).repos map[string]*refState- marked prometheus.Counter- cleared prometheus.Counter- untrusted *prometheus.CounterVec- dirty prometheus.Gauge- inflight prometheus.Gauge+ marked prometheus.Counter+ cleared prometheus.Counter+ untrusted *prometheus.CounterVec+ stepErrors prometheus.Counter+ dirty prometheus.Gauge+ inflight prometheus.Gauge+ owning prometheus.Gauge}var _ prometheus.Collector = (*Durability)(nil)@@ -130,14 +169,15 @@ type refState struct {repo RepoRefmarker stringdirty bool- // inflight counts, by key, the hook transactions between their Prepared- // and Committed phases: git may still write their table and compact the- // stack without fsync.- inflight map[Key]int- nInflight int+ // live counts, by key, the hook transactions of the process from their+ // start (Preparing, or a policy denial) to their end (Committed or+ // Aborted): any of them may still write a table or compact the stack+ // without fsync.+ live map[Key]int+ nLive intlastWrite time.Time- // synced holds the tables made durable during the current dirty period.- // Tables are immutable, so one fsync each is enough.+ // synced holds the tables made durable during the current dirty period that+ // the stack still lists. Tables are immutable, so one fsync each is enough.synced map[string]struct{}}@@ -225,21 +265,92 @@ func newDurability(locator storage.Locator, idle time.Duration) *Durability {Name: "graft_storage_ref_untrusted_total",Help: "Number of local repositories whose references could not be trusted and were moved aside to be hydrated again, by reason (reboot, sync_failed).",}, []string{"reason"}),+ stepErrors: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "graft_storage_ref_durable_step_errors_total",+ Help: "Number of durable steps that failed with a transient error (not a failed fsync) and are retried at the next step.",+ }),dirty: prometheus.NewGauge(prometheus.GaugeOpts{Name: "graft_storage_ref_dirty_repositories",Help: "Number of local repositories with reference writes that are not durable yet.",}),inflight: prometheus.NewGauge(prometheus.GaugeOpts{Name: "graft_storage_ref_writes_inflight",- Help: "Number of hook reference transactions between their Prepared and Committed phases. One that never reports Committed keeps its repository dirty.",+ Help: "Number of hook reference transactions between their start and their end. One that never ends keeps its repository dirty.",+ }),+ owning: prometheus.NewGauge(prometheus.GaugeOpts{+ Name: "graft_storage_ref_durability_owner",+ Help: "1 when this process owns the deferred durability of its storages and writes references without fsync, 0 while another process owns it.",}),}}-// Enabled reports whether the reference writes of the storage core are made-// without fsync.+// Enabled reports whether the node defers the durability of its reference+// writes once it owns it.func (d *Durability) Enabled() bool { return d != nil }+// Deferring reports whether this process owns the deferred durability, so its+// reference writes are made without fsync. It goes from false to true at most+// once.+func (d *Durability) Deferring() bool { return d != nil && d.owner.Load() }++// TryOwn takes the ownership of the deferred durability of the storages+// without waiting. It returns false when another process holds it for one of+// them; the locks taken are kept, and a later call takes the rest.+func (d *Durability) TryOwn(ctx context.Context, storages []string) (bool, error) {+ if d == nil {+ return false, nil+ }+ if d.owner.Load() {+ return true, nil+ }++ d.ownerMu.Lock()+ defer d.ownerMu.Unlock()++ for _, storageName := range storages[len(d.ownerFiles):] {+ dir, err := d.markerDir(ctx, storageName)+ if err != nil {+ return false, err+ }+ if err := d.ensureDir(dir); err != nil {+ return false, fmt.Errorf("create the dirty marker directory: %w", err)+ }+ f, err := os.OpenFile(filepath.Join(dir, ownerLockName), os.O_RDWR|os.O_CREATE, 0o600)+ if err != nil {+ return false, fmt.Errorf("open the owner lock: %w", err)+ }+ if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {+ _ = f.Close()+ if errors.Is(err, syscall.EWOULDBLOCK) {+ return false, nil+ }+ return false, fmt.Errorf("lock the owner lock: %w", err)+ }+ d.ownerFiles = append(d.ownerFiles, f)+ }++ d.owner.Store(true)+ d.owning.Set(1)+ return true, nil+}++// Close releases the ownership. The process must not write references+// without fsync afterwards; only tests and the end of the process call it.+func (d *Durability) Close() error {+ if d == nil {+ return nil+ }++ d.ownerMu.Lock()+ defer d.ownerMu.Unlock()+ var errs []error+ for _, f := range d.ownerFiles {+ errs = append(errs, f.Close())+ }+ d.ownerFiles = nil+ return errors.Join(errs...)+}+// markerPath returns the path of the dirty marker of the repository at path.func (d *Durability) markerPath(ctx context.Context, repo RepoRef, path string) (string, error) {dir, err := d.markerDir(ctx, repo.Storage)@@ -263,7 +374,7 @@ func (d *Durability) markerDir(ctx context.Context, storageName string) (string,func (d *Durability) stateLocked(path string) *refState {st := d.repos[path]if st == nil {- st = &refState{inflight: map[Key]int{}, synced: map[string]struct{}{}}+ st = &refState{live: map[Key]int{}, synced: map[string]struct{}{}}d.repos[path] = st}return st@@ -271,7 +382,8 @@ func (d *Durability) stateLocked(path string) *refState {// MarkDirty records that the repository at path is about to get reference// writes that are not durable. When it returns nil the repository has a durable-// dirty marker; the writer must not write before. The caller holds git's+// dirty marker; the writer must not write before. It is only called while the+// process owns the deferred durability (Deferring). The caller holds git's// reference lock of the repository (the voter's Prepared phase) or the syncer's// repository lock, under which Settle decides to clear the marker, and the// marker file is only written or removed under the state's file lock, so a@@ -318,9 +430,12 @@ func (d *Durability) MarkDirty(ctx context.Context, repo RepoRef, path string) ereturn nil}-// Begin records that the hook transaction key is past its Prepared phase on-// the repository at path: git may write its table and compact the stack until-// it reports Committed (End). It is called under git's reference lock.+// Begin records that the hook transaction key started on the repository at+// path: from its Preparing phase (or its policy denial, which aborts it), before+// git takes the reference lock, to its end (End), it may write a table and+// compact the stack. Every transaction begins exactly once and ends exactly+// once (Committed or Aborted), so the count of a key is the number of its live+// transactions, whichever of two transactions with the same key ends first.func (d *Durability) Begin(path string, key Key) {if d == nil {return@@ -329,15 +444,14 @@ func (d *Durability) Begin(path string, key Key) {d.mu.Lock()defer d.mu.Unlock()st := d.stateLocked(path)- st.inflight[key]++- st.nInflight+++ st.live[key]+++ st.nLive++d.inflight.Inc()}-// End records that the hook transaction key committed. A transaction aborted-// after its Prepared phase is not ended: two transactions with the same key-// cannot be told apart, and ending the wrong one would let Settle remove the-// marker while the other one still writes. Its repository stays dirty instead.+// End records that a hook transaction with key ended (Committed or Aborted).+// An end without a begin (the process started after the transaction began)+// changes nothing.func (d *Durability) End(path string, key Key) {if d == nil {return@@ -346,14 +460,14 @@ func (d *Durability) End(path string, key Key) {d.mu.Lock()defer d.mu.Unlock()st := d.repos[path]- if st == nil || st.inflight[key] == 0 {+ if st == nil || st.live[key] == 0 {return}- st.inflight[key]--- if st.inflight[key] == 0 {- delete(st.inflight, key)+ st.live[key]--+ if st.live[key] == 0 {+ delete(st.live, key)}- st.nInflight--+ st.nLive--d.inflight.Dec()}@@ -374,20 +488,43 @@ func (d *Durability) Dirty() []string {return paths}-// Settle is the durable step of the repository at path. It makes the tables-// that the stack lists, the list and the directory durable, which reports a-// write the kernel failed to do (an *UntrustedError: the caller moves the-// repository aside at once). When the repository has been idle for the idle-// period and no hook transaction is in flight, it then removes the marker:-// the decision is taken under git's reference lock, taken without waiting (a-// busy lock leaves the marker for the next step), and the marker file is-// removed after the lock is released, under the state's file lock that a-// writer's MarkDirty waits for. A repository that no longer exists (moved-// aside as a stale cache) loses its marker. The caller holds the syncer's-// repository lock, so no syncer write and no replacement of the repository-// runs meanwhile.-func (d *Durability) Settle(path string) error {+// RepoOf returns the repository whose state is kept for path.+func (d *Durability) RepoOf(path string) (RepoRef, bool) {+ if d == nil {+ return RepoRef{}, false+ }++ d.mu.Lock()+ defer d.mu.Unlock()+ st := d.repos[path]+ if st == nil || st.repo == (RepoRef{}) {+ return RepoRef{}, false+ }+ return st.repo, true+}++// CountStepError counts a durable step that failed with a transient error.+func (d *Durability) CountStepError() {if d == nil {+ return+ }+ d.stepErrors.Inc()+}++// Settle is the durable step of the repository at path; only the owner runs+// it. It makes the tables that the stack lists, the list and the directory+// durable. A failed fsync is an *UntrustedError (the caller moves the+// repository aside at once); any other error is transient. When the repository+// has been idle for the idle period, or final is set (the process stops), and+// no transaction of the process is live, it then removes the marker: the+// decision is taken under git's reference lock, taken without waiting (a busy+// lock leaves the marker for the next step), and the marker file is removed+// after the lock is released, under the state's file lock that a writer's+// MarkDirty waits for. A repository that no longer exists (moved aside as a+// stale cache) loses its marker. The caller holds the syncer's repository lock,+// so no syncer write and no replacement of the repository runs meanwhile.+func (d *Durability) Settle(path string, final bool) error {+ if !d.Deferring() {return nil}@@ -398,7 +535,7 @@ func (d *Durability) Settle(path string) error {return nil}repo := st.repo- idle := st.nInflight == 0 && time.Since(st.lastWrite) >= d.idle+ idle := st.nLive == 0 && (final || time.Since(st.lastWrite) >= d.idle)d.mu.Unlock()if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {@@ -409,13 +546,13 @@ func (d *Durability) Settle(path string) error {}if err := d.syncStack(path, st); err != nil {- return &UntrustedError{Repo: repo, Path: path, Err: err}+ return d.stepError(repo, path, err)}if !idle {return nil}- unlock, ok, err := lockStack(path)+ lock, ok, err := lockStack(path)if err != nil {return err}@@ -425,31 +562,41 @@ func (d *Durability) Settle(path string) error {cleared, err := d.settleLocked(path, st)if err != nil {- _ = unlock()- return &UntrustedError{Repo: repo, Path: path, Err: err}+ _ = lock.release()+ return d.stepError(repo, path, err)}if !cleared {- return unlock()+ return lock.release()}// The state is clean from here on, and a writer that takes git's lock now// writes a new marker: it waits for the file lock, held until the old// marker is removed.defer st.file.Unlock()- if err := unlock(); err != nil {+ if err := lock.release(); err != nil {return err}return d.removeMarker(st)}+// stepError is the error of a durable step: an *UntrustedError for a failed+// fsync, the error itself otherwise.+func (d *Durability) stepError(repo RepoRef, path string, err error) error {+ var writeback writebackError+ if errors.As(err, &writeback) {+ return &UntrustedError{Repo: repo, Path: path, Err: err}+ }+ return err+}+// settleLocked makes the tables written since the first pass durable and, when-// no hook transaction is in flight, marks the state clean with its file lock-// held (the caller releases it). The caller holds git's reference lock: no-// transaction can pass its Prepared phase meanwhile, and the transactions-// counted before have committed when the count is zero.+// no transaction of the process is live, marks the state clean with its file+// lock held (the caller releases it). The caller holds git's reference lock: no+// transaction can take it meanwhile, and the live transactions are counted from+// before they take it until their end.func (d *Durability) settleLocked(path string, st *refState) (bool, error) {d.mu.Lock()- busy := st.nInflight > 0+ busy := st.nLive > 0d.mu.Unlock()if busy {return false, nil@@ -491,19 +638,41 @@ func (d *Durability) removeMarker(st *refState) error {return nil}+// stackLock is git's reference lock of a repository, taken by the durable+// step.+type stackLock struct {+ path string+ // content is the tag and the nonce the step wrote into it, and info the file+ // it created: a lock is the step's own only when both still match (an inode+ // is reused as soon as it is freed, so the file alone does not tell).+ content []byte+ info os.FileInfo+}+// lockStack takes git's reference lock of the repository at path without// waiting: ok is false when it is held. The lock is created with its content// at once (a linked temporary file), so that a lock left by a process that-// died is recognized at startup (removeStaleStackLock).-func lockStack(path string) (unlock func() error, ok bool, err error) {+// died is recognized by the next owner (removeStaleStackLock).+func lockStack(path string) (*stackLock, bool, error) {lockPath := filepath.Join(path, "reftable", "tables.list.lock")- tmp, err := os.CreateTemp(path, ".graft-durable-step-*")+ nonce := make([]byte, 16)+ if _, err := rand.Read(nonce); err != nil {+ return nil, false, fmt.Errorf("take the reference lock: %w", err)+ }+ content := []byte(stackLockTag + hex.EncodeToString(nonce) + "\n")++ tmp, err := os.CreateTemp(path, stepTempPrefix+"*")if err != nil {return nil, false, fmt.Errorf("take the reference lock: %w", err)}defer func() { _ = os.Remove(tmp.Name()) }()- if _, err := tmp.WriteString(stackLockContent); err != nil {+ if _, err := tmp.Write(content); err != nil {+ _ = tmp.Close()+ return nil, false, fmt.Errorf("take the reference lock: %w", err)+ }+ info, err := tmp.Stat()+ if err != nil {_ = tmp.Close()return nil, false, fmt.Errorf("take the reference lock: %w", err)}@@ -517,18 +686,41 @@ func lockStack(path string) (unlock func() error, ok bool, err error) {}return nil, false, fmt.Errorf("take the reference lock: %w", err)}- return func() error {- if err := os.Remove(lockPath); err != nil {- return fmt.Errorf("release the reference lock: %w", err)- }+ return &stackLock{path: lockPath, content: content, info: info}, true, nil+}++// release removes the lock if it is still the one the step created (same file,+// same nonce): a lock that someone else removed (housekeeping's stale-lock+// cleanup) and that git or another step took again is never removed.+func (l *stackLock) release() error {+ current, err := os.Lstat(l.path)+ if errors.Is(err, fs.ErrNotExist) {+ return nil+ }+ if err != nil {+ return fmt.Errorf("release the reference lock: %w", err)+ }+ content, err := os.ReadFile(l.path)+ if errors.Is(err, fs.ErrNotExist) {return nil- }, true, nil+ }+ if err != nil {+ return fmt.Errorf("release the reference lock: %w", err)+ }+ if !os.SameFile(current, l.info) || !bytes.Equal(content, l.content) {+ return nil+ }+ if err := os.Remove(l.path); err != nil && !errors.Is(err, fs.ErrNotExist) {+ return fmt.Errorf("release the reference lock: %w", err)+ }+ return nil}// removeStaleStackLock removes the reference lock of the repository at path// when a durable step of a process that died left it: git's writes would fail// on it until housekeeping removes stale locks. A lock of git is never-// removed.+// removed. Only the owner calls it, when it takes the ownership: the durable+// steps of the process that owned the storage before have ended with it.func removeStaleStackLock(path string) error {lockPath := filepath.Join(path, "reftable", "tables.list.lock")content, err := os.ReadFile(lockPath)@@ -538,7 +730,7 @@ func removeStaleStackLock(path string) error {if err != nil {return fmt.Errorf("read the reference lock: %w", err)}- if !bytes.Equal(content, []byte(stackLockContent)) {+ if !bytes.HasPrefix(content, []byte(stackLockTag)) {return nil}if err := os.Remove(lockPath); err != nil && !errors.Is(err, fs.ErrNotExist) {@@ -547,8 +739,24 @@ func removeStaleStackLock(path string) error {return nil}+// removeStepTemporaries removes the temporary files that a durable step of a+// process that died left in the repository at path.+func removeStepTemporaries(path string) error {+ leftovers, err := filepath.Glob(filepath.Join(path, stepTempPrefix+"*"))+ if err != nil {+ return err+ }+ for _, leftover := range leftovers {+ if err := os.Remove(leftover); err != nil && !errors.Is(err, fs.ErrNotExist) {+ return fmt.Errorf("remove a temporary file of a durable step: %w", err)+ }+ }+ return nil+}+// syncStack makes durable the tables of the stack not made durable yet in this-// dirty period, then the list and the directory.+// dirty period, then the list and the directory. synced keeps only the tables+// the stack still lists.func (d *Durability) syncStack(path string, st *refState) error {reftable := filepath.Join(path, "reftable")names, err := readTablesList(filepath.Join(reftable, "tables.list"))@@ -557,12 +765,19 @@ func (d *Durability) syncStack(path string, st *refState) error {}d.mu.Lock()+ listed := make(map[string]struct{}, len(names))var pending []stringfor _, name := range names {+ listed[name] = struct{}{}if _, ok := st.synced[name]; !ok {pending = append(pending, name)}}+ for name := range st.synced {+ if _, ok := listed[name]; !ok {+ delete(st.synced, name)+ }+ }d.mu.Unlock()for _, name := range pending {@@ -618,13 +833,13 @@ func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) erroif st.dirty {d.dirty.Dec()}- d.inflight.Sub(float64(st.nInflight))+ d.inflight.Sub(float64(st.nLive))delete(d.repos, path)}return nil}-// Recovered is a dirty repository found at startup.+// Recovered is a dirty repository found by Recover.type Recovered struct {Repo RepoRefPath string@@ -633,14 +848,15 @@ type Recovered struct {Reboot bool}-// Recover reads the dirty markers of the storages. A marker of the running-// boot makes its repository dirty again, to be settled, and a reference lock-// that a durable step of the previous process left is removed. It returns-// every repository found; the caller moves aside the ones of another boot-// before serving. A marker that cannot be read is an error: its repository is-// unknown, and serving could read a torn stack.+// Recover reads the dirty markers of the storages; only the owner calls it,+// when it takes the ownership. A marker of the running boot makes its+// repository dirty again, to be settled, and what a durable step of the+// previous owner left (its reference lock, its temporary files) is removed. It+// returns every repository found; the caller moves aside the ones of another+// boot before serving. A marker that cannot be read is an error: its+// repository is unknown, and serving could read a torn stack.func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovered, error) {- if d == nil {+ if !d.Deferring() {return nil, nil}@@ -660,7 +876,10 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recoverefor _, entry := range entries {markerPath := filepath.Join(dir, entry.Name())- if !strings.HasSuffix(entry.Name(), ".json") {+ if entry.Name() == ownerLockName {+ continue+ }+ if strings.HasPrefix(entry.Name(), markerTempPrefix) {// A temporary file of a marker write that did not complete: the// write never returned, so no reference write relied on it.if err := os.Remove(markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {@@ -668,6 +887,9 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovere}continue}+ if !strings.HasSuffix(entry.Name(), ".json") {+ return nil, fmt.Errorf("unexpected file %q in the dirty marker directory", markerPath)+ }content, err := os.ReadFile(markerPath)if err != nil {@@ -688,6 +910,9 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovereif err := removeStaleStackLock(path); err != nil {return nil, err}+ if err := removeStepTemporaries(path); err != nil {+ return nil, err+ }d.mu.Lock()st := d.stateLocked(path)if !st.dirty {@@ -712,15 +937,38 @@ func (d *Durability) CountUntrusted(reason string) {d.untrusted.WithLabelValues(reason).Inc()}+// ensureDir creates the directory dir when it does not exist, and makes its+// entry and the entry of a parent it created durable.+func (d *Durability) ensureDir(dir string) error {+ if _, err := os.Stat(dir); err == nil {+ return nil+ } else if !errors.Is(err, fs.ErrNotExist) {+ return err+ }++ parent := filepath.Dir(dir)+ _, parentErr := os.Stat(parent)+ if err := os.MkdirAll(dir, 0o700); err != nil {+ return err+ }+ if err := d.syncPath(parent); err != nil {+ return err+ }+ if errors.Is(parentErr, fs.ErrNotExist) {+ return d.syncPath(filepath.Dir(parent))+ }+ return nil+}+// writeDurably writes content to path through a temporary file in the same// directory, made durable before and after the rename.func (d *Durability) writeDurably(path string, content []byte) (returnedErr error) {dir := filepath.Dir(path)- if err := os.MkdirAll(dir, 0o700); err != nil {+ if err := d.ensureDir(dir); err != nil {return err}- tmp, err := os.CreateTemp(dir, ".marker-*.tmp")+ tmp, err := os.CreateTemp(dir, markerTempPrefix+"*.tmp")if err != nil {return err}@@ -735,7 +983,7 @@ func (d *Durability) writeDurably(path string, content []byte) (returnedErr erroreturn err}if err := d.sync(tmp); err != nil {- return err+ return writebackError{err: err}}if err := tmp.Close(); err != nil {return err@@ -746,14 +994,18 @@ func (d *Durability) writeDurably(path string, content []byte) (returnedErr erroreturn d.syncPath(dir)}-// syncPath makes the file or directory at path durable.+// syncPath makes the file or directory at path durable. A failed open is+// returned as it is; a failed fsync is a writebackError.func (d *Durability) syncPath(path string) error {f, err := os.Open(path)if err != nil {return err}defer f.Close()- return d.sync(f)+ if err := d.sync(f); err != nil {+ return writebackError{err: err}+ }+ return nil}// readTablesList returns the table names of a reftable stack.@@ -779,8 +1031,10 @@ func (d *Durability) Describe(descs chan<- *prometheus.Desc) {d.marked.Describe(descs)d.cleared.Describe(descs)d.untrusted.Describe(descs)+ d.stepErrors.Describe(descs)d.dirty.Describe(descs)d.inflight.Describe(descs)+ d.owning.Describe(descs)}// Collect implements prometheus.Collector.@@ -788,6 +1042,8 @@ func (d *Durability) Collect(metrics chan<- prometheus.Metric) {d.marked.Collect(metrics)d.cleared.Collect(metrics)d.untrusted.Collect(metrics)+ d.stepErrors.Collect(metrics)d.dirty.Collect(metrics)d.inflight.Collect(metrics)+ d.owning.Collect(metrics)}diff --git a/internal/graft/corestore/localrefs/durability_test.go b/internal/graft/corestore/localrefs/durability_test.goindex 09a2b6f..74434c1 100644--- a/internal/graft/corestore/localrefs/durability_test.go+++ b/internal/graft/corestore/localrefs/durability_test.go@@ -86,10 +86,23 @@ func setupDurability(t *testing.T) durabilityFixture {}}-func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration) *Durability {+// unowned returns a durability of the fixture's storage that does not own it.+func (f durabilityFixture) unowned(t *testing.T, bootID string, idle time.Duration) *Durability {t.Helper()d, err := NewDurability(f.locator, idle, WithBootID(bootID), WithSync(f.synced.sync))require.NoError(t, err)+ t.Cleanup(func() { require.NoError(t, d.Close()) })+ return d+}++// durability returns a durability that owns the fixture's storage, as the+// process that runs the durable steps. Close it before another one owns.+func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration) *Durability {+ t.Helper()+ d := f.unowned(t, bootID, idle)+ owned, err := d.TryOwn(f.ctx, []string{f.repo.Storage})+ require.NoError(t, err)+ require.True(t, owned)return d}@@ -112,6 +125,7 @@ func TestMarkDirtyWritesOneDurableMarker(t *testing.T) {t.Parallel()f := setupDurability(t)d := f.durability(t, "boot-a", 0)+ before := len(f.synced.recorded())require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))@@ -123,20 +137,71 @@ func TestMarkDirtyWritesOneDurableMarker(t *testing.T) {require.Equal(t, []string{f.path}, d.Dirty())// The marker file and its directory were made durable before MarkDirty- // returned.- recorded := f.synced.recorded()+ // returned, once.+ recorded := f.synced.recorded()[before:]require.Len(t, recorded, 2)require.Equal(t, filepath.Dir(f.markerPath(t, d)), recorded[1])require.Equal(t, filepath.Join(f.cfg.Storages[0].Path, MarkerDirName), filepath.Dir(f.markerPath(t, d)))}+func TestOwnershipCreatesTheMarkerDirectoryDurably(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ f.durability(t, "boot-a", 0)++ dir := filepath.Join(f.cfg.Storages[0].Path, MarkerDirName)+ require.DirExists(t, dir)+ require.Contains(t, f.synced.recorded(), filepath.Dir(dir), "the new directory's entry is durable")+}++// TestOneOwnerPerStorage is the overlap of a graceful upgrade: two processes+// on one storage. Only the owner defers, settles and recovers; the other one+// writes with fsync until the owner is gone.+func TestOneOwnerPerStorage(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ storages := []string{f.repo.Storage}++ old := f.durability(t, "boot-a", 0)+ require.True(t, old.Deferring())+ require.NoError(t, old.MarkDirty(f.ctx, f.repo, f.path))+ old.Begin(f.path, key(1))++ next := f.unowned(t, "boot-a", 0)+ owned, err := next.TryOwn(f.ctx, storages)+ require.NoError(t, err)+ require.False(t, owned, "the old process owns the storage")+ require.False(t, next.Deferring())+ require.Equal(t, 0.0, testutil.ToFloat64(next.owning))++ // The next process neither recovers nor settles: the old process's+ // transaction is live, and only the old process counts it.+ found, err := next.Recover(f.ctx, storages)+ require.NoError(t, err)+ require.Empty(t, found)+ require.NoError(t, next.Settle(f.path, true))+ require.FileExists(t, f.markerPath(t, old))++ // The old process exits; the next one owns and recovers the marker.+ require.NoError(t, old.Close())+ owned, err = next.TryOwn(f.ctx, storages)+ require.NoError(t, err)+ require.True(t, owned)+ require.True(t, next.Deferring())+ found, err = next.Recover(f.ctx, storages)+ require.NoError(t, err)+ require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path}}, found)+ require.NoError(t, next.Settle(f.path, false))+ require.NoFileExists(t, f.markerPath(t, next))+}+func TestSettleMakesTheStackDurableAndRemovesTheMarker(t *testing.T) {t.Parallel()f := setupDurability(t)d := f.durability(t, "boot-a", 0)require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))- require.NoError(t, d.Settle(f.path))+ require.NoError(t, d.Settle(f.path, false))require.NoFileExists(t, f.markerPath(t, d))require.Empty(t, d.Dirty())@@ -149,8 +214,11 @@ func TestSettleMakesTheStackDurableAndRemovesTheMarker(t *testing.T) {}require.Contains(t, recorded, filepath.Join(f.path, "reftable", "tables.list"))require.Contains(t, recorded, filepath.Join(f.path, "reftable"))- // The lock taken for the removal is released.+ // The lock taken for the removal is released, and its temporary file gone.require.NoFileExists(t, filepath.Join(f.path, "reftable", "tables.list.lock"))+ leftovers, err := filepath.Glob(filepath.Join(f.path, stepTempPrefix+"*"))+ require.NoError(t, err)+ require.Empty(t, leftovers)}func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {@@ -162,33 +230,55 @@ func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {d := f.durability(t, "boot-a", time.Hour)require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))- require.NoError(t, d.Settle(f.path))+ require.NoError(t, d.Settle(f.path, false))require.FileExists(t, f.markerPath(t, d))// The stack was made durable all the same.require.Contains(t, f.synced.recorded(), filepath.Join(f.path, "reftable", f.tables(t)[0]))++ // The last step of a stopping process does not wait for idleness.+ require.NoError(t, d.Settle(f.path, true))+ require.NoFileExists(t, f.markerPath(t, d))})- t.Run("a transaction between Prepared and Committed", func(t *testing.T) {+ t.Run("a live transaction", func(t *testing.T) {t.Parallel()f := setupDurability(t)d := f.durability(t, "boot-a", 0)require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))d.Begin(f.path, key(1))- require.NoError(t, d.Settle(f.path))- require.FileExists(t, f.markerPath(t, d))+ require.NoError(t, d.Settle(f.path, true))+ require.FileExists(t, f.markerPath(t, d), "not even the last step removes it")// An end of another key does not count; the end of the transaction does.d.End(f.path, key(2))- require.NoError(t, d.Settle(f.path))+ require.NoError(t, d.Settle(f.path, false))require.FileExists(t, f.markerPath(t, d))d.End(f.path, key(1))- require.NoError(t, d.Settle(f.path))+ require.NoError(t, d.Settle(f.path, false))require.NoFileExists(t, f.markerPath(t, d))require.Equal(t, 0.0, testutil.ToFloat64(d.inflight))})+ t.Run("two live transactions with the same key", func(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ // Whichever of the two ends first (an abort before Prepared, say), the+ // other one is still counted.+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ d.Begin(f.path, key(1))+ d.Begin(f.path, key(1))+ d.End(f.path, key(1))+ require.NoError(t, d.Settle(f.path, false))+ require.FileExists(t, f.markerPath(t, d))+ d.End(f.path, key(1))+ require.NoError(t, d.Settle(f.path, false))+ require.NoFileExists(t, f.markerPath(t, d))+ })+t.Run("git holds the reference lock", func(t *testing.T) {t.Parallel()f := setupDurability(t)@@ -198,12 +288,42 @@ func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {lock := filepath.Join(f.path, "reftable", "tables.list.lock")require.NoError(t, os.WriteFile(lock, nil, 0o666))- require.NoError(t, d.Settle(f.path))+ require.NoError(t, d.Settle(f.path, false))require.FileExists(t, f.markerPath(t, d))require.FileExists(t, lock, "the durable step removed a lock it did not take")})}+// TestStackLockReleaseRemovesOnlyItsOwnLock verifies that the durable step+// never removes a reference lock that is not the one it created: one that+// someone else removed and git took again stays.+func TestStackLockReleaseRemovesOnlyItsOwnLock(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ lockPath := filepath.Join(f.path, "reftable", "tables.list.lock")++ lock, ok, err := lockStack(f.path)+ require.NoError(t, err)+ require.True(t, ok)+ _, ok, err = lockStack(f.path)+ require.NoError(t, err)+ require.False(t, ok, "the lock is exclusive")++ // Housekeeping removed the lock as stale, and another durable step took it+ // (the freed inode may well be reused for it).+ require.NoError(t, os.Remove(lockPath))+ require.NoError(t, os.WriteFile(lockPath, []byte(stackLockTag+"0123456789abcdef\n"), 0o666))+ require.NoError(t, lock.release())+ require.FileExists(t, lockPath, "a lock of another holder, even with the same tag, is not removed")++ require.NoError(t, os.Remove(lockPath))+ lock, ok, err = lockStack(f.path)+ require.NoError(t, err)+ require.True(t, ok)+ require.NoError(t, lock.release())+ require.NoFileExists(t, lockPath)+}+func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {t.Parallel()f := setupDurability(t)@@ -213,7 +333,7 @@ func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {writeback := errors.New("input/output error")f.synced.fail(writeback)- err := d.Settle(f.path)+ err := d.Settle(f.path, false)require.ErrorIs(t, err, ErrUntrusted)require.ErrorIs(t, err, writeback)var untrusted *UntrustedError@@ -225,12 +345,52 @@ func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {require.FileExists(t, f.markerPath(t, d))require.Equal(t, []string{f.path}, d.Dirty())+ f.synced.fail(nil)require.NoError(t, d.Forget(f.ctx, f.repo, f.path))require.NoFileExists(t, f.markerPath(t, d))require.Empty(t, d.Dirty())require.Equal(t, 0.0, testutil.ToFloat64(d.dirty))}+// TestSettleTransientErrorIsNotUntrusted verifies that a failure to read the+// stack (not a failed fsync) is retried: the repository is not distrusted.+func TestSettleTransientErrorIsNotUntrusted(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))++ list := filepath.Join(f.path, "reftable", "tables.list")+ require.NoError(t, os.Rename(list, list+".away"))+ err := d.Settle(f.path, false)+ require.Error(t, err)+ require.NotErrorIs(t, err, ErrUntrusted)+ require.FileExists(t, f.markerPath(t, d))++ require.NoError(t, os.Rename(list+".away", list))+ require.NoError(t, d.Settle(f.path, false))+ require.NoFileExists(t, f.markerPath(t, d))+}++// TestSettleForgetsTablesTheStackNoLongerLists verifies that the set of tables+// made durable does not grow with tables compacted away.+func TestSettleForgetsTablesTheStackNoLongerLists(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", time.Hour)+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))++ d.mu.Lock()+ d.repos[f.path].synced["0x000000000001-0x000000000001-compacted.ref"] = struct{}{}+ d.mu.Unlock()++ require.NoError(t, d.Settle(f.path, false))+ d.mu.Lock()+ defer d.mu.Unlock()+ require.Len(t, d.repos[f.path].synced, len(f.tables(t)))+ require.NotContains(t, d.repos[f.path].synced, "0x000000000001-0x000000000001-compacted.ref")+}+func TestSettleDropsTheMarkerOfARepositoryThatIsGone(t *testing.T) {t.Parallel()f := setupDurability(t)@@ -240,7 +400,7 @@ func TestSettleDropsTheMarkerOfARepositoryThatIsGone(t *testing.T) {require.NoError(t, os.Rename(f.path, f.path+".moved"))t.Cleanup(func() { require.NoError(t, os.Rename(f.path+".moved", f.path)) })- require.NoError(t, d.Settle(f.path))+ require.NoError(t, d.Settle(f.path, false))require.NoFileExists(t, f.markerPath(t, d))require.Empty(t, d.Dirty())}@@ -248,9 +408,11 @@ func TestSettleDropsTheMarkerOfARepositoryThatIsGone(t *testing.T) {func TestRecoverTellsAKernelRestartFromAProcessRestart(t *testing.T) {t.Parallel()f := setupDurability(t)+ storages := []string{f.repo.Storage}+before := f.durability(t, "boot-a", time.Hour)require.NoError(t, before.MarkDirty(f.ctx, f.repo, f.path))- storages := []string{f.repo.Storage}+ require.NoError(t, before.Close())// The same boot: the process restarted, the kernel kept every write.sameBoot := f.durability(t, "boot-a", 0)@@ -258,11 +420,12 @@ func TestRecoverTellsAKernelRestartFromAProcessRestart(t *testing.T) {require.NoError(t, err)require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path, Reboot: false}}, found)require.Equal(t, []string{f.path}, sameBoot.Dirty())- require.NoError(t, sameBoot.Settle(f.path))+ require.NoError(t, sameBoot.Settle(f.path, false))require.NoFileExists(t, f.markerPath(t, sameBoot))// Another boot: the writes may not have reached the disk.- require.NoError(t, f.durability(t, "boot-a", time.Hour).MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, sameBoot.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, sameBoot.Close())otherBoot := f.durability(t, "boot-b", 0)found, err = otherBoot.Recover(f.ctx, storages)require.NoError(t, err)@@ -270,36 +433,35 @@ func TestRecoverTellsAKernelRestartFromAProcessRestart(t *testing.T) {require.Empty(t, otherBoot.Dirty())}-// TestRecoverRemovesTheLockOfADurableStepThatDied verifies that a reference-// lock left by a durable step of a process that died is removed at startup,-// and that a lock of git is not.-func TestRecoverRemovesTheLockOfADurableStepThatDied(t *testing.T) {+// TestRecoverRemovesWhatADurableStepThatDiedLeft verifies that the reference+// lock and the temporary files of a durable step of a process that died are+// removed when the next process owns the storage, and that a lock of git is+// not.+func TestRecoverRemovesWhatADurableStepThatDiedLeft(t *testing.T) {t.Parallel()f := setupDurability(t)+ storages := []string{f.repo.Storage}lock := filepath.Join(f.path, "reftable", "tables.list.lock")- require.NoError(t, f.durability(t, "boot-a", 0).MarkDirty(f.ctx, f.repo, f.path))- unlock, ok, err := lockStack(f.path)+ dead := f.durability(t, "boot-a", 0)+ require.NoError(t, dead.MarkDirty(f.ctx, f.repo, f.path))+ _, ok, err := lockStack(f.path)require.NoError(t, err)require.True(t, ok)- _, ok, err = lockStack(f.path)- require.NoError(t, err)- require.False(t, ok, "the lock is exclusive")- entries, err := os.ReadDir(f.path)- require.NoError(t, err)- for _, entry := range entries {- require.False(t, strings.HasPrefix(entry.Name(), ".graft-durable-step-"), "temporary file %s left", entry.Name())- }+ leftover := filepath.Join(f.path, stepTempPrefix+"123")+ require.NoError(t, os.WriteFile(leftover, []byte(stackLockTag+"0123456789abcdef\n"), 0o600))+ require.NoError(t, dead.Close())- // The process dies holding the lock; the next one of the same boot removes it.- _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, []string{f.repo.Storage})+ next := f.durability(t, "boot-a", 0)+ _, err = next.Recover(f.ctx, storages)require.NoError(t, err)require.NoFileExists(t, lock)- require.Error(t, unlock(), "the lock was already removed")+ require.NoFileExists(t, leftover)+ require.NoError(t, next.Close())// A lock of git is kept.require.NoError(t, os.WriteFile(lock, []byte("0x000000000001-0x000000000003-abcdef01.ref\n"), 0o666))- _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, []string{f.repo.Storage})+ _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, storages)require.NoError(t, err)require.FileExists(t, lock)}@@ -307,20 +469,25 @@ func TestRecoverRemovesTheLockOfADurableStepThatDied(t *testing.T) {func TestRecoverRefusesACorruptMarker(t *testing.T) {t.Parallel()f := setupDurability(t)+ storages := []string{f.repo.Storage}d := f.durability(t, "boot-a", 0)require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, d.Close())marker := f.markerPath(t, d)- incomplete := filepath.Join(filepath.Dir(marker), ".marker-123.tmp")+ incomplete := filepath.Join(filepath.Dir(marker), markerTempPrefix+"123.tmp")require.NoError(t, os.WriteFile(incomplete, []byte("{"), 0o600))- found, err := f.durability(t, "boot-b", 0).Recover(f.ctx, []string{f.repo.Storage})+ next := f.durability(t, "boot-b", 0)+ found, err := next.Recover(f.ctx, storages)require.NoError(t, err)require.Len(t, found, 1)require.NoFileExists(t, incomplete, "an incomplete marker write is removed")+ require.FileExists(t, filepath.Join(filepath.Dir(marker), ownerLockName), "the owner lock is not a marker")+ require.NoError(t, next.Close())require.NoError(t, os.WriteFile(marker, []byte(`{"storage":"`), 0o600))- _, err = f.durability(t, "boot-b", 0).Recover(f.ctx, []string{f.repo.Storage})+ _, err = f.durability(t, "boot-b", 0).Recover(f.ctx, storages)require.ErrorContains(t, err, "corrupt dirty marker")}@@ -330,13 +497,18 @@ func TestNilDurabilityIsOff(t *testing.T) {var d *Durabilityrequire.False(t, d.Enabled())+ require.False(t, d.Deferring())+ owned, err := d.TryOwn(ctx, []string{"default"})+ require.NoError(t, err)+ require.False(t, owned)require.NoError(t, d.MarkDirty(ctx, RepoRef{}, "path"))- require.NoError(t, d.Settle("path"))+ require.NoError(t, d.Settle("path", true))d.Begin("path", key(1))d.End("path", key(1))found, err := d.Recover(ctx, []string{"default"})require.NoError(t, err)require.Empty(t, found)+ require.NoError(t, d.Close())}// TestTornReftableStackReadsAsAnEmptyRepository is the experiment behind thediff --git a/internal/graft/corestore/syncer/durability.go b/internal/graft/corestore/syncer/durability.goindex 32c5be5..4b468c6 100644--- a/internal/graft/corestore/syncer/durability.go+++ b/internal/graft/corestore/syncer/durability.go@@ -6,6 +6,7 @@ import ("fmt""os""path/filepath"+ "sync""time""github.com/feder-cr/git-ai/internal/graft/corestore"@@ -14,14 +15,111 @@ import ("github.com/feder-cr/git-ai/internal/structerr")+// finalStepTimeout bounds the durable steps that run when the process stops.+const finalStepTimeout = 30 * time.Second++// StartLocalReferences starts the deferred durability of the node's local+// references (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). gitaly serve+// calls it before its listeners open.+//+// When the process takes the ownership of the storages' deferred durability,+// it recovers them before returning (RecoverLocalReferences): a repository+// that had writes not yet durable when the kernel restarted is moved aside, so+// nothing reads its possibly torn stack. When another process owns them (a+// graceful upgrade overlaps the old and the new process), that process+// recovered them when it started, in this boot; this one writes references+// with fsync and tries again every DurableStepInterval, recovering when it+// takes the ownership.+//+// The returned stop ends the durable steps and runs a last one for every dirty+// repository, without waiting for it to be idle, so that a clean stop leaves+// no marker of a repository whose writes ended.+func (s *Syncer) StartLocalReferences(ctx context.Context, storages []string) (stop func(), err error) {+ d := s.core.RefDurability+ if !d.Enabled() {+ return func() {}, nil+ }++ owned, err := d.TryOwn(ctx, storages)+ if err != nil {+ return nil, fmt.Errorf("storage core: own the deferred durability: %w", err)+ }+ recovered := false+ if owned {+ if err := s.RecoverLocalReferences(ctx, storages); err != nil {+ return nil, err+ }+ recovered = true+ } else {+ s.logger.Warn("storage core: another gitaly process owns the deferred durability of the storages; references are written with fsync until it exits")+ }++ loopCtx, cancel := context.WithCancel(context.WithoutCancel(ctx))+ var wg sync.WaitGroup+ wg.Add(1)+ go func() {+ defer wg.Done()+ s.runDurableSteps(loopCtx, storages, recovered)+ }()++ return func() {+ cancel()+ wg.Wait()+ if !d.Deferring() {+ return+ }+ finalCtx, cancelFinal := context.WithTimeout(context.WithoutCancel(ctx), finalStepTimeout)+ defer cancelFinal()+ for _, path := range d.Dirty() {+ s.durableStep(finalCtx, path, true)+ }+ }, nil+}++// runDurableSteps runs, every DurableStepInterval until ctx ends, the durable+// step of every dirty repository, or, while another process owns the deferred+// durability, an attempt to take it (and the recovery once taken).+func (s *Syncer) runDurableSteps(ctx context.Context, storages []string, recovered bool) {+ d := s.core.RefDurability+ ticker := time.NewTicker(localrefs.DurableStepInterval)+ defer ticker.Stop()+ for {+ select {+ case <-ctx.Done():+ return+ case <-ticker.C:+ }++ if !recovered {+ owned, err := d.TryOwn(ctx, storages)+ if err != nil {+ s.logger.WithError(err).Error("storage core: own the deferred durability")+ continue+ }+ if !owned {+ continue+ }+ if err := s.RecoverLocalReferences(ctx, storages); err != nil {+ s.logger.WithError(err).Error("storage core: recover the local references after taking the deferred durability; retried")+ continue+ }+ recovered = true+ s.logger.Info("storage core: this process owns the deferred durability of the storages now")+ }++ for _, path := range d.Dirty() {+ s.durableStep(ctx, path, false)+ }+ }+}+// RecoverLocalReferences moves aside every local repository that had reference// writes that were not durable yet when the kernel restarted (doc/graft/PLAN7.md,// LOCAL REFERENCE WRITES RULE): its reftable stack may be torn, and git reads a// torn stack as an empty repository without an error. The next request-// hydrates it from the database and the bucket. gitaly serve calls it before-// its listeners open, so nothing reads such a repository before. A repository-// dirty from a process restart of the same boot is kept: the kernel kept every-// write, and the durable step settles it.+// hydrates it from the database and the bucket. The process owns the deferred+// durability. A repository dirty from a process restart of the same boot is+// kept: the kernel kept every write, and the durable step settles it.//// Only a cache of the storage core is moved: the database must hold the// repository with changesets, and the local repository must have its@@ -39,7 +137,7 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)continue}- cache, err := s.storageCoreCache(ctx, rec)+ cache, err := s.storageCoreCache(ctx, rec.Repo, rec.Path)if err != nil {return fmt.Errorf("storage core: recover the local references of %s:%s: %w", rec.Repo.Storage, rec.Repo.RelativePath, err)}@@ -59,11 +157,11 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)return nil}-// storageCoreCache reports whether the local repository of rec is a cache of+// storageCoreCache reports whether the local repository at path is a cache of// the storage core that a hydration can rebuild: the database holds it with// changesets, and it has a watermark.-func (s *Syncer) storageCoreCache(ctx context.Context, rec localrefs.Recovered) (bool, error) {- r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: rec.Repo.Storage, RelativePath: rec.Repo.RelativePath})+func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, path string) (bool, error) {+ r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: repo.Storage, RelativePath: repo.RelativePath})if errors.Is(err, corestore.ErrRepositoryNotFound) {return false, nil}@@ -74,12 +172,12 @@ func (s *Syncer) storageCoreCache(ctx context.Context, rec localrefs.Recovered)return false, nil}- if _, err := os.Lstat(rec.Path); errors.Is(err, os.ErrNotExist) {+ if _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) {return false, nil} else if err != nil {return false, err}- _, ok, err := gitops.ReadWatermark(rec.Path)+ _, ok, err := gitops.ReadWatermark(path)if err != nil {// A watermark that cannot be read is no proof that the storage core wrote// the repository: it is kept, as untracked() keeps it.@@ -88,48 +186,46 @@ func (s *Syncer) storageCoreCache(ctx context.Context, rec localrefs.Recovered)return ok, nil}-// RunDurableSteps runs the durable step of every dirty repository each interval-// until ctx ends (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).-func (s *Syncer) RunDurableSteps(ctx context.Context, interval time.Duration) {- if !s.core.RefDurability.Enabled() {- return- }-- ticker := time.NewTicker(interval)- defer ticker.Stop()- for {- select {- case <-ctx.Done():- return- case <-ticker.C:- }- for _, path := range s.core.RefDurability.Dirty() {- s.durableStep(ctx, path)- }- }-}-// durableStep makes the references of the repository at path durable under the-// repository lock, and moves the repository aside at once when the kernel-// reports that a write failed.-func (s *Syncer) durableStep(ctx context.Context, path string) {+// repository lock (final: without waiting for the repository to be idle). When+// the kernel reports that a write failed, the repository is moved aside at+// once, if it is a cache of the storage core; any other failure is retried at+// the next step.+func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {release, err := s.lock(ctx, path)if err != nil {return}defer release()- err = s.core.RefDurability.Settle(path)+ d := s.core.RefDurability+ err = d.Settle(path, final)+ if err == nil {+ return+ }++ logger := s.logger.WithError(err).WithField("repository_path", path)var untrusted *localrefs.UntrustedError- if errors.As(err, &untrusted) {- if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {- s.logger.WithError(err).WithField("repository_path", path).- Error("storage core: could not move aside a repository whose references may have lost writes")- }+ if !errors.As(err, &untrusted) {+ d.CountStepError()+ logger.Warn("storage core: durable step of the local references failed, retried at the next step")return}- if err != nil {- s.logger.WithError(err).WithField("repository_path", path).Error("storage core: durable step of the local references")++ cache, cacheErr := s.storageCoreCache(ctx, untrusted.Repo, path)+ if cacheErr != nil {+ d.CountStepError()+ logger.WithField("cache_check_error", cacheErr.Error()).+ Error("storage core: the references of a repository may have lost writes; whether it can be hydrated again is not known yet, retried at the next step")+ return+ }+ if !cache {+ logger.Error("storage core: the references of a repository that is not a cache of the storage core may have lost writes; it is kept in place")+ return+ }+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {+ logger.WithField("move_error", err.Error()).+ Error("storage core: could not move aside a repository whose references may have lost writes")}}@@ -146,8 +242,8 @@ func (s *Syncer) distrust(ctx context.Context, repo localrefs.RepoRef, path, rea// distrustLocked moves the local repository at path into the temporary// directory of its storage and removes it there in the background, then drops// its dirty marker: the next request hydrates it. The move is made durable-// before the marker goes, so a crash in between moves it again. The caller-// holds the repository lock.+// (both directories, and the new directory's entry) before the marker goes, so+// a crash in between moves it again. The caller holds the repository lock.func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {d := s.core.RefDurability@@ -169,8 +265,10 @@ func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, pat_ = os.Remove(stale)return structerr.NewInternal("storage core: move aside an untrusted repository: %w", err)}- if err := syncDir(filepath.Dir(path)); err != nil {- return structerr.NewInternal("storage core: make the move of an untrusted repository durable: %w", err)+ for _, dir := range []string{filepath.Dir(path), stale, tmpDir} {+ if err := syncDir(dir); err != nil {+ return structerr.NewInternal("storage core: make the move of an untrusted repository durable: %w", err)+ }}s.forgetPending(path)diff --git a/internal/graft/corestore/syncer/export_test.go b/internal/graft/corestore/syncer/export_test.goindex 162b6c8..a6f05bb 100644--- a/internal/graft/corestore/syncer/export_test.go+++ b/internal/graft/corestore/syncer/export_test.go@@ -8,8 +8,8 @@ func (s *Syncer) SetTestHook(hook func(point string)) {s.testHook = hook}-// DurableStep runs one durable step of the repository at path, as-// RunDurableSteps does every interval.-func (s *Syncer) DurableStep(ctx context.Context, path string) {- s.durableStep(ctx, path)+// DurableStep runs one durable step of the repository at path, as the+// durable-step loop does every interval (final: as the stop does).+func (s *Syncer) DurableStep(ctx context.Context, path string, final bool) {+ s.durableStep(ctx, path, final)}diff --git a/internal/graft/corestore/syncer/syncer.go b/internal/graft/corestore/syncer/syncer.goindex 429f47c..d55a668 100644--- a/internal/graft/corestore/syncer/syncer.go+++ b/internal/graft/corestore/syncer/syncer.go@@ -1379,8 +1379,10 @@ func refTransaction(ctx context.Context, repo *localrepo.Repo, w refWriter, commreturn fmt.Errorf("reference backend: %w", err)}+ // Only while this process owns the deferred durability; the syncer's+ // repository lock is held, and the ownership never goes back.var durability []gitcmd.CmdOpt- if w.core.RefDurability.Enabled() {+ if w.core.RefDurability.Deferring() {if err := w.core.RefDurability.MarkDirty(ctx, w.repo, w.path); err != nil {return err}diff --git a/internal/graft/corestore/syncer/syncer_localrefs_test.go b/internal/graft/corestore/syncer/syncer_localrefs_test.goindex 0e378ff..38d33f5 100644--- a/internal/graft/corestore/syncer/syncer_localrefs_test.go+++ b/internal/graft/corestore/syncer/syncer_localrefs_test.go@@ -7,6 +7,7 @@ import ("strings""sync/atomic""testing"+ "time""github.com/prometheus/client_golang/prometheus""github.com/stretchr/testify/require"@@ -30,7 +31,9 @@ func (e *env) markers(t *testing.T) []string {require.NoError(t, err)var names []stringfor _, entry := range entries {- names = append(names, entry.Name())+ if strings.HasSuffix(entry.Name(), ".json") {+ names = append(names, entry.Name())+ }}return names}@@ -77,7 +80,7 @@ func caughtUpDirty(t *testing.T, opts ...localrefs.DurabilityOption) (*env, coret.Helper()e := newEnv(t, envOptions{})- coretest.EnableLocalRefs(t, e.core, e.locator, opts...)+ coretest.EnableLocalRefs(t, e.core, e.cfg, opts...)src := e.newSource(t)c1 := e.writeCommit(t, src)repo := e.register(t, e.key(localRelPath), src, c1)@@ -102,7 +105,7 @@ func TestSyncerCatchUpMarksTheRepositoryDirty(t *testing.T) {e.verifySynced(t, repo)require.Equal(t, []string{e.localPath()}, e.core.RefDurability.Dirty())- e.sy.DurableStep(e.ctx, e.localPath())+ e.sy.DurableStep(e.ctx, e.localPath(), false)require.Empty(t, e.markers(t))require.Empty(t, e.core.RefDurability.Dirty())e.verifySynced(t, repo)@@ -125,7 +128,7 @@ func TestSyncerMovesAsideARepositoryDirtyAtAKernelRestart(t *testing.T) {require.Empty(t, strings.TrimSpace(string(torn)))// The node restarts after a kernel restart.- coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-b"))+ coretest.EnableLocalRefs(t, e.core, e.cfg, localrefs.WithBootID("boot-b"))e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))@@ -145,7 +148,7 @@ func TestSyncerKeepsARepositoryDirtyAtAProcessRestart(t *testing.T) {e, repo, _ := caughtUpDirty(t, localrefs.WithBootID("boot-a"))- coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-a"))+ coretest.EnableLocalRefs(t, e.core, e.cfg, localrefs.WithBootID("boot-a"))e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))@@ -153,7 +156,7 @@ func TestSyncerKeepsARepositoryDirtyAtAProcessRestart(t *testing.T) {require.Equal(t, []string{e.localPath()}, e.core.RefDurability.Dirty())e.verifySynced(t, repo)- e.sy.DurableStep(e.ctx, e.localPath())+ e.sy.DurableStep(e.ctx, e.localPath(), false)require.Empty(t, e.markers(t))}@@ -164,14 +167,14 @@ func TestSyncerRecoveryKeepsWhatIsNotACache(t *testing.T) {t.Parallel()e := newEnv(t, envOptions{})- coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-a"))+ coretest.EnableLocalRefs(t, e.core, e.cfg, localrefs.WithBootID("boot-a"))pb, path := gittest.CreateRepository(t, e.ctx, e.cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true,ReferenceBackend: git.ReferenceBackendReftables,})require.NoError(t, e.core.RefDurability.MarkDirty(e.ctx, localrefs.RepoRef{Storage: pb.GetStorageName(), RelativePath: pb.GetRelativePath()}, path))- coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-b"))+ coretest.EnableLocalRefs(t, e.core, e.cfg, localrefs.WithBootID("boot-b"))e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))@@ -196,7 +199,7 @@ func TestSyncerDurableStepMovesAsideOnAFailedWrite(t *testing.T) {}))failing.Store(true)- e.sy.DurableStep(e.ctx, e.localPath())+ e.sy.DurableStep(e.ctx, e.localPath(), false)failing.Store(false)require.NoDirExists(t, e.localPath())@@ -207,3 +210,83 @@ func TestSyncerDurableStepMovesAsideOnAFailedWrite(t *testing.T) {require.Equal(t, float64(2), syncCount(t, e.sy, "hydrated"), "the first hydration and the one after the move")e.verifySynced(t, repo)}++// TestSyncerDurableStepRetriesATransientError verifies that a durable step that+// fails without a failed fsync (the stack cannot be read for a moment) keeps+// the repository and its marker and is retried.+func TestSyncerDurableStepRetriesATransientError(t *testing.T) {+ t.Parallel()++ e, repo, _ := caughtUpDirty(t)+ list := filepath.Join(e.localPath(), "reftable", "tables.list")+ require.NoError(t, os.Rename(list, list+".away"))+ e.sy.DurableStep(e.ctx, e.localPath(), false)+ require.NoError(t, os.Rename(list+".away", list))++ require.DirExists(t, e.localPath())+ require.Len(t, e.markers(t), 1)+ require.Equal(t, float64(0), e.untrusted(t, localrefs.ReasonSyncFailed))++ e.sy.DurableStep(e.ctx, e.localPath(), false)+ require.Empty(t, e.markers(t))+ e.verifySynced(t, repo)+}++// TestSyncerSecondProcessWaitsForTheOwner is the overlap of a graceful+// upgrade: while the old process owns the deferred durability, the new one+// writes with fsync and touches no marker; once the old one is gone, the new+// one owns, recovers and settles.+func TestSyncerSecondProcessWaitsForTheOwner(t *testing.T) {+ t.Parallel()++ e, _, _ := caughtUpDirty(t)+ storages := []string{e.storageName}++ next := *e.core+ durability, err := localrefs.NewDurability(e.locator, 0, localrefs.WithBootID(coretest.TestBootID))+ require.NoError(t, err)+ next.RefDurability = durability+ t.Cleanup(func() { _ = durability.Close() })+ nextSyncer := syncer.New(&next, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)++ stop, err := nextSyncer.StartLocalReferences(e.ctx, storages)+ require.NoError(t, err)+ defer stop()+ require.False(t, durability.Deferring(), "the old process owns the storage")+ require.Len(t, e.markers(t), 1)++ // The old process exits.+ require.NoError(t, e.core.RefDurability.Close())++ // The next process tries every durable step interval.+ require.Eventually(t, func() bool { return durability.Deferring() && len(e.markers(t)) == 0 },+ 30*time.Second, 10*time.Millisecond, "the next process owns and settles the marker")+}++// TestSyncerStopSettlesEveryRepository verifies that a clean stop removes the+// marker of a repository whose writes ended, without waiting for it to be idle.+func TestSyncerStopSettlesEveryRepository(t *testing.T) {+ t.Parallel()++ e := newEnv(t, envOptions{})+ e.core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)+ durability, err := localrefs.NewDurability(e.locator, time.Hour, localrefs.WithBootID(coretest.TestBootID))+ require.NoError(t, err)+ t.Cleanup(func() { _ = durability.Close() })+ e.core.RefDurability = durability++ stop, err := e.sy.StartLocalReferences(e.ctx, []string{e.storageName})+ require.NoError(t, err)+ require.True(t, durability.Deferring())++ src := e.newSource(t)+ c1 := e.writeCommit(t, src)+ repo := e.register(t, e.key(localRelPath), src, c1)+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ e.advanceMain(t, repo.ID, src, c1)+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ require.Len(t, e.markers(t), 1, "the repository is not idle for an hour")++ stop()+ require.Empty(t, e.markers(t))+}diff --git a/internal/graft/corestore/voter/voter.go b/internal/graft/corestore/voter/voter.goindex 33d6abf..e3d67bd 100644--- a/internal/graft/corestore/voter/voter.go+++ b/internal/graft/corestore/voter/voter.go@@ -176,6 +176,8 @@ func (v *Voter) Preparing(ctx context.Context, payload gitcmd.HooksPayload, chanreturn}+ // The transaction is live from now on (it ends in Committed or Aborted).+ v.begin(ctx, payload, changes)// Deferred first, so that it runs last: the admission is waited for after// the pack work, when git is about to take the reference lock.defer v.admit(ctx, payload, changes)@@ -797,11 +799,38 @@ func (v *Voter) localRefs() bool {}// DeferredReferenceDurability reports whether the reference writes of the-// transactions this voter commits may run without fsync: authoritative mode-// with the deferred durability of the node on (doc/graft/PLAN7.md, LOCAL-// REFERENCE WRITES RULE). The hook manager asks it for UpdateWithHooks.+// transactions this voter commits may run without fsync: authoritative mode,+// and this process owns the deferred durability of the node (doc/graft/PLAN7.md,+// LOCAL REFERENCE WRITES RULE). The hook manager asks it for UpdateWithHooks.+// It goes from false to true at most once in a process.func (v *Voter) DeferredReferenceDurability() bool {- return v != nil && v.core.Authoritative() && v.core.RefDurability.Enabled()+ return v != nil && v.core.Authoritative() && v.core.RefDurability.Deferring()+}++// Denied records the start of a transaction that the standalone policy denies:+// it is live until its Aborted phase, like any other (see begin).+func (v *Voter) Denied(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {+ if v == nil || !v.core.Enabled() || payload.Repo == nil {+ return+ }+ v.begin(ctx, payload, changes)+}++// begin counts the transaction as live for the deferred durability: from+// before git takes the reference lock until Committed or Aborted, it may write+// a table or compact the stack, so the durable step keeps the repository's+// marker meanwhile.+func (v *Voter) begin(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {+ if !v.core.Authoritative() || !v.core.RefDurability.Enabled() {+ return+ }++ repoPath, err := v.repoPath(ctx, payload)+ if err != nil {+ v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path of a starting transaction")+ return+ }+ v.core.RefDurability.Begin(repoPath, localrefs.Key(pendingKey(payload.Repo, changes)))}// admit waits for the admission of the transaction to the reference lock of@@ -822,11 +851,11 @@ func (v *Voter) admit(ctx context.Context, payload gitcmd.HooksPayload, changes}// deferDurability lets git write the references of a transaction that the-// database committed without fsync (authoritative mode with deferred-// durability): the repository gets its durable dirty marker first, and the-// transaction counts as in flight until its Committed phase, because git may-// still write its table and compact the stack until then. It runs in the-// Prepared phase, under git's reference lock and before git writes anything.+// database committed without fsync (authoritative mode, and this process owns+// the deferred durability): the repository gets its durable dirty marker+// first. The transaction is live since its start (begin), so the durable step+// keeps the marker until it ends. It runs in the Prepared phase, under git's+// reference lock and before git writes anything.func (v *Voter) deferDurability(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, shadow bool) error {if shadow || !v.DeferredReferenceDurability() {return nil@@ -837,11 +866,7 @@ func (v *Voter) deferDurability(ctx context.Context, payload gitcmd.HooksPayloadreturn err}repoKey := corestore.KeyOf(payload.Repo)- if err := v.core.RefDurability.MarkDirty(ctx, localrefs.RepoRef{Storage: repoKey.Storage, RelativePath: repoKey.RelativePath}, repoPath); err != nil {- return err- }- v.core.RefDurability.Begin(repoPath, localrefs.Key(pendingKey(payload.Repo, changes)))- return nil+ return v.core.RefDurability.MarkDirty(ctx, localrefs.RepoRef{Storage: repoKey.Storage, RelativePath: repoKey.RelativePath}, repoPath)}// Aborted tells the voter that git aborted the transaction.@@ -863,14 +888,13 @@ func (v *Voter) Aborted(ctx context.Context, payload gitcmd.HooksPayload, changekey := pendingKey(payload.Repo, changes)- // A transaction aborted after its Prepared phase stays in flight for the- // deferred durability (see localrefs.Durability.End): its repository stays- // dirty.+ // git wrote nothing more of the transaction once it reports the abort.if v.localRefs() {if repoPath, err := v.repoPath(ctx, payload); err != nil {v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path after an abort, the admission ends at its bound")} else {v.core.RefAdmission.Release(repoPath, localrefs.Key(key))+ v.core.RefDurability.End(repoPath, localrefs.Key(key))}}diff --git a/internal/graft/corestore/voter/voter_localrefs_test.go b/internal/graft/corestore/voter/voter_localrefs_test.goindex 2e1901a..35a9109 100644--- a/internal/graft/corestore/voter/voter_localrefs_test.go+++ b/internal/graft/corestore/voter/voter_localrefs_test.go@@ -4,6 +4,7 @@ import ("context""os""path/filepath"+ "strings""testing""github.com/stretchr/testify/require"@@ -46,7 +47,9 @@ func (f *fixture) markers(t *testing.T) []string {require.NoError(t, err)var names []stringfor _, entry := range entries {- names = append(names, entry.Name())+ if strings.HasSuffix(entry.Name(), ".json") {+ names = append(names, entry.Name())+ }}return names}@@ -74,7 +77,7 @@ func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {// git may still write and compact until Committed: the durable step keeps// the marker.- require.NoError(t, f.core.RefDurability.Settle(f.repoPath))+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))require.Len(t, f.markers(t), 1)gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName(ref), f.a)@@ -82,14 +85,13 @@ func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {require.False(t, f.admissionHeld(t), "Committed releases the admission")require.Len(t, f.applier.calls, 1, "the local commit is still noted")- require.NoError(t, f.core.RefDurability.Settle(f.repoPath))+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))require.Empty(t, f.markers(t))}// TestLocalReferenceWritesOfAnAbortedTransaction verifies that an abort after-// Prepared releases the admission but leaves the repository dirty: the-// transaction cannot be told from another with the same input, so it is never-// counted as ended.+// Prepared releases the admission and ends the transaction: the repository is+// settled once nothing else is live.func TestLocalReferenceWritesOfAnAbortedTransaction(t *testing.T) {t.Parallel()@@ -101,8 +103,32 @@ func TestLocalReferenceWritesOfAnAbortedTransaction(t *testing.T) {f.voter.Aborted(f.ctx, f.payload, input)require.False(t, f.admissionHeld(t))- require.NoError(t, f.core.RefDurability.Settle(f.repoPath))- require.Len(t, f.markers(t), 1)+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))+ require.Empty(t, f.markers(t))+}++// TestLocalReferenceWritesOfTransactionsWithTheSameInput verifies that two+// live transactions with the same input are both counted: the one denied by+// the policy and aborted does not end the other one, which may still write.+func TestLocalReferenceWritesOfTransactionsWithTheSameInput(t *testing.T) {+ t.Parallel()++ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})+ input := line(f.zero, f.a.String(), "refs/heads/main")++ f.voter.Preparing(f.ctx, f.payload, input)+ require.NoError(t, f.prepared(input))++ // The same input from an agent the policy denies.+ f.voter.Denied(f.ctx, f.payload, input)+ f.voter.Aborted(f.ctx, f.payload, input)+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, true))+ require.Len(t, f.markers(t), 1, "the first transaction may still write")++ gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName("refs/heads/main"), f.a)+ f.voter.Committed(f.ctx, f.payload, input)+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))+ require.Empty(t, f.markers(t))}// TestLocalReferenceWritesOfARejectedTransaction verifies that a transaction@@ -131,9 +157,11 @@ func TestLocalReferenceWritesWhenTheMarkerCannotBeWritten(t *testing.T) {t.Parallel()f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})+ // The marker directory exists (the ownership created it); no file can be+ // created in it.markerDir := filepath.Join(f.cfg.Storages[0].Path, localrefs.MarkerDirName)- require.NoError(t, os.MkdirAll(filepath.Dir(markerDir), 0o755))- require.NoError(t, os.WriteFile(markerDir, nil, 0o644))+ require.NoError(t, os.Chmod(markerDir, 0o500))+ t.Cleanup(func() { require.NoError(t, os.Chmod(markerDir, 0o700)) })input := line(f.zero, f.a.String(), "refs/heads/main")err := f.prepared(input)diff --git a/internal/graft/corestore/voter/voter_test.go b/internal/graft/corestore/voter/voter_test.goindex 845d120..71e445d 100644--- a/internal/graft/corestore/voter/voter_test.go+++ b/internal/graft/corestore/voter/voter_test.go@@ -134,7 +134,7 @@ func newFixture(t *testing.T, opts fixtureOptions) *fixture {logger := testhelper.NewLogger(t)locator := config.NewLocator(cfg)if opts.localRefs {- coretest.EnableLocalRefs(t, core, locator)+ coretest.EnableLocalRefs(t, core, cfg)}cache := catfile.NewCache(cfg)t.Cleanup(cache.Stop)diff --git a/internal/graft/e2e/storage_localrefs_test.go b/internal/graft/e2e/storage_localrefs_test.goindex 768c93f..48eb1c1 100644--- a/internal/graft/e2e/storage_localrefs_test.go+++ b/internal/graft/e2e/storage_localrefs_test.go@@ -93,6 +93,22 @@ func newRealNode(t *testing.T) *realNode {}}+// sibling returns a second server of the same storage, database and bucket on+// its own socket, as the new process of a graceful upgrade is.+func (n *realNode) sibling(t *testing.T) *realNode {+ t.Helper()++ s := *n+ s.cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)+ s.clientCfg = s.cfg+ s.clientCfg.SocketPath = "unix://" + s.cfg.SocketPath+ s.cfgPath = testcfg.WriteTemporaryGitalyConfigFile(t, s.cfg)+ s.logPath = filepath.Join(t.TempDir(), "gitaly.log")+ s.setupCtx = testhelper.MergeOutgoingMetadata(s.ctx, testcfg.GitalyServersMetadataFromCfg(t, s.clientCfg))+ s.cmd, s.exited = nil, nil+ return &s+}+// start starts the server process with the extra environment env and waits// until it serves.func (n *realNode) start(t *testing.T, env ...string) {@@ -205,7 +221,9 @@ func (n *realNode) markers(t *testing.T) []string {require.NoError(t, err)var names []stringfor _, entry := range entries {- names = append(names, filepath.Join(n.storagePath(), localrefs.MarkerDirName, entry.Name()))+ if strings.HasSuffix(entry.Name(), ".json") {+ names = append(names, filepath.Join(n.storagePath(), localrefs.MarkerDirName, entry.Name()))+ }}return names}@@ -363,6 +381,47 @@ func TestStorageLocalReferencesAfterAKernelRestart(t *testing.T) {require.Contains(t, n.log(t), "moved aside a local repository whose references may have lost writes")}+// TestStorageLocalReferencesDuringAGracefulUpgrade runs a second server on+// the same storage while the first one lives, as a graceful upgrade does: the+// first one owns the deferred durability, the second writes with fsync and+// leaves the markers alone. When the first one dies, the second owns, settles+// and defers. Every write is served throughout.+func TestStorageLocalReferencesDuringAGracefulUpgrade(t *testing.T) {+ t.Parallel()++ old := newRealNode(t)+ old.start(t)+ rel := old.createRepository(t)+ base, err := commitFile(old.ctx, old.agent(t, rel), workerName(0), "main", "", "README.md", "base\n")+ require.NoError(t, err)+ written := writeBranches(t, old.agent(t, rel), old.ctx, base, "old", 3)+ written["main"] = base+ require.Len(t, old.markers(t), 1)++ next := old.sibling(t)+ next.start(t)+ require.Contains(t, next.log(t), "another gitaly process owns the deferred durability")+ for branch, commit := range writeBranches(t, next.agent(t, rel), next.ctx, base, "during", 3) {+ written[branch] = commit+ }+ next.requireServed(t, rel, written)++ // The old process owns the marker and settles it once the repository is+ // idle; the new process's writes were made with fsync and need none.+ require.Eventually(t, func() bool { return len(old.markers(t)) == 0 }, 30*time.Second, 50*time.Millisecond)++ old.kill(t)+ require.Eventually(t, func() bool {+ return strings.Contains(next.log(t), "this process owns the deferred durability of the storages now")+ }, 30*time.Second, 50*time.Millisecond)++ for branch, commit := range writeBranches(t, next.agent(t, rel), next.ctx, base, "after", 2) {+ written[branch] = commit+ }+ require.Len(t, next.markers(t), 1, "the new owner writes without fsync")+ next.requireServed(t, rel, written)+}+// TestStorageLocalReferencesStackDepthIsBounded writes many references// without fsync from concurrent writers and checks that git's compaction keeps// the stack shallow: the durable step only makes it durable, compaction stays