Git designed for AI agents.
WP76b: local reference writes of authoritative mode (admission, deferred durability)
Federico Elia310facd5b916
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.mdindex d23e7fd..43d09e6 100644--- a/doc/graft/PLAN7.md+++ b/doc/graft/PLAN7.md@@ -504,6 +504,134 @@ guarded nor swept).* disabling the roots requirement fails the sqlstore test and both reviewrace tests.+### LOCAL REFERENCE WRITES RULE (binding; WP76b, approved by the PM)++**Problem (measured).** A reftable repository has one lock for all its references+(`tables.list.lock`). In authoritative mode `UserCommitFiles` spent most of its time+waiting for it: with SQLite and a file bucket, at 50 and 100 agents, `u.prepare`+(the wait) was 643 and 1198 ms mean, `u.commit` 252 and 230 ms, the voter's Prepared+phase 19–26 ms. With `core.fsync` without `reference`, `u.commit` fell to 17 and 7 ms+and the wait to 248 and 238 ms. So the hold is git's fsync of the new table and of+`tables.list`, inside the lock. The wait that is left, about 6 holds, is git's lock+wait: a polling sleep that grows with each failed try (`reftable.lockTimeout` =+10 s). There is no queue, so newcomers win over long waiters and the lock sits free+between polls. The same unfairness starved the syncer's catch-up behind writers+(the flake of `TestSyncerConcurrentLocalWritesAreNeverReverted`).++**Root cause.** In authoritative mode the durable commit point of a reference+update is the database commit in the voter's Prepared phase; the local repository+is a cache that any node rebuilds from the database and the bucket. Each+transaction then made the same update durable a second time, inside the+repository-wide lock.++**Rule.**+1. *Admission* (`corestore/localrefs.Admission`). The reference writers of a node+ wait in a FIFO queue per local repository in front of git's lock:+ * the hook's transactions from the end of the voter's Preparing phase (after+ the pack upload, before git takes the lock) to Committed or Aborted;+ * the syncer around each catch-up `update-ref`.++ Admission orders writers only; git's lock stays the only mutual exclusion. A+ writer that waited `reftable.lockTimeout` goes on to git's lock as before+ (`bypassed`). An admission not released within the same bound ends+ (`expired`; git may die between phases). A writer whose context ends leaves+ the queue. Lock order: the syncer's repository lock, then the admission. No+ admitted writer waits for the syncer lock: `NoteCommitted` only tries it, and+ runs after the release.+2. *Deferred durability* (`corestore/localrefs.Durability`). The hook's+ `UpdateWithHooks` transactions and the syncer's catch-up write references with+ `core.fsync` without `reference`. Only the voter can turn this on+ (`DeferredReferenceDurability`: authoritative mode and the kernel's boot+ identity readable). Objects stay durable as before.+ * *Dirty marker.* Before the first such write after a durable step, the+ repository gets a marker, written durably, with the kernel's boot identity+ (`<storage>/+gitaly/graft-ref-durability/<sha256(path)>.json`, not the state+ directory, whose files the cache walker removes).+ * The hook path writes it in Prepared, after the database commit and before+ git writes anything, so only repositories the database tracks get one. If+ the marker cannot be written, Prepared fails and git aborts; catch-up+ repairs the database's write, like any local failure.+ * The syncer writes it under its repository lock.+ * A hook transaction counts as in flight from Prepared to Committed.+ Reasons from git's source: the new table is written in+ `transaction_finish` after "prepared", and auto-compaction runs inside+ `reftable_addition_commit`, before "committed". An abort after Prepared+ is not counted as an end (two transactions with the same input cannot be+ told apart), so the repository stays dirty.+ * *Durable step* (`Settle`, every second for every dirty repository, under the+ syncer's repository lock).+ * It fsyncs the tables not yet fsynced, `tables.list` and the directory.+ * When the repository has been idle for a second and nothing is in flight,+ it takes git's lock without waiting (the lock is created with a tag, by+ linking a temporary file). It then fsyncs what was written since, marks+ the state clean, and releases the lock. Last, it removes the marker,+ under a per-repository file lock that `MarkDirty` waits for.+ * A busy lock leaves the marker for the next step.+ * *Write failure.* A failed fsync in the durable step means the kernel may+ have dropped writes. The repository is moved aside at once, under the+ repository lock: the move is made durable, then the marker removed. The+ failure is logged and counted (`sync_failed`), and the next request+ hydrates the repository.+ * *Recovery* (`Syncer.RecoverLocalReferences`). `gitaly serve` runs it before+ any listener opens and before any background worker.+ * A marker of another boot means the kernel restarted with writes that may+ not have reached the disk. git reads a torn stack as an empty repository+ with exit code 0 (experiment 1, kept as+ `TestTornReftableStackReadsAsAnEmptyRepository`). So the repository is+ moved aside before anything reads it (`reboot`), and the next request+ hydrates it.+ * The move only happens when the database holds the repository with+ changesets and the local copy has a watermark; any other marker is+ dropped, keeping a repository whose local copy may be the only one.+ * A marker of the same boot is a process restart: the page cache kept+ every write, so the repository stays dirty to be settled.+ * A lock with the durable step's tag, left by a process that died, is+ removed.+ * A corrupt marker fails the startup: its repository is unknown.+ * Not Linux (no boot identity): references stay durable per transaction.+3. *Read paths.* After the recovery, no read path in authoritative mode can see+ a stack torn by a kernel restart: the gRPC listeners (unix, TCP, TLS), the+ Prometheus listener, the maintenance workers, the bundle generation, the+ tempdir cleaner and the repository counter all start after it. Within the+ same boot the stack is never torn (the page cache has every write). A failed+ writeback is caught by the durable step within a second and the repository is+ moved aside. Outside `gitaly serve`, only `gitaly-debug simulate-http-clone`,+ `gitaly-debug dump-mvcc-manifest` and the `gitaly git` passthrough, pointed+ at a repository by an operator, read references from disk. STORAGE.md says+ not to run them on a node after an unclean shutdown before the server has+ started once.++**As implemented.**+* `localrefs` (`admission.go`, `durability.go`); `Core.RefAdmission`/`RefDurability`.+* `voter` (`admit`, `deferDurability`, Committed/Aborted).+* `syncer` (`refWriter`, `durability.go`).+* `serve.go`: `openLocalReferenceWrites`, the recovery before the listeners, the+ durable-step loop.+* PM grant: `updateref.WithDeferredReferenceDurability`; the `gitcmd` constants+ `FsyncComponents`, `FsyncComponentsWithoutReferences` and+ `ReftableLockTimeoutMilliseconds`; `hook/storage_voter.go`+ (`DeferredReferenceDurability`); the optional interface asserted in+ `UpdateWithHooks`.++**Tests.**+* `localrefs`: FIFO order, a cancelled waiter, the wait bound, the hold bound,+ unique keys; the marker, the durable step (idle, in flight, git's lock held,+ failed fsync, repository gone), recovery (process restart, kernel restart,+ stale lock, corrupt marker), the torn-stack experiment.+* `voter`: committed, aborted and rejected transactions; a marker that cannot be+ written; durability per mode (off, shadow and authoritative without boot+ identity keep the fsync).+* `updateref`: the effective `core.fsync` of `update-ref`.+* `hook/updateref`: only a voter that defers turns it off.+* `syncer`: catch-up marks; reboot recovery hydrates (a read before it lists+ nothing); process-restart recovery keeps; non-caches are kept; a failed fsync+ moves aside and hydrates.+* e2e with the real `gitaly` binary: SIGKILL after writes (same boot, kept,+ settled); SIGKILL in the durable step (the lock is removed at restart, writes+ go on); a simulated power loss (torn table, other boot: moved aside, every+ write served); stack depth under 4×30 concurrent writes (git's geometric+ compaction, at most 2⌈log₂ n⌉+2 tables; measured 4 for 122 updates).+### Data model (new migration, both dialects)Migration `20261018000000_graft_storage_gc`, appended to `sqliteMigrations` and@@ -1418,6 +1546,7 @@ every entry point returns `FailedPrecondition`.| WP74b | REVIEW ROOTS RULE: `reviewroots` (Protect, Sweep, Backfill), the review service, `dbreview.WithRequiredRoots`, the two database checks | WP74 || WP75 | `gc` package | WP71–WP73 || WP76 | syncer (seed, `C_sync`), `verify_objects.go` | WP71, WP74 |+| WP76b | LOCAL REFERENCE WRITES RULE: `localrefs` (admission, deferred durability), voter, syncer, `serve.go` recovery and durable steps | WP76 || WP77 | config, `maintenance` package, `serve.go` wiring | WP74–WP76 || WP78 | proto, handlers, SDK, CLI | WP74, WP75 || WP79 | e2e, STORAGE.md and CONFIGURATION.md, benchmark | all |@@ -1478,7 +1607,11 @@ New (m3): `corestore/checkpoint/**`, `corestore/gc/**`, `corestore/maintenance/*`cmd/graft/storage.go`, `proto/storage_core.proto` (+ generated code),`config_graft_storage.go`, `doc/graft/STORAGE.md`, `corestore/dbreview/**`,`corestore/reviewroots/**`, and `internal/gitaly/service/review/**` (the REVIEW-ROOTS RULE writes only, assigned to m3 by the PM). Shared, my section only:+ROOTS RULE writes only, assigned to m3 by the PM), `corestore/localrefs/**`+(WP76b). Granted by the PM for WP76b only: `internal/git/updateref` (the deferred+durability option), `internal/git/gitcmd` (the fsync and lock-timeout constants),+`internal/gitaly/hook/storage_voter.go` (and its test double) and the updater call+in `internal/gitaly/hook/updateref/update_with_hooks.go`. Shared, my section only:`internal/cli/gitaly/serve.go`, `doc/graft/CONFIGURATION.md`. No change to the landpath: the voter is m3's; the land paths call it unchanged.@@ -1508,3 +1641,8 @@ path: the voter is m3's; the land paths call it unchanged.repository row lock.26. **One upload path**: every production upload goes through `Core.UploadPack`, andevery row that references an upload is committed under its `CommitContext`.+27. **Local references are a cache of the database**: in authoritative mode a local+ reference write without fsync happens only after its repository has a durable+ dirty marker; a marker is removed only after the stack is durable, under git's+ lock with no transaction in flight; a repository dirty at a kernel restart, or+ whose durable step failed, is moved aside and hydrated, never served.diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.mdindex 4eb2f42..c9ffbdc 100644--- a/doc/graft/STORAGE.md+++ b/doc/graft/STORAGE.md@@ -431,6 +431,45 @@ or re-import the repository from scratch (seewhose key the database no longer has; it is moved into the storage'stemporary directory, and the repository is reported missing on that node too.+- **Local reference writes and unclean shutdowns (authoritative mode).** In+ authoritative mode the database commit is the durable point of a reference+ update. A node therefore writes its local references without fsync, and makes+ them durable in a background step a second after a repository goes idle+ (`doc/graft/PLAN7.md`, LOCAL REFERENCE WRITES RULE). Until then the+ repository has a dirty marker in `<storage>/+gitaly/graft-ref-durability/`,+ which holds the kernel's boot identity.+ - *Process crash or restart* (same kernel boot): nothing is lost, because the+ page cache kept every write. The restarted node settles the markers.+ - *Kernel crash or power loss*: the references written since the last durable+ step may be lost or torn, and git reads a torn reftable stack as an empty+ repository without an error. At startup, before it opens any listener,+ `gitaly serve` moves aside every repository that was dirty under the+ previous boot, and the first request hydrates it again from the database and+ the bucket. This hydration starts from the newest checkpoint when there is+ one, so its cost is bounded by how recent the checkpoints are (the+ maintenance that creates them is WP77; until then, by the full history).+ Nothing acknowledged is lost. Cost after a power+ loss: one hydration per repository that had a write in the second before the+ loss, or that was still being written. On a busy node that can be every+ active repository. A repository is only moved when the database holds it+ with changesets and the local copy has a watermark; other markers are+ dropped.+ - *A write the kernel reports as failed* (a writeback error, same boot): the+ durable step's fsync fails, the repository is moved aside at once and logged,+ and the next request hydrates it.+ - Offline tools that read a repository from disk (`gitaly git -C <repo>`,+ `gitaly-debug simulate-http-clone`) must not be pointed at a repository of+ a node after an unclean shutdown before `gitaly serve` has started once:+ they would read the torn stack.+ - Scope: Linux only (the boot identity is `/proc/sys/kernel/random/boot_id`).+ Elsewhere, and in off and shadow modes, git makes every reference write+ durable as before.+ - The node's reference writers (hook transactions and catch-up) wait for the+ repository's reference lock in arrival order instead of git's polling.+- **A corrupt dirty marker** stops `gitaly serve` at startup with the marker's+ path: the repository it named cannot be identified. Check the storage, then+ remove the marker; the repository's next request hydrates it if it is stale.+These behaviours are proved by the end-to-end tests`internal/graft/e2e/storage_*_test.go`, with several real servers sharing onedatabase and one bucket, on SQLite and Postgres: writes on one node and reads@@ -456,6 +495,14 @@ without a single divergence.| `graft_storage_sync_total{result}` | Catch-ups before a request; results `fresh`, `caught_up`, `hydrated`, `error`. || `graft_storage_sync_seconds` | Duration of the catch-up before a request. || `graft_storage_sync_changesets_total`, `graft_storage_sync_bytes_total` | Changesets applied and pack bytes downloaded by catch-up and hydration. |+| `graft_storage_ref_admission_wait_seconds{writer}` | Wait of a local reference writer (`hook`, `syncer`) for its turn at the repository's reference lock. |+| `graft_storage_ref_admission_waiting` | Writers waiting for their turn, all repositories. |+| `graft_storage_ref_admission_bypassed_total{writer}` | Writers that went on to git's lock without their turn after `reftable.lockTimeout` (10 s). |+| `graft_storage_ref_admission_expired_total` | Turns ended by the 10 s bound instead of their release (a git process that died between its phases). |+| `graft_storage_ref_dirty_repositories` | Repositories with reference writes not yet made durable. |+| `graft_storage_ref_dirty_marked_total`, `graft_storage_ref_dirty_cleared_total` | Dirty markers written and removed by the durable step. |+| `graft_storage_ref_writes_inflight` | Hook transactions between their Prepared and Committed phases; one that never commits keeps its repository dirty. |+| `graft_storage_ref_untrusted_total{reason}` | Repositories moved aside to be hydrated again: `reboot` (dirty at a kernel restart), `sync_failed` (the durable step's fsync failed). |## Benchmarkdiff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.goindex 0d6abde..61993e3 100644--- a/internal/cli/gitaly/serve.go+++ b/internal/cli/gitaly/serve.go@@ -37,6 +37,7 @@ import ("github.com/feder-cr/git-ai/internal/gitlab""github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/gate"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"corestoreopen "github.com/feder-cr/git-ai/internal/graft/corestore/open""github.com/feder-cr/git-ai/internal/graft/corestore/reviewroots"corestoresyncer "github.com/feder-cr/git-ai/internal/graft/corestore/syncer"@@ -615,6 +616,12 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {storageGate *gate.Gate)if storageCore.Enabled() {+ if storageCore.Authoritative() {+ if err := openLocalReferenceWrites(storageCore, locator, logger); err != nil {+ return err+ }+ }+syncer := corestoresyncer.New(storageCore, logger, locator, localrepoFactory, gitCmdFactory)prometheus.MustRegister(syncer)storageSyncer = syncer@@ -624,6 +631,20 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {hookManagerOpts = append(hookManagerOpts, hook.WithStorageVoter(storageVoter))if storageCore.Authoritative() {+ // Before any listener opens: a repository whose references may be torn+ // by a kernel restart is moved aside, so nothing reads it before it is+ // hydrated again (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).+ storageNames := make([]string, 0, len(cfg.Storages))+ for _, s := range cfg.Storages {+ storageNames = append(storageNames, s.Name)+ }+ if err := syncer.RecoverLocalReferences(ctx, storageNames); err != nil {+ return err+ }+ durableCtx, stopDurableSteps := context.WithCancel(ctx)+ defer stopDurableSteps()+ go syncer.RunDurableSteps(durableCtx, localrefs.DurableStepInterval)+storageGate = gate.New(storageCore, syncer, logger, locator, localrepoFactory, protoregistry.GitalyProtoPreregistered)}@@ -892,9 +913,6 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {return waitErr}-// newPackObjectsAdaptiveLimit builds the adaptive limit backing a pack objects limiter. The limit-// only moves when the configuration asks for an adaptive one, and is otherwise pinned to the-// configured concurrency.// housekeepReviewRoots runs the backstop of the review roots after the daily// housekeeping of a repository: it writes the keep-around references that// open reviews lack and deletes those no open review uses (in authoritative@@ -921,6 +939,32 @@ func housekeepReviewRoots(ctx context.Context, logger log.Logger, core *corestorreturn nil}+// openLocalReferenceWrites sets up the admission and the deferred durability+// of the local reference writes of the storage core in authoritative mode+// (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). Without the kernel's boot+// identity (not Linux) the durability is not deferred: every reference write+// is made durable by git, as in the other modes.+func openLocalReferenceWrites(core *corestore.Core, locator storage.Locator, logger log.Logger) error {+ lockTimeout := gitcmd.ReftableLockTimeoutMilliseconds * time.Millisecond+ core.RefAdmission = localrefs.NewAdmission(lockTimeout, lockTimeout)+ prometheus.MustRegister(core.RefAdmission)++ durability, err := localrefs.NewDurability(locator, localrefs.DurableStepIdle)+ if err != nil {+ return fmt.Errorf("graft storage core: %w", err)+ }+ if durability == nil {+ logger.Info("graft storage core: the kernel exposes no boot identity, every reference write is made durable by git")+ return nil+ }+ core.RefDurability = durability+ prometheus.MustRegister(durability)+ return nil+}++// newPackObjectsAdaptiveLimit builds the adaptive limit backing a pack objects limiter. The limit+// only moves when the configuration asks for an adaptive one, and is otherwise pinned to the+// configured concurrency.func newPackObjectsAdaptiveLimit(name string, limits config.ConcurrencyLimits) *limiter.AdaptiveLimit {if !limits.Adaptive {return limiter.NewAdaptiveLimit(name, limiter.AdaptiveSetting{diff --git a/internal/git/gitcmd/command_factory.go b/internal/git/gitcmd/command_factory.goindex 946c4f8..4beb11f 100644--- a/internal/git/gitcmd/command_factory.go+++ b/internal/git/gitcmd/command_factory.go@@ -37,6 +37,16 @@ const (// Rate limiter is immediately allocated the maxBurstToken value. Burst is the maximum number of tokens// that can be consumed in a single callmaxBurstToken = 40++ // FsyncComponents are the components git makes durable on every write (`core.fsync`).+ FsyncComponents = "objects,derived-metadata,reference"+ // FsyncComponentsWithoutReferences is FsyncComponents without the references, for the+ // reference writes whose durability the storage core makes itself+ // (updateref.WithDeferredReferenceDurability).+ FsyncComponentsWithoutReferences = "objects,derived-metadata"+ // ReftableLockTimeoutMilliseconds is how long git waits for the reference lock of a reftable+ // repository (`reftable.lockTimeout`).+ ReftableLockTimeoutMilliseconds = 10000)// CommandFactory is designed to create and run git commands in a protected and fully managed manner.@@ -725,7 +735,7 @@ func (cf *ExecCommandFactory) GlobalConfiguration(ctx context.Context) ([]Config// We configure for what data should be fsynced and how that should happen.// Synchronize object files, packed-refs and loose refs to disk to lessen the// likelihood of repository corruption in case the server crashes.- ConfigPair{Key: "core.fsync", Value: "objects,derived-metadata,reference"},+ ConfigPair{Key: "core.fsync", Value: FsyncComponents},ConfigPair{Key: "core.fsyncMethod", Value: "fsync"},// When deleting references, Git needs to rewrite the `packed-refs` file to evict@@ -746,7 +756,7 @@ func (cf *ExecCommandFactory) GlobalConfiguration(ctx context.Context) ([]Config// reference transaction, so it is a shared resource like the `packed-refs` lock// above and gets the same timeout instead of Git's default of 100 milliseconds.// Context cancellation still ends the wait early.- ConfigPair{Key: "reftable.lockTimeout", Value: "10000"},+ ConfigPair{Key: "reftable.lockTimeout", Value: fmt.Sprint(ReftableLockTimeoutMilliseconds)},// Similarly, for loose references we bump the limit from 100 milliseconds to 1 second. We aim for a// lower limit here as the locking for loose references is typically a lot more fine-grained. We have// still observed lock contention around them though, but mostly in cases where the host system wasdiff --git a/internal/git/updateref/updateref.go b/internal/git/updateref/updateref.goindex 8c789ca..57192af 100644--- a/internal/git/updateref/updateref.go+++ b/internal/git/updateref/updateref.go@@ -270,10 +270,21 @@ type UpdaterOpt func(*updaterConfig)type updaterConfig struct {disableTransactions boolnoDeref bool+ deferredDurability boolrefBackend *git.ReferenceBackendrefURI string}+// WithDeferredReferenceDurability makes git write the references without fsync. Only the+// storage core's authoritative mode may set it: its database commit is the durable point of+// the transaction, and it makes the local references durable itself (doc/graft/PLAN7.md,+// LOCAL REFERENCE WRITES RULE). Objects stay durable as for every other write.+func WithDeferredReferenceDurability() UpdaterOpt {+ return func(cfg *updaterConfig) {+ cfg.deferredDurability = true+ }+}+// WithDisabledTransactions disables hooks such that no reference-transactions// are used for the updater.func WithDisabledTransactions() UpdaterOpt {@@ -337,6 +348,9 @@ func New(ctx context.Context, repo gitcmd.RepositoryExecutor, opts ...UpdaterOptif cfg.refURI != "" {cmdOpts = append(cmdOpts, gitcmd.WithEnv("GIT_REFERENCE_BACKEND="+cfg.refURI))}+ if cfg.deferredDurability {+ cmdOpts = append(cmdOpts, gitcmd.WithConfig(gitcmd.ConfigPair{Key: "core.fsync", Value: gitcmd.FsyncComponentsWithoutReferences}))+ }cmd, err := repo.Exec(ctx,diff --git a/internal/git/updateref/updateref_test.go b/internal/git/updateref/updateref_test.goindex b201a9e..df646ce 100644--- a/internal/git/updateref/updateref_test.go+++ b/internal/git/updateref/updateref_test.go@@ -8,6 +8,7 @@ import ("io/fs""os""path/filepath"+ "strings""testing""time"@@ -88,6 +89,61 @@ func TestNew_refURI(t *testing.T) {require.Equal(t, expectedErr, updater.Commit())}+// TestNew_deferredReferenceDurability verifies that only WithDeferredReferenceDurability takes the+// references out of what git makes durable, and that it keeps the objects in.+func TestNew_deferredReferenceDurability(t *testing.T) {+ t.Parallel()++ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ _, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{+ SkipCreationViaService: true,+ })+ repo := gittest.NewRepositoryPathExecutor(t, cfg, repoPath)+ commitID := gittest.WriteCommit(t, cfg, repoPath)++ // effectiveConfig returns the last value of key among the `-c key=value` arguments of a+ // git command line, which is the one git uses.+ effectiveConfig := func(args []string, key string) string {+ var effective string+ for i := 0; i+1 < len(args); i++ {+ if args[i] != "-c" {+ continue+ }+ name, value, _ := strings.Cut(args[i+1], "=")+ if strings.EqualFold(name, key) {+ effective = value+ }+ }+ return effective+ }++ for _, tc := range []struct {+ desc string+ opts []UpdaterOpt+ expected string+ }{+ {desc: "default", expected: gitcmd.FsyncComponents},+ {desc: "without transactions", opts: []UpdaterOpt{WithDisabledTransactions()}, expected: gitcmd.FsyncComponents},+ {desc: "deferred", opts: []UpdaterOpt{WithDisabledTransactions(), WithDeferredReferenceDurability()}, expected: gitcmd.FsyncComponentsWithoutReferences},+ } {+ t.Run(tc.desc, func(t *testing.T) {+ updater, err := New(ctx, repo, tc.opts...)+ require.NoError(t, err)+ defer testhelper.MustClose(t, updater)++ require.Equal(t, tc.expected, effectiveConfig(updater.cmd.Args(), "core.fsync"))++ // The writes themselves are unchanged.+ ref := git.ReferenceName("refs/heads/" + strings.ReplaceAll(tc.desc, " ", "-"))+ require.NoError(t, updater.Start())+ require.NoError(t, updater.Create(ref, commitID))+ require.NoError(t, updater.Commit())+ require.Equal(t, commitID, gittest.ResolveRevision(t, cfg, repoPath, ref.String()))+ })+ }+}+func TestUpdater_create(t *testing.T) {t.Parallel()diff --git a/internal/gitaly/hook/storage_voter.go b/internal/gitaly/hook/storage_voter.goindex 03da717..eb297fd 100644--- a/internal/gitaly/hook/storage_voter.go+++ b/internal/gitaly/hook/storage_voter.go@@ -25,6 +25,19 @@ type StorageVoter interface {// Authoritative reports whether the storage core's database replaces the// file changelog.Authoritative() bool+ // DeferredReferenceDurability reports whether reference transactions that+ // the voter commits may write their references without fsync: the database+ // commit is their durable point, and the storage core makes the local+ // references durable later (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).+ DeferredReferenceDurability() bool+}++// DeferredReferenceDurability reports whether the reference transactions that+// the manager drives itself (UpdateWithHooks) may write their references+// without fsync. It is false without a storage voter, so only the storage+// core's authoritative mode ever turns the fsync of references off.+func (m *GitLabHookManager) DeferredReferenceDurability() bool {+ return m.storageVoter != nil && m.storageVoter.DeferredReferenceDurability()}// WithStorageVoter makes the manager commit every reference transaction todiff --git a/internal/gitaly/hook/storage_voter_test.go b/internal/gitaly/hook/storage_voter_test.goindex c7da8c2..7c411ad 100644--- a/internal/gitaly/hook/storage_voter_test.go+++ b/internal/gitaly/hook/storage_voter_test.go@@ -59,6 +59,8 @@ func (v *recordingVoter) Aborted(context.Context, gitcmd.HooksPayload, []byte) {func (v *recordingVoter) Authoritative() bool { return true }+func (v *recordingVoter) DeferredReferenceDurability() bool { return false }+// TestReferenceTransactionHook_storageVoterPhases checks that the hook manager// hands every phase of a reference transaction to the storage core, the// Preparing phase included (doc/graft/PLAN6.md, LOCK AND HISTORY RULE), anddiff --git a/internal/gitaly/hook/updateref/deferred_durability_test.go b/internal/gitaly/hook/updateref/deferred_durability_test.gonew file mode 100644index 0000000..1b9d414--- /dev/null+++ b/internal/gitaly/hook/updateref/deferred_durability_test.go@@ -0,0 +1,67 @@+package updateref++import (+ "context"+ "testing"++ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/git/gitcmd"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/gitaly/config"+ "github.com/feder-cr/git-ai/internal/gitaly/hook"+ "github.com/feder-cr/git-ai/internal/gitlab"+ "github.com/feder-cr/git-ai/internal/graft/changelog"+ "github.com/feder-cr/git-ai/internal/testhelper"+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"+)++// durabilityVoter is a storage voter that only answers whether it defers the durability of the+// references.+type durabilityVoter struct {+ deferred bool+}++func (durabilityVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte) {}++func (durabilityVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {+ return nil+}++func (durabilityVoter) Committed(context.Context, gitcmd.HooksPayload, []byte) {}++func (durabilityVoter) Aborted(context.Context, gitcmd.HooksPayload, []byte) {}++func (durabilityVoter) Authoritative() bool { return true }++func (v durabilityVoter) DeferredReferenceDurability() bool { return v.deferred }++// TestDeferReferenceDurability verifies that the updater writes references without fsync only+// when the storage voter of its hook manager defers their durability: never without the storage+// core, nor when the voter does not (shadow mode, or authoritative mode where the kernel exposes+// no boot identity; the voter's own answer is tested in the voter package).+func TestDeferReferenceDurability(t *testing.T) {+ t.Parallel()++ cfg := testcfg.Build(t)+ newManager := func(opts ...hook.ManagerOption) hook.Manager {+ return hook.NewManager(cfg, config.NewLocator(cfg), testhelper.NewLogger(t), gittest.NewCommandFactory(t, cfg), nil, nil,+ gitlab.NewMockClient(t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive),+ hook.NewPostReceiveRegistry(), opts...)+ }++ for _, tc := range []struct {+ desc string+ manager hook.Manager+ expected bool+ }{+ {desc: "storage core off", manager: newManager()},+ {desc: "voter keeps the fsync", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: false}))},+ {desc: "voter defers the durability", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: true})), expected: true},+ {desc: "manager without a storage voter", manager: hook.NewMockManager(t, hook.NopPreReceive, hook.NopPostReceive, hook.NopUpdate, hook.NopReferenceTransaction)},+ } {+ t.Run(tc.desc, func(t *testing.T) {+ require.Equal(t, tc.expected, deferReferenceDurability(tc.manager))+ })+ }+}diff --git a/internal/gitaly/hook/updateref/update_with_hooks.go b/internal/gitaly/hook/updateref/update_with_hooks.goindex 407b2f9..acf21e7 100644--- a/internal/gitaly/hook/updateref/update_with_hooks.go+++ b/internal/gitaly/hook/updateref/update_with_hooks.go@@ -140,6 +140,19 @@ func (e Error) Proto() *gitalypb.ReferenceUpdateError {}}+// deferredDurability is implemented by a hook manager whose storage core writes the references+// of the transactions it commits without fsync (hook.GitLabHookManager).+type deferredDurability interface {+ DeferredReferenceDurability() bool+}++// deferReferenceDurability reports whether the reference transactions driven with hookManager+// may write their references without fsync: only when its storage voter says so.+func deferReferenceDurability(hookManager hook.Manager) bool {+ d, ok := hookManager.(deferredDurability)+ return ok && d.DeferredReferenceDurability()+}+// NewUpdaterWithHooks creates a new instance of a struct that will update a Git reference.func NewUpdaterWithHooks(cfg config.Cfg,@@ -257,7 +270,13 @@ func (u *UpdaterWithHooks) UpdateReference(// is packed, which is obviously a bad thing as Gitaly nodes may be differently packed. We// thus continue to manually drive the reference-transaction hook here, which doesn't have// this problem.- updater, err := updateref.New(ctx, repo, updateref.WithDisabledTransactions())+ updaterOpts := []updateref.UpdaterOpt{updateref.WithDisabledTransactions()}+ // The storage core's authoritative mode makes the references durable itself: its database+ // commit in the prepared phase is the durable point of the transaction.+ if deferReferenceDurability(u.hookManager) {+ updaterOpts = append(updaterOpts, updateref.WithDeferredReferenceDurability())+ }+ updater, err := updateref.New(ctx, repo, updaterOpts...)if err != nil {return fmt.Errorf("creating updater: %w", err)}diff --git a/internal/graft/corestore/core.go b/internal/graft/corestore/core.goindex 2f8cdee..df13121 100644--- a/internal/graft/corestore/core.go+++ b/internal/graft/corestore/core.go@@ -10,6 +10,7 @@ import ("time""github.com/feder-cr/git-ai/internal/gitaly/storage"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs")// BlobStore is the object storage of the storage core. Objects are immutable@@ -99,6 +100,13 @@ type Core struct {// runs in shadow mode too.GCGrace time.DurationGCMargin time.Duration+ // RefAdmission orders the local reference writers of the node and+ // RefDurability defers the durability of their writes to a durable step+ // (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). Both are set in+ // authoritative mode only; nil admits nothing and keeps every reference write+ // durable by itself.+ RefAdmission *localrefs.Admission+ RefDurability *localrefs.Durability}// WriteDeadline is W of doc/graft/PLAN7.md: a writer uploads a pack anddiff --git a/internal/graft/corestore/coretest/coretest.go b/internal/graft/corestore/coretest/coretest.goindex 93b1b95..829d77f 100644--- a/internal/graft/corestore/coretest/coretest.go+++ b/internal/graft/corestore/coretest/coretest.go@@ -20,8 +20,10 @@ import ("testing""time"+ "github.com/feder-cr/git-ai/internal/gitaly/storage""github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/blobstore"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs""github.com/feder-cr/git-ai/internal/graft/corestore/sqlstore""github.com/stretchr/testify/require")@@ -196,3 +198,20 @@ func Open(tb testing.TB, opts Options) *corestore.Core {return core}++// EnableLocalRefs gives the core the admission and the deferred durability of+// the local reference writes, as gitaly serve does in authoritative mode+// (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). The boot identity is a+// fixed test value unless opts set another one.+func EnableLocalRefs(tb testing.TB, core *corestore.Core, locator storage.Locator, opts ...localrefs.DurabilityOption) {+ tb.Helper()++ core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)+ durability, err := localrefs.NewDurability(locator, 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID)}, opts...)...)+ require.NoError(tb, err)+ core.RefDurability = durability+}++// TestBootID is the boot identity EnableLocalRefs gives the deferred+// durability by default.+const TestBootID = "test-boot"diff --git a/internal/graft/corestore/localrefs/admission.go b/internal/graft/corestore/localrefs/admission.gonew file mode 100644index 0000000..64a4536--- /dev/null+++ b/internal/graft/corestore/localrefs/admission.go@@ -0,0 +1,286 @@+// Package localrefs orders and makes durable the local reference writes of the+// storage core in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE+// WRITES RULE).+//+// A reftable repository has one lock for all its references (tables.list.lock),+// and git waits for it by polling with a growing, randomized sleep: there is no+// queue, a newcomer can take the lock before a writer that has waited for long,+// and the lock can sit free while the waiters sleep. Admission puts the writers+// of one node in a FIFO queue per repository in front of that lock. It only+// orders: git's lock stays the only mutual exclusion, so every bound of the+// admission falls back to git's own wait.+//+// Durability moves the durability of the local reference writes from each+// transaction to a durable step that runs outside the writers' path; see+// Durability.+package localrefs++import (+ "container/list"+ "context"+ "encoding/binary"+ "sync"+ "sync/atomic"+ "time"++ "github.com/prometheus/client_golang/prometheus"+)++// Key identifies the holder of an admission between Acquire and Release. The+// voter uses the pending key of the transaction (its repository and hook input).+type Key [32]byte++// Writers, the label of the admission metrics.+const (+ // WriterHook is a reference transaction of the reference-transaction hook.+ WriterHook = "hook"+ // WriterSyncer is a reference transaction or a durable step of the syncer.+ WriterSyncer = "syncer"+)++// Admission is the FIFO admission of the reference writers of one node, per+// local repository (keyed by its path). A nil *Admission admits nothing and+// never waits.+type Admission struct {+ // waitBound is how long Acquire waits before the writer goes on to git's+ // lock without being admitted. It equals git's own wait for the lock+ // (reftable.lockTimeout), so that a write that today ends in git's lock+ // timeout (another transaction on the same repository held open by the+ // same caller) does not wait forever on the admission instead.+ waitBound time.Duration+ // holdBound is how long an admission is held at most. The hook passes no+ // transaction identity and git can die between the phases of a transaction+ // without reporting Committed or Aborted, so an admission is never held+ // beyond it.+ holdBound time.Duration++ mu sync.Mutex+ queues map[string]*queue+ // unique numbers the keys of AcquireUnique.+ unique atomic.Uint64+ // onWait is called when a writer starts waiting; tests set it to know that+ // a writer is queued.+ onWait func(path string)++ wait *prometheus.HistogramVec+ waiting prometheus.Gauge+ bypassed *prometheus.CounterVec+ expired prometheus.Counter+}++var _ prometheus.Collector = (*Admission)(nil)++// queue is the admission of one repository: the holder and the writers waiting+// behind it, oldest first.+type queue struct {+ holder *ticket+ waiters *list.List+}++// ticket is one writer of a queue.+type ticket struct {+ key Key+ // admitted is closed when the ticket becomes the holder.+ admitted chan struct{}+ // element is the ticket in the waiters while it waits.+ element *list.Element+ // expiry ends the admission at the hold bound.+ expiry *time.Timer+}++// NewAdmission returns an admission with the given bounds, both positive.+func NewAdmission(waitBound, holdBound time.Duration) *Admission {+ a := &Admission{+ waitBound: waitBound,+ holdBound: holdBound,+ queues: map[string]*queue{},+ wait: prometheus.NewHistogramVec(prometheus.HistogramOpts{+ Name: "graft_storage_ref_admission_wait_seconds",+ Help: "Time a local reference writer waited for its admission to the reference lock of a repository, by writer (hook, syncer).",+ Buckets: []float64{0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},+ }, []string{"writer"}),+ waiting: prometheus.NewGauge(prometheus.GaugeOpts{+ Name: "graft_storage_ref_admission_waiting",+ Help: "Number of local reference writers waiting for their admission, all repositories.",+ }),+ bypassed: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "graft_storage_ref_admission_bypassed_total",+ Help: "Number of local reference writers that went on to git's reference lock without an admission after the wait bound, by writer.",+ }, []string{"writer"}),+ expired: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "graft_storage_ref_admission_expired_total",+ Help: "Number of admissions ended by the hold bound instead of their release (a writer that never reported its end).",+ }),+ }+ for _, writer := range []string{WriterHook, WriterSyncer} {+ a.wait.WithLabelValues(writer)+ a.bypassed.WithLabelValues(writer)+ }+ return a+}++// Acquire waits for the admission of key to the reference lock of the+// repository at path. It returns true when the writer is admitted, and must+// then call Release(path, key) when it no longer needs the lock. It returns+// false without an admission when ctx ends (the writer is gone) or the wait+// bound passes (the writer goes on to git's lock as without admission).+func (a *Admission) Acquire(ctx context.Context, path string, key Key, writer string) bool {+ if a == nil {+ return false+ }++ start := time.Now()+ t := &ticket{key: key, admitted: make(chan struct{})}++ a.mu.Lock()+ q := a.queues[path]+ if q == nil {+ q = &queue{waiters: list.New()}+ a.queues[path] = q+ }+ if q.holder == nil && q.waiters.Len() == 0 {+ a.admitLocked(path, q, t)+ a.mu.Unlock()+ a.wait.WithLabelValues(writer).Observe(time.Since(start).Seconds())+ return true+ }+ t.element = q.waiters.PushBack(t)+ a.waiting.Inc()+ a.mu.Unlock()+ if a.onWait != nil {+ a.onWait(path)+ }++ bound := time.NewTimer(a.waitBound)+ defer bound.Stop()++ var cancelled bool+ select {+ case <-t.admitted:+ a.wait.WithLabelValues(writer).Observe(time.Since(start).Seconds())+ return true+ case <-bound.C:+ case <-ctx.Done():+ cancelled = true+ }++ a.mu.Lock()+ defer a.mu.Unlock()++ select {+ case <-t.admitted:+ // Admitted while giving up.+ if cancelled {+ a.handOffLocked(path, q)+ return false+ }+ a.wait.WithLabelValues(writer).Observe(time.Since(start).Seconds())+ return true+ default:+ }++ q.waiters.Remove(t.element)+ t.element = nil+ a.waiting.Dec()+ if q.holder == nil && q.waiters.Len() == 0 {+ delete(a.queues, path)+ }+ if !cancelled {+ a.bypassed.WithLabelValues(writer).Inc()+ a.wait.WithLabelValues(writer).Observe(time.Since(start).Seconds())+ }+ return false+}++// AcquireUnique is Acquire with a key of its own. It returns the release+// function, nil when the writer was not admitted.+func (a *Admission) AcquireUnique(ctx context.Context, path, writer string) func() {+ if a == nil {+ return nil+ }++ // The keys of the voter are SHA-256 sums; a counter in the first eight bytes+ // and a marker byte that is set make a key that no other writer uses.+ var key Key+ binary.BigEndian.PutUint64(key[:8], a.unique.Add(1))+ key[31] = 0xff+ key[30] = 0xfe++ if !a.Acquire(ctx, path, key, writer) {+ return nil+ }+ return func() { a.Release(path, key) }+}++// Release ends the admission of key to the reference lock of the repository at+// path. It does nothing when key does not hold it: the admission ended at the+// hold bound, or the writer was not admitted. Two transactions with the same+// key (the same repository and hook input) are not told apart: the release of+// one may end the admission of the other, which then waits on git's lock as+// without admission.+func (a *Admission) Release(path string, key Key) {+ if a == nil {+ return+ }++ a.mu.Lock()+ defer a.mu.Unlock()++ q := a.queues[path]+ if q == nil || q.holder == nil || q.holder.key != key {+ return+ }+ q.holder.expiry.Stop()+ a.handOffLocked(path, q)+}++// admitLocked makes t the holder of q.+func (a *Admission) admitLocked(path string, q *queue, t *ticket) {+ q.holder = t+ close(t.admitted)+ t.expiry = time.AfterFunc(a.holdBound, func() { a.expire(path, t) })+}++// handOffLocked ends the admission of the holder of q and admits the oldest+// waiter, if any.+func (a *Admission) handOffLocked(path string, q *queue) {+ q.holder = nil+ front := q.waiters.Front()+ if front == nil {+ delete(a.queues, path)+ return+ }+ next := q.waiters.Remove(front).(*ticket)+ next.element = nil+ a.waiting.Dec()+ a.admitLocked(path, q, next)+}++// expire ends the admission of t at the hold bound, if t still holds it.+func (a *Admission) expire(path string, t *ticket) {+ a.mu.Lock()+ defer a.mu.Unlock()++ q := a.queues[path]+ if q == nil || q.holder != t {+ return+ }+ a.expired.Inc()+ a.handOffLocked(path, q)+}++// Describe implements prometheus.Collector.+func (a *Admission) Describe(descs chan<- *prometheus.Desc) {+ a.wait.Describe(descs)+ a.waiting.Describe(descs)+ a.bypassed.Describe(descs)+ a.expired.Describe(descs)+}++// Collect implements prometheus.Collector.+func (a *Admission) Collect(metrics chan<- prometheus.Metric) {+ a.wait.Collect(metrics)+ a.waiting.Collect(metrics)+ a.bypassed.Collect(metrics)+ a.expired.Collect(metrics)+}diff --git a/internal/graft/corestore/localrefs/admission_test.go b/internal/graft/corestore/localrefs/admission_test.gonew file mode 100644index 0000000..69d13c1--- /dev/null+++ b/internal/graft/corestore/localrefs/admission_test.go@@ -0,0 +1,173 @@+package localrefs++import (+ "context"+ "testing"+ "time"++ "github.com/prometheus/client_golang/prometheus/testutil"+ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/testhelper"+)++func TestMain(m *testing.M) {+ testhelper.Run(m)+}++func key(i byte) Key { return Key{i} }++// queued returns an admission whose onWait reports every queued writer.+func queued(waitBound, holdBound time.Duration) (*Admission, <-chan string) {+ a := NewAdmission(waitBound, holdBound)+ waiting := make(chan string, 16)+ a.onWait = func(path string) { waiting <- path }+ return a, waiting+}++// acquireAsync starts Acquire in the background, waits until the writer is+// queued, and returns the channel of its result.+func acquireAsync(ctx context.Context, t *testing.T, a *Admission, waiting <-chan string, path string, k Key) <-chan bool {+ t.Helper()+ result := make(chan bool, 1)+ go func() { result <- a.Acquire(ctx, path, k, WriterHook) }()+ require.Equal(t, path, <-waiting)+ return result+}++func TestAdmissionAdmitsInArrivalOrder(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)+ a, waiting := queued(time.Hour, time.Hour)++ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))+ results := []<-chan bool{+ acquireAsync(ctx, t, a, waiting, "repo", key(1)),+ acquireAsync(ctx, t, a, waiting, "repo", key(2)),+ acquireAsync(ctx, t, a, waiting, "repo", key(3)),+ }+ require.Equal(t, 3.0, testutil.ToFloat64(a.waiting))++ // Another repository is not ordered behind them.+ require.True(t, a.Acquire(ctx, "other", key(9), WriterHook))++ for i, result := range results {+ select {+ case <-result:+ t.Fatalf("writer %d admitted before the release of writer %d", i+1, i)+ default:+ }+ a.Release("repo", key(byte(i)))+ require.True(t, <-result, "writer %d", i+1)+ }+ a.Release("repo", key(3))+ a.Release("other", key(9))++ require.Empty(t, a.queues)+ require.Equal(t, 0.0, testutil.ToFloat64(a.waiting))+}++func TestAdmissionCancelledWriterLeavesTheQueue(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)+ a, waiting := queued(time.Hour, time.Hour)++ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))+ cancelCtx, cancel := context.WithCancel(ctx)+ cancelled := acquireAsync(cancelCtx, t, a, waiting, "repo", key(1))+ next := acquireAsync(ctx, t, a, waiting, "repo", key(2))++ cancel()+ require.False(t, <-cancelled)++ a.Release("repo", key(0))+ require.True(t, <-next)+ a.Release("repo", key(2))+ require.Empty(t, a.queues)+ require.Equal(t, 0.0, testutil.ToFloat64(a.bypassed.WithLabelValues(WriterHook)))+}++func TestAdmissionWaitBoundFallsBackToGitsLock(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)+ a := NewAdmission(20*time.Millisecond, time.Hour)++ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))+ require.False(t, a.Acquire(ctx, "repo", key(1), WriterSyncer))+ require.Equal(t, 1.0, testutil.ToFloat64(a.bypassed.WithLabelValues(WriterSyncer)))++ // The writer that went on without an admission releases nothing.+ a.Release("repo", key(1))+ require.Equal(t, key(0), a.queues["repo"].holder.key)++ a.Release("repo", key(0))+ require.Empty(t, a.queues)+}++func TestAdmissionHoldBoundEndsALostAdmission(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)+ a, waiting := queued(time.Hour, time.Hour)++ // Writer 0 never releases (git died between its phases); the hold bound+ // ends its admission, which is what its timer calls.+ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))+ second := acquireAsync(ctx, t, a, waiting, "repo", key(1))+ a.mu.Lock()+ lost := a.queues["repo"].holder+ a.mu.Unlock()+ a.expire("repo", lost)+ require.True(t, <-second)+ require.Equal(t, 1.0, testutil.ToFloat64(a.expired))++ // The late release of writer 0 does not end the admission of writer 1, and+ // a second expiry of the lost admission does nothing.+ a.Release("repo", key(0))+ a.expire("repo", lost)+ a.mu.Lock()+ require.Equal(t, key(1), a.queues["repo"].holder.key)+ a.mu.Unlock()+ require.Equal(t, 1.0, testutil.ToFloat64(a.expired))++ a.Release("repo", key(1))+ require.Empty(t, a.queues)+}++func TestAdmissionHoldBoundTimer(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)+ a, waiting := queued(time.Hour, 20*time.Millisecond)++ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))+ require.True(t, <-acquireAsync(ctx, t, a, waiting, "repo", key(1)))+ require.GreaterOrEqual(t, testutil.ToFloat64(a.expired), 1.0)+}++func TestAdmissionUniqueKeysAreOrdered(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)+ a, waiting := queued(time.Hour, time.Hour)++ release := a.AcquireUnique(ctx, "repo", WriterSyncer)+ require.NotNil(t, release)++ second := make(chan func(), 1)+ go func() { second <- a.AcquireUnique(ctx, "repo", WriterSyncer) }()+ require.Equal(t, "repo", <-waiting)++ release()+ releaseSecond := <-second+ require.NotNil(t, releaseSecond)+ releaseSecond()+ require.Empty(t, a.queues)+}++func TestNilAdmissionAdmitsNothing(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)++ var a *Admission+ require.False(t, a.Acquire(ctx, "repo", key(0), WriterHook))+ require.Nil(t, a.AcquireUnique(ctx, "repo", WriterSyncer))+ a.Release("repo", key(0))+}diff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.gonew file mode 100644index 0000000..ca0a905--- /dev/null+++ b/internal/graft/corestore/localrefs/durability.go@@ -0,0 +1,793 @@+package localrefs++import (+ "bufio"+ "bytes"+ "context"+ "crypto/sha256"+ "encoding/hex"+ "encoding/json"+ "errors"+ "fmt"+ "io/fs"+ "os"+ "path/filepath"+ "strings"+ "sync"+ "time"++ "github.com/prometheus/client_golang/prometheus"++ "github.com/feder-cr/git-ai/internal/gitaly/config"+ "github.com/feder-cr/git-ai/internal/gitaly/storage"+ "github.com/feder-cr/git-ai/proto/go/gitalypb"+)++// MarkerDirName is the directory of the dirty markers in a storage, under+// Gitaly's data directory. It is not the state directory, whose files the disk+// cache walker removes after an hour.+const MarkerDirName = config.GitalyDataPrefix + "/graft-ref-durability"++// bootIDPath is where Linux exposes the identity of the running kernel boot.+const bootIDPath = "/proc/sys/kernel/random/boot_id"++const (+ // DurableStepInterval is how often the durable step runs for every dirty+ // repository: it bounds how long a write failure the kernel reports goes+ // unnoticed.+ DurableStepInterval = time.Second+ // DurableStepIdle is how long a dirty repository must be without a reference+ // write before the durable step removes its marker.+ DurableStepIdle = time.Second+)++// CrashEnv names the test-only environment variable that makes the process+// exit (exit code 75) in the durable step, while it holds git's reference lock+// after making the stack durable and before it removes the marker: "durable_step".+// The crash tests with real binaries use it. It is read once by NewDurability.+const CrashEnv = "GRAFT_STORAGE_TEST_CRASH"++const (+ crashDurableStep = "durable_step"+ crashExitCode = 75+)++// stackLockContent is what the durable step writes into the reference lock+// it takes. git never writes it (a lock of git holds table names), so a lock+// with this content found at startup was left by a durable step that died.+const stackLockContent = "graft durable step\n"++// ErrUntrusted is returned by Settle when making the references of a+// repository durable failed: the kernel may have dropped some of their writes,+// so the local repository must not be served before it is rebuilt.+var ErrUntrusted = errors.New("storage core: the local references may have lost writes")++// UntrustedError is ErrUntrusted for one repository.+type UntrustedError struct {+ Repo RepoRef+ Path string+ Err error+}++func (e *UntrustedError) Error() string {+ return fmt.Sprintf("%v: %s: %v", ErrUntrusted, e.Path, e.Err)+}++// Unwrap returns ErrUntrusted and the cause.+func (e *UntrustedError) Unwrap() []error { return []error{ErrUntrusted, e.Err} }++// Durability defers the durability of the local reference writes of the+// storage core in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE+// WRITES RULE). The database commit of a transaction is its durable commit+// point and the local repository is a cache of it, so git writes the references+// without fsync and Durability keeps the cache safe to serve:+//+// - before the first reference write that is not durable, the repository+// gets a dirty marker, written durably, that holds the identity of the+// kernel boot (MarkDirty);+// - a durable step makes the references durable and removes the marker,+// under git's reference lock and only when no writer is between its+// Prepared and Committed phases (Settle);+// - at startup, a marker of another boot means the kernel restarted while+// the repository had writes that may never have reached the disk: its+// reftable stack may be torn, which git reads as an empty repository, so+// the repository is moved aside and hydrated again (Recover). A marker of+// the running boot is a process restart: the kernel kept every write.+//+// A nil *Durability is off: every reference write is durable by itself+// (core.fsync with references), as outside authoritative mode.+type Durability struct {+ bootID string+ locator storage.Locator+ // idle is how long a dirty repository must be without a reference write+ // before Settle removes its marker.+ idle time.Duration+ // sync makes a file or directory durable; tests replace it.+ sync func(f *os.File) error+ // crashAt is the crash point selected with CrashEnv.+ crashAt string++ mu sync.Mutex+ // repos holds the state of every repository written since the process+ // started (one entry per repository; Forget drops it).+ repos map[string]*refState++ marked prometheus.Counter+ cleared prometheus.Counter+ untrusted *prometheus.CounterVec+ dirty prometheus.Gauge+ inflight prometheus.Gauge+}++var _ prometheus.Collector = (*Durability)(nil)++// refState is what Durability knows of one local repository, by its path. Its+// fields are guarded by Durability.mu, except marker writes and removals,+// which file serializes.+type refState struct {+ // file serializes the writes and the removal of the marker file.+ file sync.Mutex+ repo RepoRef+ marker string+ dirty bool+ // inflight counts, by key, the hook transactions between their Prepared+ // and Committed phases: git may still write their table and compact the+ // stack without fsync.+ inflight map[Key]int+ nInflight int+ lastWrite time.Time+ // synced holds the tables made durable during the current dirty period.+ // Tables are immutable, so one fsync each is enough.+ synced map[string]struct{}+}++// RepoRef names a repository of a storage.+type RepoRef struct {+ Storage string `json:"storage"`+ RelativePath string `json:"relative_path"`+}++func repoOf(r RepoRef) *gitalypb.Repository {+ return &gitalypb.Repository{StorageName: r.Storage, RelativePath: r.RelativePath}+}++// marker is the content of a dirty marker.+type marker struct {+ RepoRef+ BootID string `json:"boot_id"`+}++// Reasons of graft_storage_ref_untrusted_total.+const (+ // ReasonReboot is a dirty repository found after a kernel restart.+ ReasonReboot = "reboot"+ // ReasonSyncFailed is a durable step whose fsync failed.+ ReasonSyncFailed = "sync_failed"+)++// DurabilityOption configures a Durability.+type DurabilityOption func(*Durability)++// WithBootID sets the identity of the kernel boot instead of reading it from+// the kernel. Tests use it to simulate a kernel restart.+func WithBootID(bootID string) DurabilityOption {+ return func(d *Durability) { d.bootID = bootID }+}++// WithSync replaces how a file or directory is made durable. Tests use it to+// simulate a write the kernel failed to do.+func WithSync(sync func(f *os.File) error) DurabilityOption {+ return func(d *Durability) { d.sync = sync }+}++// NewDurability returns the deferred durability of the node, or nil when the+// kernel does not expose its boot identity (not Linux) and no option sets one:+// the references are then written durably by every transaction.+func NewDurability(locator storage.Locator, idle time.Duration, opts ...DurabilityOption) (*Durability, error) {+ d := newDurability(locator, idle)+ for _, opt := range opts {+ opt(d)+ }+ if d.bootID != "" {+ return d, nil+ }++ content, err := os.ReadFile(bootIDPath)+ if errors.Is(err, fs.ErrNotExist) {+ return nil, nil+ }+ if err != nil {+ return nil, fmt.Errorf("read the boot identity: %w", err)+ }+ d.bootID = strings.TrimSpace(string(content))+ if d.bootID == "" {+ return nil, fmt.Errorf("read the boot identity: %s is empty", bootIDPath)+ }+ return d, nil+}++func newDurability(locator storage.Locator, idle time.Duration) *Durability {+ return &Durability{+ locator: locator,+ idle: idle,+ sync: (*os.File).Sync,+ crashAt: os.Getenv(CrashEnv),+ repos: map[string]*refState{},+ marked: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "graft_storage_ref_dirty_marked_total",+ Help: "Number of dirty markers written: a repository got local reference writes that are not durable yet.",+ }),+ cleared: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "graft_storage_ref_dirty_cleared_total",+ Help: "Number of dirty markers removed by a durable step.",+ }),+ untrusted: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "graft_storage_ref_untrusted_total",+ Help: "Number of local repositories whose references could not be trusted and were moved aside to be hydrated again, by reason (reboot, sync_failed).",+ }, []string{"reason"}),+ dirty: prometheus.NewGauge(prometheus.GaugeOpts{+ Name: "graft_storage_ref_dirty_repositories",+ Help: "Number of local repositories with reference writes that are not durable yet.",+ }),+ inflight: prometheus.NewGauge(prometheus.GaugeOpts{+ Name: "graft_storage_ref_writes_inflight",+ Help: "Number of hook reference transactions between their Prepared and Committed phases. One that never reports Committed keeps its repository dirty.",+ }),+ }+}++// Enabled reports whether the reference writes of the storage core are made+// without fsync.+func (d *Durability) Enabled() bool { return d != nil }++// markerPath returns the path of the dirty marker of the repository at path.+func (d *Durability) markerPath(ctx context.Context, repo RepoRef, path string) (string, error) {+ dir, err := d.markerDir(ctx, repo.Storage)+ if err != nil {+ return "", err+ }+ sum := sha256.Sum256([]byte(path))+ return filepath.Join(dir, hex.EncodeToString(sum[:])+".json"), nil+}++func (d *Durability) markerDir(ctx context.Context, storageName string) (string, error) {+ root, err := d.locator.GetStorageByName(ctx, storageName)+ if err != nil {+ return "", err+ }+ return filepath.Join(root, MarkerDirName), nil+}++// stateLocked returns the state of the repository at path, creating it. The+// caller holds d.mu.+func (d *Durability) stateLocked(path string) *refState {+ st := d.repos[path]+ if st == nil {+ st = &refState{inflight: map[Key]int{}, synced: map[string]struct{}{}}+ d.repos[path] = st+ }+ return st+}++// MarkDirty records that the repository at path is about to get reference+// writes that are not durable. When it returns nil the repository has a durable+// dirty marker; the writer must not write before. The caller holds git's+// reference lock of the repository (the voter's Prepared phase) or the syncer's+// repository lock, under which Settle decides to clear the marker, and the+// marker file is only written or removed under the state's file lock, so a+// marker is never removed after a writer relied on it.+func (d *Durability) MarkDirty(ctx context.Context, repo RepoRef, path string) error {+ if d == nil {+ return nil+ }++ d.mu.Lock()+ st := d.stateLocked(path)+ st.lastWrite = time.Now()+ d.mu.Unlock()++ st.file.Lock()+ defer st.file.Unlock()++ d.mu.Lock()+ dirty := st.dirty+ d.mu.Unlock()+ if dirty {+ return nil+ }++ markerPath, err := d.markerPath(ctx, repo, path)+ if err != nil {+ return err+ }+ content, err := json.Marshal(marker{RepoRef: repo, BootID: d.bootID})+ if err != nil {+ return fmt.Errorf("encode the dirty marker: %w", err)+ }+ if err := d.writeDurably(markerPath, content); err != nil {+ return fmt.Errorf("write the dirty marker: %w", err)+ }++ d.mu.Lock()+ defer d.mu.Unlock()+ st.dirty = true+ st.repo = repo+ st.marker = markerPath+ d.marked.Inc()+ d.dirty.Inc()+ return nil+}++// Begin records that the hook transaction key is past its Prepared phase on+// the repository at path: git may write its table and compact the stack until+// it reports Committed (End). It is called under git's reference lock.+func (d *Durability) Begin(path string, key Key) {+ if d == nil {+ return+ }++ d.mu.Lock()+ defer d.mu.Unlock()+ st := d.stateLocked(path)+ st.inflight[key]+++ st.nInflight+++ d.inflight.Inc()+}++// End records that the hook transaction key committed. A transaction aborted+// after its Prepared phase is not ended: two transactions with the same key+// cannot be told apart, and ending the wrong one would let Settle remove the+// marker while the other one still writes. Its repository stays dirty instead.+func (d *Durability) End(path string, key Key) {+ if d == nil {+ return+ }++ d.mu.Lock()+ defer d.mu.Unlock()+ st := d.repos[path]+ if st == nil || st.inflight[key] == 0 {+ return+ }+ st.inflight[key]--+ if st.inflight[key] == 0 {+ delete(st.inflight, key)+ }+ st.nInflight--+ d.inflight.Dec()+}++// Dirty returns the paths of the dirty repositories.+func (d *Durability) Dirty() []string {+ if d == nil {+ return nil+ }++ d.mu.Lock()+ defer d.mu.Unlock()+ paths := make([]string, 0, len(d.repos))+ for path, st := range d.repos {+ if st.dirty {+ paths = append(paths, path)+ }+ }+ return paths+}++// Settle is the durable step of the repository at path. It makes the tables+// that the stack lists, the list and the directory durable, which reports a+// write the kernel failed to do (an *UntrustedError: the caller moves the+// repository aside at once). When the repository has been idle for the idle+// period and no hook transaction is in flight, it then removes the marker:+// the decision is taken under git's reference lock, taken without waiting (a+// busy lock leaves the marker for the next step), and the marker file is+// removed after the lock is released, under the state's file lock that a+// writer's MarkDirty waits for. A repository that no longer exists (moved+// aside as a stale cache) loses its marker. The caller holds the syncer's+// repository lock, so no syncer write and no replacement of the repository+// runs meanwhile.+func (d *Durability) Settle(path string) error {+ if d == nil {+ return nil+ }++ d.mu.Lock()+ st := d.repos[path]+ if st == nil || !st.dirty {+ d.mu.Unlock()+ return nil+ }+ repo := st.repo+ idle := st.nInflight == 0 && time.Since(st.lastWrite) >= d.idle+ d.mu.Unlock()++ if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {+ st.file.Lock()+ defer st.file.Unlock()+ d.clear(st)+ return d.removeMarker(st)+ }++ if err := d.syncStack(path, st); err != nil {+ return &UntrustedError{Repo: repo, Path: path, Err: err}+ }+ if !idle {+ return nil+ }++ unlock, ok, err := lockStack(path)+ if err != nil {+ return err+ }+ if !ok {+ return nil+ }++ cleared, err := d.settleLocked(path, st)+ if err != nil {+ _ = unlock()+ return &UntrustedError{Repo: repo, Path: path, Err: err}+ }+ if !cleared {+ return unlock()+ }++ // The state is clean from here on, and a writer that takes git's lock now+ // writes a new marker: it waits for the file lock, held until the old+ // marker is removed.+ defer st.file.Unlock()+ if err := unlock(); err != nil {+ return err+ }+ return d.removeMarker(st)+}++// settleLocked makes the tables written since the first pass durable and, when+// no hook transaction is in flight, marks the state clean with its file lock+// held (the caller releases it). The caller holds git's reference lock: no+// transaction can pass its Prepared phase meanwhile, and the transactions+// counted before have committed when the count is zero.+func (d *Durability) settleLocked(path string, st *refState) (bool, error) {+ d.mu.Lock()+ busy := st.nInflight > 0+ d.mu.Unlock()+ if busy {+ return false, nil+ }++ if err := d.syncStack(path, st); err != nil {+ return false, err+ }+ if d.crashAt == crashDurableStep {+ os.Exit(crashExitCode)+ }++ st.file.Lock()+ d.clear(st)+ return true, nil+}++// clear marks the state clean. The caller holds the state's file lock.+func (d *Durability) clear(st *refState) {+ d.mu.Lock()+ defer d.mu.Unlock()+ if st.dirty {+ st.dirty = false+ d.cleared.Inc()+ d.dirty.Dec()+ }+ st.synced = map[string]struct{}{}+}++// removeMarker removes the marker file of a clean state. The caller holds the+// state's file lock.+func (d *Durability) removeMarker(st *refState) error {+ if st.marker == "" {+ return nil+ }+ if err := os.Remove(st.marker); err != nil && !errors.Is(err, fs.ErrNotExist) {+ return fmt.Errorf("remove the dirty marker: %w", err)+ }+ return nil+}++// lockStack takes git's reference lock of the repository at path without+// waiting: ok is false when it is held. The lock is created with its content+// at once (a linked temporary file), so that a lock left by a process that+// died is recognized at startup (removeStaleStackLock).+func lockStack(path string) (unlock func() error, ok bool, err error) {+ lockPath := filepath.Join(path, "reftable", "tables.list.lock")++ tmp, err := os.CreateTemp(path, ".graft-durable-step-*")+ if err != nil {+ return nil, false, fmt.Errorf("take the reference lock: %w", err)+ }+ defer func() { _ = os.Remove(tmp.Name()) }()+ if _, err := tmp.WriteString(stackLockContent); err != nil {+ _ = tmp.Close()+ return nil, false, fmt.Errorf("take the reference lock: %w", err)+ }+ if err := tmp.Close(); err != nil {+ return nil, false, fmt.Errorf("take the reference lock: %w", err)+ }++ if err := os.Link(tmp.Name(), lockPath); err != nil {+ if errors.Is(err, fs.ErrExist) {+ return nil, false, nil+ }+ return nil, false, fmt.Errorf("take the reference lock: %w", err)+ }+ return func() error {+ if err := os.Remove(lockPath); err != nil {+ return fmt.Errorf("release the reference lock: %w", err)+ }+ return nil+ }, true, nil+}++// removeStaleStackLock removes the reference lock of the repository at path+// when a durable step of a process that died left it: git's writes would fail+// on it until housekeeping removes stale locks. A lock of git is never+// removed.+func removeStaleStackLock(path string) error {+ lockPath := filepath.Join(path, "reftable", "tables.list.lock")+ content, err := os.ReadFile(lockPath)+ if errors.Is(err, fs.ErrNotExist) {+ return nil+ }+ if err != nil {+ return fmt.Errorf("read the reference lock: %w", err)+ }+ if !bytes.Equal(content, []byte(stackLockContent)) {+ return nil+ }+ if err := os.Remove(lockPath); err != nil && !errors.Is(err, fs.ErrNotExist) {+ return fmt.Errorf("remove a stale reference lock: %w", err)+ }+ return nil+}++// syncStack makes durable the tables of the stack not made durable yet in this+// dirty period, then the list and the directory.+func (d *Durability) syncStack(path string, st *refState) error {+ reftable := filepath.Join(path, "reftable")+ names, err := readTablesList(filepath.Join(reftable, "tables.list"))+ if err != nil {+ return fmt.Errorf("read the table list: %w", err)+ }++ d.mu.Lock()+ var pending []string+ for _, name := range names {+ if _, ok := st.synced[name]; !ok {+ pending = append(pending, name)+ }+ }+ d.mu.Unlock()++ for _, name := range pending {+ if err := d.syncPath(filepath.Join(reftable, name)); err != nil {+ if errors.Is(err, fs.ErrNotExist) {+ // Compacted away after the list was read: the next list names+ // the table that replaced it.+ continue+ }+ return err+ }+ d.mu.Lock()+ st.synced[name] = struct{}{}+ d.mu.Unlock()+ }++ for _, p := range []string{filepath.Join(reftable, "tables.list"), reftable} {+ if err := d.syncPath(p); err != nil {+ return err+ }+ }+ return nil+}++// Forget drops the state and the marker of the repository at path, which the+// caller moved aside under the syncer's repository lock. A writer that still+// writes to the moved repository writes to a directory that is removed.+func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) error {+ if d == nil {+ return nil+ }++ markerPath, err := d.markerPath(ctx, repo, path)+ if err != nil {+ return err+ }++ d.mu.Lock()+ st := d.repos[path]+ d.mu.Unlock()+ if st != nil {+ st.file.Lock()+ defer st.file.Unlock()+ }++ if err := os.Remove(markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {+ return fmt.Errorf("remove the dirty marker: %w", err)+ }++ d.mu.Lock()+ defer d.mu.Unlock()+ if st != nil && d.repos[path] == st {+ if st.dirty {+ d.dirty.Dec()+ }+ d.inflight.Sub(float64(st.nInflight))+ delete(d.repos, path)+ }+ return nil+}++// Recovered is a dirty repository found at startup.+type Recovered struct {+ Repo RepoRef+ Path string+ // Reboot reports that the marker is of another kernel boot: the repository+ // must be moved aside before anything reads it.+ Reboot bool+}++// Recover reads the dirty markers of the storages. A marker of the running+// boot makes its repository dirty again, to be settled, and a reference lock+// that a durable step of the previous process left is removed. It returns+// every repository found; the caller moves aside the ones of another boot+// before serving. A marker that cannot be read is an error: its repository is+// unknown, and serving could read a torn stack.+func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovered, error) {+ if d == nil {+ return nil, nil+ }++ var found []Recovered+ for _, storageName := range storages {+ dir, err := d.markerDir(ctx, storageName)+ if err != nil {+ return nil, err+ }+ entries, err := os.ReadDir(dir)+ if errors.Is(err, fs.ErrNotExist) {+ continue+ }+ if err != nil {+ return nil, fmt.Errorf("read the dirty markers of storage %q: %w", storageName, err)+ }++ for _, entry := range entries {+ markerPath := filepath.Join(dir, entry.Name())+ if !strings.HasSuffix(entry.Name(), ".json") {+ // A temporary file of a marker write that did not complete: the+ // write never returned, so no reference write relied on it.+ if err := os.Remove(markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {+ return nil, fmt.Errorf("remove an incomplete dirty marker: %w", err)+ }+ continue+ }++ content, err := os.ReadFile(markerPath)+ if err != nil {+ return nil, fmt.Errorf("read the dirty marker %q: %w", markerPath, err)+ }+ var m marker+ if err := json.Unmarshal(content, &m); err != nil || m.Storage != storageName || m.RelativePath == "" || m.BootID == "" {+ return nil, fmt.Errorf("corrupt dirty marker %q: the repository it names cannot be trusted and is unknown", markerPath)+ }++ path, err := d.locator.GetRepoPath(ctx, repoOf(m.RepoRef), storage.WithRepositoryVerificationSkipped())+ if err != nil {+ return nil, fmt.Errorf("dirty marker %q: %w", markerPath, err)+ }++ reboot := m.BootID != d.bootID+ if !reboot {+ if err := removeStaleStackLock(path); err != nil {+ return nil, err+ }+ d.mu.Lock()+ st := d.stateLocked(path)+ if !st.dirty {+ st.dirty = true+ st.repo = m.RepoRef+ st.marker = markerPath+ d.dirty.Inc()+ }+ d.mu.Unlock()+ }+ found = append(found, Recovered{Repo: m.RepoRef, Path: path, Reboot: reboot})+ }+ }+ return found, nil+}++// CountUntrusted counts a repository moved aside for reason.+func (d *Durability) CountUntrusted(reason string) {+ if d == nil {+ return+ }+ d.untrusted.WithLabelValues(reason).Inc()+}++// writeDurably writes content to path through a temporary file in the same+// directory, made durable before and after the rename.+func (d *Durability) writeDurably(path string, content []byte) (returnedErr error) {+ dir := filepath.Dir(path)+ if err := os.MkdirAll(dir, 0o700); err != nil {+ return err+ }++ tmp, err := os.CreateTemp(dir, ".marker-*.tmp")+ if err != nil {+ return err+ }+ defer func() {+ if returnedErr != nil {+ _ = tmp.Close()+ _ = os.Remove(tmp.Name())+ }+ }()++ if _, err := tmp.Write(content); err != nil {+ return err+ }+ if err := d.sync(tmp); err != nil {+ return err+ }+ if err := tmp.Close(); err != nil {+ return err+ }+ if err := os.Rename(tmp.Name(), path); err != nil {+ return err+ }+ return d.syncPath(dir)+}++// syncPath makes the file or directory at path durable.+func (d *Durability) syncPath(path string) error {+ f, err := os.Open(path)+ if err != nil {+ return err+ }+ defer f.Close()+ return d.sync(f)+}++// readTablesList returns the table names of a reftable stack.+func readTablesList(path string) ([]string, error) {+ f, err := os.Open(path)+ if err != nil {+ return nil, err+ }+ defer f.Close()++ var names []string+ scanner := bufio.NewScanner(f)+ for scanner.Scan() {+ if name := strings.TrimSpace(scanner.Text()); name != "" {+ names = append(names, name)+ }+ }+ return names, scanner.Err()+}++// Describe implements prometheus.Collector.+func (d *Durability) Describe(descs chan<- *prometheus.Desc) {+ d.marked.Describe(descs)+ d.cleared.Describe(descs)+ d.untrusted.Describe(descs)+ d.dirty.Describe(descs)+ d.inflight.Describe(descs)+}++// Collect implements prometheus.Collector.+func (d *Durability) Collect(metrics chan<- prometheus.Metric) {+ d.marked.Collect(metrics)+ d.cleared.Collect(metrics)+ d.untrusted.Collect(metrics)+ d.dirty.Collect(metrics)+ d.inflight.Collect(metrics)+}diff --git a/internal/graft/corestore/localrefs/durability_test.go b/internal/graft/corestore/localrefs/durability_test.gonew file mode 100644index 0000000..09a2b6f--- /dev/null+++ b/internal/graft/corestore/localrefs/durability_test.go@@ -0,0 +1,365 @@+package localrefs++import (+ "context"+ "errors"+ "os"+ "path/filepath"+ "strings"+ "sync"+ "testing"+ "time"++ "github.com/prometheus/client_golang/prometheus/testutil"+ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/gitaly/config"+ "github.com/feder-cr/git-ai/internal/gitaly/storage"+ "github.com/feder-cr/git-ai/internal/testhelper"+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"+)++// durabilityFixture is a reftable repository with a few references, written+// one transaction each so that its stack has several tables.+type durabilityFixture struct {+ ctx context.Context+ cfg config.Cfg+ locator storage.Locator+ repo RepoRef+ path string+ commit git.ObjectID+ synced *syncRecorder+}++// syncRecorder records the paths made durable, and fails them while failWith+// is set.+type syncRecorder struct {+ mu sync.Mutex+ paths []string+ failWith error+}++func (s *syncRecorder) sync(f *os.File) error {+ s.mu.Lock()+ defer s.mu.Unlock()+ if s.failWith != nil {+ return s.failWith+ }+ s.paths = append(s.paths, f.Name())+ return f.Sync()+}++func (s *syncRecorder) recorded() []string {+ s.mu.Lock()+ defer s.mu.Unlock()+ return append([]string(nil), s.paths...)+}++func (s *syncRecorder) fail(err error) {+ s.mu.Lock()+ defer s.mu.Unlock()+ s.failWith = err+}++func setupDurability(t *testing.T) durabilityFixture {+ t.Helper()+ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ pb, path := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{+ SkipCreationViaService: true,+ ReferenceBackend: git.ReferenceBackendReftables,+ })+ commit := gittest.WriteCommit(t, cfg, path)+ for _, name := range []string{"refs/heads/main", "refs/heads/a", "refs/heads/b"} {+ gittest.WriteRef(t, cfg, path, git.ReferenceName(name), commit)+ }+ return durabilityFixture{+ ctx: ctx,+ cfg: cfg,+ locator: config.NewLocator(cfg),+ repo: RepoRef{Storage: pb.GetStorageName(), RelativePath: pb.GetRelativePath()},+ path: path,+ commit: commit,+ synced: &syncRecorder{},+ }+}++func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration) *Durability {+ t.Helper()+ d, err := NewDurability(f.locator, idle, WithBootID(bootID), WithSync(f.synced.sync))+ require.NoError(t, err)+ return d+}++func (f durabilityFixture) markerPath(t *testing.T, d *Durability) string {+ t.Helper()+ p, err := d.markerPath(f.ctx, f.repo, f.path)+ require.NoError(t, err)+ return p+}++func (f durabilityFixture) tables(t *testing.T) []string {+ t.Helper()+ names, err := readTablesList(filepath.Join(f.path, "reftable", "tables.list"))+ require.NoError(t, err)+ require.NotEmpty(t, names)+ return names+}++func TestMarkDirtyWritesOneDurableMarker(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))++ content, err := os.ReadFile(f.markerPath(t, d))+ require.NoError(t, err)+ require.JSONEq(t, `{"storage":"`+f.repo.Storage+`","relative_path":"`+f.repo.RelativePath+`","boot_id":"boot-a"}`, string(content))+ require.Equal(t, 1.0, testutil.ToFloat64(d.marked))+ require.Equal(t, []string{f.path}, d.Dirty())++ // The marker file and its directory were made durable before MarkDirty+ // returned.+ recorded := f.synced.recorded()+ require.Len(t, recorded, 2)+ require.Equal(t, filepath.Dir(f.markerPath(t, d)), recorded[1])+ require.Equal(t, filepath.Join(f.cfg.Storages[0].Path, MarkerDirName), filepath.Dir(f.markerPath(t, d)))+}++func TestSettleMakesTheStackDurableAndRemovesTheMarker(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, d.Settle(f.path))++ require.NoFileExists(t, f.markerPath(t, d))+ require.Empty(t, d.Dirty())+ require.Equal(t, 1.0, testutil.ToFloat64(d.cleared))+ require.Equal(t, 0.0, testutil.ToFloat64(d.dirty))++ recorded := f.synced.recorded()+ for _, table := range f.tables(t) {+ require.Contains(t, recorded, filepath.Join(f.path, "reftable", table))+ }+ require.Contains(t, recorded, filepath.Join(f.path, "reftable", "tables.list"))+ require.Contains(t, recorded, filepath.Join(f.path, "reftable"))+ // The lock taken for the removal is released.+ require.NoFileExists(t, filepath.Join(f.path, "reftable", "tables.list.lock"))+}++func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {+ t.Parallel()++ t.Run("not idle", func(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", time.Hour)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, d.Settle(f.path))+ require.FileExists(t, f.markerPath(t, d))+ // The stack was made durable all the same.+ require.Contains(t, f.synced.recorded(), filepath.Join(f.path, "reftable", f.tables(t)[0]))+ })++ t.Run("a transaction between Prepared and Committed", func(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ d.Begin(f.path, key(1))+ require.NoError(t, d.Settle(f.path))+ require.FileExists(t, f.markerPath(t, d))++ // An end of another key does not count; the end of the transaction does.+ d.End(f.path, key(2))+ require.NoError(t, d.Settle(f.path))+ require.FileExists(t, f.markerPath(t, d))++ d.End(f.path, key(1))+ require.NoError(t, d.Settle(f.path))+ require.NoFileExists(t, f.markerPath(t, d))+ require.Equal(t, 0.0, testutil.ToFloat64(d.inflight))+ })++ t.Run("git holds the reference lock", func(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ lock := filepath.Join(f.path, "reftable", "tables.list.lock")+ require.NoError(t, os.WriteFile(lock, nil, 0o666))++ require.NoError(t, d.Settle(f.path))+ require.FileExists(t, f.markerPath(t, d))+ require.FileExists(t, lock, "the durable step removed a lock it did not take")+ })+}++func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ writeback := errors.New("input/output error")+ f.synced.fail(writeback)++ err := d.Settle(f.path)+ require.ErrorIs(t, err, ErrUntrusted)+ require.ErrorIs(t, err, writeback)+ var untrusted *UntrustedError+ require.ErrorAs(t, err, &untrusted)+ require.Equal(t, f.repo, untrusted.Repo)+ require.Equal(t, f.path, untrusted.Path)++ // The repository stays dirty until the caller moves it aside.+ require.FileExists(t, f.markerPath(t, d))+ require.Equal(t, []string{f.path}, d.Dirty())++ require.NoError(t, d.Forget(f.ctx, f.repo, f.path))+ require.NoFileExists(t, f.markerPath(t, d))+ require.Empty(t, d.Dirty())+ require.Equal(t, 0.0, testutil.ToFloat64(d.dirty))+}++func TestSettleDropsTheMarkerOfARepositoryThatIsGone(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)++ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))+ require.NoError(t, os.Rename(f.path, f.path+".moved"))+ t.Cleanup(func() { require.NoError(t, os.Rename(f.path+".moved", f.path)) })++ require.NoError(t, d.Settle(f.path))+ require.NoFileExists(t, f.markerPath(t, d))+ require.Empty(t, d.Dirty())+}++func TestRecoverTellsAKernelRestartFromAProcessRestart(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ before := f.durability(t, "boot-a", time.Hour)+ require.NoError(t, before.MarkDirty(f.ctx, f.repo, f.path))+ storages := []string{f.repo.Storage}++ // The same boot: the process restarted, the kernel kept every write.+ sameBoot := f.durability(t, "boot-a", 0)+ found, err := sameBoot.Recover(f.ctx, storages)+ require.NoError(t, err)+ require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path, Reboot: false}}, found)+ require.Equal(t, []string{f.path}, sameBoot.Dirty())+ require.NoError(t, sameBoot.Settle(f.path))+ require.NoFileExists(t, f.markerPath(t, sameBoot))++ // Another boot: the writes may not have reached the disk.+ require.NoError(t, f.durability(t, "boot-a", time.Hour).MarkDirty(f.ctx, f.repo, f.path))+ otherBoot := f.durability(t, "boot-b", 0)+ found, err = otherBoot.Recover(f.ctx, storages)+ require.NoError(t, err)+ require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path, Reboot: true}}, found)+ require.Empty(t, otherBoot.Dirty())+}++// TestRecoverRemovesTheLockOfADurableStepThatDied verifies that a reference+// lock left by a durable step of a process that died is removed at startup,+// and that a lock of git is not.+func TestRecoverRemovesTheLockOfADurableStepThatDied(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ lock := filepath.Join(f.path, "reftable", "tables.list.lock")++ require.NoError(t, f.durability(t, "boot-a", 0).MarkDirty(f.ctx, f.repo, f.path))+ unlock, ok, err := lockStack(f.path)+ require.NoError(t, err)+ require.True(t, ok)+ _, ok, err = lockStack(f.path)+ require.NoError(t, err)+ require.False(t, ok, "the lock is exclusive")+ entries, err := os.ReadDir(f.path)+ require.NoError(t, err)+ for _, entry := range entries {+ require.False(t, strings.HasPrefix(entry.Name(), ".graft-durable-step-"), "temporary file %s left", entry.Name())+ }++ // The process dies holding the lock; the next one of the same boot removes it.+ _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, []string{f.repo.Storage})+ require.NoError(t, err)+ require.NoFileExists(t, lock)+ require.Error(t, unlock(), "the lock was already removed")++ // A lock of git is kept.+ require.NoError(t, os.WriteFile(lock, []byte("0x000000000001-0x000000000003-abcdef01.ref\n"), 0o666))+ _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, []string{f.repo.Storage})+ require.NoError(t, err)+ require.FileExists(t, lock)+}++func TestRecoverRefusesACorruptMarker(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)+ d := f.durability(t, "boot-a", 0)+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))++ marker := f.markerPath(t, d)+ incomplete := filepath.Join(filepath.Dir(marker), ".marker-123.tmp")+ require.NoError(t, os.WriteFile(incomplete, []byte("{"), 0o600))++ found, err := f.durability(t, "boot-b", 0).Recover(f.ctx, []string{f.repo.Storage})+ require.NoError(t, err)+ require.Len(t, found, 1)+ require.NoFileExists(t, incomplete, "an incomplete marker write is removed")++ require.NoError(t, os.WriteFile(marker, []byte(`{"storage":"`), 0o600))+ _, err = f.durability(t, "boot-b", 0).Recover(f.ctx, []string{f.repo.Storage})+ require.ErrorContains(t, err, "corrupt dirty marker")+}++func TestNilDurabilityIsOff(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)++ var d *Durability+ require.False(t, d.Enabled())+ require.NoError(t, d.MarkDirty(ctx, RepoRef{}, "path"))+ require.NoError(t, d.Settle("path"))+ d.Begin("path", key(1))+ d.End("path", key(1))+ found, err := d.Recover(ctx, []string{"default"})+ require.NoError(t, err)+ require.Empty(t, found)+}++// TestTornReftableStackReadsAsAnEmptyRepository is the experiment behind the+// LOCAL REFERENCE WRITES RULE (doc/graft/PLAN7.md): a reftable stack that a+// crash tore (a table whose data never reached the disk) is read by git as a+// repository without references, without an error. Only writes fail. So a torn+// stack must never be served: nothing in git would tell.+func TestTornReftableStackReadsAsAnEmptyRepository(t *testing.T) {+ t.Parallel()+ f := setupDurability(t)++ refs := gittest.Exec(t, f.cfg, "-C", f.path, "for-each-ref", "--format=%(refname)")+ require.Equal(t, "refs/heads/a\nrefs/heads/b\nrefs/heads/main", strings.TrimSpace(string(refs)))++ tables := f.tables(t)+ newest := filepath.Join(f.path, "reftable", tables[len(tables)-1])+ require.NoError(t, os.Truncate(newest, 0))++ refs = gittest.Exec(t, f.cfg, "-C", f.path, "for-each-ref", "--format=%(refname)")+ require.Empty(t, strings.TrimSpace(string(refs)), "git reports the references of a torn stack")++ cmd := gittest.NewCommand(t, f.cfg, "-C", f.path, "update-ref", "refs/heads/new", f.commit.String())+ output, err := cmd.CombinedOutput()+ require.Error(t, err)+ require.Contains(t, string(output), "corrupt reftable file")+}diff --git a/internal/graft/corestore/syncer/durability.go b/internal/graft/corestore/syncer/durability.gonew file mode 100644index 0000000..32c5be5--- /dev/null+++ b/internal/graft/corestore/syncer/durability.go@@ -0,0 +1,200 @@+package syncer++import (+ "context"+ "errors"+ "fmt"+ "os"+ "path/filepath"+ "time"++ "github.com/feder-cr/git-ai/internal/graft/corestore"+ "github.com/feder-cr/git-ai/internal/graft/corestore/gitops"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"+ "github.com/feder-cr/git-ai/internal/structerr"+)++// RecoverLocalReferences moves aside every local repository that had reference+// writes that were not durable yet when the kernel restarted (doc/graft/PLAN7.md,+// LOCAL REFERENCE WRITES RULE): its reftable stack may be torn, and git reads a+// torn stack as an empty repository without an error. The next request+// hydrates it from the database and the bucket. gitaly serve calls it before+// its listeners open, so nothing reads such a repository before. A repository+// dirty from a process restart of the same boot is kept: the kernel kept every+// write, and the durable step settles it.+//+// Only a cache of the storage core is moved: the database must hold the+// repository with changesets, and the local repository must have its+// watermark. Any other marker (the repository was removed, or replaced by one+// being imported, whose local copy may be the only one) is dropped.+func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string) error {+ d := s.core.RefDurability+ found, err := d.Recover(ctx, storages)+ if err != nil {+ return fmt.Errorf("storage core: recover the local references: %w", err)+ }++ for _, rec := range found {+ if !rec.Reboot {+ continue+ }++ cache, err := s.storageCoreCache(ctx, rec)+ if err != nil {+ return fmt.Errorf("storage core: recover the local references of %s:%s: %w", rec.Repo.Storage, rec.Repo.RelativePath, err)+ }+ if !cache {+ s.logger.WithField("repository_path", rec.Path).+ Warn("storage core: dropped the dirty marker of a local repository that is not a cache of the storage core")+ if err := d.Forget(ctx, rec.Repo, rec.Path); err != nil {+ return err+ }+ continue+ }++ if err := s.distrust(ctx, rec.Repo, rec.Path, localrefs.ReasonReboot, nil); err != nil {+ return err+ }+ }+ return nil+}++// storageCoreCache reports whether the local repository of rec is a cache of+// the storage core that a hydration can rebuild: the database holds it with+// changesets, and it has a watermark.+func (s *Syncer) storageCoreCache(ctx context.Context, rec localrefs.Recovered) (bool, error) {+ r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: rec.Repo.Storage, RelativePath: rec.Repo.RelativePath})+ if errors.Is(err, corestore.ErrRepositoryNotFound) {+ return false, nil+ }+ if err != nil {+ return false, err+ }+ if r.HeadSeq == 0 {+ return false, nil+ }++ if _, err := os.Lstat(rec.Path); errors.Is(err, os.ErrNotExist) {+ return false, nil+ } else if err != nil {+ return false, err+ }+ _, ok, err := gitops.ReadWatermark(rec.Path)+ if err != nil {+ // A watermark that cannot be read is no proof that the storage core wrote+ // the repository: it is kept, as untracked() keeps it.+ return false, nil+ }+ return ok, nil+}++// RunDurableSteps runs the durable step of every dirty repository each interval+// until ctx ends (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).+func (s *Syncer) RunDurableSteps(ctx context.Context, interval time.Duration) {+ if !s.core.RefDurability.Enabled() {+ return+ }++ ticker := time.NewTicker(interval)+ defer ticker.Stop()+ for {+ select {+ case <-ctx.Done():+ return+ case <-ticker.C:+ }+ for _, path := range s.core.RefDurability.Dirty() {+ s.durableStep(ctx, path)+ }+ }+}++// durableStep makes the references of the repository at path durable under the+// repository lock, and moves the repository aside at once when the kernel+// reports that a write failed.+func (s *Syncer) durableStep(ctx context.Context, path string) {+ release, err := s.lock(ctx, path)+ if err != nil {+ return+ }+ defer release()++ err = s.core.RefDurability.Settle(path)+ var untrusted *localrefs.UntrustedError+ if errors.As(err, &untrusted) {+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {+ s.logger.WithError(err).WithField("repository_path", path).+ Error("storage core: could not move aside a repository whose references may have lost writes")+ }+ return+ }+ if err != nil {+ s.logger.WithError(err).WithField("repository_path", path).Error("storage core: durable step of the local references")+ }+}++// distrust is distrustLocked under the repository lock.+func (s *Syncer) distrust(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {+ release, err := s.lock(ctx, path)+ if err != nil {+ return err+ }+ defer release()+ return s.distrustLocked(ctx, repo, path, reason, cause)+}++// distrustLocked moves the local repository at path into the temporary+// directory of its storage and removes it there in the background, then drops+// its dirty marker: the next request hydrates it. The move is made durable+// before the marker goes, so a crash in between moves it again. The caller+// holds the repository lock.+func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {+ d := s.core.RefDurability++ if _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) {+ return d.Forget(ctx, repo, path)+ } else if err != nil {+ return structerr.NewInternal("storage core: stat repository: %w", err)+ }++ tmpDir, err := s.tmpDir(repo.Storage)+ if err != nil {+ return err+ }+ stale, err := os.MkdirTemp(tmpDir, tmpStalePattern)+ if err != nil {+ return structerr.NewInternal("storage core: create stale directory: %w", err)+ }+ if err := os.Rename(path, filepath.Join(stale, "repo.git")); err != nil {+ _ = os.Remove(stale)+ return structerr.NewInternal("storage core: move aside an untrusted repository: %w", err)+ }+ if err := syncDir(filepath.Dir(path)); err != nil {+ return structerr.NewInternal("storage core: make the move of an untrusted repository durable: %w", err)+ }++ s.forgetPending(path)+ if err := d.Forget(ctx, repo, path); err != nil {+ return err+ }+ d.CountUntrusted(reason)++ logger := s.logger.WithField("repository_path", path).WithField("stale_path", stale).WithField("reason", reason)+ if cause != nil {+ logger = logger.WithError(cause)+ }+ logger.Error("storage core: moved aside a local repository whose references may have lost writes, the next request hydrates it")++ go func() { _ = os.RemoveAll(stale) }()+ return nil+}++// syncDir makes the entries of the directory at path durable.+func syncDir(path string) error {+ f, err := os.Open(path)+ if err != nil {+ return err+ }+ defer f.Close()+ return f.Sync()+}diff --git a/internal/graft/corestore/syncer/export_test.go b/internal/graft/corestore/syncer/export_test.goindex b9b2bfc..162b6c8 100644--- a/internal/graft/corestore/syncer/export_test.go+++ b/internal/graft/corestore/syncer/export_test.go@@ -1,7 +1,15 @@package syncer+import "context"+// SetTestHook installs a function called at the named points of a catch-up// ("fetched", "checked"). Only for tests; set it before the first EnsureFresh.func (s *Syncer) SetTestHook(hook func(point string)) {s.testHook = hook}++// DurableStep runs one durable step of the repository at path, as+// RunDurableSteps does every interval.+func (s *Syncer) DurableStep(ctx context.Context, path string) {+ s.durableStep(ctx, path)+}diff --git a/internal/graft/corestore/syncer/syncer.go b/internal/graft/corestore/syncer/syncer.goindex 3c91015..429f47c 100644--- a/internal/graft/corestore/syncer/syncer.go+++ b/internal/graft/corestore/syncer/syncer.go@@ -66,6 +66,7 @@ import ("github.com/feder-cr/git-ai/internal/graft/changelog""github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/gitops"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"reviewstore "github.com/feder-cr/git-ai/internal/graft/review/store""github.com/feder-cr/git-ai/internal/log""github.com/feder-cr/git-ai/internal/structerr"@@ -1161,7 +1162,8 @@ func (s *Syncer) applyLocked(ctx context.Context, repo *localrepo.Repo, hash gitreturn applyOutcome{done: true}, nil}- if err := applyMoves(ctx, repo, hash, moves); err != nil {+ w := refWriter{core: s.core, repo: localrefs.RepoRef{Storage: r.Key.Storage, RelativePath: r.Key.RelativePath}, path: path}+ if err := applyMoves(ctx, repo, w, hash, moves); err != nil {var txErr *refTxErrorif errors.As(err, &txErr) && txErr.locked {return applyOutcome{lockErr: err}, nil@@ -1225,7 +1227,7 @@ func movedSince(ctx context.Context, repo *localrepo.Repo, moves map[string]move//// The references other than HEAD are moved first, HEAD alone afterwards: git// refuses one transaction that updates HEAD and the reference it points to.-func applyMoves(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash, moves map[string]move) error {+func applyMoves(ctx context.Context, repo *localrepo.Repo, w refWriter, hash git.ObjectHash, moves map[string]move) error {var names []stringmoveHead := falsefor name, m := range moves {@@ -1246,12 +1248,12 @@ func applyMoves(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash,sort.Strings(names)if len(names) > 0 {- if err := applyChecked(ctx, repo, hash, names, moves); err != nil {+ if err := applyChecked(ctx, repo, w, hash, names, moves); err != nil {return err}}if moveHead {- return applyChecked(ctx, repo, hash, []string{"HEAD"}, moves)+ return applyChecked(ctx, repo, w, hash, []string{"HEAD"}, moves)}return nil}@@ -1264,7 +1266,7 @@ func applyMoves(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash,// removed and a created name (refs/a and refs/a/b). Between the two// transactions a removed reference is absent; the second transaction requires// that, so a local write in between still fails it.-func applyChecked(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash, names []string, moves map[string]move) error {+func applyChecked(ctx context.Context, repo *localrepo.Repo, w refWriter, hash git.ObjectHash, names []string, moves map[string]move) error {var removals, others []stringtwoSteps := falsefor _, name := range names {@@ -1281,17 +1283,28 @@ func applyChecked(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash}if !twoSteps {- err := refTransaction(ctx, repo, append(append([]string{}, removals...), others...))+ err := refTransaction(ctx, repo, w, append(append([]string{}, removals...), others...))var txErr *refTxErrorif err == nil || !errors.As(err, &txErr) || !txErr.fdConflict || len(removals) == 0 || len(others) == 0 {return err}}- if err := refTransaction(ctx, repo, removals); err != nil {+ if err := refTransaction(ctx, repo, w, removals); err != nil {return fmt.Errorf("apply removals: %w", err)}- return refTransaction(ctx, repo, others)+ return refTransaction(ctx, repo, w, others)+}++// refWriter runs the reference transactions of the catch-up on the local+// repository at path, under the syncer's repository lock: admitted to git's+// reference lock like the hook's writers, and without fsync when the node+// defers the durability of its reference writes (doc/graft/PLAN7.md, LOCAL+// REFERENCE WRITES RULE).+type refWriter struct {+ core *corestore.Core+ repo localrefs.RepoRef+ path string}// nul terminates the fields of update-ref -z.@@ -1353,8 +1366,10 @@ func (e *refTxError) Unwrap() error { return e.err }// refTransaction runs the commands (casSteps) as one transaction of// `git update-ref -z --stdin --no-deref`, without hooks (the database has the// changes already: no reference-transaction hook may run for them). An empty-// command list does nothing.-func refTransaction(ctx context.Context, repo *localrepo.Repo, commands []string) error {+// command list does nothing. With deferred durability the repository gets its+// dirty marker before git writes; the caller holds the syncer's repository+// lock, which the durable step holds too.+func refTransaction(ctx context.Context, repo *localrepo.Repo, w refWriter, commands []string) error {if len(commands) == 0 {return nil}@@ -1364,6 +1379,17 @@ func refTransaction(ctx context.Context, repo *localrepo.Repo, commands []stringreturn fmt.Errorf("reference backend: %w", err)}+ var durability []gitcmd.CmdOpt+ if w.core.RefDurability.Enabled() {+ if err := w.core.RefDurability.MarkDirty(ctx, w.repo, w.path); err != nil {+ return err+ }+ durability = append(durability, gitcmd.WithConfig(gitcmd.ConfigPair{Key: "core.fsync", Value: gitcmd.FsyncComponentsWithoutReferences}))+ }+ if release := w.core.RefAdmission.AcquireUnique(ctx, w.path, localrefs.WriterSyncer); release != nil {+ defer release()+ }+var stdin bytes.Bufferstdin.WriteString("start" + nul)for _, c := range commands {@@ -1381,10 +1407,12 @@ func refTransaction(ctx context.Context, repo *localrepo.Repo, commands []stringgitcmd.Flag{Name: "--no-deref"},},},- gitcmd.WithDisabledHooks(),- gitcmd.WithStdin(&stdin),- gitcmd.WithStdout(io.Discard),- gitcmd.WithStderr(&stderr),+ append([]gitcmd.CmdOpt{+ gitcmd.WithDisabledHooks(),+ gitcmd.WithStdin(&stdin),+ gitcmd.WithStdout(io.Discard),+ gitcmd.WithStderr(&stderr),+ }, durability...)...,)if err == nil {return nildiff --git a/internal/graft/corestore/syncer/syncer_localrefs_test.go b/internal/graft/corestore/syncer/syncer_localrefs_test.gonew file mode 100644index 0000000..0e378ff--- /dev/null+++ b/internal/graft/corestore/syncer/syncer_localrefs_test.go@@ -0,0 +1,209 @@+package syncer_test++import (+ "errors"+ "os"+ "path/filepath"+ "strings"+ "sync/atomic"+ "testing"++ "github.com/prometheus/client_golang/prometheus"+ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/graft/corestore"+ "github.com/feder-cr/git-ai/internal/graft/corestore/coretest"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"+ "github.com/feder-cr/git-ai/internal/graft/corestore/syncer"+)++// markers returns the dirty markers of the node's storage.+func (e *env) markers(t *testing.T) []string {+ t.Helper()++ entries, err := os.ReadDir(filepath.Join(e.storagePath, localrefs.MarkerDirName))+ if os.IsNotExist(err) {+ return nil+ }+ require.NoError(t, err)+ var names []string+ for _, entry := range entries {+ names = append(names, entry.Name())+ }+ return names+}++// untrusted returns graft_storage_ref_untrusted_total{reason} of the node.+func (e *env) untrusted(t *testing.T, reason string) float64 {+ t.Helper()++ reg := prometheus.NewPedanticRegistry()+ require.NoError(t, reg.Register(e.core.RefDurability))+ families, err := reg.Gather()+ require.NoError(t, err)+ for _, family := range families {+ if family.GetName() != "graft_storage_ref_untrusted_total" {+ continue+ }+ for _, metric := range family.GetMetric() {+ for _, label := range metric.GetLabel() {+ if label.GetName() == "reason" && label.GetValue() == reason {+ return metric.GetCounter().GetValue()+ }+ }+ }+ }+ return 0+}++// tearNewestTable truncates the newest table of the local repository's stack:+// what a kernel restart can leave of a table written without fsync.+func (e *env) tearNewestTable(t *testing.T) {+ t.Helper()++ list, err := os.ReadFile(filepath.Join(e.localPath(), "reftable", "tables.list"))+ require.NoError(t, err)+ names := strings.Fields(string(list))+ require.NotEmpty(t, names)+ require.NoError(t, os.Truncate(filepath.Join(e.localPath(), "reftable", names[len(names)-1]), 0))+}++// caughtUpDirty returns a node whose local repository was hydrated, then caught+// up with one changeset: its references were written without fsync, so the+// repository is dirty.+func caughtUpDirty(t *testing.T, opts ...localrefs.DurabilityOption) (*env, corestore.Repository, git.ObjectID) {+ t.Helper()++ e := newEnv(t, envOptions{})+ coretest.EnableLocalRefs(t, e.core, e.locator, opts...)+ src := e.newSource(t)+ c1 := e.writeCommit(t, src)+ repo := e.register(t, e.key(localRelPath), src, c1)++ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ require.Empty(t, e.markers(t), "a hydration writes its references durably")++ c2 := e.advanceMain(t, repo.ID, src, c1)+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ require.Equal(t, float64(1), syncCount(t, e.sy, "caught_up"))+ require.Len(t, e.markers(t), 1, "the catch-up marks the repository before writing without fsync")++ repo, err := e.core.Meta.GetRepository(e.ctx, repo.Key)+ require.NoError(t, err)+ return e, repo, c2+}++func TestSyncerCatchUpMarksTheRepositoryDirty(t *testing.T) {+ t.Parallel()++ e, repo, _ := caughtUpDirty(t)+ e.verifySynced(t, repo)+ require.Equal(t, []string{e.localPath()}, e.core.RefDurability.Dirty())++ e.sy.DurableStep(e.ctx, e.localPath())+ require.Empty(t, e.markers(t))+ require.Empty(t, e.core.RefDurability.Dirty())+ e.verifySynced(t, repo)+}++// TestSyncerMovesAsideARepositoryDirtyAtAKernelRestart is the recovery of the+// LOCAL REFERENCE WRITES RULE: a repository with references written without+// fsync when the kernel restarted is moved aside before anything reads it, and+// hydrated again. Without the recovery the torn stack reads as an empty+// repository, without an error.+func TestSyncerMovesAsideARepositoryDirtyAtAKernelRestart(t *testing.T) {+ t.Parallel()++ e, repo, _ := caughtUpDirty(t, localrefs.WithBootID("boot-a"))+ e.tearNewestTable(t)++ // A read before the recovery returns the wrong state, and nothing reports+ // an error: git lists no references.+ torn := gittest.Exec(t, e.cfg, "-C", e.localPath(), "for-each-ref", "--format=%(refname)")+ require.Empty(t, strings.TrimSpace(string(torn)))++ // The node restarts after a kernel restart.+ coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-b"))+ e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)+ require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))++ require.NoDirExists(t, e.localPath())+ require.Empty(t, e.markers(t))+ require.Equal(t, float64(1), e.untrusted(t, localrefs.ReasonReboot))++ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ require.Equal(t, float64(1), syncCount(t, e.sy, "hydrated"), "the restarted node hydrates the repository moved aside")+ e.verifySynced(t, repo)+}++// TestSyncerKeepsARepositoryDirtyAtAProcessRestart verifies that a restart of+// the process alone keeps the repository: the kernel kept every write.+func TestSyncerKeepsARepositoryDirtyAtAProcessRestart(t *testing.T) {+ t.Parallel()++ e, repo, _ := caughtUpDirty(t, localrefs.WithBootID("boot-a"))++ coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-a"))+ e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)+ require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))++ require.DirExists(t, e.localPath())+ require.Equal(t, []string{e.localPath()}, e.core.RefDurability.Dirty())+ e.verifySynced(t, repo)++ e.sy.DurableStep(e.ctx, e.localPath())+ require.Empty(t, e.markers(t))+}++// TestSyncerRecoveryKeepsWhatIsNotACache verifies that the recovery never moves+// a local repository that a hydration cannot rebuild: one the database does+// not hold. Its marker is dropped.+func TestSyncerRecoveryKeepsWhatIsNotACache(t *testing.T) {+ t.Parallel()++ e := newEnv(t, envOptions{})+ coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-a"))+ pb, path := gittest.CreateRepository(t, e.ctx, e.cfg, gittest.CreateRepositoryConfig{+ SkipCreationViaService: true,+ ReferenceBackend: git.ReferenceBackendReftables,+ })+ require.NoError(t, e.core.RefDurability.MarkDirty(e.ctx, localrefs.RepoRef{Storage: pb.GetStorageName(), RelativePath: pb.GetRelativePath()}, path))++ coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-b"))+ e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)+ require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))++ require.DirExists(t, path)+ require.Empty(t, e.markers(t))+ require.Equal(t, float64(0), e.untrusted(t, localrefs.ReasonReboot))+}++// TestSyncerDurableStepMovesAsideOnAFailedWrite verifies the runtime half of+// the rule: when the kernel reports that a write of the references failed, the+// repository is moved aside at once, without waiting for a restart, and the+// next request hydrates it.+func TestSyncerDurableStepMovesAsideOnAFailedWrite(t *testing.T) {+ t.Parallel()++ var failing atomic.Bool+ e, repo, _ := caughtUpDirty(t, localrefs.WithSync(func(f *os.File) error {+ if failing.Load() {+ return errors.New("input/output error")+ }+ return f.Sync()+ }))++ failing.Store(true)+ e.sy.DurableStep(e.ctx, e.localPath())+ failing.Store(false)++ require.NoDirExists(t, e.localPath())+ require.Empty(t, e.markers(t))+ require.Equal(t, float64(1), e.untrusted(t, localrefs.ReasonSyncFailed))++ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))+ require.Equal(t, float64(2), syncCount(t, e.sy, "hydrated"), "the first hydration and the one after the move")+ e.verifySynced(t, repo)+}diff --git a/internal/graft/corestore/voter/voter.go b/internal/graft/corestore/voter/voter.goindex 9a382ec..33d6abf 100644--- a/internal/graft/corestore/voter/voter.go+++ b/internal/graft/corestore/voter/voter.go@@ -41,6 +41,7 @@ import ("github.com/feder-cr/git-ai/internal/graft/changelog""github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/gitops"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs""github.com/feder-cr/git-ai/internal/log""github.com/feder-cr/git-ai/internal/structerr""github.com/feder-cr/git-ai/proto/go/gitalypb"@@ -175,6 +176,10 @@ func (v *Voter) Preparing(ctx context.Context, payload gitcmd.HooksPayload, chanreturn}+ // Deferred first, so that it runs last: the admission is waited for after+ // the pack work, when git is about to take the reference lock.+ defer v.admit(ctx, payload, changes)+start := time.Now()defer func() { v.metrics.preparing.Observe(time.Since(start).Seconds()) }()@@ -520,6 +525,10 @@ func (v *Voter) prepare(ctx context.Context, payload gitcmd.HooksPayload, change}if res.NoOp {+ // git still writes the values the database already holds.+ if err := v.deferDurability(ctx, payload, changes, shadow); err != nil {+ return v.unavailable(err)+ }result = resultNoOpreturn nil}@@ -535,6 +544,13 @@ func (v *Voter) prepare(ctx context.Context, payload gitcmd.HooksPayload, changeseq: res.Seq,}, time.Now())+ // After the entry, so that a failure here counts as a local failure in the+ // Aborted phase: the database holds the changeset, catch-up repairs the+ // local repository.+ if err := v.deferDurability(ctx, payload, changes, shadow); err != nil {+ return v.unavailable(err)+ }+if err := v.fault(faultAfterCommit); err != nil {return fmt.Errorf("storage core: %w", err)}@@ -741,23 +757,91 @@ func (v *Voter) Committed(ctx context.Context, payload gitcmd.HooksPayload, chanreturn}- entry, ok := v.pending.take(pendingKey(payload.Repo, changes), time.Now())+ key := pendingKey(payload.Repo, changes)+ entry, ok := v.pending.take(key, time.Now())+ if !v.localRefs() && (!ok || v.applier == nil) {+ return+ }++ repoPath, err := v.repoPath(ctx, payload)+ if err != nil {+ v.logger.WithError(err).ErrorContext(ctx, "storage core: resolving the repository path after a commit")+ return+ }++ // git finished the transaction, its auto-compaction included, and released+ // the reference lock.+ v.core.RefAdmission.Release(repoPath, localrefs.Key(key))+ v.core.RefDurability.End(repoPath, localrefs.Key(key))+if !ok || v.applier == nil {return}+ v.applier.NoteCommitted(ctx, repoPath, entry.repoID, entry.seq)+}- // The path is built from the storage and the relative path only, so that a- // quarantine of the hooks payload does not end up in it.- repoPath, err := v.locator.GetRepoPath(ctx, &gitalypb.Repository{+// repoPath returns the path of the repository of the transaction. It is built+// from the storage and the relative path only, so that a quarantine of the+// hooks payload does not end up in it.+func (v *Voter) repoPath(ctx context.Context, payload gitcmd.HooksPayload) (string, error) {+ return v.locator.GetRepoPath(ctx, &gitalypb.Repository{StorageName: payload.Repo.GetStorageName(),RelativePath: payload.Repo.GetRelativePath(),}, storage.WithRepositoryVerificationSkipped())+}++// localRefs reports whether the local reference writes of the node are+// admitted or their durability deferred (authoritative mode).+func (v *Voter) localRefs() bool {+ return v.core.Authoritative() && (v.core.RefAdmission != nil || v.core.RefDurability.Enabled())+}++// DeferredReferenceDurability reports whether the reference writes of the+// transactions this voter commits may run without fsync: authoritative mode+// with the deferred durability of the node on (doc/graft/PLAN7.md, LOCAL+// REFERENCE WRITES RULE). The hook manager asks it for UpdateWithHooks.+func (v *Voter) DeferredReferenceDurability() bool {+ return v != nil && v.core.Authoritative() && v.core.RefDurability.Enabled()+}++// admit waits for the admission of the transaction to the reference lock of+// its repository (authoritative mode). Committed or Aborted releases it. It+// never fails the transaction: without an admission git waits for its lock as+// before.+func (v *Voter) admit(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {+ if !v.core.Authoritative() || v.core.RefAdmission == nil {+ return+ }++ repoPath, err := v.repoPath(ctx, payload)if err != nil {- v.logger.WithError(err).ErrorContext(ctx, "storage core: resolving the repository path after a commit")+ v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path for the admission, the transaction goes on without it")return}+ v.core.RefAdmission.Acquire(ctx, repoPath, localrefs.Key(pendingKey(payload.Repo, changes)), localrefs.WriterHook)+}- v.applier.NoteCommitted(ctx, repoPath, entry.repoID, entry.seq)+// deferDurability lets git write the references of a transaction that the+// database committed without fsync (authoritative mode with deferred+// durability): the repository gets its durable dirty marker first, and the+// transaction counts as in flight until its Committed phase, because git may+// still write its table and compact the stack until then. It runs in the+// Prepared phase, under git's reference lock and before git writes anything.+func (v *Voter) deferDurability(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, shadow bool) error {+ if shadow || !v.DeferredReferenceDurability() {+ return nil+ }++ repoPath, err := v.repoPath(ctx, payload)+ if err != nil {+ return err+ }+ repoKey := corestore.KeyOf(payload.Repo)+ if err := v.core.RefDurability.MarkDirty(ctx, localrefs.RepoRef{Storage: repoKey.Storage, RelativePath: repoKey.RelativePath}, repoPath); err != nil {+ return err+ }+ v.core.RefDurability.Begin(repoPath, localrefs.Key(pendingKey(payload.Repo, changes)))+ return nil}// Aborted tells the voter that git aborted the transaction.@@ -779,6 +863,17 @@ func (v *Voter) Aborted(ctx context.Context, payload gitcmd.HooksPayload, changekey := pendingKey(payload.Repo, changes)+ // A transaction aborted after its Prepared phase stays in flight for the+ // deferred durability (see localrefs.Durability.End): its repository stays+ // dirty.+ if v.localRefs() {+ if repoPath, err := v.repoPath(ctx, payload); err != nil {+ v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path after an abort, the admission ends at its bound")+ } else {+ v.core.RefAdmission.Release(repoPath, localrefs.Key(key))+ }+ }+// A transaction aborted before its Prepared phase (git did not get the// locks) leaves the pack of its Preparing phase unused: an orphan in the// bucket, like the pack of any failed write.diff --git a/internal/graft/corestore/voter/voter_localrefs_test.go b/internal/graft/corestore/voter/voter_localrefs_test.gonew file mode 100644index 0000000..2e1901a--- /dev/null+++ b/internal/graft/corestore/voter/voter_localrefs_test.go@@ -0,0 +1,181 @@+package voter_test++import (+ "context"+ "os"+ "path/filepath"+ "testing"++ "github.com/stretchr/testify/require"+ "google.golang.org/grpc/codes"+ "google.golang.org/grpc/status"++ "github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/git/localrepo"+ "github.com/feder-cr/git-ai/internal/graft/corestore"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"+ "github.com/feder-cr/git-ai/internal/graft/corestore/voter"+ "github.com/feder-cr/git-ai/internal/testhelper"+)++// admissionHeld reports whether a writer holds the admission of the fixture's+// repository: an Acquire with a cancelled context is admitted at once when+// nobody holds it, and gives up otherwise.+func (f *fixture) admissionHeld(t *testing.T) bool {+ t.Helper()++ ctx, cancel := context.WithCancel(f.ctx)+ cancel()+ probe := localrefs.Key{0xaa, 0xbb}+ if f.core.RefAdmission.Acquire(ctx, f.repoPath, probe, localrefs.WriterSyncer) {+ f.core.RefAdmission.Release(f.repoPath, probe)+ return false+ }+ return true+}++// markers returns the dirty markers of the fixture's storage.+func (f *fixture) markers(t *testing.T) []string {+ t.Helper()++ entries, err := os.ReadDir(filepath.Join(f.cfg.Storages[0].Path, localrefs.MarkerDirName))+ if os.IsNotExist(err) {+ return nil+ }+ require.NoError(t, err)+ var names []string+ for _, entry := range entries {+ names = append(names, entry.Name())+ }+ return names+}++// TestLocalReferenceWritesOfACommittedTransaction follows one authoritative+// transaction through the LOCAL REFERENCE WRITES RULE (doc/graft/PLAN7.md):+// admitted at the end of Preparing, marked dirty and in flight after the+// database commit in Prepared, released and ended in Committed.+func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {+ t.Parallel()++ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})+ const ref = "refs/heads/main"+ input := line(f.zero, f.a.String(), ref)++ require.True(t, f.voter.DeferredReferenceDurability())+ require.False(t, f.admissionHeld(t))++ f.voter.Preparing(f.ctx, f.payload, input)+ require.True(t, f.admissionHeld(t), "the transaction is admitted at the end of Preparing")+ require.Empty(t, f.markers(t), "nothing is dirty before the database commit")++ require.NoError(t, f.prepared(input))+ require.Len(t, f.markers(t), 1)++ // git may still write and compact until Committed: the durable step keeps+ // the marker.+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath))+ require.Len(t, f.markers(t), 1)++ gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName(ref), f.a)+ f.voter.Committed(f.ctx, f.payload, input)+ require.False(t, f.admissionHeld(t), "Committed releases the admission")+ require.Len(t, f.applier.calls, 1, "the local commit is still noted")++ require.NoError(t, f.core.RefDurability.Settle(f.repoPath))+ require.Empty(t, f.markers(t))+}++// TestLocalReferenceWritesOfAnAbortedTransaction verifies that an abort after+// Prepared releases the admission but leaves the repository dirty: the+// transaction cannot be told from another with the same input, so it is never+// counted as ended.+func TestLocalReferenceWritesOfAnAbortedTransaction(t *testing.T) {+ t.Parallel()++ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})+ input := line(f.zero, f.a.String(), "refs/heads/main")++ f.voter.Preparing(f.ctx, f.payload, input)+ require.NoError(t, f.prepared(input))+ f.voter.Aborted(f.ctx, f.payload, input)+ require.False(t, f.admissionHeld(t))++ require.NoError(t, f.core.RefDurability.Settle(f.repoPath))+ require.Len(t, f.markers(t), 1)+}++// TestLocalReferenceWritesOfARejectedTransaction verifies that a transaction+// the database rejects marks nothing: git writes nothing for it.+func TestLocalReferenceWritesOfARejectedTransaction(t *testing.T) {+ t.Parallel()++ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})+ const ref = "refs/heads/main"+ f.seed(t, ref, corestore.Target(f.b.String()))+ input := line(f.a.String(), f.c.String(), ref)++ f.voter.Preparing(f.ctx, f.payload, input)+ requireMismatch(t, f.prepared(input), ref, f.a.String(), f.b.String())+ f.voter.Aborted(f.ctx, f.payload, input)++ require.False(t, f.admissionHeld(t))+ require.Empty(t, f.markers(t))+}++// TestLocalReferenceWritesWhenTheMarkerCannotBeWritten verifies that git does+// not write references without fsync when the repository cannot be marked+// dirty: the transaction is aborted after the database commit, which catch-up+// repairs like any local failure.+func TestLocalReferenceWritesWhenTheMarkerCannotBeWritten(t *testing.T) {+ t.Parallel()++ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})+ markerDir := filepath.Join(f.cfg.Storages[0].Path, localrefs.MarkerDirName)+ require.NoError(t, os.MkdirAll(filepath.Dir(markerDir), 0o755))+ require.NoError(t, os.WriteFile(markerDir, nil, 0o644))++ input := line(f.zero, f.a.String(), "refs/heads/main")+ err := f.prepared(input)+ require.Equal(t, codes.Unavailable, status.Code(err), "error %v", err)+ require.ErrorContains(t, err, "write the dirty marker")+ require.Equal(t, uint64(1), f.headSeq(t), "the database committed the changeset before the marker")+}++// TestDeferredReferenceDurabilityByMode verifies that references are written+// without fsync in authoritative mode with deferred durability only: off and+// shadow modes, and authoritative mode without the boot identity, keep git's+// fsync of references.+func TestDeferredReferenceDurabilityByMode(t *testing.T) {+ t.Parallel()++ var nilVoter *voter.Voter+ require.False(t, nilVoter.DeferredReferenceDurability())++ off := voter.New(&corestore.Core{Mode: corestore.ModeOff}, testhelper.NewLogger(t), localrepo.Factory{}, nil, nil)+ require.False(t, off.DeferredReferenceDurability())++ shadow := newFixture(t, fixtureOptions{mode: corestore.ModeShadow, localRefs: true})+ require.False(t, shadow.voter.DeferredReferenceDurability())++ authoritative := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative})+ require.False(t, authoritative.voter.DeferredReferenceDurability())++ deferred := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})+ require.True(t, deferred.voter.DeferredReferenceDurability())+}++// TestLocalReferenceWritesNotInShadowMode verifies that shadow mode, where git+// is the source of truth, neither admits nor marks.+func TestLocalReferenceWritesNotInShadowMode(t *testing.T) {+ t.Parallel()++ f := newFixture(t, fixtureOptions{mode: corestore.ModeShadow, localRefs: true})+ input := line(f.zero, f.a.String(), "refs/heads/main")++ f.voter.Preparing(f.ctx, f.payload, input)+ require.False(t, f.admissionHeld(t))+ require.NoError(t, f.prepared(input))+ require.Empty(t, f.markers(t))+ f.voter.Committed(f.ctx, f.payload, input)+}diff --git a/internal/graft/corestore/voter/voter_test.go b/internal/graft/corestore/voter/voter_test.goindex b2f8b41..845d120 100644--- a/internal/graft/corestore/voter/voter_test.go+++ b/internal/graft/corestore/voter/voter_test.go@@ -90,6 +90,9 @@ type fixtureOptions struct {// files makes the repository use the files reference backend instead of// the package's default, reftable (see TestMain).files bool+ // localRefs gives the core the admission and the deferred durability of+ // the local reference writes (coretest.EnableLocalRefs).+ localRefs bool}func newFixture(t *testing.T, opts fixtureOptions) *fixture {@@ -130,6 +133,9 @@ func newFixture(t *testing.T, opts fixtureOptions) *fixture {logger := testhelper.NewLogger(t)locator := config.NewLocator(cfg)+ if opts.localRefs {+ coretest.EnableLocalRefs(t, core, locator)+ }cache := catfile.NewCache(cfg)t.Cleanup(cache.Stop)factory := localrepo.NewFactory(logger, locator, gittest.NewCommandFactory(t, cfg), cache)diff --git a/internal/graft/e2e/storage_localrefs_test.go b/internal/graft/e2e/storage_localrefs_test.gonew file mode 100644index 0000000..768c93f--- /dev/null+++ b/internal/graft/e2e/storage_localrefs_test.go@@ -0,0 +1,421 @@+package e2e++import (+ "context"+ "encoding/json"+ "errors"+ "fmt"+ "math"+ "os"+ "os/exec"+ "path/filepath"+ "strings"+ "sync"+ "syscall"+ "testing"+ "time"++ "github.com/stretchr/testify/require"+ "google.golang.org/grpc"+ healthpb "google.golang.org/grpc/health/grpc_health_v1"++ "github.com/feder-cr/git-ai/internal/featureflag"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/gitaly/config"+ "github.com/feder-cr/git-ai/internal/graft/corestore/coretest"+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"+ "github.com/feder-cr/git-ai/internal/graft/corestore/sqlstore"+ "github.com/feder-cr/git-ai/internal/testhelper"+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"+ "github.com/feder-cr/git-ai/sdk"+)++// The tests of the LOCAL REFERENCE WRITES RULE (doc/graft/PLAN7.md) with real+// binaries: a `gitaly serve` process in authoritative mode is killed with+// SIGKILL at the points of the rule and started again on the same storage,+// database and bucket.++// realNode is a gitaly server process of one node.+type realNode struct {+ // cfg is the configuration the process reads; clientCfg is the same with the+ // socket as a dialable address.+ cfg config.Cfg+ clientCfg config.Cfg+ bin string+ cfgPath string+ logPath string+ tokens map[string]string+ ctx context.Context+ setupCtx context.Context++ cmd *exec.Cmd+ // exited is closed when the process has exited and cmd.Wait returned.+ exited chan struct{}+}++func newRealNode(t *testing.T) *realNode {+ t.Helper()++ graft, tokens := workerGraft(1)+ cfg := testcfg.Build(t)+ cfg.Auth.Token = sharedTestToken+ cfg.Graft = unbatched(graft)+ cfg.Graft.Standalone = true+ cfg.Graft.Storage = config.GraftStorage{+ Mode: "authoritative",+ Database: coretest.DatabaseURL(t, sqlstore.SQLite),+ Bucket: coretest.BucketURL(t, false),+ NodeID: "node-a",+ }+ cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)+ testcfg.BuildGitalySSH(t, cfg)+ testcfg.BuildGitalyHooks(t, cfg)++ clientCfg := cfg+ clientCfg.SocketPath = "unix://" + cfg.SocketPath+ // The requests to a separate server process carry no feature flags of the+ // test binary, as a production client's do not: testhelper.Context marks+ // them as explicit, and the server would refuse a flag of its own that the+ // test binary does not define.+ base, cancel := context.WithCancel(context.Background())+ t.Cleanup(cancel)+ ctx := featureflag.OutgoingCtxWithFeatureFlag(base, featureflag.GPGSigning, false)++ return &realNode{+ cfg: cfg,+ clientCfg: clientCfg,+ bin: testcfg.BuildGitaly(t, cfg),+ cfgPath: testcfg.WriteTemporaryGitalyConfigFile(t, cfg),+ logPath: filepath.Join(t.TempDir(), "gitaly.log"),+ tokens: tokens,+ ctx: ctx,+ setupCtx: testhelper.MergeOutgoingMetadata(ctx, testcfg.GitalyServersMetadataFromCfg(t, clientCfg)),+ }+}++// start starts the server process with the extra environment env and waits+// until it serves.+func (n *realNode) start(t *testing.T, env ...string) {+ t.Helper()++ logFile, err := os.OpenFile(n.logPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)+ require.NoError(t, err)+ t.Cleanup(func() { _ = logFile.Close() })++ cmd := exec.Command(n.bin, "serve", n.cfgPath)+ cmd.Env = append(os.Environ(), env...)+ cmd.Stdout = logFile+ cmd.Stderr = logFile+ require.NoError(t, cmd.Start())++ exited := make(chan struct{})+ go func() {+ _ = cmd.Wait()+ close(exited)+ }()+ n.cmd, n.exited = cmd, exited+ t.Cleanup(func() {+ select {+ case <-exited:+ default:+ _ = cmd.Process.Signal(syscall.SIGKILL)+ <-exited+ }+ })++ conn := gittest.DialService(t, n.ctx, n.clientCfg)+ ready, cancel := context.WithTimeout(n.ctx, 2*time.Minute)+ defer cancel()+ // Any answer (Unimplemented included) means the server serves.+ _, err = healthpb.NewHealthClient(conn).Check(ready, &healthpb.HealthCheckRequest{}, grpc.WaitForReady(true))+ if ready.Err() != nil {+ t.Fatalf("the server did not start: %v\n%s", err, n.log(t))+ }+}++// kill kills the server process with SIGKILL and waits for its end.+func (n *realNode) kill(t *testing.T) {+ t.Helper()+ require.NoError(t, n.cmd.Process.Signal(syscall.SIGKILL))+ <-n.exited+}++// waitExit waits for the server process to exit by itself and returns its exit+// code.+func (n *realNode) waitExit(t *testing.T) int {+ t.Helper()++ select {+ case <-n.exited:+ case <-time.After(2 * time.Minute):+ t.Fatalf("the server did not exit\n%s", n.log(t))+ }+ return n.cmd.ProcessState.ExitCode()+}++func (n *realNode) log(t *testing.T) string {+ t.Helper()+ content, err := os.ReadFile(n.logPath)+ require.NoError(t, err)+ return string(content)+}++func (n *realNode) storagePath() string { return n.cfg.Storages[0].Path }++// createRepository creates an empty repository through the server and returns+// its relative path. In authoritative mode the gate registers it.+func (n *realNode) createRepository(t *testing.T) string {+ t.Helper()+ repo, _ := gittest.CreateRepository(t, n.setupCtx, n.clientCfg)+ return repo.GetRelativePath()+}++func (n *realNode) repoPath(relPath string) string {+ return filepath.Join(n.storagePath(), relPath)+}++// agent returns a handle whose requests carry the worker's token.+func (n *realNode) agent(t *testing.T, relPath string) *sdk.Repo {+ t.Helper()++ conn, err := sdk.Dial(n.ctx, n.clientCfg.SocketPath, sdk.WithStorage(n.cfg.Storages[0].Name), sdk.WithAgentToken(n.tokens[workerName(0)]))+ require.NoError(t, err)+ t.Cleanup(func() { _ = conn.Close() })+ return conn.Repo(relPath)+}++// operator returns a handle whose requests carry the shared token.+func (n *realNode) operator(t *testing.T, relPath string) *sdk.Repo {+ t.Helper()++ conn, err := sdk.Dial(n.ctx, n.clientCfg.SocketPath, sdk.WithStorage(n.cfg.Storages[0].Name), sdk.WithToken(sharedTestToken))+ require.NoError(t, err)+ t.Cleanup(func() { _ = conn.Close() })+ return conn.Repo(relPath)+}++// markers returns the dirty markers of the node's storage.+func (n *realNode) markers(t *testing.T) []string {+ t.Helper()++ entries, err := os.ReadDir(filepath.Join(n.storagePath(), localrefs.MarkerDirName))+ if errors.Is(err, os.ErrNotExist) {+ return nil+ }+ require.NoError(t, err)+ var names []string+ for _, entry := range entries {+ names = append(names, filepath.Join(n.storagePath(), localrefs.MarkerDirName, entry.Name()))+ }+ return names+}++// tables returns the tables of the stack of the node's local repository.+func (n *realNode) tables(t *testing.T, relPath string) []string {+ t.Helper()+ list, err := os.ReadFile(filepath.Join(n.repoPath(relPath), "reftable", "tables.list"))+ require.NoError(t, err)+ return strings.Fields(string(list))+}++// requireServed checks, through the server, that every branch of want is+// served with its commit and that the node's local copy equals the database at+// its head.+func (n *realNode) requireServed(t *testing.T, relPath string, want map[string]string) {+ t.Helper()++ onA := n.agent(t, relPath)+ for branch, commit := range want {+ got, err := onA.ResolveRevision(n.ctx, branch)+ require.NoError(t, err, "branch %s", branch)+ require.Equal(t, commit, got, "branch %s", branch)+ }++ verified, err := n.operator(t, relPath).StorageVerify(n.ctx, true)+ require.NoError(t, err)+ require.Empty(t, verified.Divergences, "local references must equal the database references")+ require.True(t, verified.ConnectivityOK, "connectivity: %s", verified.ConnectivityError)+}++// writeBranches commits one file on each of count new branches and returns+// the commits by branch.+func writeBranches(t *testing.T, onA *sdk.Repo, ctx context.Context, base string, prefix string, count int) map[string]string {+ t.Helper()++ written := map[string]string{}+ for i := 0; i < count; i++ {+ branch := fmt.Sprintf("%s/%d", prefix, i)+ commit, err := commitFile(ctx, onA, workerName(0), branch, base, branch+".txt", branch+"\n")+ require.NoError(t, err)+ written[branch] = commit+ }+ return written+}++// TestStorageLocalReferencesAfterAProcessKill kills the server right after+// reference writes made without fsync, before the durable step: the kernel+// kept every write, so the restarted node keeps its repository, serves every+// write and settles the marker.+func TestStorageLocalReferencesAfterAProcessKill(t *testing.T) {+ t.Parallel()++ n := newRealNode(t)+ n.start(t)+ rel := n.createRepository(t)+ onA := n.agent(t, rel)+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")+ require.NoError(t, err)+ written := writeBranches(t, onA, n.ctx, base, "before-kill", 5)+ written["main"] = base++ // The last write is less than the idle period old: the marker is there.+ require.Len(t, n.markers(t), 1)+ n.kill(t)+ require.Len(t, n.markers(t), 1, "the marker survives the process")++ n.start(t)+ n.requireServed(t, rel, written)+ require.DirExists(t, n.repoPath(rel), "a process restart keeps the repository")++ // The durable step runs every second and removes the marker of an idle+ // repository.+ require.Eventually(t, func() bool { return len(n.markers(t)) == 0 }, 30*time.Second, 50*time.Millisecond)+ n.requireServed(t, rel, written)+}++// TestStorageLocalReferencesAfterAKillInTheDurableStep kills the server while+// its durable step holds git's reference lock: the restarted node removes the+// lock it left, so writes go on at once, and serves every write.+func TestStorageLocalReferencesAfterAKillInTheDurableStep(t *testing.T) {+ t.Parallel()++ n := newRealNode(t)+ n.start(t)+ rel := n.createRepository(t)+ onA := n.agent(t, rel)+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")+ require.NoError(t, err)+ written := writeBranches(t, onA, n.ctx, base, "before-crash", 3)+ written["main"] = base+ require.Len(t, n.markers(t), 1)+ n.kill(t)++ // Restarted with the crash point, the node finds the repository dirty and+ // idle: its first durable step dies holding the lock.+ n.start(t, localrefs.CrashEnv+"=durable_step")+ require.Equal(t, 75, n.waitExit(t), "the process dies in the durable step")+ lock := filepath.Join(n.repoPath(rel), "reftable", "tables.list.lock")+ require.FileExists(t, lock)+ require.Len(t, n.markers(t), 1, "the marker is removed only after the lock")++ n.start(t)+ require.NoFileExists(t, lock, "the restarted node removes the lock its durable step left")+ n.requireServed(t, rel, written)++ more := writeBranches(t, n.agent(t, rel), n.ctx, base, "after-restart", 2)+ for branch, commit := range more {+ written[branch] = commit+ }+ n.requireServed(t, rel, written)+}++// TestStorageLocalReferencesAfterAKernelRestart simulates what a kernel+// restart (power loss) can leave of reference writes made without fsync: the+// newest table of the stack is empty. Read directly, the repository then looks+// empty, without an error. The restarted node moves it aside before serving+// and hydrates it again, so every write is served.+func TestStorageLocalReferencesAfterAKernelRestart(t *testing.T) {+ t.Parallel()++ n := newRealNode(t)+ n.start(t)+ rel := n.createRepository(t)+ onA := n.agent(t, rel)+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")+ require.NoError(t, err)+ written := writeBranches(t, onA, n.ctx, base, "before-power-loss", 5)+ written["main"] = base++ markers := n.markers(t)+ require.Len(t, markers, 1)+ n.kill(t)++ // The power loss: the newest table never reached the disk, and the kernel+ // that boots next has another boot identity.+ tables := n.tables(t, rel)+ require.NoError(t, os.Truncate(filepath.Join(n.repoPath(rel), "reftable", tables[len(tables)-1]), 0))+ content, err := os.ReadFile(markers[0])+ require.NoError(t, err)+ var m map[string]string+ require.NoError(t, json.Unmarshal(content, &m))+ m["boot_id"] = "the-boot-before-the-power-loss"+ content, err = json.Marshal(m)+ require.NoError(t, err)+ require.NoError(t, os.WriteFile(markers[0], content, 0o600))++ // What a server without the rule would serve.+ torn := gittest.Exec(t, n.cfg, "-C", n.repoPath(rel), "for-each-ref", "--format=%(refname)")+ require.Empty(t, strings.TrimSpace(string(torn)), "git reads the torn stack as an empty repository")++ n.start(t)+ require.Empty(t, n.markers(t))+ n.requireServed(t, rel, written)+ require.Contains(t, n.log(t), "moved aside a local repository whose references may have lost writes")+}++// TestStorageLocalReferencesStackDepthIsBounded writes many references+// without fsync from concurrent writers and checks that git's compaction keeps+// the stack shallow: the durable step only makes it durable, compaction stays+// git's.+func TestStorageLocalReferencesStackDepthIsBounded(t *testing.T) {+ t.Parallel()++ const writers, perWriter = 4, 30+ n := newRealNode(t)+ n.start(t)+ rel := n.createRepository(t)+ onA := n.agent(t, rel)+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")+ require.NoError(t, err)++ var (+ mu sync.Mutex+ maxDepth int+ wg sync.WaitGroup+ )+ errs := make(chan error, writers)+ for w := 0; w < writers; w++ {+ wg.Add(1)+ go func(w int) {+ defer wg.Done()+ for i := 0; i < perWriter; i++ {+ branch := fmt.Sprintf("load/%d/%d", w, i)+ if _, err := commitFile(n.ctx, onA, workerName(0), branch, base, branch+".txt", branch+"\n"); err != nil {+ errs <- err+ return+ }+ list, err := os.ReadFile(filepath.Join(n.repoPath(rel), "reftable", "tables.list"))+ if err != nil {+ errs <- err+ return+ }+ mu.Lock()+ maxDepth = max(maxDepth, len(strings.Fields(string(list))))+ mu.Unlock()+ }+ }(w)+ }+ wg.Wait()+ close(errs)+ for err := range errs {+ require.NoError(t, err)+ }++ // git compacts the stack geometrically (factor 2): about log2 of the number+ // of updates. Concurrent writers can make a compaction lose its race and+ // leave a few more tables for a while.+ updates := writers*perWriter + 2+ bound := 2*int(math.Ceil(math.Log2(float64(updates)))) + 2+ t.Logf("maximum stack depth %d over %d reference updates (bound %d)", maxDepth, updates, bound)+ require.LessOrEqual(t, maxDepth, bound)+}