Skip to content

Git designed for AI agents.

WP76b: local reference writes of authoritative mode (admission, deferred durability)

Federico Elia310facd5b916
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.md
index d23e7fd..43d09e6 100644
--- a/doc/graft/PLAN7.md
+++ b/doc/graft/PLAN7.md
@@ -504,6 +504,134 @@ guarded nor swept).
* disabling the roots requirement fails the sqlstore test and both review
race tests.
+### LOCAL REFERENCE WRITES RULE (binding; WP76b, approved by the PM)
+
+**Problem (measured).** A reftable repository has one lock for all its references
+(`tables.list.lock`). In authoritative mode `UserCommitFiles` spent most of its time
+waiting for it: with SQLite and a file bucket, at 50 and 100 agents, `u.prepare`
+(the wait) was 643 and 1198 ms mean, `u.commit` 252 and 230 ms, the voter's Prepared
+phase 19–26 ms. With `core.fsync` without `reference`, `u.commit` fell to 17 and 7 ms
+and the wait to 248 and 238 ms. So the hold is git's fsync of the new table and of
+`tables.list`, inside the lock. The wait that is left, about 6 holds, is git's lock
+wait: a polling sleep that grows with each failed try (`reftable.lockTimeout` =
+10 s). There is no queue, so newcomers win over long waiters and the lock sits free
+between polls. The same unfairness starved the syncer's catch-up behind writers
+(the flake of `TestSyncerConcurrentLocalWritesAreNeverReverted`).
+
+**Root cause.** In authoritative mode the durable commit point of a reference
+update is the database commit in the voter's Prepared phase; the local repository
+is a cache that any node rebuilds from the database and the bucket. Each
+transaction then made the same update durable a second time, inside the
+repository-wide lock.
+
+**Rule.**
+1. *Admission* (`corestore/localrefs.Admission`). The reference writers of a node
+ wait in a FIFO queue per local repository in front of git's lock:
+ * the hook's transactions from the end of the voter's Preparing phase (after
+ the pack upload, before git takes the lock) to Committed or Aborted;
+ * the syncer around each catch-up `update-ref`.
+
+ Admission orders writers only; git's lock stays the only mutual exclusion. A
+ writer that waited `reftable.lockTimeout` goes on to git's lock as before
+ (`bypassed`). An admission not released within the same bound ends
+ (`expired`; git may die between phases). A writer whose context ends leaves
+ the queue. Lock order: the syncer's repository lock, then the admission. No
+ admitted writer waits for the syncer lock: `NoteCommitted` only tries it, and
+ runs after the release.
+2. *Deferred durability* (`corestore/localrefs.Durability`). The hook's
+ `UpdateWithHooks` transactions and the syncer's catch-up write references with
+ `core.fsync` without `reference`. Only the voter can turn this on
+ (`DeferredReferenceDurability`: authoritative mode and the kernel's boot
+ identity readable). Objects stay durable as before.
+ * *Dirty marker.* Before the first such write after a durable step, the
+ repository gets a marker, written durably, with the kernel's boot identity
+ (`<storage>/+gitaly/graft-ref-durability/<sha256(path)>.json`, not the state
+ directory, whose files the cache walker removes).
+ * The hook path writes it in Prepared, after the database commit and before
+ git writes anything, so only repositories the database tracks get one. If
+ the marker cannot be written, Prepared fails and git aborts; catch-up
+ repairs the database's write, like any local failure.
+ * The syncer writes it under its repository lock.
+ * A hook transaction counts as in flight from Prepared to Committed.
+ Reasons from git's source: the new table is written in
+ `transaction_finish` after "prepared", and auto-compaction runs inside
+ `reftable_addition_commit`, before "committed". An abort after Prepared
+ is not counted as an end (two transactions with the same input cannot be
+ told apart), so the repository stays dirty.
+ * *Durable step* (`Settle`, every second for every dirty repository, under the
+ syncer's repository lock).
+ * It fsyncs the tables not yet fsynced, `tables.list` and the directory.
+ * When the repository has been idle for a second and nothing is in flight,
+ it takes git's lock without waiting (the lock is created with a tag, by
+ linking a temporary file). It then fsyncs what was written since, marks
+ the state clean, and releases the lock. Last, it removes the marker,
+ under a per-repository file lock that `MarkDirty` waits for.
+ * A busy lock leaves the marker for the next step.
+ * *Write failure.* A failed fsync in the durable step means the kernel may
+ have dropped writes. The repository is moved aside at once, under the
+ repository lock: the move is made durable, then the marker removed. The
+ failure is logged and counted (`sync_failed`), and the next request
+ hydrates the repository.
+ * *Recovery* (`Syncer.RecoverLocalReferences`). `gitaly serve` runs it before
+ any listener opens and before any background worker.
+ * A marker of another boot means the kernel restarted with writes that may
+ not have reached the disk. git reads a torn stack as an empty repository
+ with exit code 0 (experiment 1, kept as
+ `TestTornReftableStackReadsAsAnEmptyRepository`). So the repository is
+ moved aside before anything reads it (`reboot`), and the next request
+ hydrates it.
+ * The move only happens when the database holds the repository with
+ changesets and the local copy has a watermark; any other marker is
+ dropped, keeping a repository whose local copy may be the only one.
+ * A marker of the same boot is a process restart: the page cache kept
+ every write, so the repository stays dirty to be settled.
+ * A lock with the durable step's tag, left by a process that died, is
+ removed.
+ * A corrupt marker fails the startup: its repository is unknown.
+ * Not Linux (no boot identity): references stay durable per transaction.
+3. *Read paths.* After the recovery, no read path in authoritative mode can see
+ a stack torn by a kernel restart: the gRPC listeners (unix, TCP, TLS), the
+ Prometheus listener, the maintenance workers, the bundle generation, the
+ tempdir cleaner and the repository counter all start after it. Within the
+ same boot the stack is never torn (the page cache has every write). A failed
+ writeback is caught by the durable step within a second and the repository is
+ moved aside. Outside `gitaly serve`, only `gitaly-debug simulate-http-clone`,
+ `gitaly-debug dump-mvcc-manifest` and the `gitaly git` passthrough, pointed
+ at a repository by an operator, read references from disk. STORAGE.md says
+ not to run them on a node after an unclean shutdown before the server has
+ started once.
+
+**As implemented.**
+* `localrefs` (`admission.go`, `durability.go`); `Core.RefAdmission`/`RefDurability`.
+* `voter` (`admit`, `deferDurability`, Committed/Aborted).
+* `syncer` (`refWriter`, `durability.go`).
+* `serve.go`: `openLocalReferenceWrites`, the recovery before the listeners, the
+ durable-step loop.
+* PM grant: `updateref.WithDeferredReferenceDurability`; the `gitcmd` constants
+ `FsyncComponents`, `FsyncComponentsWithoutReferences` and
+ `ReftableLockTimeoutMilliseconds`; `hook/storage_voter.go`
+ (`DeferredReferenceDurability`); the optional interface asserted in
+ `UpdateWithHooks`.
+
+**Tests.**
+* `localrefs`: FIFO order, a cancelled waiter, the wait bound, the hold bound,
+ unique keys; the marker, the durable step (idle, in flight, git's lock held,
+ failed fsync, repository gone), recovery (process restart, kernel restart,
+ stale lock, corrupt marker), the torn-stack experiment.
+* `voter`: committed, aborted and rejected transactions; a marker that cannot be
+ written; durability per mode (off, shadow and authoritative without boot
+ identity keep the fsync).
+* `updateref`: the effective `core.fsync` of `update-ref`.
+* `hook/updateref`: only a voter that defers turns it off.
+* `syncer`: catch-up marks; reboot recovery hydrates (a read before it lists
+ nothing); process-restart recovery keeps; non-caches are kept; a failed fsync
+ moves aside and hydrates.
+* e2e with the real `gitaly` binary: SIGKILL after writes (same boot, kept,
+ settled); SIGKILL in the durable step (the lock is removed at restart, writes
+ go on); a simulated power loss (torn table, other boot: moved aside, every
+ write served); stack depth under 4×30 concurrent writes (git's geometric
+ compaction, at most 2⌈log₂ n⌉+2 tables; measured 4 for 122 updates).
+
### Data model (new migration, both dialects)
Migration `20261018000000_graft_storage_gc`, appended to `sqliteMigrations` and
@@ -1418,6 +1546,7 @@ every entry point returns `FailedPrecondition`.
| WP74b | REVIEW ROOTS RULE: `reviewroots` (Protect, Sweep, Backfill), the review service, `dbreview.WithRequiredRoots`, the two database checks | WP74 |
| WP75 | `gc` package | WP71–WP73 |
| WP76 | syncer (seed, `C_sync`), `verify_objects.go` | WP71, WP74 |
+| WP76b | LOCAL REFERENCE WRITES RULE: `localrefs` (admission, deferred durability), voter, syncer, `serve.go` recovery and durable steps | WP76 |
| WP77 | config, `maintenance` package, `serve.go` wiring | WP74–WP76 |
| WP78 | proto, handlers, SDK, CLI | WP74, WP75 |
| WP79 | e2e, STORAGE.md and CONFIGURATION.md, benchmark | all |
@@ -1478,7 +1607,11 @@ New (m3): `corestore/checkpoint/**`, `corestore/gc/**`, `corestore/maintenance/*
`cmd/graft/storage.go`, `proto/storage_core.proto` (+ generated code),
`config_graft_storage.go`, `doc/graft/STORAGE.md`, `corestore/dbreview/**`,
`corestore/reviewroots/**`, and `internal/gitaly/service/review/**` (the REVIEW
-ROOTS RULE writes only, assigned to m3 by the PM). Shared, my section only:
+ROOTS RULE writes only, assigned to m3 by the PM), `corestore/localrefs/**`
+(WP76b). Granted by the PM for WP76b only: `internal/git/updateref` (the deferred
+durability option), `internal/git/gitcmd` (the fsync and lock-timeout constants),
+`internal/gitaly/hook/storage_voter.go` (and its test double) and the updater call
+in `internal/gitaly/hook/updateref/update_with_hooks.go`. Shared, my section only:
`internal/cli/gitaly/serve.go`, `doc/graft/CONFIGURATION.md`. No change to the land
path: the voter is m3's; the land paths call it unchanged.
@@ -1508,3 +1641,8 @@ path: the voter is m3's; the land paths call it unchanged.
repository row lock.
26. **One upload path**: every production upload goes through `Core.UploadPack`, and
every row that references an upload is committed under its `CommitContext`.
+27. **Local references are a cache of the database**: in authoritative mode a local
+ reference write without fsync happens only after its repository has a durable
+ dirty marker; a marker is removed only after the stack is durable, under git's
+ lock with no transaction in flight; a repository dirty at a kernel restart, or
+ whose durable step failed, is moved aside and hydrated, never served.
diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.md
index 4eb2f42..c9ffbdc 100644
--- a/doc/graft/STORAGE.md
+++ b/doc/graft/STORAGE.md
@@ -431,6 +431,45 @@ or re-import the repository from scratch (see
whose key the database no longer has; it is moved into the storage's
temporary directory, and the repository is reported missing on that node too.
+- **Local reference writes and unclean shutdowns (authoritative mode).** In
+ authoritative mode the database commit is the durable point of a reference
+ update. A node therefore writes its local references without fsync, and makes
+ them durable in a background step a second after a repository goes idle
+ (`doc/graft/PLAN7.md`, LOCAL REFERENCE WRITES RULE). Until then the
+ repository has a dirty marker in `<storage>/+gitaly/graft-ref-durability/`,
+ which holds the kernel's boot identity.
+ - *Process crash or restart* (same kernel boot): nothing is lost, because the
+ page cache kept every write. The restarted node settles the markers.
+ - *Kernel crash or power loss*: the references written since the last durable
+ step may be lost or torn, and git reads a torn reftable stack as an empty
+ repository without an error. At startup, before it opens any listener,
+ `gitaly serve` moves aside every repository that was dirty under the
+ previous boot, and the first request hydrates it again from the database and
+ the bucket. This hydration starts from the newest checkpoint when there is
+ one, so its cost is bounded by how recent the checkpoints are (the
+ maintenance that creates them is WP77; until then, by the full history).
+ Nothing acknowledged is lost. Cost after a power
+ loss: one hydration per repository that had a write in the second before the
+ loss, or that was still being written. On a busy node that can be every
+ active repository. A repository is only moved when the database holds it
+ with changesets and the local copy has a watermark; other markers are
+ dropped.
+ - *A write the kernel reports as failed* (a writeback error, same boot): the
+ durable step's fsync fails, the repository is moved aside at once and logged,
+ and the next request hydrates it.
+ - Offline tools that read a repository from disk (`gitaly git -C <repo>`,
+ `gitaly-debug simulate-http-clone`) must not be pointed at a repository of
+ a node after an unclean shutdown before `gitaly serve` has started once:
+ they would read the torn stack.
+ - Scope: Linux only (the boot identity is `/proc/sys/kernel/random/boot_id`).
+ Elsewhere, and in off and shadow modes, git makes every reference write
+ durable as before.
+ - The node's reference writers (hook transactions and catch-up) wait for the
+ repository's reference lock in arrival order instead of git's polling.
+- **A corrupt dirty marker** stops `gitaly serve` at startup with the marker's
+ path: the repository it named cannot be identified. Check the storage, then
+ remove the marker; the repository's next request hydrates it if it is stale.
+
These behaviours are proved by the end-to-end tests
`internal/graft/e2e/storage_*_test.go`, with several real servers sharing one
database and one bucket, on SQLite and Postgres: writes on one node and reads
@@ -456,6 +495,14 @@ without a single divergence.
| `graft_storage_sync_total{result}` | Catch-ups before a request; results `fresh`, `caught_up`, `hydrated`, `error`. |
| `graft_storage_sync_seconds` | Duration of the catch-up before a request. |
| `graft_storage_sync_changesets_total`, `graft_storage_sync_bytes_total` | Changesets applied and pack bytes downloaded by catch-up and hydration. |
+| `graft_storage_ref_admission_wait_seconds{writer}` | Wait of a local reference writer (`hook`, `syncer`) for its turn at the repository's reference lock. |
+| `graft_storage_ref_admission_waiting` | Writers waiting for their turn, all repositories. |
+| `graft_storage_ref_admission_bypassed_total{writer}` | Writers that went on to git's lock without their turn after `reftable.lockTimeout` (10 s). |
+| `graft_storage_ref_admission_expired_total` | Turns ended by the 10 s bound instead of their release (a git process that died between its phases). |
+| `graft_storage_ref_dirty_repositories` | Repositories with reference writes not yet made durable. |
+| `graft_storage_ref_dirty_marked_total`, `graft_storage_ref_dirty_cleared_total` | Dirty markers written and removed by the durable step. |
+| `graft_storage_ref_writes_inflight` | Hook transactions between their Prepared and Committed phases; one that never commits keeps its repository dirty. |
+| `graft_storage_ref_untrusted_total{reason}` | Repositories moved aside to be hydrated again: `reboot` (dirty at a kernel restart), `sync_failed` (the durable step's fsync failed). |
## Benchmark
diff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.go
index 0d6abde..61993e3 100644
--- a/internal/cli/gitaly/serve.go
+++ b/internal/cli/gitaly/serve.go
@@ -37,6 +37,7 @@ import (
"github.com/feder-cr/git-ai/internal/gitlab"
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/gate"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
corestoreopen "github.com/feder-cr/git-ai/internal/graft/corestore/open"
"github.com/feder-cr/git-ai/internal/graft/corestore/reviewroots"
corestoresyncer "github.com/feder-cr/git-ai/internal/graft/corestore/syncer"
@@ -615,6 +616,12 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
storageGate *gate.Gate
)
if storageCore.Enabled() {
+ if storageCore.Authoritative() {
+ if err := openLocalReferenceWrites(storageCore, locator, logger); err != nil {
+ return err
+ }
+ }
+
syncer := corestoresyncer.New(storageCore, logger, locator, localrepoFactory, gitCmdFactory)
prometheus.MustRegister(syncer)
storageSyncer = syncer
@@ -624,6 +631,20 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
hookManagerOpts = append(hookManagerOpts, hook.WithStorageVoter(storageVoter))
if storageCore.Authoritative() {
+ // Before any listener opens: a repository whose references may be torn
+ // by a kernel restart is moved aside, so nothing reads it before it is
+ // hydrated again (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).
+ storageNames := make([]string, 0, len(cfg.Storages))
+ for _, s := range cfg.Storages {
+ storageNames = append(storageNames, s.Name)
+ }
+ if err := syncer.RecoverLocalReferences(ctx, storageNames); err != nil {
+ return err
+ }
+ durableCtx, stopDurableSteps := context.WithCancel(ctx)
+ defer stopDurableSteps()
+ go syncer.RunDurableSteps(durableCtx, localrefs.DurableStepInterval)
+
storageGate = gate.New(storageCore, syncer, logger, locator, localrepoFactory, protoregistry.GitalyProtoPreregistered)
}
@@ -892,9 +913,6 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
return waitErr
}
-// newPackObjectsAdaptiveLimit builds the adaptive limit backing a pack objects limiter. The limit
-// only moves when the configuration asks for an adaptive one, and is otherwise pinned to the
-// configured concurrency.
// housekeepReviewRoots runs the backstop of the review roots after the daily
// housekeeping of a repository: it writes the keep-around references that
// open reviews lack and deletes those no open review uses (in authoritative
@@ -921,6 +939,32 @@ func housekeepReviewRoots(ctx context.Context, logger log.Logger, core *corestor
return nil
}
+// openLocalReferenceWrites sets up the admission and the deferred durability
+// of the local reference writes of the storage core in authoritative mode
+// (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). Without the kernel's boot
+// identity (not Linux) the durability is not deferred: every reference write
+// is made durable by git, as in the other modes.
+func openLocalReferenceWrites(core *corestore.Core, locator storage.Locator, logger log.Logger) error {
+ lockTimeout := gitcmd.ReftableLockTimeoutMilliseconds * time.Millisecond
+ core.RefAdmission = localrefs.NewAdmission(lockTimeout, lockTimeout)
+ prometheus.MustRegister(core.RefAdmission)
+
+ durability, err := localrefs.NewDurability(locator, localrefs.DurableStepIdle)
+ if err != nil {
+ return fmt.Errorf("graft storage core: %w", err)
+ }
+ if durability == nil {
+ logger.Info("graft storage core: the kernel exposes no boot identity, every reference write is made durable by git")
+ return nil
+ }
+ core.RefDurability = durability
+ prometheus.MustRegister(durability)
+ return nil
+}
+
+// newPackObjectsAdaptiveLimit builds the adaptive limit backing a pack objects limiter. The limit
+// only moves when the configuration asks for an adaptive one, and is otherwise pinned to the
+// configured concurrency.
func newPackObjectsAdaptiveLimit(name string, limits config.ConcurrencyLimits) *limiter.AdaptiveLimit {
if !limits.Adaptive {
return limiter.NewAdaptiveLimit(name, limiter.AdaptiveSetting{
diff --git a/internal/git/gitcmd/command_factory.go b/internal/git/gitcmd/command_factory.go
index 946c4f8..4beb11f 100644
--- a/internal/git/gitcmd/command_factory.go
+++ b/internal/git/gitcmd/command_factory.go
@@ -37,6 +37,16 @@ const (
// Rate limiter is immediately allocated the maxBurstToken value. Burst is the maximum number of tokens
// that can be consumed in a single call
maxBurstToken = 40
+
+ // FsyncComponents are the components git makes durable on every write (`core.fsync`).
+ FsyncComponents = "objects,derived-metadata,reference"
+ // FsyncComponentsWithoutReferences is FsyncComponents without the references, for the
+ // reference writes whose durability the storage core makes itself
+ // (updateref.WithDeferredReferenceDurability).
+ FsyncComponentsWithoutReferences = "objects,derived-metadata"
+ // ReftableLockTimeoutMilliseconds is how long git waits for the reference lock of a reftable
+ // repository (`reftable.lockTimeout`).
+ ReftableLockTimeoutMilliseconds = 10000
)
// CommandFactory is designed to create and run git commands in a protected and fully managed manner.
@@ -725,7 +735,7 @@ func (cf *ExecCommandFactory) GlobalConfiguration(ctx context.Context) ([]Config
// We configure for what data should be fsynced and how that should happen.
// Synchronize object files, packed-refs and loose refs to disk to lessen the
// likelihood of repository corruption in case the server crashes.
- ConfigPair{Key: "core.fsync", Value: "objects,derived-metadata,reference"},
+ ConfigPair{Key: "core.fsync", Value: FsyncComponents},
ConfigPair{Key: "core.fsyncMethod", Value: "fsync"},
// When deleting references, Git needs to rewrite the `packed-refs` file to evict
@@ -746,7 +756,7 @@ func (cf *ExecCommandFactory) GlobalConfiguration(ctx context.Context) ([]Config
// reference transaction, so it is a shared resource like the `packed-refs` lock
// above and gets the same timeout instead of Git's default of 100 milliseconds.
// Context cancellation still ends the wait early.
- ConfigPair{Key: "reftable.lockTimeout", Value: "10000"},
+ ConfigPair{Key: "reftable.lockTimeout", Value: fmt.Sprint(ReftableLockTimeoutMilliseconds)},
// Similarly, for loose references we bump the limit from 100 milliseconds to 1 second. We aim for a
// lower limit here as the locking for loose references is typically a lot more fine-grained. We have
// still observed lock contention around them though, but mostly in cases where the host system was
diff --git a/internal/git/updateref/updateref.go b/internal/git/updateref/updateref.go
index 8c789ca..57192af 100644
--- a/internal/git/updateref/updateref.go
+++ b/internal/git/updateref/updateref.go
@@ -270,10 +270,21 @@ type UpdaterOpt func(*updaterConfig)
type updaterConfig struct {
disableTransactions bool
noDeref bool
+ deferredDurability bool
refBackend *git.ReferenceBackend
refURI string
}
+// WithDeferredReferenceDurability makes git write the references without fsync. Only the
+// storage core's authoritative mode may set it: its database commit is the durable point of
+// the transaction, and it makes the local references durable itself (doc/graft/PLAN7.md,
+// LOCAL REFERENCE WRITES RULE). Objects stay durable as for every other write.
+func WithDeferredReferenceDurability() UpdaterOpt {
+ return func(cfg *updaterConfig) {
+ cfg.deferredDurability = true
+ }
+}
+
// WithDisabledTransactions disables hooks such that no reference-transactions
// are used for the updater.
func WithDisabledTransactions() UpdaterOpt {
@@ -337,6 +348,9 @@ func New(ctx context.Context, repo gitcmd.RepositoryExecutor, opts ...UpdaterOpt
if cfg.refURI != "" {
cmdOpts = append(cmdOpts, gitcmd.WithEnv("GIT_REFERENCE_BACKEND="+cfg.refURI))
}
+ if cfg.deferredDurability {
+ cmdOpts = append(cmdOpts, gitcmd.WithConfig(gitcmd.ConfigPair{Key: "core.fsync", Value: gitcmd.FsyncComponentsWithoutReferences}))
+ }
cmd, err := repo.Exec(
ctx,
diff --git a/internal/git/updateref/updateref_test.go b/internal/git/updateref/updateref_test.go
index b201a9e..df646ce 100644
--- a/internal/git/updateref/updateref_test.go
+++ b/internal/git/updateref/updateref_test.go
@@ -8,6 +8,7 @@ import (
"io/fs"
"os"
"path/filepath"
+ "strings"
"testing"
"time"
@@ -88,6 +89,61 @@ func TestNew_refURI(t *testing.T) {
require.Equal(t, expectedErr, updater.Commit())
}
+// TestNew_deferredReferenceDurability verifies that only WithDeferredReferenceDurability takes the
+// references out of what git makes durable, and that it keeps the objects in.
+func TestNew_deferredReferenceDurability(t *testing.T) {
+ t.Parallel()
+
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+ _, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{
+ SkipCreationViaService: true,
+ })
+ repo := gittest.NewRepositoryPathExecutor(t, cfg, repoPath)
+ commitID := gittest.WriteCommit(t, cfg, repoPath)
+
+ // effectiveConfig returns the last value of key among the `-c key=value` arguments of a
+ // git command line, which is the one git uses.
+ effectiveConfig := func(args []string, key string) string {
+ var effective string
+ for i := 0; i+1 < len(args); i++ {
+ if args[i] != "-c" {
+ continue
+ }
+ name, value, _ := strings.Cut(args[i+1], "=")
+ if strings.EqualFold(name, key) {
+ effective = value
+ }
+ }
+ return effective
+ }
+
+ for _, tc := range []struct {
+ desc string
+ opts []UpdaterOpt
+ expected string
+ }{
+ {desc: "default", expected: gitcmd.FsyncComponents},
+ {desc: "without transactions", opts: []UpdaterOpt{WithDisabledTransactions()}, expected: gitcmd.FsyncComponents},
+ {desc: "deferred", opts: []UpdaterOpt{WithDisabledTransactions(), WithDeferredReferenceDurability()}, expected: gitcmd.FsyncComponentsWithoutReferences},
+ } {
+ t.Run(tc.desc, func(t *testing.T) {
+ updater, err := New(ctx, repo, tc.opts...)
+ require.NoError(t, err)
+ defer testhelper.MustClose(t, updater)
+
+ require.Equal(t, tc.expected, effectiveConfig(updater.cmd.Args(), "core.fsync"))
+
+ // The writes themselves are unchanged.
+ ref := git.ReferenceName("refs/heads/" + strings.ReplaceAll(tc.desc, " ", "-"))
+ require.NoError(t, updater.Start())
+ require.NoError(t, updater.Create(ref, commitID))
+ require.NoError(t, updater.Commit())
+ require.Equal(t, commitID, gittest.ResolveRevision(t, cfg, repoPath, ref.String()))
+ })
+ }
+}
+
func TestUpdater_create(t *testing.T) {
t.Parallel()
diff --git a/internal/gitaly/hook/storage_voter.go b/internal/gitaly/hook/storage_voter.go
index 03da717..eb297fd 100644
--- a/internal/gitaly/hook/storage_voter.go
+++ b/internal/gitaly/hook/storage_voter.go
@@ -25,6 +25,19 @@ type StorageVoter interface {
// Authoritative reports whether the storage core's database replaces the
// file changelog.
Authoritative() bool
+ // DeferredReferenceDurability reports whether reference transactions that
+ // the voter commits may write their references without fsync: the database
+ // commit is their durable point, and the storage core makes the local
+ // references durable later (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).
+ DeferredReferenceDurability() bool
+}
+
+// DeferredReferenceDurability reports whether the reference transactions that
+// the manager drives itself (UpdateWithHooks) may write their references
+// without fsync. It is false without a storage voter, so only the storage
+// core's authoritative mode ever turns the fsync of references off.
+func (m *GitLabHookManager) DeferredReferenceDurability() bool {
+ return m.storageVoter != nil && m.storageVoter.DeferredReferenceDurability()
}
// WithStorageVoter makes the manager commit every reference transaction to
diff --git a/internal/gitaly/hook/storage_voter_test.go b/internal/gitaly/hook/storage_voter_test.go
index c7da8c2..7c411ad 100644
--- a/internal/gitaly/hook/storage_voter_test.go
+++ b/internal/gitaly/hook/storage_voter_test.go
@@ -59,6 +59,8 @@ func (v *recordingVoter) Aborted(context.Context, gitcmd.HooksPayload, []byte) {
func (v *recordingVoter) Authoritative() bool { return true }
+func (v *recordingVoter) DeferredReferenceDurability() bool { return false }
+
// TestReferenceTransactionHook_storageVoterPhases checks that the hook manager
// hands every phase of a reference transaction to the storage core, the
// Preparing phase included (doc/graft/PLAN6.md, LOCK AND HISTORY RULE), and
diff --git a/internal/gitaly/hook/updateref/deferred_durability_test.go b/internal/gitaly/hook/updateref/deferred_durability_test.go
new file mode 100644
index 0000000..1b9d414
--- /dev/null
+++ b/internal/gitaly/hook/updateref/deferred_durability_test.go
@@ -0,0 +1,67 @@
+package updateref
+
+import (
+ "context"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git/gitcmd"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/internal/gitaly/hook"
+ "github.com/feder-cr/git-ai/internal/gitlab"
+ "github.com/feder-cr/git-ai/internal/graft/changelog"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+)
+
+// durabilityVoter is a storage voter that only answers whether it defers the durability of the
+// references.
+type durabilityVoter struct {
+ deferred bool
+}
+
+func (durabilityVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte) {}
+
+func (durabilityVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {
+ return nil
+}
+
+func (durabilityVoter) Committed(context.Context, gitcmd.HooksPayload, []byte) {}
+
+func (durabilityVoter) Aborted(context.Context, gitcmd.HooksPayload, []byte) {}
+
+func (durabilityVoter) Authoritative() bool { return true }
+
+func (v durabilityVoter) DeferredReferenceDurability() bool { return v.deferred }
+
+// TestDeferReferenceDurability verifies that the updater writes references without fsync only
+// when the storage voter of its hook manager defers their durability: never without the storage
+// core, nor when the voter does not (shadow mode, or authoritative mode where the kernel exposes
+// no boot identity; the voter's own answer is tested in the voter package).
+func TestDeferReferenceDurability(t *testing.T) {
+ t.Parallel()
+
+ cfg := testcfg.Build(t)
+ newManager := func(opts ...hook.ManagerOption) hook.Manager {
+ return hook.NewManager(cfg, config.NewLocator(cfg), testhelper.NewLogger(t), gittest.NewCommandFactory(t, cfg), nil, nil,
+ gitlab.NewMockClient(t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive),
+ hook.NewPostReceiveRegistry(), opts...)
+ }
+
+ for _, tc := range []struct {
+ desc string
+ manager hook.Manager
+ expected bool
+ }{
+ {desc: "storage core off", manager: newManager()},
+ {desc: "voter keeps the fsync", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: false}))},
+ {desc: "voter defers the durability", manager: newManager(hook.WithStorageVoter(durabilityVoter{deferred: true})), expected: true},
+ {desc: "manager without a storage voter", manager: hook.NewMockManager(t, hook.NopPreReceive, hook.NopPostReceive, hook.NopUpdate, hook.NopReferenceTransaction)},
+ } {
+ t.Run(tc.desc, func(t *testing.T) {
+ require.Equal(t, tc.expected, deferReferenceDurability(tc.manager))
+ })
+ }
+}
diff --git a/internal/gitaly/hook/updateref/update_with_hooks.go b/internal/gitaly/hook/updateref/update_with_hooks.go
index 407b2f9..acf21e7 100644
--- a/internal/gitaly/hook/updateref/update_with_hooks.go
+++ b/internal/gitaly/hook/updateref/update_with_hooks.go
@@ -140,6 +140,19 @@ func (e Error) Proto() *gitalypb.ReferenceUpdateError {
}
}
+// deferredDurability is implemented by a hook manager whose storage core writes the references
+// of the transactions it commits without fsync (hook.GitLabHookManager).
+type deferredDurability interface {
+ DeferredReferenceDurability() bool
+}
+
+// deferReferenceDurability reports whether the reference transactions driven with hookManager
+// may write their references without fsync: only when its storage voter says so.
+func deferReferenceDurability(hookManager hook.Manager) bool {
+ d, ok := hookManager.(deferredDurability)
+ return ok && d.DeferredReferenceDurability()
+}
+
// NewUpdaterWithHooks creates a new instance of a struct that will update a Git reference.
func NewUpdaterWithHooks(
cfg config.Cfg,
@@ -257,7 +270,13 @@ func (u *UpdaterWithHooks) UpdateReference(
// is packed, which is obviously a bad thing as Gitaly nodes may be differently packed. We
// thus continue to manually drive the reference-transaction hook here, which doesn't have
// this problem.
- updater, err := updateref.New(ctx, repo, updateref.WithDisabledTransactions())
+ updaterOpts := []updateref.UpdaterOpt{updateref.WithDisabledTransactions()}
+ // The storage core's authoritative mode makes the references durable itself: its database
+ // commit in the prepared phase is the durable point of the transaction.
+ if deferReferenceDurability(u.hookManager) {
+ updaterOpts = append(updaterOpts, updateref.WithDeferredReferenceDurability())
+ }
+ updater, err := updateref.New(ctx, repo, updaterOpts...)
if err != nil {
return fmt.Errorf("creating updater: %w", err)
}
diff --git a/internal/graft/corestore/core.go b/internal/graft/corestore/core.go
index 2f8cdee..df13121 100644
--- a/internal/graft/corestore/core.go
+++ b/internal/graft/corestore/core.go
@@ -10,6 +10,7 @@ import (
"time"
"github.com/feder-cr/git-ai/internal/gitaly/storage"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
)
// BlobStore is the object storage of the storage core. Objects are immutable
@@ -99,6 +100,13 @@ type Core struct {
// runs in shadow mode too.
GCGrace time.Duration
GCMargin time.Duration
+ // RefAdmission orders the local reference writers of the node and
+ // RefDurability defers the durability of their writes to a durable step
+ // (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). Both are set in
+ // authoritative mode only; nil admits nothing and keeps every reference write
+ // durable by itself.
+ RefAdmission *localrefs.Admission
+ RefDurability *localrefs.Durability
}
// WriteDeadline is W of doc/graft/PLAN7.md: a writer uploads a pack and
diff --git a/internal/graft/corestore/coretest/coretest.go b/internal/graft/corestore/coretest/coretest.go
index 93b1b95..829d77f 100644
--- a/internal/graft/corestore/coretest/coretest.go
+++ b/internal/graft/corestore/coretest/coretest.go
@@ -20,8 +20,10 @@ import (
"testing"
"time"
+ "github.com/feder-cr/git-ai/internal/gitaly/storage"
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/blobstore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
"github.com/feder-cr/git-ai/internal/graft/corestore/sqlstore"
"github.com/stretchr/testify/require"
)
@@ -196,3 +198,20 @@ func Open(tb testing.TB, opts Options) *corestore.Core {
return core
}
+
+// EnableLocalRefs gives the core the admission and the deferred durability of
+// the local reference writes, as gitaly serve does in authoritative mode
+// (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). The boot identity is a
+// fixed test value unless opts set another one.
+func EnableLocalRefs(tb testing.TB, core *corestore.Core, locator storage.Locator, opts ...localrefs.DurabilityOption) {
+ tb.Helper()
+
+ core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)
+ durability, err := localrefs.NewDurability(locator, 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID)}, opts...)...)
+ require.NoError(tb, err)
+ core.RefDurability = durability
+}
+
+// TestBootID is the boot identity EnableLocalRefs gives the deferred
+// durability by default.
+const TestBootID = "test-boot"
diff --git a/internal/graft/corestore/localrefs/admission.go b/internal/graft/corestore/localrefs/admission.go
new file mode 100644
index 0000000..64a4536
--- /dev/null
+++ b/internal/graft/corestore/localrefs/admission.go
@@ -0,0 +1,286 @@
+// Package localrefs orders and makes durable the local reference writes of the
+// storage core in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE
+// WRITES RULE).
+//
+// A reftable repository has one lock for all its references (tables.list.lock),
+// and git waits for it by polling with a growing, randomized sleep: there is no
+// queue, a newcomer can take the lock before a writer that has waited for long,
+// and the lock can sit free while the waiters sleep. Admission puts the writers
+// of one node in a FIFO queue per repository in front of that lock. It only
+// orders: git's lock stays the only mutual exclusion, so every bound of the
+// admission falls back to git's own wait.
+//
+// Durability moves the durability of the local reference writes from each
+// transaction to a durable step that runs outside the writers' path; see
+// Durability.
+package localrefs
+
+import (
+ "container/list"
+ "context"
+ "encoding/binary"
+ "sync"
+ "sync/atomic"
+ "time"
+
+ "github.com/prometheus/client_golang/prometheus"
+)
+
+// Key identifies the holder of an admission between Acquire and Release. The
+// voter uses the pending key of the transaction (its repository and hook input).
+type Key [32]byte
+
+// Writers, the label of the admission metrics.
+const (
+ // WriterHook is a reference transaction of the reference-transaction hook.
+ WriterHook = "hook"
+ // WriterSyncer is a reference transaction or a durable step of the syncer.
+ WriterSyncer = "syncer"
+)
+
+// Admission is the FIFO admission of the reference writers of one node, per
+// local repository (keyed by its path). A nil *Admission admits nothing and
+// never waits.
+type Admission struct {
+ // waitBound is how long Acquire waits before the writer goes on to git's
+ // lock without being admitted. It equals git's own wait for the lock
+ // (reftable.lockTimeout), so that a write that today ends in git's lock
+ // timeout (another transaction on the same repository held open by the
+ // same caller) does not wait forever on the admission instead.
+ waitBound time.Duration
+ // holdBound is how long an admission is held at most. The hook passes no
+ // transaction identity and git can die between the phases of a transaction
+ // without reporting Committed or Aborted, so an admission is never held
+ // beyond it.
+ holdBound time.Duration
+
+ mu sync.Mutex
+ queues map[string]*queue
+ // unique numbers the keys of AcquireUnique.
+ unique atomic.Uint64
+ // onWait is called when a writer starts waiting; tests set it to know that
+ // a writer is queued.
+ onWait func(path string)
+
+ wait *prometheus.HistogramVec
+ waiting prometheus.Gauge
+ bypassed *prometheus.CounterVec
+ expired prometheus.Counter
+}
+
+var _ prometheus.Collector = (*Admission)(nil)
+
+// queue is the admission of one repository: the holder and the writers waiting
+// behind it, oldest first.
+type queue struct {
+ holder *ticket
+ waiters *list.List
+}
+
+// ticket is one writer of a queue.
+type ticket struct {
+ key Key
+ // admitted is closed when the ticket becomes the holder.
+ admitted chan struct{}
+ // element is the ticket in the waiters while it waits.
+ element *list.Element
+ // expiry ends the admission at the hold bound.
+ expiry *time.Timer
+}
+
+// NewAdmission returns an admission with the given bounds, both positive.
+func NewAdmission(waitBound, holdBound time.Duration) *Admission {
+ a := &Admission{
+ waitBound: waitBound,
+ holdBound: holdBound,
+ queues: map[string]*queue{},
+ wait: prometheus.NewHistogramVec(prometheus.HistogramOpts{
+ Name: "graft_storage_ref_admission_wait_seconds",
+ Help: "Time a local reference writer waited for its admission to the reference lock of a repository, by writer (hook, syncer).",
+ Buckets: []float64{0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},
+ }, []string{"writer"}),
+ waiting: prometheus.NewGauge(prometheus.GaugeOpts{
+ Name: "graft_storage_ref_admission_waiting",
+ Help: "Number of local reference writers waiting for their admission, all repositories.",
+ }),
+ bypassed: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "graft_storage_ref_admission_bypassed_total",
+ Help: "Number of local reference writers that went on to git's reference lock without an admission after the wait bound, by writer.",
+ }, []string{"writer"}),
+ expired: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_storage_ref_admission_expired_total",
+ Help: "Number of admissions ended by the hold bound instead of their release (a writer that never reported its end).",
+ }),
+ }
+ for _, writer := range []string{WriterHook, WriterSyncer} {
+ a.wait.WithLabelValues(writer)
+ a.bypassed.WithLabelValues(writer)
+ }
+ return a
+}
+
+// Acquire waits for the admission of key to the reference lock of the
+// repository at path. It returns true when the writer is admitted, and must
+// then call Release(path, key) when it no longer needs the lock. It returns
+// false without an admission when ctx ends (the writer is gone) or the wait
+// bound passes (the writer goes on to git's lock as without admission).
+func (a *Admission) Acquire(ctx context.Context, path string, key Key, writer string) bool {
+ if a == nil {
+ return false
+ }
+
+ start := time.Now()
+ t := &ticket{key: key, admitted: make(chan struct{})}
+
+ a.mu.Lock()
+ q := a.queues[path]
+ if q == nil {
+ q = &queue{waiters: list.New()}
+ a.queues[path] = q
+ }
+ if q.holder == nil && q.waiters.Len() == 0 {
+ a.admitLocked(path, q, t)
+ a.mu.Unlock()
+ a.wait.WithLabelValues(writer).Observe(time.Since(start).Seconds())
+ return true
+ }
+ t.element = q.waiters.PushBack(t)
+ a.waiting.Inc()
+ a.mu.Unlock()
+ if a.onWait != nil {
+ a.onWait(path)
+ }
+
+ bound := time.NewTimer(a.waitBound)
+ defer bound.Stop()
+
+ var cancelled bool
+ select {
+ case <-t.admitted:
+ a.wait.WithLabelValues(writer).Observe(time.Since(start).Seconds())
+ return true
+ case <-bound.C:
+ case <-ctx.Done():
+ cancelled = true
+ }
+
+ a.mu.Lock()
+ defer a.mu.Unlock()
+
+ select {
+ case <-t.admitted:
+ // Admitted while giving up.
+ if cancelled {
+ a.handOffLocked(path, q)
+ return false
+ }
+ a.wait.WithLabelValues(writer).Observe(time.Since(start).Seconds())
+ return true
+ default:
+ }
+
+ q.waiters.Remove(t.element)
+ t.element = nil
+ a.waiting.Dec()
+ if q.holder == nil && q.waiters.Len() == 0 {
+ delete(a.queues, path)
+ }
+ if !cancelled {
+ a.bypassed.WithLabelValues(writer).Inc()
+ a.wait.WithLabelValues(writer).Observe(time.Since(start).Seconds())
+ }
+ return false
+}
+
+// AcquireUnique is Acquire with a key of its own. It returns the release
+// function, nil when the writer was not admitted.
+func (a *Admission) AcquireUnique(ctx context.Context, path, writer string) func() {
+ if a == nil {
+ return nil
+ }
+
+ // The keys of the voter are SHA-256 sums; a counter in the first eight bytes
+ // and a marker byte that is set make a key that no other writer uses.
+ var key Key
+ binary.BigEndian.PutUint64(key[:8], a.unique.Add(1))
+ key[31] = 0xff
+ key[30] = 0xfe
+
+ if !a.Acquire(ctx, path, key, writer) {
+ return nil
+ }
+ return func() { a.Release(path, key) }
+}
+
+// Release ends the admission of key to the reference lock of the repository at
+// path. It does nothing when key does not hold it: the admission ended at the
+// hold bound, or the writer was not admitted. Two transactions with the same
+// key (the same repository and hook input) are not told apart: the release of
+// one may end the admission of the other, which then waits on git's lock as
+// without admission.
+func (a *Admission) Release(path string, key Key) {
+ if a == nil {
+ return
+ }
+
+ a.mu.Lock()
+ defer a.mu.Unlock()
+
+ q := a.queues[path]
+ if q == nil || q.holder == nil || q.holder.key != key {
+ return
+ }
+ q.holder.expiry.Stop()
+ a.handOffLocked(path, q)
+}
+
+// admitLocked makes t the holder of q.
+func (a *Admission) admitLocked(path string, q *queue, t *ticket) {
+ q.holder = t
+ close(t.admitted)
+ t.expiry = time.AfterFunc(a.holdBound, func() { a.expire(path, t) })
+}
+
+// handOffLocked ends the admission of the holder of q and admits the oldest
+// waiter, if any.
+func (a *Admission) handOffLocked(path string, q *queue) {
+ q.holder = nil
+ front := q.waiters.Front()
+ if front == nil {
+ delete(a.queues, path)
+ return
+ }
+ next := q.waiters.Remove(front).(*ticket)
+ next.element = nil
+ a.waiting.Dec()
+ a.admitLocked(path, q, next)
+}
+
+// expire ends the admission of t at the hold bound, if t still holds it.
+func (a *Admission) expire(path string, t *ticket) {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+
+ q := a.queues[path]
+ if q == nil || q.holder != t {
+ return
+ }
+ a.expired.Inc()
+ a.handOffLocked(path, q)
+}
+
+// Describe implements prometheus.Collector.
+func (a *Admission) Describe(descs chan<- *prometheus.Desc) {
+ a.wait.Describe(descs)
+ a.waiting.Describe(descs)
+ a.bypassed.Describe(descs)
+ a.expired.Describe(descs)
+}
+
+// Collect implements prometheus.Collector.
+func (a *Admission) Collect(metrics chan<- prometheus.Metric) {
+ a.wait.Collect(metrics)
+ a.waiting.Collect(metrics)
+ a.bypassed.Collect(metrics)
+ a.expired.Collect(metrics)
+}
diff --git a/internal/graft/corestore/localrefs/admission_test.go b/internal/graft/corestore/localrefs/admission_test.go
new file mode 100644
index 0000000..69d13c1
--- /dev/null
+++ b/internal/graft/corestore/localrefs/admission_test.go
@@ -0,0 +1,173 @@
+package localrefs
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ "github.com/prometheus/client_golang/prometheus/testutil"
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/testhelper"
+)
+
+func TestMain(m *testing.M) {
+ testhelper.Run(m)
+}
+
+func key(i byte) Key { return Key{i} }
+
+// queued returns an admission whose onWait reports every queued writer.
+func queued(waitBound, holdBound time.Duration) (*Admission, <-chan string) {
+ a := NewAdmission(waitBound, holdBound)
+ waiting := make(chan string, 16)
+ a.onWait = func(path string) { waiting <- path }
+ return a, waiting
+}
+
+// acquireAsync starts Acquire in the background, waits until the writer is
+// queued, and returns the channel of its result.
+func acquireAsync(ctx context.Context, t *testing.T, a *Admission, waiting <-chan string, path string, k Key) <-chan bool {
+ t.Helper()
+ result := make(chan bool, 1)
+ go func() { result <- a.Acquire(ctx, path, k, WriterHook) }()
+ require.Equal(t, path, <-waiting)
+ return result
+}
+
+func TestAdmissionAdmitsInArrivalOrder(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ a, waiting := queued(time.Hour, time.Hour)
+
+ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))
+ results := []<-chan bool{
+ acquireAsync(ctx, t, a, waiting, "repo", key(1)),
+ acquireAsync(ctx, t, a, waiting, "repo", key(2)),
+ acquireAsync(ctx, t, a, waiting, "repo", key(3)),
+ }
+ require.Equal(t, 3.0, testutil.ToFloat64(a.waiting))
+
+ // Another repository is not ordered behind them.
+ require.True(t, a.Acquire(ctx, "other", key(9), WriterHook))
+
+ for i, result := range results {
+ select {
+ case <-result:
+ t.Fatalf("writer %d admitted before the release of writer %d", i+1, i)
+ default:
+ }
+ a.Release("repo", key(byte(i)))
+ require.True(t, <-result, "writer %d", i+1)
+ }
+ a.Release("repo", key(3))
+ a.Release("other", key(9))
+
+ require.Empty(t, a.queues)
+ require.Equal(t, 0.0, testutil.ToFloat64(a.waiting))
+}
+
+func TestAdmissionCancelledWriterLeavesTheQueue(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ a, waiting := queued(time.Hour, time.Hour)
+
+ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))
+ cancelCtx, cancel := context.WithCancel(ctx)
+ cancelled := acquireAsync(cancelCtx, t, a, waiting, "repo", key(1))
+ next := acquireAsync(ctx, t, a, waiting, "repo", key(2))
+
+ cancel()
+ require.False(t, <-cancelled)
+
+ a.Release("repo", key(0))
+ require.True(t, <-next)
+ a.Release("repo", key(2))
+ require.Empty(t, a.queues)
+ require.Equal(t, 0.0, testutil.ToFloat64(a.bypassed.WithLabelValues(WriterHook)))
+}
+
+func TestAdmissionWaitBoundFallsBackToGitsLock(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ a := NewAdmission(20*time.Millisecond, time.Hour)
+
+ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))
+ require.False(t, a.Acquire(ctx, "repo", key(1), WriterSyncer))
+ require.Equal(t, 1.0, testutil.ToFloat64(a.bypassed.WithLabelValues(WriterSyncer)))
+
+ // The writer that went on without an admission releases nothing.
+ a.Release("repo", key(1))
+ require.Equal(t, key(0), a.queues["repo"].holder.key)
+
+ a.Release("repo", key(0))
+ require.Empty(t, a.queues)
+}
+
+func TestAdmissionHoldBoundEndsALostAdmission(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ a, waiting := queued(time.Hour, time.Hour)
+
+ // Writer 0 never releases (git died between its phases); the hold bound
+ // ends its admission, which is what its timer calls.
+ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))
+ second := acquireAsync(ctx, t, a, waiting, "repo", key(1))
+ a.mu.Lock()
+ lost := a.queues["repo"].holder
+ a.mu.Unlock()
+ a.expire("repo", lost)
+ require.True(t, <-second)
+ require.Equal(t, 1.0, testutil.ToFloat64(a.expired))
+
+ // The late release of writer 0 does not end the admission of writer 1, and
+ // a second expiry of the lost admission does nothing.
+ a.Release("repo", key(0))
+ a.expire("repo", lost)
+ a.mu.Lock()
+ require.Equal(t, key(1), a.queues["repo"].holder.key)
+ a.mu.Unlock()
+ require.Equal(t, 1.0, testutil.ToFloat64(a.expired))
+
+ a.Release("repo", key(1))
+ require.Empty(t, a.queues)
+}
+
+func TestAdmissionHoldBoundTimer(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ a, waiting := queued(time.Hour, 20*time.Millisecond)
+
+ require.True(t, a.Acquire(ctx, "repo", key(0), WriterHook))
+ require.True(t, <-acquireAsync(ctx, t, a, waiting, "repo", key(1)))
+ require.GreaterOrEqual(t, testutil.ToFloat64(a.expired), 1.0)
+}
+
+func TestAdmissionUniqueKeysAreOrdered(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ a, waiting := queued(time.Hour, time.Hour)
+
+ release := a.AcquireUnique(ctx, "repo", WriterSyncer)
+ require.NotNil(t, release)
+
+ second := make(chan func(), 1)
+ go func() { second <- a.AcquireUnique(ctx, "repo", WriterSyncer) }()
+ require.Equal(t, "repo", <-waiting)
+
+ release()
+ releaseSecond := <-second
+ require.NotNil(t, releaseSecond)
+ releaseSecond()
+ require.Empty(t, a.queues)
+}
+
+func TestNilAdmissionAdmitsNothing(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+
+ var a *Admission
+ require.False(t, a.Acquire(ctx, "repo", key(0), WriterHook))
+ require.Nil(t, a.AcquireUnique(ctx, "repo", WriterSyncer))
+ a.Release("repo", key(0))
+}
diff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.go
new file mode 100644
index 0000000..ca0a905
--- /dev/null
+++ b/internal/graft/corestore/localrefs/durability.go
@@ -0,0 +1,793 @@
+package localrefs
+
+import (
+ "bufio"
+ "bytes"
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/prometheus/client_golang/prometheus"
+
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/internal/gitaly/storage"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
+)
+
+// MarkerDirName is the directory of the dirty markers in a storage, under
+// Gitaly's data directory. It is not the state directory, whose files the disk
+// cache walker removes after an hour.
+const MarkerDirName = config.GitalyDataPrefix + "/graft-ref-durability"
+
+// bootIDPath is where Linux exposes the identity of the running kernel boot.
+const bootIDPath = "/proc/sys/kernel/random/boot_id"
+
+const (
+ // DurableStepInterval is how often the durable step runs for every dirty
+ // repository: it bounds how long a write failure the kernel reports goes
+ // unnoticed.
+ DurableStepInterval = time.Second
+ // DurableStepIdle is how long a dirty repository must be without a reference
+ // write before the durable step removes its marker.
+ DurableStepIdle = time.Second
+)
+
+// CrashEnv names the test-only environment variable that makes the process
+// exit (exit code 75) in the durable step, while it holds git's reference lock
+// after making the stack durable and before it removes the marker: "durable_step".
+// The crash tests with real binaries use it. It is read once by NewDurability.
+const CrashEnv = "GRAFT_STORAGE_TEST_CRASH"
+
+const (
+ crashDurableStep = "durable_step"
+ crashExitCode = 75
+)
+
+// stackLockContent is what the durable step writes into the reference lock
+// it takes. git never writes it (a lock of git holds table names), so a lock
+// with this content found at startup was left by a durable step that died.
+const stackLockContent = "graft durable step\n"
+
+// ErrUntrusted is returned by Settle when making the references of a
+// repository durable failed: the kernel may have dropped some of their writes,
+// so the local repository must not be served before it is rebuilt.
+var ErrUntrusted = errors.New("storage core: the local references may have lost writes")
+
+// UntrustedError is ErrUntrusted for one repository.
+type UntrustedError struct {
+ Repo RepoRef
+ Path string
+ Err error
+}
+
+func (e *UntrustedError) Error() string {
+ return fmt.Sprintf("%v: %s: %v", ErrUntrusted, e.Path, e.Err)
+}
+
+// Unwrap returns ErrUntrusted and the cause.
+func (e *UntrustedError) Unwrap() []error { return []error{ErrUntrusted, e.Err} }
+
+// Durability defers the durability of the local reference writes of the
+// storage core in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE
+// WRITES RULE). The database commit of a transaction is its durable commit
+// point and the local repository is a cache of it, so git writes the references
+// without fsync and Durability keeps the cache safe to serve:
+//
+// - before the first reference write that is not durable, the repository
+// gets a dirty marker, written durably, that holds the identity of the
+// kernel boot (MarkDirty);
+// - a durable step makes the references durable and removes the marker,
+// under git's reference lock and only when no writer is between its
+// Prepared and Committed phases (Settle);
+// - at startup, a marker of another boot means the kernel restarted while
+// the repository had writes that may never have reached the disk: its
+// reftable stack may be torn, which git reads as an empty repository, so
+// the repository is moved aside and hydrated again (Recover). A marker of
+// the running boot is a process restart: the kernel kept every write.
+//
+// A nil *Durability is off: every reference write is durable by itself
+// (core.fsync with references), as outside authoritative mode.
+type Durability struct {
+ bootID string
+ locator storage.Locator
+ // idle is how long a dirty repository must be without a reference write
+ // before Settle removes its marker.
+ idle time.Duration
+ // sync makes a file or directory durable; tests replace it.
+ sync func(f *os.File) error
+ // crashAt is the crash point selected with CrashEnv.
+ crashAt string
+
+ mu sync.Mutex
+ // repos holds the state of every repository written since the process
+ // started (one entry per repository; Forget drops it).
+ repos map[string]*refState
+
+ marked prometheus.Counter
+ cleared prometheus.Counter
+ untrusted *prometheus.CounterVec
+ dirty prometheus.Gauge
+ inflight prometheus.Gauge
+}
+
+var _ prometheus.Collector = (*Durability)(nil)
+
+// refState is what Durability knows of one local repository, by its path. Its
+// fields are guarded by Durability.mu, except marker writes and removals,
+// which file serializes.
+type refState struct {
+ // file serializes the writes and the removal of the marker file.
+ file sync.Mutex
+ repo RepoRef
+ marker string
+ dirty bool
+ // inflight counts, by key, the hook transactions between their Prepared
+ // and Committed phases: git may still write their table and compact the
+ // stack without fsync.
+ inflight map[Key]int
+ nInflight int
+ lastWrite time.Time
+ // synced holds the tables made durable during the current dirty period.
+ // Tables are immutable, so one fsync each is enough.
+ synced map[string]struct{}
+}
+
+// RepoRef names a repository of a storage.
+type RepoRef struct {
+ Storage string `json:"storage"`
+ RelativePath string `json:"relative_path"`
+}
+
+func repoOf(r RepoRef) *gitalypb.Repository {
+ return &gitalypb.Repository{StorageName: r.Storage, RelativePath: r.RelativePath}
+}
+
+// marker is the content of a dirty marker.
+type marker struct {
+ RepoRef
+ BootID string `json:"boot_id"`
+}
+
+// Reasons of graft_storage_ref_untrusted_total.
+const (
+ // ReasonReboot is a dirty repository found after a kernel restart.
+ ReasonReboot = "reboot"
+ // ReasonSyncFailed is a durable step whose fsync failed.
+ ReasonSyncFailed = "sync_failed"
+)
+
+// DurabilityOption configures a Durability.
+type DurabilityOption func(*Durability)
+
+// WithBootID sets the identity of the kernel boot instead of reading it from
+// the kernel. Tests use it to simulate a kernel restart.
+func WithBootID(bootID string) DurabilityOption {
+ return func(d *Durability) { d.bootID = bootID }
+}
+
+// WithSync replaces how a file or directory is made durable. Tests use it to
+// simulate a write the kernel failed to do.
+func WithSync(sync func(f *os.File) error) DurabilityOption {
+ return func(d *Durability) { d.sync = sync }
+}
+
+// NewDurability returns the deferred durability of the node, or nil when the
+// kernel does not expose its boot identity (not Linux) and no option sets one:
+// the references are then written durably by every transaction.
+func NewDurability(locator storage.Locator, idle time.Duration, opts ...DurabilityOption) (*Durability, error) {
+ d := newDurability(locator, idle)
+ for _, opt := range opts {
+ opt(d)
+ }
+ if d.bootID != "" {
+ return d, nil
+ }
+
+ content, err := os.ReadFile(bootIDPath)
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil, nil
+ }
+ if err != nil {
+ return nil, fmt.Errorf("read the boot identity: %w", err)
+ }
+ d.bootID = strings.TrimSpace(string(content))
+ if d.bootID == "" {
+ return nil, fmt.Errorf("read the boot identity: %s is empty", bootIDPath)
+ }
+ return d, nil
+}
+
+func newDurability(locator storage.Locator, idle time.Duration) *Durability {
+ return &Durability{
+ locator: locator,
+ idle: idle,
+ sync: (*os.File).Sync,
+ crashAt: os.Getenv(CrashEnv),
+ repos: map[string]*refState{},
+ marked: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_storage_ref_dirty_marked_total",
+ Help: "Number of dirty markers written: a repository got local reference writes that are not durable yet.",
+ }),
+ cleared: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_storage_ref_dirty_cleared_total",
+ Help: "Number of dirty markers removed by a durable step.",
+ }),
+ untrusted: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "graft_storage_ref_untrusted_total",
+ Help: "Number of local repositories whose references could not be trusted and were moved aside to be hydrated again, by reason (reboot, sync_failed).",
+ }, []string{"reason"}),
+ dirty: prometheus.NewGauge(prometheus.GaugeOpts{
+ Name: "graft_storage_ref_dirty_repositories",
+ Help: "Number of local repositories with reference writes that are not durable yet.",
+ }),
+ inflight: prometheus.NewGauge(prometheus.GaugeOpts{
+ Name: "graft_storage_ref_writes_inflight",
+ Help: "Number of hook reference transactions between their Prepared and Committed phases. One that never reports Committed keeps its repository dirty.",
+ }),
+ }
+}
+
+// Enabled reports whether the reference writes of the storage core are made
+// without fsync.
+func (d *Durability) Enabled() bool { return d != nil }
+
+// markerPath returns the path of the dirty marker of the repository at path.
+func (d *Durability) markerPath(ctx context.Context, repo RepoRef, path string) (string, error) {
+ dir, err := d.markerDir(ctx, repo.Storage)
+ if err != nil {
+ return "", err
+ }
+ sum := sha256.Sum256([]byte(path))
+ return filepath.Join(dir, hex.EncodeToString(sum[:])+".json"), nil
+}
+
+func (d *Durability) markerDir(ctx context.Context, storageName string) (string, error) {
+ root, err := d.locator.GetStorageByName(ctx, storageName)
+ if err != nil {
+ return "", err
+ }
+ return filepath.Join(root, MarkerDirName), nil
+}
+
+// stateLocked returns the state of the repository at path, creating it. The
+// caller holds d.mu.
+func (d *Durability) stateLocked(path string) *refState {
+ st := d.repos[path]
+ if st == nil {
+ st = &refState{inflight: map[Key]int{}, synced: map[string]struct{}{}}
+ d.repos[path] = st
+ }
+ return st
+}
+
+// MarkDirty records that the repository at path is about to get reference
+// writes that are not durable. When it returns nil the repository has a durable
+// dirty marker; the writer must not write before. The caller holds git's
+// reference lock of the repository (the voter's Prepared phase) or the syncer's
+// repository lock, under which Settle decides to clear the marker, and the
+// marker file is only written or removed under the state's file lock, so a
+// marker is never removed after a writer relied on it.
+func (d *Durability) MarkDirty(ctx context.Context, repo RepoRef, path string) error {
+ if d == nil {
+ return nil
+ }
+
+ d.mu.Lock()
+ st := d.stateLocked(path)
+ st.lastWrite = time.Now()
+ d.mu.Unlock()
+
+ st.file.Lock()
+ defer st.file.Unlock()
+
+ d.mu.Lock()
+ dirty := st.dirty
+ d.mu.Unlock()
+ if dirty {
+ return nil
+ }
+
+ markerPath, err := d.markerPath(ctx, repo, path)
+ if err != nil {
+ return err
+ }
+ content, err := json.Marshal(marker{RepoRef: repo, BootID: d.bootID})
+ if err != nil {
+ return fmt.Errorf("encode the dirty marker: %w", err)
+ }
+ if err := d.writeDurably(markerPath, content); err != nil {
+ return fmt.Errorf("write the dirty marker: %w", err)
+ }
+
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ st.dirty = true
+ st.repo = repo
+ st.marker = markerPath
+ d.marked.Inc()
+ d.dirty.Inc()
+ return nil
+}
+
+// Begin records that the hook transaction key is past its Prepared phase on
+// the repository at path: git may write its table and compact the stack until
+// it reports Committed (End). It is called under git's reference lock.
+func (d *Durability) Begin(path string, key Key) {
+ if d == nil {
+ return
+ }
+
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ st := d.stateLocked(path)
+ st.inflight[key]++
+ st.nInflight++
+ d.inflight.Inc()
+}
+
+// End records that the hook transaction key committed. A transaction aborted
+// after its Prepared phase is not ended: two transactions with the same key
+// cannot be told apart, and ending the wrong one would let Settle remove the
+// marker while the other one still writes. Its repository stays dirty instead.
+func (d *Durability) End(path string, key Key) {
+ if d == nil {
+ return
+ }
+
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ st := d.repos[path]
+ if st == nil || st.inflight[key] == 0 {
+ return
+ }
+ st.inflight[key]--
+ if st.inflight[key] == 0 {
+ delete(st.inflight, key)
+ }
+ st.nInflight--
+ d.inflight.Dec()
+}
+
+// Dirty returns the paths of the dirty repositories.
+func (d *Durability) Dirty() []string {
+ if d == nil {
+ return nil
+ }
+
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ paths := make([]string, 0, len(d.repos))
+ for path, st := range d.repos {
+ if st.dirty {
+ paths = append(paths, path)
+ }
+ }
+ return paths
+}
+
+// Settle is the durable step of the repository at path. It makes the tables
+// that the stack lists, the list and the directory durable, which reports a
+// write the kernel failed to do (an *UntrustedError: the caller moves the
+// repository aside at once). When the repository has been idle for the idle
+// period and no hook transaction is in flight, it then removes the marker:
+// the decision is taken under git's reference lock, taken without waiting (a
+// busy lock leaves the marker for the next step), and the marker file is
+// removed after the lock is released, under the state's file lock that a
+// writer's MarkDirty waits for. A repository that no longer exists (moved
+// aside as a stale cache) loses its marker. The caller holds the syncer's
+// repository lock, so no syncer write and no replacement of the repository
+// runs meanwhile.
+func (d *Durability) Settle(path string) error {
+ if d == nil {
+ return nil
+ }
+
+ d.mu.Lock()
+ st := d.repos[path]
+ if st == nil || !st.dirty {
+ d.mu.Unlock()
+ return nil
+ }
+ repo := st.repo
+ idle := st.nInflight == 0 && time.Since(st.lastWrite) >= d.idle
+ d.mu.Unlock()
+
+ if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {
+ st.file.Lock()
+ defer st.file.Unlock()
+ d.clear(st)
+ return d.removeMarker(st)
+ }
+
+ if err := d.syncStack(path, st); err != nil {
+ return &UntrustedError{Repo: repo, Path: path, Err: err}
+ }
+ if !idle {
+ return nil
+ }
+
+ unlock, ok, err := lockStack(path)
+ if err != nil {
+ return err
+ }
+ if !ok {
+ return nil
+ }
+
+ cleared, err := d.settleLocked(path, st)
+ if err != nil {
+ _ = unlock()
+ return &UntrustedError{Repo: repo, Path: path, Err: err}
+ }
+ if !cleared {
+ return unlock()
+ }
+
+ // The state is clean from here on, and a writer that takes git's lock now
+ // writes a new marker: it waits for the file lock, held until the old
+ // marker is removed.
+ defer st.file.Unlock()
+ if err := unlock(); err != nil {
+ return err
+ }
+ return d.removeMarker(st)
+}
+
+// settleLocked makes the tables written since the first pass durable and, when
+// no hook transaction is in flight, marks the state clean with its file lock
+// held (the caller releases it). The caller holds git's reference lock: no
+// transaction can pass its Prepared phase meanwhile, and the transactions
+// counted before have committed when the count is zero.
+func (d *Durability) settleLocked(path string, st *refState) (bool, error) {
+ d.mu.Lock()
+ busy := st.nInflight > 0
+ d.mu.Unlock()
+ if busy {
+ return false, nil
+ }
+
+ if err := d.syncStack(path, st); err != nil {
+ return false, err
+ }
+ if d.crashAt == crashDurableStep {
+ os.Exit(crashExitCode)
+ }
+
+ st.file.Lock()
+ d.clear(st)
+ return true, nil
+}
+
+// clear marks the state clean. The caller holds the state's file lock.
+func (d *Durability) clear(st *refState) {
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ if st.dirty {
+ st.dirty = false
+ d.cleared.Inc()
+ d.dirty.Dec()
+ }
+ st.synced = map[string]struct{}{}
+}
+
+// removeMarker removes the marker file of a clean state. The caller holds the
+// state's file lock.
+func (d *Durability) removeMarker(st *refState) error {
+ if st.marker == "" {
+ return nil
+ }
+ if err := os.Remove(st.marker); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return fmt.Errorf("remove the dirty marker: %w", err)
+ }
+ return nil
+}
+
+// lockStack takes git's reference lock of the repository at path without
+// waiting: ok is false when it is held. The lock is created with its content
+// at once (a linked temporary file), so that a lock left by a process that
+// died is recognized at startup (removeStaleStackLock).
+func lockStack(path string) (unlock func() error, ok bool, err error) {
+ lockPath := filepath.Join(path, "reftable", "tables.list.lock")
+
+ tmp, err := os.CreateTemp(path, ".graft-durable-step-*")
+ if err != nil {
+ return nil, false, fmt.Errorf("take the reference lock: %w", err)
+ }
+ defer func() { _ = os.Remove(tmp.Name()) }()
+ if _, err := tmp.WriteString(stackLockContent); err != nil {
+ _ = tmp.Close()
+ return nil, false, fmt.Errorf("take the reference lock: %w", err)
+ }
+ if err := tmp.Close(); err != nil {
+ return nil, false, fmt.Errorf("take the reference lock: %w", err)
+ }
+
+ if err := os.Link(tmp.Name(), lockPath); err != nil {
+ if errors.Is(err, fs.ErrExist) {
+ return nil, false, nil
+ }
+ return nil, false, fmt.Errorf("take the reference lock: %w", err)
+ }
+ return func() error {
+ if err := os.Remove(lockPath); err != nil {
+ return fmt.Errorf("release the reference lock: %w", err)
+ }
+ return nil
+ }, true, nil
+}
+
+// removeStaleStackLock removes the reference lock of the repository at path
+// when a durable step of a process that died left it: git's writes would fail
+// on it until housekeeping removes stale locks. A lock of git is never
+// removed.
+func removeStaleStackLock(path string) error {
+ lockPath := filepath.Join(path, "reftable", "tables.list.lock")
+ content, err := os.ReadFile(lockPath)
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ if err != nil {
+ return fmt.Errorf("read the reference lock: %w", err)
+ }
+ if !bytes.Equal(content, []byte(stackLockContent)) {
+ return nil
+ }
+ if err := os.Remove(lockPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return fmt.Errorf("remove a stale reference lock: %w", err)
+ }
+ return nil
+}
+
+// syncStack makes durable the tables of the stack not made durable yet in this
+// dirty period, then the list and the directory.
+func (d *Durability) syncStack(path string, st *refState) error {
+ reftable := filepath.Join(path, "reftable")
+ names, err := readTablesList(filepath.Join(reftable, "tables.list"))
+ if err != nil {
+ return fmt.Errorf("read the table list: %w", err)
+ }
+
+ d.mu.Lock()
+ var pending []string
+ for _, name := range names {
+ if _, ok := st.synced[name]; !ok {
+ pending = append(pending, name)
+ }
+ }
+ d.mu.Unlock()
+
+ for _, name := range pending {
+ if err := d.syncPath(filepath.Join(reftable, name)); err != nil {
+ if errors.Is(err, fs.ErrNotExist) {
+ // Compacted away after the list was read: the next list names
+ // the table that replaced it.
+ continue
+ }
+ return err
+ }
+ d.mu.Lock()
+ st.synced[name] = struct{}{}
+ d.mu.Unlock()
+ }
+
+ for _, p := range []string{filepath.Join(reftable, "tables.list"), reftable} {
+ if err := d.syncPath(p); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// Forget drops the state and the marker of the repository at path, which the
+// caller moved aside under the syncer's repository lock. A writer that still
+// writes to the moved repository writes to a directory that is removed.
+func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) error {
+ if d == nil {
+ return nil
+ }
+
+ markerPath, err := d.markerPath(ctx, repo, path)
+ if err != nil {
+ return err
+ }
+
+ d.mu.Lock()
+ st := d.repos[path]
+ d.mu.Unlock()
+ if st != nil {
+ st.file.Lock()
+ defer st.file.Unlock()
+ }
+
+ if err := os.Remove(markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return fmt.Errorf("remove the dirty marker: %w", err)
+ }
+
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ if st != nil && d.repos[path] == st {
+ if st.dirty {
+ d.dirty.Dec()
+ }
+ d.inflight.Sub(float64(st.nInflight))
+ delete(d.repos, path)
+ }
+ return nil
+}
+
+// Recovered is a dirty repository found at startup.
+type Recovered struct {
+ Repo RepoRef
+ Path string
+ // Reboot reports that the marker is of another kernel boot: the repository
+ // must be moved aside before anything reads it.
+ Reboot bool
+}
+
+// Recover reads the dirty markers of the storages. A marker of the running
+// boot makes its repository dirty again, to be settled, and a reference lock
+// that a durable step of the previous process left is removed. It returns
+// every repository found; the caller moves aside the ones of another boot
+// before serving. A marker that cannot be read is an error: its repository is
+// unknown, and serving could read a torn stack.
+func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovered, error) {
+ if d == nil {
+ return nil, nil
+ }
+
+ var found []Recovered
+ for _, storageName := range storages {
+ dir, err := d.markerDir(ctx, storageName)
+ if err != nil {
+ return nil, err
+ }
+ entries, err := os.ReadDir(dir)
+ if errors.Is(err, fs.ErrNotExist) {
+ continue
+ }
+ if err != nil {
+ return nil, fmt.Errorf("read the dirty markers of storage %q: %w", storageName, err)
+ }
+
+ for _, entry := range entries {
+ markerPath := filepath.Join(dir, entry.Name())
+ if !strings.HasSuffix(entry.Name(), ".json") {
+ // A temporary file of a marker write that did not complete: the
+ // write never returned, so no reference write relied on it.
+ if err := os.Remove(markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return nil, fmt.Errorf("remove an incomplete dirty marker: %w", err)
+ }
+ continue
+ }
+
+ content, err := os.ReadFile(markerPath)
+ if err != nil {
+ return nil, fmt.Errorf("read the dirty marker %q: %w", markerPath, err)
+ }
+ var m marker
+ if err := json.Unmarshal(content, &m); err != nil || m.Storage != storageName || m.RelativePath == "" || m.BootID == "" {
+ return nil, fmt.Errorf("corrupt dirty marker %q: the repository it names cannot be trusted and is unknown", markerPath)
+ }
+
+ path, err := d.locator.GetRepoPath(ctx, repoOf(m.RepoRef), storage.WithRepositoryVerificationSkipped())
+ if err != nil {
+ return nil, fmt.Errorf("dirty marker %q: %w", markerPath, err)
+ }
+
+ reboot := m.BootID != d.bootID
+ if !reboot {
+ if err := removeStaleStackLock(path); err != nil {
+ return nil, err
+ }
+ d.mu.Lock()
+ st := d.stateLocked(path)
+ if !st.dirty {
+ st.dirty = true
+ st.repo = m.RepoRef
+ st.marker = markerPath
+ d.dirty.Inc()
+ }
+ d.mu.Unlock()
+ }
+ found = append(found, Recovered{Repo: m.RepoRef, Path: path, Reboot: reboot})
+ }
+ }
+ return found, nil
+}
+
+// CountUntrusted counts a repository moved aside for reason.
+func (d *Durability) CountUntrusted(reason string) {
+ if d == nil {
+ return
+ }
+ d.untrusted.WithLabelValues(reason).Inc()
+}
+
+// writeDurably writes content to path through a temporary file in the same
+// directory, made durable before and after the rename.
+func (d *Durability) writeDurably(path string, content []byte) (returnedErr error) {
+ dir := filepath.Dir(path)
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ return err
+ }
+
+ tmp, err := os.CreateTemp(dir, ".marker-*.tmp")
+ if err != nil {
+ return err
+ }
+ defer func() {
+ if returnedErr != nil {
+ _ = tmp.Close()
+ _ = os.Remove(tmp.Name())
+ }
+ }()
+
+ if _, err := tmp.Write(content); err != nil {
+ return err
+ }
+ if err := d.sync(tmp); err != nil {
+ return err
+ }
+ if err := tmp.Close(); err != nil {
+ return err
+ }
+ if err := os.Rename(tmp.Name(), path); err != nil {
+ return err
+ }
+ return d.syncPath(dir)
+}
+
+// syncPath makes the file or directory at path durable.
+func (d *Durability) syncPath(path string) error {
+ f, err := os.Open(path)
+ if err != nil {
+ return err
+ }
+ defer f.Close()
+ return d.sync(f)
+}
+
+// readTablesList returns the table names of a reftable stack.
+func readTablesList(path string) ([]string, error) {
+ f, err := os.Open(path)
+ if err != nil {
+ return nil, err
+ }
+ defer f.Close()
+
+ var names []string
+ scanner := bufio.NewScanner(f)
+ for scanner.Scan() {
+ if name := strings.TrimSpace(scanner.Text()); name != "" {
+ names = append(names, name)
+ }
+ }
+ return names, scanner.Err()
+}
+
+// Describe implements prometheus.Collector.
+func (d *Durability) Describe(descs chan<- *prometheus.Desc) {
+ d.marked.Describe(descs)
+ d.cleared.Describe(descs)
+ d.untrusted.Describe(descs)
+ d.dirty.Describe(descs)
+ d.inflight.Describe(descs)
+}
+
+// Collect implements prometheus.Collector.
+func (d *Durability) Collect(metrics chan<- prometheus.Metric) {
+ d.marked.Collect(metrics)
+ d.cleared.Collect(metrics)
+ d.untrusted.Collect(metrics)
+ d.dirty.Collect(metrics)
+ d.inflight.Collect(metrics)
+}
diff --git a/internal/graft/corestore/localrefs/durability_test.go b/internal/graft/corestore/localrefs/durability_test.go
new file mode 100644
index 0000000..09a2b6f
--- /dev/null
+++ b/internal/graft/corestore/localrefs/durability_test.go
@@ -0,0 +1,365 @@
+package localrefs
+
+import (
+ "context"
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/prometheus/client_golang/prometheus/testutil"
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/internal/gitaly/storage"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+)
+
+// durabilityFixture is a reftable repository with a few references, written
+// one transaction each so that its stack has several tables.
+type durabilityFixture struct {
+ ctx context.Context
+ cfg config.Cfg
+ locator storage.Locator
+ repo RepoRef
+ path string
+ commit git.ObjectID
+ synced *syncRecorder
+}
+
+// syncRecorder records the paths made durable, and fails them while failWith
+// is set.
+type syncRecorder struct {
+ mu sync.Mutex
+ paths []string
+ failWith error
+}
+
+func (s *syncRecorder) sync(f *os.File) error {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ if s.failWith != nil {
+ return s.failWith
+ }
+ s.paths = append(s.paths, f.Name())
+ return f.Sync()
+}
+
+func (s *syncRecorder) recorded() []string {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return append([]string(nil), s.paths...)
+}
+
+func (s *syncRecorder) fail(err error) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ s.failWith = err
+}
+
+func setupDurability(t *testing.T) durabilityFixture {
+ t.Helper()
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+ pb, path := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{
+ SkipCreationViaService: true,
+ ReferenceBackend: git.ReferenceBackendReftables,
+ })
+ commit := gittest.WriteCommit(t, cfg, path)
+ for _, name := range []string{"refs/heads/main", "refs/heads/a", "refs/heads/b"} {
+ gittest.WriteRef(t, cfg, path, git.ReferenceName(name), commit)
+ }
+ return durabilityFixture{
+ ctx: ctx,
+ cfg: cfg,
+ locator: config.NewLocator(cfg),
+ repo: RepoRef{Storage: pb.GetStorageName(), RelativePath: pb.GetRelativePath()},
+ path: path,
+ commit: commit,
+ synced: &syncRecorder{},
+ }
+}
+
+func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration) *Durability {
+ t.Helper()
+ d, err := NewDurability(f.locator, idle, WithBootID(bootID), WithSync(f.synced.sync))
+ require.NoError(t, err)
+ return d
+}
+
+func (f durabilityFixture) markerPath(t *testing.T, d *Durability) string {
+ t.Helper()
+ p, err := d.markerPath(f.ctx, f.repo, f.path)
+ require.NoError(t, err)
+ return p
+}
+
+func (f durabilityFixture) tables(t *testing.T) []string {
+ t.Helper()
+ names, err := readTablesList(filepath.Join(f.path, "reftable", "tables.list"))
+ require.NoError(t, err)
+ require.NotEmpty(t, names)
+ return names
+}
+
+func TestMarkDirtyWritesOneDurableMarker(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+
+ content, err := os.ReadFile(f.markerPath(t, d))
+ require.NoError(t, err)
+ require.JSONEq(t, `{"storage":"`+f.repo.Storage+`","relative_path":"`+f.repo.RelativePath+`","boot_id":"boot-a"}`, string(content))
+ require.Equal(t, 1.0, testutil.ToFloat64(d.marked))
+ require.Equal(t, []string{f.path}, d.Dirty())
+
+ // The marker file and its directory were made durable before MarkDirty
+ // returned.
+ recorded := f.synced.recorded()
+ require.Len(t, recorded, 2)
+ require.Equal(t, filepath.Dir(f.markerPath(t, d)), recorded[1])
+ require.Equal(t, filepath.Join(f.cfg.Storages[0].Path, MarkerDirName), filepath.Dir(f.markerPath(t, d)))
+}
+
+func TestSettleMakesTheStackDurableAndRemovesTheMarker(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, d.Settle(f.path))
+
+ require.NoFileExists(t, f.markerPath(t, d))
+ require.Empty(t, d.Dirty())
+ require.Equal(t, 1.0, testutil.ToFloat64(d.cleared))
+ require.Equal(t, 0.0, testutil.ToFloat64(d.dirty))
+
+ recorded := f.synced.recorded()
+ for _, table := range f.tables(t) {
+ require.Contains(t, recorded, filepath.Join(f.path, "reftable", table))
+ }
+ require.Contains(t, recorded, filepath.Join(f.path, "reftable", "tables.list"))
+ require.Contains(t, recorded, filepath.Join(f.path, "reftable"))
+ // The lock taken for the removal is released.
+ require.NoFileExists(t, filepath.Join(f.path, "reftable", "tables.list.lock"))
+}
+
+func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {
+ t.Parallel()
+
+ t.Run("not idle", func(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", time.Hour)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, d.Settle(f.path))
+ require.FileExists(t, f.markerPath(t, d))
+ // The stack was made durable all the same.
+ require.Contains(t, f.synced.recorded(), filepath.Join(f.path, "reftable", f.tables(t)[0]))
+ })
+
+ t.Run("a transaction between Prepared and Committed", func(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ d.Begin(f.path, key(1))
+ require.NoError(t, d.Settle(f.path))
+ require.FileExists(t, f.markerPath(t, d))
+
+ // An end of another key does not count; the end of the transaction does.
+ d.End(f.path, key(2))
+ require.NoError(t, d.Settle(f.path))
+ require.FileExists(t, f.markerPath(t, d))
+
+ d.End(f.path, key(1))
+ require.NoError(t, d.Settle(f.path))
+ require.NoFileExists(t, f.markerPath(t, d))
+ require.Equal(t, 0.0, testutil.ToFloat64(d.inflight))
+ })
+
+ t.Run("git holds the reference lock", func(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ lock := filepath.Join(f.path, "reftable", "tables.list.lock")
+ require.NoError(t, os.WriteFile(lock, nil, 0o666))
+
+ require.NoError(t, d.Settle(f.path))
+ require.FileExists(t, f.markerPath(t, d))
+ require.FileExists(t, lock, "the durable step removed a lock it did not take")
+ })
+}
+
+func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ writeback := errors.New("input/output error")
+ f.synced.fail(writeback)
+
+ err := d.Settle(f.path)
+ require.ErrorIs(t, err, ErrUntrusted)
+ require.ErrorIs(t, err, writeback)
+ var untrusted *UntrustedError
+ require.ErrorAs(t, err, &untrusted)
+ require.Equal(t, f.repo, untrusted.Repo)
+ require.Equal(t, f.path, untrusted.Path)
+
+ // The repository stays dirty until the caller moves it aside.
+ require.FileExists(t, f.markerPath(t, d))
+ require.Equal(t, []string{f.path}, d.Dirty())
+
+ require.NoError(t, d.Forget(f.ctx, f.repo, f.path))
+ require.NoFileExists(t, f.markerPath(t, d))
+ require.Empty(t, d.Dirty())
+ require.Equal(t, 0.0, testutil.ToFloat64(d.dirty))
+}
+
+func TestSettleDropsTheMarkerOfARepositoryThatIsGone(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, os.Rename(f.path, f.path+".moved"))
+ t.Cleanup(func() { require.NoError(t, os.Rename(f.path+".moved", f.path)) })
+
+ require.NoError(t, d.Settle(f.path))
+ require.NoFileExists(t, f.markerPath(t, d))
+ require.Empty(t, d.Dirty())
+}
+
+func TestRecoverTellsAKernelRestartFromAProcessRestart(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ before := f.durability(t, "boot-a", time.Hour)
+ require.NoError(t, before.MarkDirty(f.ctx, f.repo, f.path))
+ storages := []string{f.repo.Storage}
+
+ // The same boot: the process restarted, the kernel kept every write.
+ sameBoot := f.durability(t, "boot-a", 0)
+ found, err := sameBoot.Recover(f.ctx, storages)
+ require.NoError(t, err)
+ require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path, Reboot: false}}, found)
+ require.Equal(t, []string{f.path}, sameBoot.Dirty())
+ require.NoError(t, sameBoot.Settle(f.path))
+ require.NoFileExists(t, f.markerPath(t, sameBoot))
+
+ // Another boot: the writes may not have reached the disk.
+ require.NoError(t, f.durability(t, "boot-a", time.Hour).MarkDirty(f.ctx, f.repo, f.path))
+ otherBoot := f.durability(t, "boot-b", 0)
+ found, err = otherBoot.Recover(f.ctx, storages)
+ require.NoError(t, err)
+ require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path, Reboot: true}}, found)
+ require.Empty(t, otherBoot.Dirty())
+}
+
+// TestRecoverRemovesTheLockOfADurableStepThatDied verifies that a reference
+// lock left by a durable step of a process that died is removed at startup,
+// and that a lock of git is not.
+func TestRecoverRemovesTheLockOfADurableStepThatDied(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ lock := filepath.Join(f.path, "reftable", "tables.list.lock")
+
+ require.NoError(t, f.durability(t, "boot-a", 0).MarkDirty(f.ctx, f.repo, f.path))
+ unlock, ok, err := lockStack(f.path)
+ require.NoError(t, err)
+ require.True(t, ok)
+ _, ok, err = lockStack(f.path)
+ require.NoError(t, err)
+ require.False(t, ok, "the lock is exclusive")
+ entries, err := os.ReadDir(f.path)
+ require.NoError(t, err)
+ for _, entry := range entries {
+ require.False(t, strings.HasPrefix(entry.Name(), ".graft-durable-step-"), "temporary file %s left", entry.Name())
+ }
+
+ // The process dies holding the lock; the next one of the same boot removes it.
+ _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, []string{f.repo.Storage})
+ require.NoError(t, err)
+ require.NoFileExists(t, lock)
+ require.Error(t, unlock(), "the lock was already removed")
+
+ // A lock of git is kept.
+ require.NoError(t, os.WriteFile(lock, []byte("0x000000000001-0x000000000003-abcdef01.ref\n"), 0o666))
+ _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, []string{f.repo.Storage})
+ require.NoError(t, err)
+ require.FileExists(t, lock)
+}
+
+func TestRecoverRefusesACorruptMarker(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+
+ marker := f.markerPath(t, d)
+ incomplete := filepath.Join(filepath.Dir(marker), ".marker-123.tmp")
+ require.NoError(t, os.WriteFile(incomplete, []byte("{"), 0o600))
+
+ found, err := f.durability(t, "boot-b", 0).Recover(f.ctx, []string{f.repo.Storage})
+ require.NoError(t, err)
+ require.Len(t, found, 1)
+ require.NoFileExists(t, incomplete, "an incomplete marker write is removed")
+
+ require.NoError(t, os.WriteFile(marker, []byte(`{"storage":"`), 0o600))
+ _, err = f.durability(t, "boot-b", 0).Recover(f.ctx, []string{f.repo.Storage})
+ require.ErrorContains(t, err, "corrupt dirty marker")
+}
+
+func TestNilDurabilityIsOff(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+
+ var d *Durability
+ require.False(t, d.Enabled())
+ require.NoError(t, d.MarkDirty(ctx, RepoRef{}, "path"))
+ require.NoError(t, d.Settle("path"))
+ d.Begin("path", key(1))
+ d.End("path", key(1))
+ found, err := d.Recover(ctx, []string{"default"})
+ require.NoError(t, err)
+ require.Empty(t, found)
+}
+
+// TestTornReftableStackReadsAsAnEmptyRepository is the experiment behind the
+// LOCAL REFERENCE WRITES RULE (doc/graft/PLAN7.md): a reftable stack that a
+// crash tore (a table whose data never reached the disk) is read by git as a
+// repository without references, without an error. Only writes fail. So a torn
+// stack must never be served: nothing in git would tell.
+func TestTornReftableStackReadsAsAnEmptyRepository(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+
+ refs := gittest.Exec(t, f.cfg, "-C", f.path, "for-each-ref", "--format=%(refname)")
+ require.Equal(t, "refs/heads/a\nrefs/heads/b\nrefs/heads/main", strings.TrimSpace(string(refs)))
+
+ tables := f.tables(t)
+ newest := filepath.Join(f.path, "reftable", tables[len(tables)-1])
+ require.NoError(t, os.Truncate(newest, 0))
+
+ refs = gittest.Exec(t, f.cfg, "-C", f.path, "for-each-ref", "--format=%(refname)")
+ require.Empty(t, strings.TrimSpace(string(refs)), "git reports the references of a torn stack")
+
+ cmd := gittest.NewCommand(t, f.cfg, "-C", f.path, "update-ref", "refs/heads/new", f.commit.String())
+ output, err := cmd.CombinedOutput()
+ require.Error(t, err)
+ require.Contains(t, string(output), "corrupt reftable file")
+}
diff --git a/internal/graft/corestore/syncer/durability.go b/internal/graft/corestore/syncer/durability.go
new file mode 100644
index 0000000..32c5be5
--- /dev/null
+++ b/internal/graft/corestore/syncer/durability.go
@@ -0,0 +1,200 @@
+package syncer
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+ "time"
+
+ "github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/gitops"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
+ "github.com/feder-cr/git-ai/internal/structerr"
+)
+
+// RecoverLocalReferences moves aside every local repository that had reference
+// writes that were not durable yet when the kernel restarted (doc/graft/PLAN7.md,
+// LOCAL REFERENCE WRITES RULE): its reftable stack may be torn, and git reads a
+// torn stack as an empty repository without an error. The next request
+// hydrates it from the database and the bucket. gitaly serve calls it before
+// its listeners open, so nothing reads such a repository before. A repository
+// dirty from a process restart of the same boot is kept: the kernel kept every
+// write, and the durable step settles it.
+//
+// Only a cache of the storage core is moved: the database must hold the
+// repository with changesets, and the local repository must have its
+// watermark. Any other marker (the repository was removed, or replaced by one
+// being imported, whose local copy may be the only one) is dropped.
+func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string) error {
+ d := s.core.RefDurability
+ found, err := d.Recover(ctx, storages)
+ if err != nil {
+ return fmt.Errorf("storage core: recover the local references: %w", err)
+ }
+
+ for _, rec := range found {
+ if !rec.Reboot {
+ continue
+ }
+
+ cache, err := s.storageCoreCache(ctx, rec)
+ if err != nil {
+ return fmt.Errorf("storage core: recover the local references of %s:%s: %w", rec.Repo.Storage, rec.Repo.RelativePath, err)
+ }
+ if !cache {
+ s.logger.WithField("repository_path", rec.Path).
+ Warn("storage core: dropped the dirty marker of a local repository that is not a cache of the storage core")
+ if err := d.Forget(ctx, rec.Repo, rec.Path); err != nil {
+ return err
+ }
+ continue
+ }
+
+ if err := s.distrust(ctx, rec.Repo, rec.Path, localrefs.ReasonReboot, nil); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// storageCoreCache reports whether the local repository of rec is a cache of
+// the storage core that a hydration can rebuild: the database holds it with
+// changesets, and it has a watermark.
+func (s *Syncer) storageCoreCache(ctx context.Context, rec localrefs.Recovered) (bool, error) {
+ r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: rec.Repo.Storage, RelativePath: rec.Repo.RelativePath})
+ if errors.Is(err, corestore.ErrRepositoryNotFound) {
+ return false, nil
+ }
+ if err != nil {
+ return false, err
+ }
+ if r.HeadSeq == 0 {
+ return false, nil
+ }
+
+ if _, err := os.Lstat(rec.Path); errors.Is(err, os.ErrNotExist) {
+ return false, nil
+ } else if err != nil {
+ return false, err
+ }
+ _, ok, err := gitops.ReadWatermark(rec.Path)
+ if err != nil {
+ // A watermark that cannot be read is no proof that the storage core wrote
+ // the repository: it is kept, as untracked() keeps it.
+ return false, nil
+ }
+ return ok, nil
+}
+
+// RunDurableSteps runs the durable step of every dirty repository each interval
+// until ctx ends (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).
+func (s *Syncer) RunDurableSteps(ctx context.Context, interval time.Duration) {
+ if !s.core.RefDurability.Enabled() {
+ return
+ }
+
+ ticker := time.NewTicker(interval)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ }
+ for _, path := range s.core.RefDurability.Dirty() {
+ s.durableStep(ctx, path)
+ }
+ }
+}
+
+// durableStep makes the references of the repository at path durable under the
+// repository lock, and moves the repository aside at once when the kernel
+// reports that a write failed.
+func (s *Syncer) durableStep(ctx context.Context, path string) {
+ release, err := s.lock(ctx, path)
+ if err != nil {
+ return
+ }
+ defer release()
+
+ err = s.core.RefDurability.Settle(path)
+ var untrusted *localrefs.UntrustedError
+ if errors.As(err, &untrusted) {
+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {
+ s.logger.WithError(err).WithField("repository_path", path).
+ Error("storage core: could not move aside a repository whose references may have lost writes")
+ }
+ return
+ }
+ if err != nil {
+ s.logger.WithError(err).WithField("repository_path", path).Error("storage core: durable step of the local references")
+ }
+}
+
+// distrust is distrustLocked under the repository lock.
+func (s *Syncer) distrust(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {
+ release, err := s.lock(ctx, path)
+ if err != nil {
+ return err
+ }
+ defer release()
+ return s.distrustLocked(ctx, repo, path, reason, cause)
+}
+
+// distrustLocked moves the local repository at path into the temporary
+// directory of its storage and removes it there in the background, then drops
+// its dirty marker: the next request hydrates it. The move is made durable
+// before the marker goes, so a crash in between moves it again. The caller
+// holds the repository lock.
+func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {
+ d := s.core.RefDurability
+
+ if _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) {
+ return d.Forget(ctx, repo, path)
+ } else if err != nil {
+ return structerr.NewInternal("storage core: stat repository: %w", err)
+ }
+
+ tmpDir, err := s.tmpDir(repo.Storage)
+ if err != nil {
+ return err
+ }
+ stale, err := os.MkdirTemp(tmpDir, tmpStalePattern)
+ if err != nil {
+ return structerr.NewInternal("storage core: create stale directory: %w", err)
+ }
+ if err := os.Rename(path, filepath.Join(stale, "repo.git")); err != nil {
+ _ = os.Remove(stale)
+ return structerr.NewInternal("storage core: move aside an untrusted repository: %w", err)
+ }
+ if err := syncDir(filepath.Dir(path)); err != nil {
+ return structerr.NewInternal("storage core: make the move of an untrusted repository durable: %w", err)
+ }
+
+ s.forgetPending(path)
+ if err := d.Forget(ctx, repo, path); err != nil {
+ return err
+ }
+ d.CountUntrusted(reason)
+
+ logger := s.logger.WithField("repository_path", path).WithField("stale_path", stale).WithField("reason", reason)
+ if cause != nil {
+ logger = logger.WithError(cause)
+ }
+ logger.Error("storage core: moved aside a local repository whose references may have lost writes, the next request hydrates it")
+
+ go func() { _ = os.RemoveAll(stale) }()
+ return nil
+}
+
+// syncDir makes the entries of the directory at path durable.
+func syncDir(path string) error {
+ f, err := os.Open(path)
+ if err != nil {
+ return err
+ }
+ defer f.Close()
+ return f.Sync()
+}
diff --git a/internal/graft/corestore/syncer/export_test.go b/internal/graft/corestore/syncer/export_test.go
index b9b2bfc..162b6c8 100644
--- a/internal/graft/corestore/syncer/export_test.go
+++ b/internal/graft/corestore/syncer/export_test.go
@@ -1,7 +1,15 @@
package syncer
+import "context"
+
// SetTestHook installs a function called at the named points of a catch-up
// ("fetched", "checked"). Only for tests; set it before the first EnsureFresh.
func (s *Syncer) SetTestHook(hook func(point string)) {
s.testHook = hook
}
+
+// DurableStep runs one durable step of the repository at path, as
+// RunDurableSteps does every interval.
+func (s *Syncer) DurableStep(ctx context.Context, path string) {
+ s.durableStep(ctx, path)
+}
diff --git a/internal/graft/corestore/syncer/syncer.go b/internal/graft/corestore/syncer/syncer.go
index 3c91015..429f47c 100644
--- a/internal/graft/corestore/syncer/syncer.go
+++ b/internal/graft/corestore/syncer/syncer.go
@@ -66,6 +66,7 @@ import (
"github.com/feder-cr/git-ai/internal/graft/changelog"
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/gitops"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
reviewstore "github.com/feder-cr/git-ai/internal/graft/review/store"
"github.com/feder-cr/git-ai/internal/log"
"github.com/feder-cr/git-ai/internal/structerr"
@@ -1161,7 +1162,8 @@ func (s *Syncer) applyLocked(ctx context.Context, repo *localrepo.Repo, hash git
return applyOutcome{done: true}, nil
}
- if err := applyMoves(ctx, repo, hash, moves); err != nil {
+ w := refWriter{core: s.core, repo: localrefs.RepoRef{Storage: r.Key.Storage, RelativePath: r.Key.RelativePath}, path: path}
+ if err := applyMoves(ctx, repo, w, hash, moves); err != nil {
var txErr *refTxError
if errors.As(err, &txErr) && txErr.locked {
return applyOutcome{lockErr: err}, nil
@@ -1225,7 +1227,7 @@ func movedSince(ctx context.Context, repo *localrepo.Repo, moves map[string]move
//
// The references other than HEAD are moved first, HEAD alone afterwards: git
// refuses one transaction that updates HEAD and the reference it points to.
-func applyMoves(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash, moves map[string]move) error {
+func applyMoves(ctx context.Context, repo *localrepo.Repo, w refWriter, hash git.ObjectHash, moves map[string]move) error {
var names []string
moveHead := false
for name, m := range moves {
@@ -1246,12 +1248,12 @@ func applyMoves(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash,
sort.Strings(names)
if len(names) > 0 {
- if err := applyChecked(ctx, repo, hash, names, moves); err != nil {
+ if err := applyChecked(ctx, repo, w, hash, names, moves); err != nil {
return err
}
}
if moveHead {
- return applyChecked(ctx, repo, hash, []string{"HEAD"}, moves)
+ return applyChecked(ctx, repo, w, hash, []string{"HEAD"}, moves)
}
return nil
}
@@ -1264,7 +1266,7 @@ func applyMoves(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash,
// removed and a created name (refs/a and refs/a/b). Between the two
// transactions a removed reference is absent; the second transaction requires
// that, so a local write in between still fails it.
-func applyChecked(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash, names []string, moves map[string]move) error {
+func applyChecked(ctx context.Context, repo *localrepo.Repo, w refWriter, hash git.ObjectHash, names []string, moves map[string]move) error {
var removals, others []string
twoSteps := false
for _, name := range names {
@@ -1281,17 +1283,28 @@ func applyChecked(ctx context.Context, repo *localrepo.Repo, hash git.ObjectHash
}
if !twoSteps {
- err := refTransaction(ctx, repo, append(append([]string{}, removals...), others...))
+ err := refTransaction(ctx, repo, w, append(append([]string{}, removals...), others...))
var txErr *refTxError
if err == nil || !errors.As(err, &txErr) || !txErr.fdConflict || len(removals) == 0 || len(others) == 0 {
return err
}
}
- if err := refTransaction(ctx, repo, removals); err != nil {
+ if err := refTransaction(ctx, repo, w, removals); err != nil {
return fmt.Errorf("apply removals: %w", err)
}
- return refTransaction(ctx, repo, others)
+ return refTransaction(ctx, repo, w, others)
+}
+
+// refWriter runs the reference transactions of the catch-up on the local
+// repository at path, under the syncer's repository lock: admitted to git's
+// reference lock like the hook's writers, and without fsync when the node
+// defers the durability of its reference writes (doc/graft/PLAN7.md, LOCAL
+// REFERENCE WRITES RULE).
+type refWriter struct {
+ core *corestore.Core
+ repo localrefs.RepoRef
+ path string
}
// nul terminates the fields of update-ref -z.
@@ -1353,8 +1366,10 @@ func (e *refTxError) Unwrap() error { return e.err }
// refTransaction runs the commands (casSteps) as one transaction of
// `git update-ref -z --stdin --no-deref`, without hooks (the database has the
// changes already: no reference-transaction hook may run for them). An empty
-// command list does nothing.
-func refTransaction(ctx context.Context, repo *localrepo.Repo, commands []string) error {
+// command list does nothing. With deferred durability the repository gets its
+// dirty marker before git writes; the caller holds the syncer's repository
+// lock, which the durable step holds too.
+func refTransaction(ctx context.Context, repo *localrepo.Repo, w refWriter, commands []string) error {
if len(commands) == 0 {
return nil
}
@@ -1364,6 +1379,17 @@ func refTransaction(ctx context.Context, repo *localrepo.Repo, commands []string
return fmt.Errorf("reference backend: %w", err)
}
+ var durability []gitcmd.CmdOpt
+ if w.core.RefDurability.Enabled() {
+ if err := w.core.RefDurability.MarkDirty(ctx, w.repo, w.path); err != nil {
+ return err
+ }
+ durability = append(durability, gitcmd.WithConfig(gitcmd.ConfigPair{Key: "core.fsync", Value: gitcmd.FsyncComponentsWithoutReferences}))
+ }
+ if release := w.core.RefAdmission.AcquireUnique(ctx, w.path, localrefs.WriterSyncer); release != nil {
+ defer release()
+ }
+
var stdin bytes.Buffer
stdin.WriteString("start" + nul)
for _, c := range commands {
@@ -1381,10 +1407,12 @@ func refTransaction(ctx context.Context, repo *localrepo.Repo, commands []string
gitcmd.Flag{Name: "--no-deref"},
},
},
- gitcmd.WithDisabledHooks(),
- gitcmd.WithStdin(&stdin),
- gitcmd.WithStdout(io.Discard),
- gitcmd.WithStderr(&stderr),
+ append([]gitcmd.CmdOpt{
+ gitcmd.WithDisabledHooks(),
+ gitcmd.WithStdin(&stdin),
+ gitcmd.WithStdout(io.Discard),
+ gitcmd.WithStderr(&stderr),
+ }, durability...)...,
)
if err == nil {
return nil
diff --git a/internal/graft/corestore/syncer/syncer_localrefs_test.go b/internal/graft/corestore/syncer/syncer_localrefs_test.go
new file mode 100644
index 0000000..0e378ff
--- /dev/null
+++ b/internal/graft/corestore/syncer/syncer_localrefs_test.go
@@ -0,0 +1,209 @@
+package syncer_test
+
+import (
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "sync/atomic"
+ "testing"
+
+ "github.com/prometheus/client_golang/prometheus"
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/coretest"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/syncer"
+)
+
+// markers returns the dirty markers of the node's storage.
+func (e *env) markers(t *testing.T) []string {
+ t.Helper()
+
+ entries, err := os.ReadDir(filepath.Join(e.storagePath, localrefs.MarkerDirName))
+ if os.IsNotExist(err) {
+ return nil
+ }
+ require.NoError(t, err)
+ var names []string
+ for _, entry := range entries {
+ names = append(names, entry.Name())
+ }
+ return names
+}
+
+// untrusted returns graft_storage_ref_untrusted_total{reason} of the node.
+func (e *env) untrusted(t *testing.T, reason string) float64 {
+ t.Helper()
+
+ reg := prometheus.NewPedanticRegistry()
+ require.NoError(t, reg.Register(e.core.RefDurability))
+ families, err := reg.Gather()
+ require.NoError(t, err)
+ for _, family := range families {
+ if family.GetName() != "graft_storage_ref_untrusted_total" {
+ continue
+ }
+ for _, metric := range family.GetMetric() {
+ for _, label := range metric.GetLabel() {
+ if label.GetName() == "reason" && label.GetValue() == reason {
+ return metric.GetCounter().GetValue()
+ }
+ }
+ }
+ }
+ return 0
+}
+
+// tearNewestTable truncates the newest table of the local repository's stack:
+// what a kernel restart can leave of a table written without fsync.
+func (e *env) tearNewestTable(t *testing.T) {
+ t.Helper()
+
+ list, err := os.ReadFile(filepath.Join(e.localPath(), "reftable", "tables.list"))
+ require.NoError(t, err)
+ names := strings.Fields(string(list))
+ require.NotEmpty(t, names)
+ require.NoError(t, os.Truncate(filepath.Join(e.localPath(), "reftable", names[len(names)-1]), 0))
+}
+
+// caughtUpDirty returns a node whose local repository was hydrated, then caught
+// up with one changeset: its references were written without fsync, so the
+// repository is dirty.
+func caughtUpDirty(t *testing.T, opts ...localrefs.DurabilityOption) (*env, corestore.Repository, git.ObjectID) {
+ t.Helper()
+
+ e := newEnv(t, envOptions{})
+ coretest.EnableLocalRefs(t, e.core, e.locator, opts...)
+ src := e.newSource(t)
+ c1 := e.writeCommit(t, src)
+ repo := e.register(t, e.key(localRelPath), src, c1)
+
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ require.Empty(t, e.markers(t), "a hydration writes its references durably")
+
+ c2 := e.advanceMain(t, repo.ID, src, c1)
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ require.Equal(t, float64(1), syncCount(t, e.sy, "caught_up"))
+ require.Len(t, e.markers(t), 1, "the catch-up marks the repository before writing without fsync")
+
+ repo, err := e.core.Meta.GetRepository(e.ctx, repo.Key)
+ require.NoError(t, err)
+ return e, repo, c2
+}
+
+func TestSyncerCatchUpMarksTheRepositoryDirty(t *testing.T) {
+ t.Parallel()
+
+ e, repo, _ := caughtUpDirty(t)
+ e.verifySynced(t, repo)
+ require.Equal(t, []string{e.localPath()}, e.core.RefDurability.Dirty())
+
+ e.sy.DurableStep(e.ctx, e.localPath())
+ require.Empty(t, e.markers(t))
+ require.Empty(t, e.core.RefDurability.Dirty())
+ e.verifySynced(t, repo)
+}
+
+// TestSyncerMovesAsideARepositoryDirtyAtAKernelRestart is the recovery of the
+// LOCAL REFERENCE WRITES RULE: a repository with references written without
+// fsync when the kernel restarted is moved aside before anything reads it, and
+// hydrated again. Without the recovery the torn stack reads as an empty
+// repository, without an error.
+func TestSyncerMovesAsideARepositoryDirtyAtAKernelRestart(t *testing.T) {
+ t.Parallel()
+
+ e, repo, _ := caughtUpDirty(t, localrefs.WithBootID("boot-a"))
+ e.tearNewestTable(t)
+
+ // A read before the recovery returns the wrong state, and nothing reports
+ // an error: git lists no references.
+ torn := gittest.Exec(t, e.cfg, "-C", e.localPath(), "for-each-ref", "--format=%(refname)")
+ require.Empty(t, strings.TrimSpace(string(torn)))
+
+ // The node restarts after a kernel restart.
+ coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-b"))
+ e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)
+ require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))
+
+ require.NoDirExists(t, e.localPath())
+ require.Empty(t, e.markers(t))
+ require.Equal(t, float64(1), e.untrusted(t, localrefs.ReasonReboot))
+
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ require.Equal(t, float64(1), syncCount(t, e.sy, "hydrated"), "the restarted node hydrates the repository moved aside")
+ e.verifySynced(t, repo)
+}
+
+// TestSyncerKeepsARepositoryDirtyAtAProcessRestart verifies that a restart of
+// the process alone keeps the repository: the kernel kept every write.
+func TestSyncerKeepsARepositoryDirtyAtAProcessRestart(t *testing.T) {
+ t.Parallel()
+
+ e, repo, _ := caughtUpDirty(t, localrefs.WithBootID("boot-a"))
+
+ coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-a"))
+ e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)
+ require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))
+
+ require.DirExists(t, e.localPath())
+ require.Equal(t, []string{e.localPath()}, e.core.RefDurability.Dirty())
+ e.verifySynced(t, repo)
+
+ e.sy.DurableStep(e.ctx, e.localPath())
+ require.Empty(t, e.markers(t))
+}
+
+// TestSyncerRecoveryKeepsWhatIsNotACache verifies that the recovery never moves
+// a local repository that a hydration cannot rebuild: one the database does
+// not hold. Its marker is dropped.
+func TestSyncerRecoveryKeepsWhatIsNotACache(t *testing.T) {
+ t.Parallel()
+
+ e := newEnv(t, envOptions{})
+ coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-a"))
+ pb, path := gittest.CreateRepository(t, e.ctx, e.cfg, gittest.CreateRepositoryConfig{
+ SkipCreationViaService: true,
+ ReferenceBackend: git.ReferenceBackendReftables,
+ })
+ require.NoError(t, e.core.RefDurability.MarkDirty(e.ctx, localrefs.RepoRef{Storage: pb.GetStorageName(), RelativePath: pb.GetRelativePath()}, path))
+
+ coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-b"))
+ e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)
+ require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))
+
+ require.DirExists(t, path)
+ require.Empty(t, e.markers(t))
+ require.Equal(t, float64(0), e.untrusted(t, localrefs.ReasonReboot))
+}
+
+// TestSyncerDurableStepMovesAsideOnAFailedWrite verifies the runtime half of
+// the rule: when the kernel reports that a write of the references failed, the
+// repository is moved aside at once, without waiting for a restart, and the
+// next request hydrates it.
+func TestSyncerDurableStepMovesAsideOnAFailedWrite(t *testing.T) {
+ t.Parallel()
+
+ var failing atomic.Bool
+ e, repo, _ := caughtUpDirty(t, localrefs.WithSync(func(f *os.File) error {
+ if failing.Load() {
+ return errors.New("input/output error")
+ }
+ return f.Sync()
+ }))
+
+ failing.Store(true)
+ e.sy.DurableStep(e.ctx, e.localPath())
+ failing.Store(false)
+
+ require.NoDirExists(t, e.localPath())
+ require.Empty(t, e.markers(t))
+ require.Equal(t, float64(1), e.untrusted(t, localrefs.ReasonSyncFailed))
+
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ require.Equal(t, float64(2), syncCount(t, e.sy, "hydrated"), "the first hydration and the one after the move")
+ e.verifySynced(t, repo)
+}
diff --git a/internal/graft/corestore/voter/voter.go b/internal/graft/corestore/voter/voter.go
index 9a382ec..33d6abf 100644
--- a/internal/graft/corestore/voter/voter.go
+++ b/internal/graft/corestore/voter/voter.go
@@ -41,6 +41,7 @@ import (
"github.com/feder-cr/git-ai/internal/graft/changelog"
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/gitops"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
"github.com/feder-cr/git-ai/internal/log"
"github.com/feder-cr/git-ai/internal/structerr"
"github.com/feder-cr/git-ai/proto/go/gitalypb"
@@ -175,6 +176,10 @@ func (v *Voter) Preparing(ctx context.Context, payload gitcmd.HooksPayload, chan
return
}
+ // Deferred first, so that it runs last: the admission is waited for after
+ // the pack work, when git is about to take the reference lock.
+ defer v.admit(ctx, payload, changes)
+
start := time.Now()
defer func() { v.metrics.preparing.Observe(time.Since(start).Seconds()) }()
@@ -520,6 +525,10 @@ func (v *Voter) prepare(ctx context.Context, payload gitcmd.HooksPayload, change
}
if res.NoOp {
+ // git still writes the values the database already holds.
+ if err := v.deferDurability(ctx, payload, changes, shadow); err != nil {
+ return v.unavailable(err)
+ }
result = resultNoOp
return nil
}
@@ -535,6 +544,13 @@ func (v *Voter) prepare(ctx context.Context, payload gitcmd.HooksPayload, change
seq: res.Seq,
}, time.Now())
+ // After the entry, so that a failure here counts as a local failure in the
+ // Aborted phase: the database holds the changeset, catch-up repairs the
+ // local repository.
+ if err := v.deferDurability(ctx, payload, changes, shadow); err != nil {
+ return v.unavailable(err)
+ }
+
if err := v.fault(faultAfterCommit); err != nil {
return fmt.Errorf("storage core: %w", err)
}
@@ -741,23 +757,91 @@ func (v *Voter) Committed(ctx context.Context, payload gitcmd.HooksPayload, chan
return
}
- entry, ok := v.pending.take(pendingKey(payload.Repo, changes), time.Now())
+ key := pendingKey(payload.Repo, changes)
+ entry, ok := v.pending.take(key, time.Now())
+ if !v.localRefs() && (!ok || v.applier == nil) {
+ return
+ }
+
+ repoPath, err := v.repoPath(ctx, payload)
+ if err != nil {
+ v.logger.WithError(err).ErrorContext(ctx, "storage core: resolving the repository path after a commit")
+ return
+ }
+
+ // git finished the transaction, its auto-compaction included, and released
+ // the reference lock.
+ v.core.RefAdmission.Release(repoPath, localrefs.Key(key))
+ v.core.RefDurability.End(repoPath, localrefs.Key(key))
+
if !ok || v.applier == nil {
return
}
+ v.applier.NoteCommitted(ctx, repoPath, entry.repoID, entry.seq)
+}
- // The path is built from the storage and the relative path only, so that a
- // quarantine of the hooks payload does not end up in it.
- repoPath, err := v.locator.GetRepoPath(ctx, &gitalypb.Repository{
+// repoPath returns the path of the repository of the transaction. It is built
+// from the storage and the relative path only, so that a quarantine of the
+// hooks payload does not end up in it.
+func (v *Voter) repoPath(ctx context.Context, payload gitcmd.HooksPayload) (string, error) {
+ return v.locator.GetRepoPath(ctx, &gitalypb.Repository{
StorageName: payload.Repo.GetStorageName(),
RelativePath: payload.Repo.GetRelativePath(),
}, storage.WithRepositoryVerificationSkipped())
+}
+
+// localRefs reports whether the local reference writes of the node are
+// admitted or their durability deferred (authoritative mode).
+func (v *Voter) localRefs() bool {
+ return v.core.Authoritative() && (v.core.RefAdmission != nil || v.core.RefDurability.Enabled())
+}
+
+// DeferredReferenceDurability reports whether the reference writes of the
+// transactions this voter commits may run without fsync: authoritative mode
+// with the deferred durability of the node on (doc/graft/PLAN7.md, LOCAL
+// REFERENCE WRITES RULE). The hook manager asks it for UpdateWithHooks.
+func (v *Voter) DeferredReferenceDurability() bool {
+ return v != nil && v.core.Authoritative() && v.core.RefDurability.Enabled()
+}
+
+// admit waits for the admission of the transaction to the reference lock of
+// its repository (authoritative mode). Committed or Aborted releases it. It
+// never fails the transaction: without an admission git waits for its lock as
+// before.
+func (v *Voter) admit(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {
+ if !v.core.Authoritative() || v.core.RefAdmission == nil {
+ return
+ }
+
+ repoPath, err := v.repoPath(ctx, payload)
if err != nil {
- v.logger.WithError(err).ErrorContext(ctx, "storage core: resolving the repository path after a commit")
+ v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path for the admission, the transaction goes on without it")
return
}
+ v.core.RefAdmission.Acquire(ctx, repoPath, localrefs.Key(pendingKey(payload.Repo, changes)), localrefs.WriterHook)
+}
- v.applier.NoteCommitted(ctx, repoPath, entry.repoID, entry.seq)
+// deferDurability lets git write the references of a transaction that the
+// database committed without fsync (authoritative mode with deferred
+// durability): the repository gets its durable dirty marker first, and the
+// transaction counts as in flight until its Committed phase, because git may
+// still write its table and compact the stack until then. It runs in the
+// Prepared phase, under git's reference lock and before git writes anything.
+func (v *Voter) deferDurability(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, shadow bool) error {
+ if shadow || !v.DeferredReferenceDurability() {
+ return nil
+ }
+
+ repoPath, err := v.repoPath(ctx, payload)
+ if err != nil {
+ return err
+ }
+ repoKey := corestore.KeyOf(payload.Repo)
+ if err := v.core.RefDurability.MarkDirty(ctx, localrefs.RepoRef{Storage: repoKey.Storage, RelativePath: repoKey.RelativePath}, repoPath); err != nil {
+ return err
+ }
+ v.core.RefDurability.Begin(repoPath, localrefs.Key(pendingKey(payload.Repo, changes)))
+ return nil
}
// Aborted tells the voter that git aborted the transaction.
@@ -779,6 +863,17 @@ func (v *Voter) Aborted(ctx context.Context, payload gitcmd.HooksPayload, change
key := pendingKey(payload.Repo, changes)
+ // A transaction aborted after its Prepared phase stays in flight for the
+ // deferred durability (see localrefs.Durability.End): its repository stays
+ // dirty.
+ if v.localRefs() {
+ if repoPath, err := v.repoPath(ctx, payload); err != nil {
+ v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path after an abort, the admission ends at its bound")
+ } else {
+ v.core.RefAdmission.Release(repoPath, localrefs.Key(key))
+ }
+ }
+
// A transaction aborted before its Prepared phase (git did not get the
// locks) leaves the pack of its Preparing phase unused: an orphan in the
// bucket, like the pack of any failed write.
diff --git a/internal/graft/corestore/voter/voter_localrefs_test.go b/internal/graft/corestore/voter/voter_localrefs_test.go
new file mode 100644
index 0000000..2e1901a
--- /dev/null
+++ b/internal/graft/corestore/voter/voter_localrefs_test.go
@@ -0,0 +1,181 @@
+package voter_test
+
+import (
+ "context"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/git/localrepo"
+ "github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/voter"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+)
+
+// admissionHeld reports whether a writer holds the admission of the fixture's
+// repository: an Acquire with a cancelled context is admitted at once when
+// nobody holds it, and gives up otherwise.
+func (f *fixture) admissionHeld(t *testing.T) bool {
+ t.Helper()
+
+ ctx, cancel := context.WithCancel(f.ctx)
+ cancel()
+ probe := localrefs.Key{0xaa, 0xbb}
+ if f.core.RefAdmission.Acquire(ctx, f.repoPath, probe, localrefs.WriterSyncer) {
+ f.core.RefAdmission.Release(f.repoPath, probe)
+ return false
+ }
+ return true
+}
+
+// markers returns the dirty markers of the fixture's storage.
+func (f *fixture) markers(t *testing.T) []string {
+ t.Helper()
+
+ entries, err := os.ReadDir(filepath.Join(f.cfg.Storages[0].Path, localrefs.MarkerDirName))
+ if os.IsNotExist(err) {
+ return nil
+ }
+ require.NoError(t, err)
+ var names []string
+ for _, entry := range entries {
+ names = append(names, entry.Name())
+ }
+ return names
+}
+
+// TestLocalReferenceWritesOfACommittedTransaction follows one authoritative
+// transaction through the LOCAL REFERENCE WRITES RULE (doc/graft/PLAN7.md):
+// admitted at the end of Preparing, marked dirty and in flight after the
+// database commit in Prepared, released and ended in Committed.
+func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
+ const ref = "refs/heads/main"
+ input := line(f.zero, f.a.String(), ref)
+
+ require.True(t, f.voter.DeferredReferenceDurability())
+ require.False(t, f.admissionHeld(t))
+
+ f.voter.Preparing(f.ctx, f.payload, input)
+ require.True(t, f.admissionHeld(t), "the transaction is admitted at the end of Preparing")
+ require.Empty(t, f.markers(t), "nothing is dirty before the database commit")
+
+ require.NoError(t, f.prepared(input))
+ require.Len(t, f.markers(t), 1)
+
+ // git may still write and compact until Committed: the durable step keeps
+ // the marker.
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath))
+ require.Len(t, f.markers(t), 1)
+
+ gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName(ref), f.a)
+ f.voter.Committed(f.ctx, f.payload, input)
+ require.False(t, f.admissionHeld(t), "Committed releases the admission")
+ require.Len(t, f.applier.calls, 1, "the local commit is still noted")
+
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath))
+ require.Empty(t, f.markers(t))
+}
+
+// TestLocalReferenceWritesOfAnAbortedTransaction verifies that an abort after
+// Prepared releases the admission but leaves the repository dirty: the
+// transaction cannot be told from another with the same input, so it is never
+// counted as ended.
+func TestLocalReferenceWritesOfAnAbortedTransaction(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
+ input := line(f.zero, f.a.String(), "refs/heads/main")
+
+ f.voter.Preparing(f.ctx, f.payload, input)
+ require.NoError(t, f.prepared(input))
+ f.voter.Aborted(f.ctx, f.payload, input)
+ require.False(t, f.admissionHeld(t))
+
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath))
+ require.Len(t, f.markers(t), 1)
+}
+
+// TestLocalReferenceWritesOfARejectedTransaction verifies that a transaction
+// the database rejects marks nothing: git writes nothing for it.
+func TestLocalReferenceWritesOfARejectedTransaction(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
+ const ref = "refs/heads/main"
+ f.seed(t, ref, corestore.Target(f.b.String()))
+ input := line(f.a.String(), f.c.String(), ref)
+
+ f.voter.Preparing(f.ctx, f.payload, input)
+ requireMismatch(t, f.prepared(input), ref, f.a.String(), f.b.String())
+ f.voter.Aborted(f.ctx, f.payload, input)
+
+ require.False(t, f.admissionHeld(t))
+ require.Empty(t, f.markers(t))
+}
+
+// TestLocalReferenceWritesWhenTheMarkerCannotBeWritten verifies that git does
+// not write references without fsync when the repository cannot be marked
+// dirty: the transaction is aborted after the database commit, which catch-up
+// repairs like any local failure.
+func TestLocalReferenceWritesWhenTheMarkerCannotBeWritten(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
+ markerDir := filepath.Join(f.cfg.Storages[0].Path, localrefs.MarkerDirName)
+ require.NoError(t, os.MkdirAll(filepath.Dir(markerDir), 0o755))
+ require.NoError(t, os.WriteFile(markerDir, nil, 0o644))
+
+ input := line(f.zero, f.a.String(), "refs/heads/main")
+ err := f.prepared(input)
+ require.Equal(t, codes.Unavailable, status.Code(err), "error %v", err)
+ require.ErrorContains(t, err, "write the dirty marker")
+ require.Equal(t, uint64(1), f.headSeq(t), "the database committed the changeset before the marker")
+}
+
+// TestDeferredReferenceDurabilityByMode verifies that references are written
+// without fsync in authoritative mode with deferred durability only: off and
+// shadow modes, and authoritative mode without the boot identity, keep git's
+// fsync of references.
+func TestDeferredReferenceDurabilityByMode(t *testing.T) {
+ t.Parallel()
+
+ var nilVoter *voter.Voter
+ require.False(t, nilVoter.DeferredReferenceDurability())
+
+ off := voter.New(&corestore.Core{Mode: corestore.ModeOff}, testhelper.NewLogger(t), localrepo.Factory{}, nil, nil)
+ require.False(t, off.DeferredReferenceDurability())
+
+ shadow := newFixture(t, fixtureOptions{mode: corestore.ModeShadow, localRefs: true})
+ require.False(t, shadow.voter.DeferredReferenceDurability())
+
+ authoritative := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative})
+ require.False(t, authoritative.voter.DeferredReferenceDurability())
+
+ deferred := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
+ require.True(t, deferred.voter.DeferredReferenceDurability())
+}
+
+// TestLocalReferenceWritesNotInShadowMode verifies that shadow mode, where git
+// is the source of truth, neither admits nor marks.
+func TestLocalReferenceWritesNotInShadowMode(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t, fixtureOptions{mode: corestore.ModeShadow, localRefs: true})
+ input := line(f.zero, f.a.String(), "refs/heads/main")
+
+ f.voter.Preparing(f.ctx, f.payload, input)
+ require.False(t, f.admissionHeld(t))
+ require.NoError(t, f.prepared(input))
+ require.Empty(t, f.markers(t))
+ f.voter.Committed(f.ctx, f.payload, input)
+}
diff --git a/internal/graft/corestore/voter/voter_test.go b/internal/graft/corestore/voter/voter_test.go
index b2f8b41..845d120 100644
--- a/internal/graft/corestore/voter/voter_test.go
+++ b/internal/graft/corestore/voter/voter_test.go
@@ -90,6 +90,9 @@ type fixtureOptions struct {
// files makes the repository use the files reference backend instead of
// the package's default, reftable (see TestMain).
files bool
+ // localRefs gives the core the admission and the deferred durability of
+ // the local reference writes (coretest.EnableLocalRefs).
+ localRefs bool
}
func newFixture(t *testing.T, opts fixtureOptions) *fixture {
@@ -130,6 +133,9 @@ func newFixture(t *testing.T, opts fixtureOptions) *fixture {
logger := testhelper.NewLogger(t)
locator := config.NewLocator(cfg)
+ if opts.localRefs {
+ coretest.EnableLocalRefs(t, core, locator)
+ }
cache := catfile.NewCache(cfg)
t.Cleanup(cache.Stop)
factory := localrepo.NewFactory(logger, locator, gittest.NewCommandFactory(t, cfg), cache)
diff --git a/internal/graft/e2e/storage_localrefs_test.go b/internal/graft/e2e/storage_localrefs_test.go
new file mode 100644
index 0000000..768c93f
--- /dev/null
+++ b/internal/graft/e2e/storage_localrefs_test.go
@@ -0,0 +1,421 @@
+package e2e
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "math"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "sync"
+ "syscall"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/grpc"
+ healthpb "google.golang.org/grpc/health/grpc_health_v1"
+
+ "github.com/feder-cr/git-ai/internal/featureflag"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/coretest"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/sqlstore"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+ "github.com/feder-cr/git-ai/sdk"
+)
+
+// The tests of the LOCAL REFERENCE WRITES RULE (doc/graft/PLAN7.md) with real
+// binaries: a `gitaly serve` process in authoritative mode is killed with
+// SIGKILL at the points of the rule and started again on the same storage,
+// database and bucket.
+
+// realNode is a gitaly server process of one node.
+type realNode struct {
+ // cfg is the configuration the process reads; clientCfg is the same with the
+ // socket as a dialable address.
+ cfg config.Cfg
+ clientCfg config.Cfg
+ bin string
+ cfgPath string
+ logPath string
+ tokens map[string]string
+ ctx context.Context
+ setupCtx context.Context
+
+ cmd *exec.Cmd
+ // exited is closed when the process has exited and cmd.Wait returned.
+ exited chan struct{}
+}
+
+func newRealNode(t *testing.T) *realNode {
+ t.Helper()
+
+ graft, tokens := workerGraft(1)
+ cfg := testcfg.Build(t)
+ cfg.Auth.Token = sharedTestToken
+ cfg.Graft = unbatched(graft)
+ cfg.Graft.Standalone = true
+ cfg.Graft.Storage = config.GraftStorage{
+ Mode: "authoritative",
+ Database: coretest.DatabaseURL(t, sqlstore.SQLite),
+ Bucket: coretest.BucketURL(t, false),
+ NodeID: "node-a",
+ }
+ cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)
+ testcfg.BuildGitalySSH(t, cfg)
+ testcfg.BuildGitalyHooks(t, cfg)
+
+ clientCfg := cfg
+ clientCfg.SocketPath = "unix://" + cfg.SocketPath
+ // The requests to a separate server process carry no feature flags of the
+ // test binary, as a production client's do not: testhelper.Context marks
+ // them as explicit, and the server would refuse a flag of its own that the
+ // test binary does not define.
+ base, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ ctx := featureflag.OutgoingCtxWithFeatureFlag(base, featureflag.GPGSigning, false)
+
+ return &realNode{
+ cfg: cfg,
+ clientCfg: clientCfg,
+ bin: testcfg.BuildGitaly(t, cfg),
+ cfgPath: testcfg.WriteTemporaryGitalyConfigFile(t, cfg),
+ logPath: filepath.Join(t.TempDir(), "gitaly.log"),
+ tokens: tokens,
+ ctx: ctx,
+ setupCtx: testhelper.MergeOutgoingMetadata(ctx, testcfg.GitalyServersMetadataFromCfg(t, clientCfg)),
+ }
+}
+
+// start starts the server process with the extra environment env and waits
+// until it serves.
+func (n *realNode) start(t *testing.T, env ...string) {
+ t.Helper()
+
+ logFile, err := os.OpenFile(n.logPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = logFile.Close() })
+
+ cmd := exec.Command(n.bin, "serve", n.cfgPath)
+ cmd.Env = append(os.Environ(), env...)
+ cmd.Stdout = logFile
+ cmd.Stderr = logFile
+ require.NoError(t, cmd.Start())
+
+ exited := make(chan struct{})
+ go func() {
+ _ = cmd.Wait()
+ close(exited)
+ }()
+ n.cmd, n.exited = cmd, exited
+ t.Cleanup(func() {
+ select {
+ case <-exited:
+ default:
+ _ = cmd.Process.Signal(syscall.SIGKILL)
+ <-exited
+ }
+ })
+
+ conn := gittest.DialService(t, n.ctx, n.clientCfg)
+ ready, cancel := context.WithTimeout(n.ctx, 2*time.Minute)
+ defer cancel()
+ // Any answer (Unimplemented included) means the server serves.
+ _, err = healthpb.NewHealthClient(conn).Check(ready, &healthpb.HealthCheckRequest{}, grpc.WaitForReady(true))
+ if ready.Err() != nil {
+ t.Fatalf("the server did not start: %v\n%s", err, n.log(t))
+ }
+}
+
+// kill kills the server process with SIGKILL and waits for its end.
+func (n *realNode) kill(t *testing.T) {
+ t.Helper()
+ require.NoError(t, n.cmd.Process.Signal(syscall.SIGKILL))
+ <-n.exited
+}
+
+// waitExit waits for the server process to exit by itself and returns its exit
+// code.
+func (n *realNode) waitExit(t *testing.T) int {
+ t.Helper()
+
+ select {
+ case <-n.exited:
+ case <-time.After(2 * time.Minute):
+ t.Fatalf("the server did not exit\n%s", n.log(t))
+ }
+ return n.cmd.ProcessState.ExitCode()
+}
+
+func (n *realNode) log(t *testing.T) string {
+ t.Helper()
+ content, err := os.ReadFile(n.logPath)
+ require.NoError(t, err)
+ return string(content)
+}
+
+func (n *realNode) storagePath() string { return n.cfg.Storages[0].Path }
+
+// createRepository creates an empty repository through the server and returns
+// its relative path. In authoritative mode the gate registers it.
+func (n *realNode) createRepository(t *testing.T) string {
+ t.Helper()
+ repo, _ := gittest.CreateRepository(t, n.setupCtx, n.clientCfg)
+ return repo.GetRelativePath()
+}
+
+func (n *realNode) repoPath(relPath string) string {
+ return filepath.Join(n.storagePath(), relPath)
+}
+
+// agent returns a handle whose requests carry the worker's token.
+func (n *realNode) agent(t *testing.T, relPath string) *sdk.Repo {
+ t.Helper()
+
+ conn, err := sdk.Dial(n.ctx, n.clientCfg.SocketPath, sdk.WithStorage(n.cfg.Storages[0].Name), sdk.WithAgentToken(n.tokens[workerName(0)]))
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = conn.Close() })
+ return conn.Repo(relPath)
+}
+
+// operator returns a handle whose requests carry the shared token.
+func (n *realNode) operator(t *testing.T, relPath string) *sdk.Repo {
+ t.Helper()
+
+ conn, err := sdk.Dial(n.ctx, n.clientCfg.SocketPath, sdk.WithStorage(n.cfg.Storages[0].Name), sdk.WithToken(sharedTestToken))
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = conn.Close() })
+ return conn.Repo(relPath)
+}
+
+// markers returns the dirty markers of the node's storage.
+func (n *realNode) markers(t *testing.T) []string {
+ t.Helper()
+
+ entries, err := os.ReadDir(filepath.Join(n.storagePath(), localrefs.MarkerDirName))
+ if errors.Is(err, os.ErrNotExist) {
+ return nil
+ }
+ require.NoError(t, err)
+ var names []string
+ for _, entry := range entries {
+ names = append(names, filepath.Join(n.storagePath(), localrefs.MarkerDirName, entry.Name()))
+ }
+ return names
+}
+
+// tables returns the tables of the stack of the node's local repository.
+func (n *realNode) tables(t *testing.T, relPath string) []string {
+ t.Helper()
+ list, err := os.ReadFile(filepath.Join(n.repoPath(relPath), "reftable", "tables.list"))
+ require.NoError(t, err)
+ return strings.Fields(string(list))
+}
+
+// requireServed checks, through the server, that every branch of want is
+// served with its commit and that the node's local copy equals the database at
+// its head.
+func (n *realNode) requireServed(t *testing.T, relPath string, want map[string]string) {
+ t.Helper()
+
+ onA := n.agent(t, relPath)
+ for branch, commit := range want {
+ got, err := onA.ResolveRevision(n.ctx, branch)
+ require.NoError(t, err, "branch %s", branch)
+ require.Equal(t, commit, got, "branch %s", branch)
+ }
+
+ verified, err := n.operator(t, relPath).StorageVerify(n.ctx, true)
+ require.NoError(t, err)
+ require.Empty(t, verified.Divergences, "local references must equal the database references")
+ require.True(t, verified.ConnectivityOK, "connectivity: %s", verified.ConnectivityError)
+}
+
+// writeBranches commits one file on each of count new branches and returns
+// the commits by branch.
+func writeBranches(t *testing.T, onA *sdk.Repo, ctx context.Context, base string, prefix string, count int) map[string]string {
+ t.Helper()
+
+ written := map[string]string{}
+ for i := 0; i < count; i++ {
+ branch := fmt.Sprintf("%s/%d", prefix, i)
+ commit, err := commitFile(ctx, onA, workerName(0), branch, base, branch+".txt", branch+"\n")
+ require.NoError(t, err)
+ written[branch] = commit
+ }
+ return written
+}
+
+// TestStorageLocalReferencesAfterAProcessKill kills the server right after
+// reference writes made without fsync, before the durable step: the kernel
+// kept every write, so the restarted node keeps its repository, serves every
+// write and settles the marker.
+func TestStorageLocalReferencesAfterAProcessKill(t *testing.T) {
+ t.Parallel()
+
+ n := newRealNode(t)
+ n.start(t)
+ rel := n.createRepository(t)
+ onA := n.agent(t, rel)
+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")
+ require.NoError(t, err)
+ written := writeBranches(t, onA, n.ctx, base, "before-kill", 5)
+ written["main"] = base
+
+ // The last write is less than the idle period old: the marker is there.
+ require.Len(t, n.markers(t), 1)
+ n.kill(t)
+ require.Len(t, n.markers(t), 1, "the marker survives the process")
+
+ n.start(t)
+ n.requireServed(t, rel, written)
+ require.DirExists(t, n.repoPath(rel), "a process restart keeps the repository")
+
+ // The durable step runs every second and removes the marker of an idle
+ // repository.
+ require.Eventually(t, func() bool { return len(n.markers(t)) == 0 }, 30*time.Second, 50*time.Millisecond)
+ n.requireServed(t, rel, written)
+}
+
+// TestStorageLocalReferencesAfterAKillInTheDurableStep kills the server while
+// its durable step holds git's reference lock: the restarted node removes the
+// lock it left, so writes go on at once, and serves every write.
+func TestStorageLocalReferencesAfterAKillInTheDurableStep(t *testing.T) {
+ t.Parallel()
+
+ n := newRealNode(t)
+ n.start(t)
+ rel := n.createRepository(t)
+ onA := n.agent(t, rel)
+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")
+ require.NoError(t, err)
+ written := writeBranches(t, onA, n.ctx, base, "before-crash", 3)
+ written["main"] = base
+ require.Len(t, n.markers(t), 1)
+ n.kill(t)
+
+ // Restarted with the crash point, the node finds the repository dirty and
+ // idle: its first durable step dies holding the lock.
+ n.start(t, localrefs.CrashEnv+"=durable_step")
+ require.Equal(t, 75, n.waitExit(t), "the process dies in the durable step")
+ lock := filepath.Join(n.repoPath(rel), "reftable", "tables.list.lock")
+ require.FileExists(t, lock)
+ require.Len(t, n.markers(t), 1, "the marker is removed only after the lock")
+
+ n.start(t)
+ require.NoFileExists(t, lock, "the restarted node removes the lock its durable step left")
+ n.requireServed(t, rel, written)
+
+ more := writeBranches(t, n.agent(t, rel), n.ctx, base, "after-restart", 2)
+ for branch, commit := range more {
+ written[branch] = commit
+ }
+ n.requireServed(t, rel, written)
+}
+
+// TestStorageLocalReferencesAfterAKernelRestart simulates what a kernel
+// restart (power loss) can leave of reference writes made without fsync: the
+// newest table of the stack is empty. Read directly, the repository then looks
+// empty, without an error. The restarted node moves it aside before serving
+// and hydrates it again, so every write is served.
+func TestStorageLocalReferencesAfterAKernelRestart(t *testing.T) {
+ t.Parallel()
+
+ n := newRealNode(t)
+ n.start(t)
+ rel := n.createRepository(t)
+ onA := n.agent(t, rel)
+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")
+ require.NoError(t, err)
+ written := writeBranches(t, onA, n.ctx, base, "before-power-loss", 5)
+ written["main"] = base
+
+ markers := n.markers(t)
+ require.Len(t, markers, 1)
+ n.kill(t)
+
+ // The power loss: the newest table never reached the disk, and the kernel
+ // that boots next has another boot identity.
+ tables := n.tables(t, rel)
+ require.NoError(t, os.Truncate(filepath.Join(n.repoPath(rel), "reftable", tables[len(tables)-1]), 0))
+ content, err := os.ReadFile(markers[0])
+ require.NoError(t, err)
+ var m map[string]string
+ require.NoError(t, json.Unmarshal(content, &m))
+ m["boot_id"] = "the-boot-before-the-power-loss"
+ content, err = json.Marshal(m)
+ require.NoError(t, err)
+ require.NoError(t, os.WriteFile(markers[0], content, 0o600))
+
+ // What a server without the rule would serve.
+ torn := gittest.Exec(t, n.cfg, "-C", n.repoPath(rel), "for-each-ref", "--format=%(refname)")
+ require.Empty(t, strings.TrimSpace(string(torn)), "git reads the torn stack as an empty repository")
+
+ n.start(t)
+ require.Empty(t, n.markers(t))
+ n.requireServed(t, rel, written)
+ require.Contains(t, n.log(t), "moved aside a local repository whose references may have lost writes")
+}
+
+// TestStorageLocalReferencesStackDepthIsBounded writes many references
+// without fsync from concurrent writers and checks that git's compaction keeps
+// the stack shallow: the durable step only makes it durable, compaction stays
+// git's.
+func TestStorageLocalReferencesStackDepthIsBounded(t *testing.T) {
+ t.Parallel()
+
+ const writers, perWriter = 4, 30
+ n := newRealNode(t)
+ n.start(t)
+ rel := n.createRepository(t)
+ onA := n.agent(t, rel)
+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")
+ require.NoError(t, err)
+
+ var (
+ mu sync.Mutex
+ maxDepth int
+ wg sync.WaitGroup
+ )
+ errs := make(chan error, writers)
+ for w := 0; w < writers; w++ {
+ wg.Add(1)
+ go func(w int) {
+ defer wg.Done()
+ for i := 0; i < perWriter; i++ {
+ branch := fmt.Sprintf("load/%d/%d", w, i)
+ if _, err := commitFile(n.ctx, onA, workerName(0), branch, base, branch+".txt", branch+"\n"); err != nil {
+ errs <- err
+ return
+ }
+ list, err := os.ReadFile(filepath.Join(n.repoPath(rel), "reftable", "tables.list"))
+ if err != nil {
+ errs <- err
+ return
+ }
+ mu.Lock()
+ maxDepth = max(maxDepth, len(strings.Fields(string(list))))
+ mu.Unlock()
+ }
+ }(w)
+ }
+ wg.Wait()
+ close(errs)
+ for err := range errs {
+ require.NoError(t, err)
+ }
+
+ // git compacts the stack geometrically (factor 2): about log2 of the number
+ // of updates. Concurrent writers can make a compaction lose its race and
+ // leave a few more tables for a while.
+ updates := writers*perWriter + 2
+ bound := 2*int(math.Ceil(math.Log2(float64(updates)))) + 2
+ t.Logf("maximum stack depth %d over %d reference updates (bound %d)", maxDepth, updates, bound)
+ require.LessOrEqual(t, maxDepth, bound)
+}