Skip to content

Git designed for AI agents.

WP76b follow-up: one owner per storage, exact live count, review fixes

Federico Elia8953e8e0b973
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.md
index d3e15d3..89c2705 100644
--- a/doc/graft/PLAN7.md
+++ b/doc/graft/PLAN7.md
@@ -541,8 +541,31 @@ repository-wide lock.
2. *Deferred durability* (`corestore/localrefs.Durability`). The hook's
`UpdateWithHooks` transactions and the syncer's catch-up write references with
`core.fsync` without `reference`. Only the voter can turn this on
- (`DeferredReferenceDurability`: authoritative mode and the kernel's boot
- identity readable). Objects stay durable as before.
+ (`DeferredReferenceDurability`: authoritative mode, the kernel's boot identity
+ readable, and this process owns the storage's deferred durability). Objects
+ stay durable as before.
+ * *One owner per storage.* The deferred durability of a storage belongs to the
+ process that holds an exclusive lock (flock) on
+ `<storage>/+gitaly/graft-ref-durability/owner.lock` for its whole life.
+ * Only the owner writes references without fsync, runs the durable steps,
+ recovers, and removes a durable step's leftovers. A second process on the
+ storage (a graceful upgrade overlaps the old and the new process) writes
+ references with fsync and touches no marker or lock. It tries every second
+ to take the ownership, and recovers when it does.
+ * Ownership only goes from false to true within a process. So a write
+ decided as deferred always runs in the owner.
+ * Why this is safe:
+ * The unfsynced writes, the live transactions and the syncer lock that
+ the durable step reasons about all belong to one process.
+ * A non-owner's writes are durable by themselves. Its compactions fsync
+ the new table and list before they delete the tables they replace.
+ * A new owner takes over only after the old process exited (the flock is
+ released at exit), so no durable step of the old process is still
+ running.
+ * While an old process lives, no foreign-boot marker can exist: that
+ process recovered at its own start in this boot.
+ * Measured with real binaries: a second `gitaly serve` during the first's
+ life, then the first killed.
* *Dirty marker.* Before the first such write after a durable step, the
repository gets a marker, written durably, with the kernel's boot identity
(`<storage>/+gitaly/graft-ref-durability/<sha256(path)>.json`, not the state
@@ -552,28 +575,49 @@ repository-wide lock.
the marker cannot be written, Prepared fails and git aborts; catch-up
repairs the database's write, like any local failure.
* The syncer writes it under its repository lock.
- * A hook transaction counts as in flight from Prepared to Committed.
- Reasons from git's source: the new table is written in
- `transaction_finish` after "prepared", and auto-compaction runs inside
- `reftable_addition_commit`, before "committed". An abort after Prepared
- is not counted as an end (two transactions with the same input cannot be
- told apart), so the repository stays dirty.
+ * A hook transaction is live from its start to its end, and the durable
+ step never clears the marker while any is live.
+ * Start: the voter's Preparing, or `Denied` for a transaction the policy
+ denies.
+ * End: Committed or Aborted.
+ * Every transaction starts once and ends once, so the count per input is
+ the number of live transactions, whichever of two with the same input
+ ends first.
+ * Reasons from git's source: the new table is written in
+ `transaction_finish` after "prepared", and auto-compaction runs inside
+ `reftable_addition_commit`, before "committed".
+ * A transaction whose end is never reported keeps its repository dirty
+ (conservative).
* *Durable step* (`Settle`, every second for every dirty repository, under the
syncer's repository lock).
* It fsyncs the tables not yet fsynced, `tables.list` and the directory.
- * When the repository has been idle for a second and nothing is in flight,
- it takes git's lock without waiting (the lock is created with a tag, by
- linking a temporary file). It then fsyncs what was written since, marks
- the state clean, and releases the lock. Last, it removes the marker,
- under a per-repository file lock that `MarkDirty` waits for.
+ * When the repository has been idle for a second, or the process stops,
+ and nothing is live, the step clears the marker:
+ * It takes git's lock without waiting. The lock is created by linking a
+ temporary file that holds a tag and a random nonce.
+ * It fsyncs what was written since, marks the state clean, and releases
+ the lock. The release removes the lock only if it is still the same
+ file with the same nonce: an inode is reused as soon as it is freed, so
+ a lock that someone else replaced is never removed.
+ * Last, it removes the marker, under a per-repository file lock that
+ `MarkDirty` waits for.
* A busy lock leaves the marker for the next step.
- * *Write failure.* A failed fsync in the durable step means the kernel may
- have dropped writes. The repository is moved aside at once, under the
- repository lock: the move is made durable, then the marker removed. The
- failure is logged and counted (`sync_failed`), and the next request
- hydrates the repository.
+ * The set of fsynced tables keeps only the tables the stack still lists.
+ * A graceful stop runs a last step for every dirty repository after the
+ servers stopped, so a clean reboot does not rehydrate them.
+ * *Write failure.* Only a failed fsync means the kernel may have dropped
+ writes. The repository is then moved aside at once, under the repository
+ lock: the move (both directories and the new entry) is made durable, then
+ the marker removed. This happens only when the repository is a cache of the
+ storage core, the same check as the recovery. The failure is logged and
+ counted (`sync_failed`), and the next request hydrates the repository. Any
+ other error (opening a file, reading the list) is transient: counted
+ (`graft_storage_ref_durable_step_errors_total`) and retried at the next
+ step.
* *Recovery* (`Syncer.RecoverLocalReferences`). `gitaly serve` runs it before
- any listener opens and before any background worker.
+ any listener opens and before any background worker, when it owns the
+ storage (`Syncer.StartLocalReferences`); otherwise it runs when the process
+ takes the ownership.
* A marker of another boot means the kernel restarted with writes that may
not have reached the disk. git reads a torn stack as an empty repository
with exit code 0 (experiment 1, kept as
@@ -585,8 +629,8 @@ repository-wide lock.
dropped, keeping a repository whose local copy may be the only one.
* A marker of the same boot is a process restart: the page cache kept
every write, so the repository stays dirty to be settled.
- * A lock with the durable step's tag, left by a process that died, is
- removed.
+ * A lock with the durable step's tag and the step's temporary files, left
+ by a process that died, are removed.
* A corrupt marker fails the startup: its repository is unknown.
* Not Linux (no boot identity): references stay durable per transaction.
3. *Read paths.* After the recovery, no read path in authoritative mode can see
@@ -605,32 +649,62 @@ repository-wide lock.
* `localrefs` (`admission.go`, `durability.go`); `Core.RefAdmission`/`RefDurability`.
* `voter` (`admit`, `deferDurability`, Committed/Aborted).
* `syncer` (`refWriter`, `durability.go`).
-* `serve.go`: `openLocalReferenceWrites`, the recovery before the listeners, the
- durable-step loop.
+* `serve.go`: `openLocalReferenceWrites`; `StartLocalReferences` before the
+ listeners (ownership, recovery, durable-step loop), and its stop, deferred so
+ that it runs after the servers stopped.
* PM grant: `updateref.WithDeferredReferenceDurability`; the `gitcmd` constants
`FsyncComponents`, `FsyncComponentsWithoutReferences` and
`ReftableLockTimeoutMilliseconds`; `hook/storage_voter.go`
- (`DeferredReferenceDurability`); the optional interface asserted in
+ (`DeferredReferenceDurability`, `Denied`); the optional interface asserted in
`UpdateWithHooks`.
+* Follow-up after the independent review of 310facd:
+ * MAJOR: process-local state with overlapping processes. Fixed by one owner
+ per storage.
+ * The unlock is ownership-checked (nonce and file).
+ * An abort ends its transaction (the live count).
+ * The fsynced-table set is pruned.
+ * A final step runs on a graceful stop.
+ * Only a failed fsync distrusts a repository, and only after the cache check.
+ * The directory fsyncs (the marker directory's parent; the stale and temporary
+ directories of a move) are added.
+ * The step's temporary files are cleaned at recovery.
**Tests.**
-* `localrefs`: FIFO order, a cancelled waiter, the wait bound, the hold bound,
- unique keys; the marker, the durable step (idle, in flight, git's lock held,
- failed fsync, repository gone), recovery (process restart, kernel restart,
- stale lock, corrupt marker), the torn-stack experiment.
-* `voter`: committed, aborted and rejected transactions; a marker that cannot be
+* `localrefs`:
+ * admission: FIFO order, a cancelled waiter, the wait bound, the hold bound,
+ unique keys;
+ * ownership: one owner per storage; a non-owner neither defers, settles nor
+ recovers; ownership taken after the owner closes; the marker directory
+ created durably;
+ * durable step: the marker; idle, live transactions (including two with the
+ same key), git's lock held, the final step; the checked release (a
+ replaced lock with a reused inode is kept); failed fsync versus a transient
+ error; pruning; repository gone;
+ * recovery: process restart, kernel restart, stale lock and temporary files,
+ corrupt marker;
+ * the torn-stack experiment.
+* `voter`: committed, aborted (now settled) and rejected transactions; a denied
+ and an allowed transaction with the same input; a marker that cannot be
written; durability per mode (off, shadow and authoritative without boot
identity keep the fsync).
* `updateref`: the effective `core.fsync` of `update-ref`.
* `hook/updateref`: only a voter that defers turns it off.
* `syncer`: catch-up marks; reboot recovery hydrates (a read before it lists
nothing); process-restart recovery keeps; non-caches are kept; a failed fsync
- moves aside and hydrates.
-* e2e with the real `gitaly` binary: SIGKILL after writes (same boot, kept,
- settled); SIGKILL in the durable step (the lock is removed at restart, writes
- go on); a simulated power loss (torn table, other boot: moved aside, every
- write served); stack depth under 4×30 concurrent writes (git's geometric
- compaction, at most 2⌈log₂ n⌉+2 tables; measured 4 for 122 updates).
+ moves aside and hydrates; a transient error is retried; a second process waits
+ for the owner, then owns and settles; the stop settles a repository that is
+ not idle.
+* e2e with the real `gitaly` binary:
+ * SIGKILL after writes: same boot, kept, settled;
+ * SIGKILL in the durable step: the lock is removed at restart and writes go
+ on;
+ * a simulated power loss (torn table, other boot): moved aside, every write
+ served;
+ * a graceful-upgrade overlap: a second `gitaly serve` on the same storage
+ writes with fsync while the first owns; after a SIGKILL of the first, the
+ second owns and defers;
+ * stack depth under 4×30 concurrent writes: git's geometric compaction, at
+ most 2⌈log₂ n⌉+2 tables; measured 4 for 122 updates.
**Measured** (2026-10-19, same method as the problem's numbers, alternating
before = b665f56 and after = WP76b, two runs each, load 3–7; mean ms, 50 / 100
diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.md
index b616dec..29cb166 100644
--- a/doc/graft/STORAGE.md
+++ b/doc/graft/STORAGE.md
@@ -456,7 +456,18 @@ or re-import the repository from scratch (see
dropped.
- *A write the kernel reports as failed* (a writeback error, same boot): the
durable step's fsync fails, the repository is moved aside at once and logged,
- and the next request hydrates it.
+ and the next request hydrates it. Other errors of the step (a file that
+ cannot be opened for a moment) are retried every second and counted.
+ - *Clean stop*: after the servers stopped, `gitaly serve` makes every dirty
+ repository durable and removes its marker, so a clean reboot hydrates
+ nothing.
+ - *Graceful upgrade* (`GITALY_UPGRADES_ENABLED`): the new process serves while
+ the old one drains. One process per storage owns the deferred durability, by
+ an exclusive lock on `<storage>/+gitaly/graft-ref-durability/owner.lock`.
+ Until the old process exits, the new one writes references with fsync, as
+ in the other modes; then it takes over (`graft_storage_ref_durability_owner`).
+ During the overlap two processes also open an SQLite database, and their
+ writes meet in SQLite's own lock wait, see Limits; Postgres is not affected.
- Offline tools that read a repository from disk (`gitaly git -C <repo>`,
`gitaly-debug simulate-http-clone`) must not be pointed at a repository of
a node after an unclean shutdown before `gitaly serve` has started once:
@@ -501,7 +512,9 @@ without a single divergence.
| `graft_storage_ref_admission_expired_total` | Turns ended by the 10 s bound instead of their release (a git process that died between its phases). |
| `graft_storage_ref_dirty_repositories` | Repositories with reference writes not yet made durable. |
| `graft_storage_ref_dirty_marked_total`, `graft_storage_ref_dirty_cleared_total` | Dirty markers written and removed by the durable step. |
-| `graft_storage_ref_writes_inflight` | Hook transactions between their Prepared and Committed phases; one that never commits keeps its repository dirty. |
+| `graft_storage_ref_writes_inflight` | Hook transactions between their start and their end (Committed or Aborted); one whose end is never reported keeps its repository dirty. |
+| `graft_storage_ref_durability_owner` | 1 when this process owns the deferred durability of its storages, 0 while another process (the old one of a graceful upgrade) does. |
+| `graft_storage_ref_durable_step_errors_total` | Durable steps that failed with a transient error (not a failed fsync); retried every second. |
| `graft_storage_ref_untrusted_total{reason}` | Repositories moved aside to be hydrated again: `reboot` (dirty at a kernel restart), `sync_failed` (the durable step's fsync failed). |
## Benchmark
@@ -614,9 +627,12 @@ show:
after a power loss it is always hydrated again, even though its references
were made durable less than a second earlier: the marker does not record which
writes are already durable.
-- **SQLite serves one node.** SQLite has a single write lock. The writes of
- one gitaly process take it in the order they arrive, but several processes on
- one SQLite file (several nodes on one host) meet in SQLite's own lock wait,
+- **SQLite serves one node.** Only `gitaly serve` opens the database: the
+ `graft` CLI and the other tools go through the server. SQLite has a single
+ write lock. The writes of one gitaly process take it in the order they
+ arrive, but several processes on one SQLite file (several nodes on one host,
+ or the old and the new process during a graceful upgrade) meet in SQLite's
+ own lock wait,
which keeps no queue: under steady contention an unlucky write can wait past
the 10 s busy timeout and fail. A deployment with more than one node uses
Postgres; SQLite is for one node, or for tests.
diff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.go
index 61993e3..116a080 100644
--- a/internal/cli/gitaly/serve.go
+++ b/internal/cli/gitaly/serve.go
@@ -638,12 +638,13 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
for _, s := range cfg.Storages {
storageNames = append(storageNames, s.Name)
}
- if err := syncer.RecoverLocalReferences(ctx, storageNames); err != nil {
+ stopLocalReferences, err := syncer.StartLocalReferences(ctx, storageNames)
+ if err != nil {
return err
}
- durableCtx, stopDurableSteps := context.WithCancel(ctx)
- defer stopDurableSteps()
- go syncer.RunDurableSteps(durableCtx, localrefs.DurableStepInterval)
+ // Deferred here, so it runs after the servers stopped: a last durable step
+ // settles every repository whose writes ended.
+ defer stopLocalReferences()
storageGate = gate.New(storageCore, syncer, logger, locator, localrepoFactory, protoregistry.GitalyProtoPreregistered)
}
diff --git a/internal/gitaly/hook/storage_voter.go b/internal/gitaly/hook/storage_voter.go
index eb297fd..a0fe77a 100644
--- a/internal/gitaly/hook/storage_voter.go
+++ b/internal/gitaly/hook/storage_voter.go
@@ -15,6 +15,11 @@ type StorageVoter interface {
// its pack) before git takes the reference locks. It never fails the
// transaction: when it did not complete, Prepared does that work itself.
Preparing(ctx context.Context, payload gitcmd.HooksPayload, changes []byte)
+ // Denied is called instead of Preparing for a transaction that the
+ // standalone access policy denies: it prepares nothing, and the transaction
+ // is aborted in its Prepared phase. Every transaction gets exactly one of
+ // Preparing and Denied before its Aborted or Committed phase.
+ Denied(ctx context.Context, payload gitcmd.HooksPayload, changes []byte)
// Prepared commits the transaction to the storage core while git holds
// the reference locks. An error aborts the transaction.
Prepared(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, meta changelog.Meta) error
@@ -65,6 +70,7 @@ func (m *GitLabHookManager) storagePreparing(ctx context.Context, payload gitcmd
}
if message, err := m.graftPolicyDecision(env, changes); err != nil || message != "" {
+ m.storageVoter.Denied(ctx, payload, changes)
return
}
diff --git a/internal/gitaly/hook/storage_voter_test.go b/internal/gitaly/hook/storage_voter_test.go
index 7c411ad..abb3694 100644
--- a/internal/gitaly/hook/storage_voter_test.go
+++ b/internal/gitaly/hook/storage_voter_test.go
@@ -44,6 +44,10 @@ func (v *recordingVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte)
v.record("preparing")
}
+func (v *recordingVoter) Denied(context.Context, gitcmd.HooksPayload, []byte) {
+ v.record("denied")
+}
+
func (v *recordingVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {
v.record("prepared")
return nil
@@ -82,9 +86,11 @@ func TestReferenceTransactionHook_storageVoterPhases(t *testing.T) {
want: []string{"preparing", "prepared", "committed"},
},
{
+ // The voter learns of its start (Denied), so that it counts the
+ // transaction until its abort, but prepares nothing.
desc: "a denied transaction is neither prepared nor committed",
policy: []config.GraftPolicyRule{{Refs: []string{"refs/heads/main"}, Allow: false}},
- want: nil,
+ want: []string{"denied"},
denied: true,
},
} {
diff --git a/internal/gitaly/hook/updateref/deferred_durability_test.go b/internal/gitaly/hook/updateref/deferred_durability_test.go
index 1b9d414..a381daf 100644
--- a/internal/gitaly/hook/updateref/deferred_durability_test.go
+++ b/internal/gitaly/hook/updateref/deferred_durability_test.go
@@ -24,6 +24,8 @@ type durabilityVoter struct {
func (durabilityVoter) Preparing(context.Context, gitcmd.HooksPayload, []byte) {}
+func (durabilityVoter) Denied(context.Context, gitcmd.HooksPayload, []byte) {}
+
func (durabilityVoter) Prepared(context.Context, gitcmd.HooksPayload, []byte, changelog.Meta) error {
return nil
}
diff --git a/internal/graft/corestore/coretest/coretest.go b/internal/graft/corestore/coretest/coretest.go
index 829d77f..f7e1da8 100644
--- a/internal/graft/corestore/coretest/coretest.go
+++ b/internal/graft/corestore/coretest/coretest.go
@@ -20,7 +20,7 @@ import (
"testing"
"time"
- "github.com/feder-cr/git-ai/internal/gitaly/storage"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/blobstore"
"github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
@@ -200,18 +200,35 @@ func Open(tb testing.TB, opts Options) *corestore.Core {
}
// EnableLocalRefs gives the core the admission and the deferred durability of
-// the local reference writes, as gitaly serve does in authoritative mode
-// (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). The boot identity is a
-// fixed test value unless opts set another one.
-func EnableLocalRefs(tb testing.TB, core *corestore.Core, locator storage.Locator, opts ...localrefs.DurabilityOption) {
+// the local reference writes, owning the storages of cfg, as gitaly serve does
+// in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). The
+// boot identity is a fixed test value unless opts set another one. A deferred
+// durability the core had is closed first, as when its process exits.
+func EnableLocalRefs(tb testing.TB, core *corestore.Core, cfg config.Cfg, opts ...localrefs.DurabilityOption) {
tb.Helper()
+ if core.RefDurability != nil {
+ require.NoError(tb, core.RefDurability.Close())
+ }
core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)
- durability, err := localrefs.NewDurability(locator, 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID)}, opts...)...)
+ durability, err := localrefs.NewDurability(config.NewLocator(cfg), 0, append([]localrefs.DurabilityOption{localrefs.WithBootID(TestBootID)}, opts...)...)
+ require.NoError(tb, err)
+ owned, err := durability.TryOwn(context.Background(), StorageNames(cfg))
require.NoError(tb, err)
+ require.True(tb, owned, "another durability of the test owns the storages")
+ tb.Cleanup(func() { _ = durability.Close() })
core.RefDurability = durability
}
+// StorageNames returns the names of the storages of cfg.
+func StorageNames(cfg config.Cfg) []string {
+ names := make([]string, 0, len(cfg.Storages))
+ for _, s := range cfg.Storages {
+ names = append(names, s.Name)
+ }
+ return names
+}
+
// TestBootID is the boot identity EnableLocalRefs gives the deferred
// durability by default.
const TestBootID = "test-boot"
diff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.go
index ca0a905..66147b1 100644
--- a/internal/graft/corestore/localrefs/durability.go
+++ b/internal/graft/corestore/localrefs/durability.go
@@ -4,6 +4,7 @@ import (
"bufio"
"bytes"
"context"
+ "crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
@@ -14,6 +15,8 @@ import (
"path/filepath"
"strings"
"sync"
+ "sync/atomic"
+ "syscall"
"time"
"github.com/prometheus/client_golang/prometheus"
@@ -28,13 +31,25 @@ import (
// cache walker removes after an hour.
const MarkerDirName = config.GitalyDataPrefix + "/graft-ref-durability"
+// ownerLockName is the file in MarkerDirName whose exclusive lock makes a
+// process the owner of the storage's deferred durability.
+const ownerLockName = "owner.lock"
+
+// markerTempPrefix prefixes the temporary files of marker writes.
+const markerTempPrefix = ".marker-"
+
+// stepTempPrefix prefixes the temporary files the durable step links its
+// reference lock from, in the repository directory.
+const stepTempPrefix = ".graft-durable-step-"
+
// bootIDPath is where Linux exposes the identity of the running kernel boot.
const bootIDPath = "/proc/sys/kernel/random/boot_id"
const (
// DurableStepInterval is how often the durable step runs for every dirty
// repository: it bounds how long a write failure the kernel reports goes
- // unnoticed.
+ // unnoticed. A process that does not own the deferred durability tries to
+ // take it as often.
DurableStepInterval = time.Second
// DurableStepIdle is how long a dirty repository must be without a reference
// write before the durable step removes its marker.
@@ -52,14 +67,16 @@ const (
crashExitCode = 75
)
-// stackLockContent is what the durable step writes into the reference lock
-// it takes. git never writes it (a lock of git holds table names), so a lock
-// with this content found at startup was left by a durable step that died.
-const stackLockContent = "graft durable step\n"
+// stackLockTag starts the content of the reference lock the durable step
+// takes, followed by a random nonce of that lock. git never writes it (a lock
+// of git holds table names), so a lock with this tag found by a new owner was
+// left by a durable step that died, and a step removes a lock only when it
+// still holds its own nonce.
+const stackLockTag = "graft durable step "
-// ErrUntrusted is returned by Settle when making the references of a
-// repository durable failed: the kernel may have dropped some of their writes,
-// so the local repository must not be served before it is rebuilt.
+// ErrUntrusted is returned by Settle when the kernel reported that making the
+// references of a repository durable failed: it may have dropped some of their
+// writes, so the local repository must not be served before it is rebuilt.
var ErrUntrusted = errors.New("storage core: the local references may have lost writes")
// UntrustedError is ErrUntrusted for one repository.
@@ -76,23 +93,37 @@ func (e *UntrustedError) Error() string {
// Unwrap returns ErrUntrusted and the cause.
func (e *UntrustedError) Unwrap() []error { return []error{ErrUntrusted, e.Err} }
+// writebackError is an fsync that failed: the kernel reports a write it could
+// not make durable. Every other error of the durable step (opening a file,
+// reading the table list) is transient and retried at the next step.
+type writebackError struct{ err error }
+
+func (e writebackError) Error() string { return "fsync: " + e.err.Error() }
+func (e writebackError) Unwrap() error { return e.err }
+
// Durability defers the durability of the local reference writes of the
// storage core in authoritative mode (doc/graft/PLAN7.md, LOCAL REFERENCE
// WRITES RULE). The database commit of a transaction is its durable commit
// point and the local repository is a cache of it, so git writes the references
// without fsync and Durability keeps the cache safe to serve:
//
+// - one process per storage owns the deferred durability (an exclusive lock
+// held for the life of the process); only the owner writes references
+// without fsync, runs the durable steps and touches the durable step's
+// locks. Another process on the same storage (a graceful upgrade overlaps
+// the old and the new process) writes them with fsync until it owns;
// - before the first reference write that is not durable, the repository
// gets a dirty marker, written durably, that holds the identity of the
// kernel boot (MarkDirty);
// - a durable step makes the references durable and removes the marker,
-// under git's reference lock and only when no writer is between its
-// Prepared and Committed phases (Settle);
-// - at startup, a marker of another boot means the kernel restarted while
-// the repository had writes that may never have reached the disk: its
-// reftable stack may be torn, which git reads as an empty repository, so
-// the repository is moved aside and hydrated again (Recover). A marker of
-// the running boot is a process restart: the kernel kept every write.
+// under git's reference lock and only when no transaction of the process is
+// live (Settle);
+// - when it takes the ownership, a marker of another boot means the kernel
+// restarted while the repository had writes that may never have reached the
+// disk: its reftable stack may be torn, which git reads as an empty
+// repository, so the repository is moved aside and hydrated again
+// (Recover). A marker of the running boot is a process restart: the kernel
+// kept every write.
//
// A nil *Durability is off: every reference write is durable by itself
// (core.fsync with references), as outside authoritative mode.
@@ -107,16 +138,24 @@ type Durability struct {
// crashAt is the crash point selected with CrashEnv.
crashAt string
+ // owner is set once the process owns the deferred durability of its
+ // storages; it never goes back. ownerFiles hold the owner locks.
+ owner atomic.Bool
+ ownerMu sync.Mutex
+ ownerFiles []*os.File
+
mu sync.Mutex
// repos holds the state of every repository written since the process
// started (one entry per repository; Forget drops it).
repos map[string]*refState
- marked prometheus.Counter
- cleared prometheus.Counter
- untrusted *prometheus.CounterVec
- dirty prometheus.Gauge
- inflight prometheus.Gauge
+ marked prometheus.Counter
+ cleared prometheus.Counter
+ untrusted *prometheus.CounterVec
+ stepErrors prometheus.Counter
+ dirty prometheus.Gauge
+ inflight prometheus.Gauge
+ owning prometheus.Gauge
}
var _ prometheus.Collector = (*Durability)(nil)
@@ -130,14 +169,15 @@ type refState struct {
repo RepoRef
marker string
dirty bool
- // inflight counts, by key, the hook transactions between their Prepared
- // and Committed phases: git may still write their table and compact the
- // stack without fsync.
- inflight map[Key]int
- nInflight int
+ // live counts, by key, the hook transactions of the process from their
+ // start (Preparing, or a policy denial) to their end (Committed or
+ // Aborted): any of them may still write a table or compact the stack
+ // without fsync.
+ live map[Key]int
+ nLive int
lastWrite time.Time
- // synced holds the tables made durable during the current dirty period.
- // Tables are immutable, so one fsync each is enough.
+ // synced holds the tables made durable during the current dirty period that
+ // the stack still lists. Tables are immutable, so one fsync each is enough.
synced map[string]struct{}
}
@@ -225,21 +265,92 @@ func newDurability(locator storage.Locator, idle time.Duration) *Durability {
Name: "graft_storage_ref_untrusted_total",
Help: "Number of local repositories whose references could not be trusted and were moved aside to be hydrated again, by reason (reboot, sync_failed).",
}, []string{"reason"}),
+ stepErrors: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_storage_ref_durable_step_errors_total",
+ Help: "Number of durable steps that failed with a transient error (not a failed fsync) and are retried at the next step.",
+ }),
dirty: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "graft_storage_ref_dirty_repositories",
Help: "Number of local repositories with reference writes that are not durable yet.",
}),
inflight: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "graft_storage_ref_writes_inflight",
- Help: "Number of hook reference transactions between their Prepared and Committed phases. One that never reports Committed keeps its repository dirty.",
+ Help: "Number of hook reference transactions between their start and their end. One that never ends keeps its repository dirty.",
+ }),
+ owning: prometheus.NewGauge(prometheus.GaugeOpts{
+ Name: "graft_storage_ref_durability_owner",
+ Help: "1 when this process owns the deferred durability of its storages and writes references without fsync, 0 while another process owns it.",
}),
}
}
-// Enabled reports whether the reference writes of the storage core are made
-// without fsync.
+// Enabled reports whether the node defers the durability of its reference
+// writes once it owns it.
func (d *Durability) Enabled() bool { return d != nil }
+// Deferring reports whether this process owns the deferred durability, so its
+// reference writes are made without fsync. It goes from false to true at most
+// once.
+func (d *Durability) Deferring() bool { return d != nil && d.owner.Load() }
+
+// TryOwn takes the ownership of the deferred durability of the storages
+// without waiting. It returns false when another process holds it for one of
+// them; the locks taken are kept, and a later call takes the rest.
+func (d *Durability) TryOwn(ctx context.Context, storages []string) (bool, error) {
+ if d == nil {
+ return false, nil
+ }
+ if d.owner.Load() {
+ return true, nil
+ }
+
+ d.ownerMu.Lock()
+ defer d.ownerMu.Unlock()
+
+ for _, storageName := range storages[len(d.ownerFiles):] {
+ dir, err := d.markerDir(ctx, storageName)
+ if err != nil {
+ return false, err
+ }
+ if err := d.ensureDir(dir); err != nil {
+ return false, fmt.Errorf("create the dirty marker directory: %w", err)
+ }
+ f, err := os.OpenFile(filepath.Join(dir, ownerLockName), os.O_RDWR|os.O_CREATE, 0o600)
+ if err != nil {
+ return false, fmt.Errorf("open the owner lock: %w", err)
+ }
+ if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
+ _ = f.Close()
+ if errors.Is(err, syscall.EWOULDBLOCK) {
+ return false, nil
+ }
+ return false, fmt.Errorf("lock the owner lock: %w", err)
+ }
+ d.ownerFiles = append(d.ownerFiles, f)
+ }
+
+ d.owner.Store(true)
+ d.owning.Set(1)
+ return true, nil
+}
+
+// Close releases the ownership. The process must not write references
+// without fsync afterwards; only tests and the end of the process call it.
+func (d *Durability) Close() error {
+ if d == nil {
+ return nil
+ }
+
+ d.ownerMu.Lock()
+ defer d.ownerMu.Unlock()
+ var errs []error
+ for _, f := range d.ownerFiles {
+ errs = append(errs, f.Close())
+ }
+ d.ownerFiles = nil
+ return errors.Join(errs...)
+}
+
// markerPath returns the path of the dirty marker of the repository at path.
func (d *Durability) markerPath(ctx context.Context, repo RepoRef, path string) (string, error) {
dir, err := d.markerDir(ctx, repo.Storage)
@@ -263,7 +374,7 @@ func (d *Durability) markerDir(ctx context.Context, storageName string) (string,
func (d *Durability) stateLocked(path string) *refState {
st := d.repos[path]
if st == nil {
- st = &refState{inflight: map[Key]int{}, synced: map[string]struct{}{}}
+ st = &refState{live: map[Key]int{}, synced: map[string]struct{}{}}
d.repos[path] = st
}
return st
@@ -271,7 +382,8 @@ func (d *Durability) stateLocked(path string) *refState {
// MarkDirty records that the repository at path is about to get reference
// writes that are not durable. When it returns nil the repository has a durable
-// dirty marker; the writer must not write before. The caller holds git's
+// dirty marker; the writer must not write before. It is only called while the
+// process owns the deferred durability (Deferring). The caller holds git's
// reference lock of the repository (the voter's Prepared phase) or the syncer's
// repository lock, under which Settle decides to clear the marker, and the
// marker file is only written or removed under the state's file lock, so a
@@ -318,9 +430,12 @@ func (d *Durability) MarkDirty(ctx context.Context, repo RepoRef, path string) e
return nil
}
-// Begin records that the hook transaction key is past its Prepared phase on
-// the repository at path: git may write its table and compact the stack until
-// it reports Committed (End). It is called under git's reference lock.
+// Begin records that the hook transaction key started on the repository at
+// path: from its Preparing phase (or its policy denial, which aborts it), before
+// git takes the reference lock, to its end (End), it may write a table and
+// compact the stack. Every transaction begins exactly once and ends exactly
+// once (Committed or Aborted), so the count of a key is the number of its live
+// transactions, whichever of two transactions with the same key ends first.
func (d *Durability) Begin(path string, key Key) {
if d == nil {
return
@@ -329,15 +444,14 @@ func (d *Durability) Begin(path string, key Key) {
d.mu.Lock()
defer d.mu.Unlock()
st := d.stateLocked(path)
- st.inflight[key]++
- st.nInflight++
+ st.live[key]++
+ st.nLive++
d.inflight.Inc()
}
-// End records that the hook transaction key committed. A transaction aborted
-// after its Prepared phase is not ended: two transactions with the same key
-// cannot be told apart, and ending the wrong one would let Settle remove the
-// marker while the other one still writes. Its repository stays dirty instead.
+// End records that a hook transaction with key ended (Committed or Aborted).
+// An end without a begin (the process started after the transaction began)
+// changes nothing.
func (d *Durability) End(path string, key Key) {
if d == nil {
return
@@ -346,14 +460,14 @@ func (d *Durability) End(path string, key Key) {
d.mu.Lock()
defer d.mu.Unlock()
st := d.repos[path]
- if st == nil || st.inflight[key] == 0 {
+ if st == nil || st.live[key] == 0 {
return
}
- st.inflight[key]--
- if st.inflight[key] == 0 {
- delete(st.inflight, key)
+ st.live[key]--
+ if st.live[key] == 0 {
+ delete(st.live, key)
}
- st.nInflight--
+ st.nLive--
d.inflight.Dec()
}
@@ -374,20 +488,43 @@ func (d *Durability) Dirty() []string {
return paths
}
-// Settle is the durable step of the repository at path. It makes the tables
-// that the stack lists, the list and the directory durable, which reports a
-// write the kernel failed to do (an *UntrustedError: the caller moves the
-// repository aside at once). When the repository has been idle for the idle
-// period and no hook transaction is in flight, it then removes the marker:
-// the decision is taken under git's reference lock, taken without waiting (a
-// busy lock leaves the marker for the next step), and the marker file is
-// removed after the lock is released, under the state's file lock that a
-// writer's MarkDirty waits for. A repository that no longer exists (moved
-// aside as a stale cache) loses its marker. The caller holds the syncer's
-// repository lock, so no syncer write and no replacement of the repository
-// runs meanwhile.
-func (d *Durability) Settle(path string) error {
+// RepoOf returns the repository whose state is kept for path.
+func (d *Durability) RepoOf(path string) (RepoRef, bool) {
+ if d == nil {
+ return RepoRef{}, false
+ }
+
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ st := d.repos[path]
+ if st == nil || st.repo == (RepoRef{}) {
+ return RepoRef{}, false
+ }
+ return st.repo, true
+}
+
+// CountStepError counts a durable step that failed with a transient error.
+func (d *Durability) CountStepError() {
if d == nil {
+ return
+ }
+ d.stepErrors.Inc()
+}
+
+// Settle is the durable step of the repository at path; only the owner runs
+// it. It makes the tables that the stack lists, the list and the directory
+// durable. A failed fsync is an *UntrustedError (the caller moves the
+// repository aside at once); any other error is transient. When the repository
+// has been idle for the idle period, or final is set (the process stops), and
+// no transaction of the process is live, it then removes the marker: the
+// decision is taken under git's reference lock, taken without waiting (a busy
+// lock leaves the marker for the next step), and the marker file is removed
+// after the lock is released, under the state's file lock that a writer's
+// MarkDirty waits for. A repository that no longer exists (moved aside as a
+// stale cache) loses its marker. The caller holds the syncer's repository lock,
+// so no syncer write and no replacement of the repository runs meanwhile.
+func (d *Durability) Settle(path string, final bool) error {
+ if !d.Deferring() {
return nil
}
@@ -398,7 +535,7 @@ func (d *Durability) Settle(path string) error {
return nil
}
repo := st.repo
- idle := st.nInflight == 0 && time.Since(st.lastWrite) >= d.idle
+ idle := st.nLive == 0 && (final || time.Since(st.lastWrite) >= d.idle)
d.mu.Unlock()
if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {
@@ -409,13 +546,13 @@ func (d *Durability) Settle(path string) error {
}
if err := d.syncStack(path, st); err != nil {
- return &UntrustedError{Repo: repo, Path: path, Err: err}
+ return d.stepError(repo, path, err)
}
if !idle {
return nil
}
- unlock, ok, err := lockStack(path)
+ lock, ok, err := lockStack(path)
if err != nil {
return err
}
@@ -425,31 +562,41 @@ func (d *Durability) Settle(path string) error {
cleared, err := d.settleLocked(path, st)
if err != nil {
- _ = unlock()
- return &UntrustedError{Repo: repo, Path: path, Err: err}
+ _ = lock.release()
+ return d.stepError(repo, path, err)
}
if !cleared {
- return unlock()
+ return lock.release()
}
// The state is clean from here on, and a writer that takes git's lock now
// writes a new marker: it waits for the file lock, held until the old
// marker is removed.
defer st.file.Unlock()
- if err := unlock(); err != nil {
+ if err := lock.release(); err != nil {
return err
}
return d.removeMarker(st)
}
+// stepError is the error of a durable step: an *UntrustedError for a failed
+// fsync, the error itself otherwise.
+func (d *Durability) stepError(repo RepoRef, path string, err error) error {
+ var writeback writebackError
+ if errors.As(err, &writeback) {
+ return &UntrustedError{Repo: repo, Path: path, Err: err}
+ }
+ return err
+}
+
// settleLocked makes the tables written since the first pass durable and, when
-// no hook transaction is in flight, marks the state clean with its file lock
-// held (the caller releases it). The caller holds git's reference lock: no
-// transaction can pass its Prepared phase meanwhile, and the transactions
-// counted before have committed when the count is zero.
+// no transaction of the process is live, marks the state clean with its file
+// lock held (the caller releases it). The caller holds git's reference lock: no
+// transaction can take it meanwhile, and the live transactions are counted from
+// before they take it until their end.
func (d *Durability) settleLocked(path string, st *refState) (bool, error) {
d.mu.Lock()
- busy := st.nInflight > 0
+ busy := st.nLive > 0
d.mu.Unlock()
if busy {
return false, nil
@@ -491,19 +638,41 @@ func (d *Durability) removeMarker(st *refState) error {
return nil
}
+// stackLock is git's reference lock of a repository, taken by the durable
+// step.
+type stackLock struct {
+ path string
+ // content is the tag and the nonce the step wrote into it, and info the file
+ // it created: a lock is the step's own only when both still match (an inode
+ // is reused as soon as it is freed, so the file alone does not tell).
+ content []byte
+ info os.FileInfo
+}
+
// lockStack takes git's reference lock of the repository at path without
// waiting: ok is false when it is held. The lock is created with its content
// at once (a linked temporary file), so that a lock left by a process that
-// died is recognized at startup (removeStaleStackLock).
-func lockStack(path string) (unlock func() error, ok bool, err error) {
+// died is recognized by the next owner (removeStaleStackLock).
+func lockStack(path string) (*stackLock, bool, error) {
lockPath := filepath.Join(path, "reftable", "tables.list.lock")
- tmp, err := os.CreateTemp(path, ".graft-durable-step-*")
+ nonce := make([]byte, 16)
+ if _, err := rand.Read(nonce); err != nil {
+ return nil, false, fmt.Errorf("take the reference lock: %w", err)
+ }
+ content := []byte(stackLockTag + hex.EncodeToString(nonce) + "\n")
+
+ tmp, err := os.CreateTemp(path, stepTempPrefix+"*")
if err != nil {
return nil, false, fmt.Errorf("take the reference lock: %w", err)
}
defer func() { _ = os.Remove(tmp.Name()) }()
- if _, err := tmp.WriteString(stackLockContent); err != nil {
+ if _, err := tmp.Write(content); err != nil {
+ _ = tmp.Close()
+ return nil, false, fmt.Errorf("take the reference lock: %w", err)
+ }
+ info, err := tmp.Stat()
+ if err != nil {
_ = tmp.Close()
return nil, false, fmt.Errorf("take the reference lock: %w", err)
}
@@ -517,18 +686,41 @@ func lockStack(path string) (unlock func() error, ok bool, err error) {
}
return nil, false, fmt.Errorf("take the reference lock: %w", err)
}
- return func() error {
- if err := os.Remove(lockPath); err != nil {
- return fmt.Errorf("release the reference lock: %w", err)
- }
+ return &stackLock{path: lockPath, content: content, info: info}, true, nil
+}
+
+// release removes the lock if it is still the one the step created (same file,
+// same nonce): a lock that someone else removed (housekeeping's stale-lock
+// cleanup) and that git or another step took again is never removed.
+func (l *stackLock) release() error {
+ current, err := os.Lstat(l.path)
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ if err != nil {
+ return fmt.Errorf("release the reference lock: %w", err)
+ }
+ content, err := os.ReadFile(l.path)
+ if errors.Is(err, fs.ErrNotExist) {
return nil
- }, true, nil
+ }
+ if err != nil {
+ return fmt.Errorf("release the reference lock: %w", err)
+ }
+ if !os.SameFile(current, l.info) || !bytes.Equal(content, l.content) {
+ return nil
+ }
+ if err := os.Remove(l.path); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return fmt.Errorf("release the reference lock: %w", err)
+ }
+ return nil
}
// removeStaleStackLock removes the reference lock of the repository at path
// when a durable step of a process that died left it: git's writes would fail
// on it until housekeeping removes stale locks. A lock of git is never
-// removed.
+// removed. Only the owner calls it, when it takes the ownership: the durable
+// steps of the process that owned the storage before have ended with it.
func removeStaleStackLock(path string) error {
lockPath := filepath.Join(path, "reftable", "tables.list.lock")
content, err := os.ReadFile(lockPath)
@@ -538,7 +730,7 @@ func removeStaleStackLock(path string) error {
if err != nil {
return fmt.Errorf("read the reference lock: %w", err)
}
- if !bytes.Equal(content, []byte(stackLockContent)) {
+ if !bytes.HasPrefix(content, []byte(stackLockTag)) {
return nil
}
if err := os.Remove(lockPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
@@ -547,8 +739,24 @@ func removeStaleStackLock(path string) error {
return nil
}
+// removeStepTemporaries removes the temporary files that a durable step of a
+// process that died left in the repository at path.
+func removeStepTemporaries(path string) error {
+ leftovers, err := filepath.Glob(filepath.Join(path, stepTempPrefix+"*"))
+ if err != nil {
+ return err
+ }
+ for _, leftover := range leftovers {
+ if err := os.Remove(leftover); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return fmt.Errorf("remove a temporary file of a durable step: %w", err)
+ }
+ }
+ return nil
+}
+
// syncStack makes durable the tables of the stack not made durable yet in this
-// dirty period, then the list and the directory.
+// dirty period, then the list and the directory. synced keeps only the tables
+// the stack still lists.
func (d *Durability) syncStack(path string, st *refState) error {
reftable := filepath.Join(path, "reftable")
names, err := readTablesList(filepath.Join(reftable, "tables.list"))
@@ -557,12 +765,19 @@ func (d *Durability) syncStack(path string, st *refState) error {
}
d.mu.Lock()
+ listed := make(map[string]struct{}, len(names))
var pending []string
for _, name := range names {
+ listed[name] = struct{}{}
if _, ok := st.synced[name]; !ok {
pending = append(pending, name)
}
}
+ for name := range st.synced {
+ if _, ok := listed[name]; !ok {
+ delete(st.synced, name)
+ }
+ }
d.mu.Unlock()
for _, name := range pending {
@@ -618,13 +833,13 @@ func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) erro
if st.dirty {
d.dirty.Dec()
}
- d.inflight.Sub(float64(st.nInflight))
+ d.inflight.Sub(float64(st.nLive))
delete(d.repos, path)
}
return nil
}
-// Recovered is a dirty repository found at startup.
+// Recovered is a dirty repository found by Recover.
type Recovered struct {
Repo RepoRef
Path string
@@ -633,14 +848,15 @@ type Recovered struct {
Reboot bool
}
-// Recover reads the dirty markers of the storages. A marker of the running
-// boot makes its repository dirty again, to be settled, and a reference lock
-// that a durable step of the previous process left is removed. It returns
-// every repository found; the caller moves aside the ones of another boot
-// before serving. A marker that cannot be read is an error: its repository is
-// unknown, and serving could read a torn stack.
+// Recover reads the dirty markers of the storages; only the owner calls it,
+// when it takes the ownership. A marker of the running boot makes its
+// repository dirty again, to be settled, and what a durable step of the
+// previous owner left (its reference lock, its temporary files) is removed. It
+// returns every repository found; the caller moves aside the ones of another
+// boot before serving. A marker that cannot be read is an error: its
+// repository is unknown, and serving could read a torn stack.
func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovered, error) {
- if d == nil {
+ if !d.Deferring() {
return nil, nil
}
@@ -660,7 +876,10 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovere
for _, entry := range entries {
markerPath := filepath.Join(dir, entry.Name())
- if !strings.HasSuffix(entry.Name(), ".json") {
+ if entry.Name() == ownerLockName {
+ continue
+ }
+ if strings.HasPrefix(entry.Name(), markerTempPrefix) {
// A temporary file of a marker write that did not complete: the
// write never returned, so no reference write relied on it.
if err := os.Remove(markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
@@ -668,6 +887,9 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovere
}
continue
}
+ if !strings.HasSuffix(entry.Name(), ".json") {
+ return nil, fmt.Errorf("unexpected file %q in the dirty marker directory", markerPath)
+ }
content, err := os.ReadFile(markerPath)
if err != nil {
@@ -688,6 +910,9 @@ func (d *Durability) Recover(ctx context.Context, storages []string) ([]Recovere
if err := removeStaleStackLock(path); err != nil {
return nil, err
}
+ if err := removeStepTemporaries(path); err != nil {
+ return nil, err
+ }
d.mu.Lock()
st := d.stateLocked(path)
if !st.dirty {
@@ -712,15 +937,38 @@ func (d *Durability) CountUntrusted(reason string) {
d.untrusted.WithLabelValues(reason).Inc()
}
+// ensureDir creates the directory dir when it does not exist, and makes its
+// entry and the entry of a parent it created durable.
+func (d *Durability) ensureDir(dir string) error {
+ if _, err := os.Stat(dir); err == nil {
+ return nil
+ } else if !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+
+ parent := filepath.Dir(dir)
+ _, parentErr := os.Stat(parent)
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ return err
+ }
+ if err := d.syncPath(parent); err != nil {
+ return err
+ }
+ if errors.Is(parentErr, fs.ErrNotExist) {
+ return d.syncPath(filepath.Dir(parent))
+ }
+ return nil
+}
+
// writeDurably writes content to path through a temporary file in the same
// directory, made durable before and after the rename.
func (d *Durability) writeDurably(path string, content []byte) (returnedErr error) {
dir := filepath.Dir(path)
- if err := os.MkdirAll(dir, 0o700); err != nil {
+ if err := d.ensureDir(dir); err != nil {
return err
}
- tmp, err := os.CreateTemp(dir, ".marker-*.tmp")
+ tmp, err := os.CreateTemp(dir, markerTempPrefix+"*.tmp")
if err != nil {
return err
}
@@ -735,7 +983,7 @@ func (d *Durability) writeDurably(path string, content []byte) (returnedErr erro
return err
}
if err := d.sync(tmp); err != nil {
- return err
+ return writebackError{err: err}
}
if err := tmp.Close(); err != nil {
return err
@@ -746,14 +994,18 @@ func (d *Durability) writeDurably(path string, content []byte) (returnedErr erro
return d.syncPath(dir)
}
-// syncPath makes the file or directory at path durable.
+// syncPath makes the file or directory at path durable. A failed open is
+// returned as it is; a failed fsync is a writebackError.
func (d *Durability) syncPath(path string) error {
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
- return d.sync(f)
+ if err := d.sync(f); err != nil {
+ return writebackError{err: err}
+ }
+ return nil
}
// readTablesList returns the table names of a reftable stack.
@@ -779,8 +1031,10 @@ func (d *Durability) Describe(descs chan<- *prometheus.Desc) {
d.marked.Describe(descs)
d.cleared.Describe(descs)
d.untrusted.Describe(descs)
+ d.stepErrors.Describe(descs)
d.dirty.Describe(descs)
d.inflight.Describe(descs)
+ d.owning.Describe(descs)
}
// Collect implements prometheus.Collector.
@@ -788,6 +1042,8 @@ func (d *Durability) Collect(metrics chan<- prometheus.Metric) {
d.marked.Collect(metrics)
d.cleared.Collect(metrics)
d.untrusted.Collect(metrics)
+ d.stepErrors.Collect(metrics)
d.dirty.Collect(metrics)
d.inflight.Collect(metrics)
+ d.owning.Collect(metrics)
}
diff --git a/internal/graft/corestore/localrefs/durability_test.go b/internal/graft/corestore/localrefs/durability_test.go
index 09a2b6f..74434c1 100644
--- a/internal/graft/corestore/localrefs/durability_test.go
+++ b/internal/graft/corestore/localrefs/durability_test.go
@@ -86,10 +86,23 @@ func setupDurability(t *testing.T) durabilityFixture {
}
}
-func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration) *Durability {
+// unowned returns a durability of the fixture's storage that does not own it.
+func (f durabilityFixture) unowned(t *testing.T, bootID string, idle time.Duration) *Durability {
t.Helper()
d, err := NewDurability(f.locator, idle, WithBootID(bootID), WithSync(f.synced.sync))
require.NoError(t, err)
+ t.Cleanup(func() { require.NoError(t, d.Close()) })
+ return d
+}
+
+// durability returns a durability that owns the fixture's storage, as the
+// process that runs the durable steps. Close it before another one owns.
+func (f durabilityFixture) durability(t *testing.T, bootID string, idle time.Duration) *Durability {
+ t.Helper()
+ d := f.unowned(t, bootID, idle)
+ owned, err := d.TryOwn(f.ctx, []string{f.repo.Storage})
+ require.NoError(t, err)
+ require.True(t, owned)
return d
}
@@ -112,6 +125,7 @@ func TestMarkDirtyWritesOneDurableMarker(t *testing.T) {
t.Parallel()
f := setupDurability(t)
d := f.durability(t, "boot-a", 0)
+ before := len(f.synced.recorded())
require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
@@ -123,20 +137,71 @@ func TestMarkDirtyWritesOneDurableMarker(t *testing.T) {
require.Equal(t, []string{f.path}, d.Dirty())
// The marker file and its directory were made durable before MarkDirty
- // returned.
- recorded := f.synced.recorded()
+ // returned, once.
+ recorded := f.synced.recorded()[before:]
require.Len(t, recorded, 2)
require.Equal(t, filepath.Dir(f.markerPath(t, d)), recorded[1])
require.Equal(t, filepath.Join(f.cfg.Storages[0].Path, MarkerDirName), filepath.Dir(f.markerPath(t, d)))
}
+func TestOwnershipCreatesTheMarkerDirectoryDurably(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ f.durability(t, "boot-a", 0)
+
+ dir := filepath.Join(f.cfg.Storages[0].Path, MarkerDirName)
+ require.DirExists(t, dir)
+ require.Contains(t, f.synced.recorded(), filepath.Dir(dir), "the new directory's entry is durable")
+}
+
+// TestOneOwnerPerStorage is the overlap of a graceful upgrade: two processes
+// on one storage. Only the owner defers, settles and recovers; the other one
+// writes with fsync until the owner is gone.
+func TestOneOwnerPerStorage(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ storages := []string{f.repo.Storage}
+
+ old := f.durability(t, "boot-a", 0)
+ require.True(t, old.Deferring())
+ require.NoError(t, old.MarkDirty(f.ctx, f.repo, f.path))
+ old.Begin(f.path, key(1))
+
+ next := f.unowned(t, "boot-a", 0)
+ owned, err := next.TryOwn(f.ctx, storages)
+ require.NoError(t, err)
+ require.False(t, owned, "the old process owns the storage")
+ require.False(t, next.Deferring())
+ require.Equal(t, 0.0, testutil.ToFloat64(next.owning))
+
+ // The next process neither recovers nor settles: the old process's
+ // transaction is live, and only the old process counts it.
+ found, err := next.Recover(f.ctx, storages)
+ require.NoError(t, err)
+ require.Empty(t, found)
+ require.NoError(t, next.Settle(f.path, true))
+ require.FileExists(t, f.markerPath(t, old))
+
+ // The old process exits; the next one owns and recovers the marker.
+ require.NoError(t, old.Close())
+ owned, err = next.TryOwn(f.ctx, storages)
+ require.NoError(t, err)
+ require.True(t, owned)
+ require.True(t, next.Deferring())
+ found, err = next.Recover(f.ctx, storages)
+ require.NoError(t, err)
+ require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path}}, found)
+ require.NoError(t, next.Settle(f.path, false))
+ require.NoFileExists(t, f.markerPath(t, next))
+}
+
func TestSettleMakesTheStackDurableAndRemovesTheMarker(t *testing.T) {
t.Parallel()
f := setupDurability(t)
d := f.durability(t, "boot-a", 0)
require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
- require.NoError(t, d.Settle(f.path))
+ require.NoError(t, d.Settle(f.path, false))
require.NoFileExists(t, f.markerPath(t, d))
require.Empty(t, d.Dirty())
@@ -149,8 +214,11 @@ func TestSettleMakesTheStackDurableAndRemovesTheMarker(t *testing.T) {
}
require.Contains(t, recorded, filepath.Join(f.path, "reftable", "tables.list"))
require.Contains(t, recorded, filepath.Join(f.path, "reftable"))
- // The lock taken for the removal is released.
+ // The lock taken for the removal is released, and its temporary file gone.
require.NoFileExists(t, filepath.Join(f.path, "reftable", "tables.list.lock"))
+ leftovers, err := filepath.Glob(filepath.Join(f.path, stepTempPrefix+"*"))
+ require.NoError(t, err)
+ require.Empty(t, leftovers)
}
func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {
@@ -162,33 +230,55 @@ func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {
d := f.durability(t, "boot-a", time.Hour)
require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
- require.NoError(t, d.Settle(f.path))
+ require.NoError(t, d.Settle(f.path, false))
require.FileExists(t, f.markerPath(t, d))
// The stack was made durable all the same.
require.Contains(t, f.synced.recorded(), filepath.Join(f.path, "reftable", f.tables(t)[0]))
+
+ // The last step of a stopping process does not wait for idleness.
+ require.NoError(t, d.Settle(f.path, true))
+ require.NoFileExists(t, f.markerPath(t, d))
})
- t.Run("a transaction between Prepared and Committed", func(t *testing.T) {
+ t.Run("a live transaction", func(t *testing.T) {
t.Parallel()
f := setupDurability(t)
d := f.durability(t, "boot-a", 0)
require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
d.Begin(f.path, key(1))
- require.NoError(t, d.Settle(f.path))
- require.FileExists(t, f.markerPath(t, d))
+ require.NoError(t, d.Settle(f.path, true))
+ require.FileExists(t, f.markerPath(t, d), "not even the last step removes it")
// An end of another key does not count; the end of the transaction does.
d.End(f.path, key(2))
- require.NoError(t, d.Settle(f.path))
+ require.NoError(t, d.Settle(f.path, false))
require.FileExists(t, f.markerPath(t, d))
d.End(f.path, key(1))
- require.NoError(t, d.Settle(f.path))
+ require.NoError(t, d.Settle(f.path, false))
require.NoFileExists(t, f.markerPath(t, d))
require.Equal(t, 0.0, testutil.ToFloat64(d.inflight))
})
+ t.Run("two live transactions with the same key", func(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+
+ // Whichever of the two ends first (an abort before Prepared, say), the
+ // other one is still counted.
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ d.Begin(f.path, key(1))
+ d.Begin(f.path, key(1))
+ d.End(f.path, key(1))
+ require.NoError(t, d.Settle(f.path, false))
+ require.FileExists(t, f.markerPath(t, d))
+ d.End(f.path, key(1))
+ require.NoError(t, d.Settle(f.path, false))
+ require.NoFileExists(t, f.markerPath(t, d))
+ })
+
t.Run("git holds the reference lock", func(t *testing.T) {
t.Parallel()
f := setupDurability(t)
@@ -198,12 +288,42 @@ func TestSettleKeepsTheMarkerWhileWritesMayGoOn(t *testing.T) {
lock := filepath.Join(f.path, "reftable", "tables.list.lock")
require.NoError(t, os.WriteFile(lock, nil, 0o666))
- require.NoError(t, d.Settle(f.path))
+ require.NoError(t, d.Settle(f.path, false))
require.FileExists(t, f.markerPath(t, d))
require.FileExists(t, lock, "the durable step removed a lock it did not take")
})
}
+// TestStackLockReleaseRemovesOnlyItsOwnLock verifies that the durable step
+// never removes a reference lock that is not the one it created: one that
+// someone else removed and git took again stays.
+func TestStackLockReleaseRemovesOnlyItsOwnLock(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ lockPath := filepath.Join(f.path, "reftable", "tables.list.lock")
+
+ lock, ok, err := lockStack(f.path)
+ require.NoError(t, err)
+ require.True(t, ok)
+ _, ok, err = lockStack(f.path)
+ require.NoError(t, err)
+ require.False(t, ok, "the lock is exclusive")
+
+ // Housekeeping removed the lock as stale, and another durable step took it
+ // (the freed inode may well be reused for it).
+ require.NoError(t, os.Remove(lockPath))
+ require.NoError(t, os.WriteFile(lockPath, []byte(stackLockTag+"0123456789abcdef\n"), 0o666))
+ require.NoError(t, lock.release())
+ require.FileExists(t, lockPath, "a lock of another holder, even with the same tag, is not removed")
+
+ require.NoError(t, os.Remove(lockPath))
+ lock, ok, err = lockStack(f.path)
+ require.NoError(t, err)
+ require.True(t, ok)
+ require.NoError(t, lock.release())
+ require.NoFileExists(t, lockPath)
+}
+
func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {
t.Parallel()
f := setupDurability(t)
@@ -213,7 +333,7 @@ func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {
writeback := errors.New("input/output error")
f.synced.fail(writeback)
- err := d.Settle(f.path)
+ err := d.Settle(f.path, false)
require.ErrorIs(t, err, ErrUntrusted)
require.ErrorIs(t, err, writeback)
var untrusted *UntrustedError
@@ -225,12 +345,52 @@ func TestSettleReportsAFailedWriteAsUntrusted(t *testing.T) {
require.FileExists(t, f.markerPath(t, d))
require.Equal(t, []string{f.path}, d.Dirty())
+ f.synced.fail(nil)
require.NoError(t, d.Forget(f.ctx, f.repo, f.path))
require.NoFileExists(t, f.markerPath(t, d))
require.Empty(t, d.Dirty())
require.Equal(t, 0.0, testutil.ToFloat64(d.dirty))
}
+// TestSettleTransientErrorIsNotUntrusted verifies that a failure to read the
+// stack (not a failed fsync) is retried: the repository is not distrusted.
+func TestSettleTransientErrorIsNotUntrusted(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+
+ list := filepath.Join(f.path, "reftable", "tables.list")
+ require.NoError(t, os.Rename(list, list+".away"))
+ err := d.Settle(f.path, false)
+ require.Error(t, err)
+ require.NotErrorIs(t, err, ErrUntrusted)
+ require.FileExists(t, f.markerPath(t, d))
+
+ require.NoError(t, os.Rename(list+".away", list))
+ require.NoError(t, d.Settle(f.path, false))
+ require.NoFileExists(t, f.markerPath(t, d))
+}
+
+// TestSettleForgetsTablesTheStackNoLongerLists verifies that the set of tables
+// made durable does not grow with tables compacted away.
+func TestSettleForgetsTablesTheStackNoLongerLists(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", time.Hour)
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+
+ d.mu.Lock()
+ d.repos[f.path].synced["0x000000000001-0x000000000001-compacted.ref"] = struct{}{}
+ d.mu.Unlock()
+
+ require.NoError(t, d.Settle(f.path, false))
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ require.Len(t, d.repos[f.path].synced, len(f.tables(t)))
+ require.NotContains(t, d.repos[f.path].synced, "0x000000000001-0x000000000001-compacted.ref")
+}
+
func TestSettleDropsTheMarkerOfARepositoryThatIsGone(t *testing.T) {
t.Parallel()
f := setupDurability(t)
@@ -240,7 +400,7 @@ func TestSettleDropsTheMarkerOfARepositoryThatIsGone(t *testing.T) {
require.NoError(t, os.Rename(f.path, f.path+".moved"))
t.Cleanup(func() { require.NoError(t, os.Rename(f.path+".moved", f.path)) })
- require.NoError(t, d.Settle(f.path))
+ require.NoError(t, d.Settle(f.path, false))
require.NoFileExists(t, f.markerPath(t, d))
require.Empty(t, d.Dirty())
}
@@ -248,9 +408,11 @@ func TestSettleDropsTheMarkerOfARepositoryThatIsGone(t *testing.T) {
func TestRecoverTellsAKernelRestartFromAProcessRestart(t *testing.T) {
t.Parallel()
f := setupDurability(t)
+ storages := []string{f.repo.Storage}
+
before := f.durability(t, "boot-a", time.Hour)
require.NoError(t, before.MarkDirty(f.ctx, f.repo, f.path))
- storages := []string{f.repo.Storage}
+ require.NoError(t, before.Close())
// The same boot: the process restarted, the kernel kept every write.
sameBoot := f.durability(t, "boot-a", 0)
@@ -258,11 +420,12 @@ func TestRecoverTellsAKernelRestartFromAProcessRestart(t *testing.T) {
require.NoError(t, err)
require.Equal(t, []Recovered{{Repo: f.repo, Path: f.path, Reboot: false}}, found)
require.Equal(t, []string{f.path}, sameBoot.Dirty())
- require.NoError(t, sameBoot.Settle(f.path))
+ require.NoError(t, sameBoot.Settle(f.path, false))
require.NoFileExists(t, f.markerPath(t, sameBoot))
// Another boot: the writes may not have reached the disk.
- require.NoError(t, f.durability(t, "boot-a", time.Hour).MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, sameBoot.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, sameBoot.Close())
otherBoot := f.durability(t, "boot-b", 0)
found, err = otherBoot.Recover(f.ctx, storages)
require.NoError(t, err)
@@ -270,36 +433,35 @@ func TestRecoverTellsAKernelRestartFromAProcessRestart(t *testing.T) {
require.Empty(t, otherBoot.Dirty())
}
-// TestRecoverRemovesTheLockOfADurableStepThatDied verifies that a reference
-// lock left by a durable step of a process that died is removed at startup,
-// and that a lock of git is not.
-func TestRecoverRemovesTheLockOfADurableStepThatDied(t *testing.T) {
+// TestRecoverRemovesWhatADurableStepThatDiedLeft verifies that the reference
+// lock and the temporary files of a durable step of a process that died are
+// removed when the next process owns the storage, and that a lock of git is
+// not.
+func TestRecoverRemovesWhatADurableStepThatDiedLeft(t *testing.T) {
t.Parallel()
f := setupDurability(t)
+ storages := []string{f.repo.Storage}
lock := filepath.Join(f.path, "reftable", "tables.list.lock")
- require.NoError(t, f.durability(t, "boot-a", 0).MarkDirty(f.ctx, f.repo, f.path))
- unlock, ok, err := lockStack(f.path)
+ dead := f.durability(t, "boot-a", 0)
+ require.NoError(t, dead.MarkDirty(f.ctx, f.repo, f.path))
+ _, ok, err := lockStack(f.path)
require.NoError(t, err)
require.True(t, ok)
- _, ok, err = lockStack(f.path)
- require.NoError(t, err)
- require.False(t, ok, "the lock is exclusive")
- entries, err := os.ReadDir(f.path)
- require.NoError(t, err)
- for _, entry := range entries {
- require.False(t, strings.HasPrefix(entry.Name(), ".graft-durable-step-"), "temporary file %s left", entry.Name())
- }
+ leftover := filepath.Join(f.path, stepTempPrefix+"123")
+ require.NoError(t, os.WriteFile(leftover, []byte(stackLockTag+"0123456789abcdef\n"), 0o600))
+ require.NoError(t, dead.Close())
- // The process dies holding the lock; the next one of the same boot removes it.
- _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, []string{f.repo.Storage})
+ next := f.durability(t, "boot-a", 0)
+ _, err = next.Recover(f.ctx, storages)
require.NoError(t, err)
require.NoFileExists(t, lock)
- require.Error(t, unlock(), "the lock was already removed")
+ require.NoFileExists(t, leftover)
+ require.NoError(t, next.Close())
// A lock of git is kept.
require.NoError(t, os.WriteFile(lock, []byte("0x000000000001-0x000000000003-abcdef01.ref\n"), 0o666))
- _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, []string{f.repo.Storage})
+ _, err = f.durability(t, "boot-a", 0).Recover(f.ctx, storages)
require.NoError(t, err)
require.FileExists(t, lock)
}
@@ -307,20 +469,25 @@ func TestRecoverRemovesTheLockOfADurableStepThatDied(t *testing.T) {
func TestRecoverRefusesACorruptMarker(t *testing.T) {
t.Parallel()
f := setupDurability(t)
+ storages := []string{f.repo.Storage}
d := f.durability(t, "boot-a", 0)
require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+ require.NoError(t, d.Close())
marker := f.markerPath(t, d)
- incomplete := filepath.Join(filepath.Dir(marker), ".marker-123.tmp")
+ incomplete := filepath.Join(filepath.Dir(marker), markerTempPrefix+"123.tmp")
require.NoError(t, os.WriteFile(incomplete, []byte("{"), 0o600))
- found, err := f.durability(t, "boot-b", 0).Recover(f.ctx, []string{f.repo.Storage})
+ next := f.durability(t, "boot-b", 0)
+ found, err := next.Recover(f.ctx, storages)
require.NoError(t, err)
require.Len(t, found, 1)
require.NoFileExists(t, incomplete, "an incomplete marker write is removed")
+ require.FileExists(t, filepath.Join(filepath.Dir(marker), ownerLockName), "the owner lock is not a marker")
+ require.NoError(t, next.Close())
require.NoError(t, os.WriteFile(marker, []byte(`{"storage":"`), 0o600))
- _, err = f.durability(t, "boot-b", 0).Recover(f.ctx, []string{f.repo.Storage})
+ _, err = f.durability(t, "boot-b", 0).Recover(f.ctx, storages)
require.ErrorContains(t, err, "corrupt dirty marker")
}
@@ -330,13 +497,18 @@ func TestNilDurabilityIsOff(t *testing.T) {
var d *Durability
require.False(t, d.Enabled())
+ require.False(t, d.Deferring())
+ owned, err := d.TryOwn(ctx, []string{"default"})
+ require.NoError(t, err)
+ require.False(t, owned)
require.NoError(t, d.MarkDirty(ctx, RepoRef{}, "path"))
- require.NoError(t, d.Settle("path"))
+ require.NoError(t, d.Settle("path", true))
d.Begin("path", key(1))
d.End("path", key(1))
found, err := d.Recover(ctx, []string{"default"})
require.NoError(t, err)
require.Empty(t, found)
+ require.NoError(t, d.Close())
}
// TestTornReftableStackReadsAsAnEmptyRepository is the experiment behind the
diff --git a/internal/graft/corestore/syncer/durability.go b/internal/graft/corestore/syncer/durability.go
index 32c5be5..4b468c6 100644
--- a/internal/graft/corestore/syncer/durability.go
+++ b/internal/graft/corestore/syncer/durability.go
@@ -6,6 +6,7 @@ import (
"fmt"
"os"
"path/filepath"
+ "sync"
"time"
"github.com/feder-cr/git-ai/internal/graft/corestore"
@@ -14,14 +15,111 @@ import (
"github.com/feder-cr/git-ai/internal/structerr"
)
+// finalStepTimeout bounds the durable steps that run when the process stops.
+const finalStepTimeout = 30 * time.Second
+
+// StartLocalReferences starts the deferred durability of the node's local
+// references (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE). gitaly serve
+// calls it before its listeners open.
+//
+// When the process takes the ownership of the storages' deferred durability,
+// it recovers them before returning (RecoverLocalReferences): a repository
+// that had writes not yet durable when the kernel restarted is moved aside, so
+// nothing reads its possibly torn stack. When another process owns them (a
+// graceful upgrade overlaps the old and the new process), that process
+// recovered them when it started, in this boot; this one writes references
+// with fsync and tries again every DurableStepInterval, recovering when it
+// takes the ownership.
+//
+// The returned stop ends the durable steps and runs a last one for every dirty
+// repository, without waiting for it to be idle, so that a clean stop leaves
+// no marker of a repository whose writes ended.
+func (s *Syncer) StartLocalReferences(ctx context.Context, storages []string) (stop func(), err error) {
+ d := s.core.RefDurability
+ if !d.Enabled() {
+ return func() {}, nil
+ }
+
+ owned, err := d.TryOwn(ctx, storages)
+ if err != nil {
+ return nil, fmt.Errorf("storage core: own the deferred durability: %w", err)
+ }
+ recovered := false
+ if owned {
+ if err := s.RecoverLocalReferences(ctx, storages); err != nil {
+ return nil, err
+ }
+ recovered = true
+ } else {
+ s.logger.Warn("storage core: another gitaly process owns the deferred durability of the storages; references are written with fsync until it exits")
+ }
+
+ loopCtx, cancel := context.WithCancel(context.WithoutCancel(ctx))
+ var wg sync.WaitGroup
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ s.runDurableSteps(loopCtx, storages, recovered)
+ }()
+
+ return func() {
+ cancel()
+ wg.Wait()
+ if !d.Deferring() {
+ return
+ }
+ finalCtx, cancelFinal := context.WithTimeout(context.WithoutCancel(ctx), finalStepTimeout)
+ defer cancelFinal()
+ for _, path := range d.Dirty() {
+ s.durableStep(finalCtx, path, true)
+ }
+ }, nil
+}
+
+// runDurableSteps runs, every DurableStepInterval until ctx ends, the durable
+// step of every dirty repository, or, while another process owns the deferred
+// durability, an attempt to take it (and the recovery once taken).
+func (s *Syncer) runDurableSteps(ctx context.Context, storages []string, recovered bool) {
+ d := s.core.RefDurability
+ ticker := time.NewTicker(localrefs.DurableStepInterval)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ }
+
+ if !recovered {
+ owned, err := d.TryOwn(ctx, storages)
+ if err != nil {
+ s.logger.WithError(err).Error("storage core: own the deferred durability")
+ continue
+ }
+ if !owned {
+ continue
+ }
+ if err := s.RecoverLocalReferences(ctx, storages); err != nil {
+ s.logger.WithError(err).Error("storage core: recover the local references after taking the deferred durability; retried")
+ continue
+ }
+ recovered = true
+ s.logger.Info("storage core: this process owns the deferred durability of the storages now")
+ }
+
+ for _, path := range d.Dirty() {
+ s.durableStep(ctx, path, false)
+ }
+ }
+}
+
// RecoverLocalReferences moves aside every local repository that had reference
// writes that were not durable yet when the kernel restarted (doc/graft/PLAN7.md,
// LOCAL REFERENCE WRITES RULE): its reftable stack may be torn, and git reads a
// torn stack as an empty repository without an error. The next request
-// hydrates it from the database and the bucket. gitaly serve calls it before
-// its listeners open, so nothing reads such a repository before. A repository
-// dirty from a process restart of the same boot is kept: the kernel kept every
-// write, and the durable step settles it.
+// hydrates it from the database and the bucket. The process owns the deferred
+// durability. A repository dirty from a process restart of the same boot is
+// kept: the kernel kept every write, and the durable step settles it.
//
// Only a cache of the storage core is moved: the database must hold the
// repository with changesets, and the local repository must have its
@@ -39,7 +137,7 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)
continue
}
- cache, err := s.storageCoreCache(ctx, rec)
+ cache, err := s.storageCoreCache(ctx, rec.Repo, rec.Path)
if err != nil {
return fmt.Errorf("storage core: recover the local references of %s:%s: %w", rec.Repo.Storage, rec.Repo.RelativePath, err)
}
@@ -59,11 +157,11 @@ func (s *Syncer) RecoverLocalReferences(ctx context.Context, storages []string)
return nil
}
-// storageCoreCache reports whether the local repository of rec is a cache of
+// storageCoreCache reports whether the local repository at path is a cache of
// the storage core that a hydration can rebuild: the database holds it with
// changesets, and it has a watermark.
-func (s *Syncer) storageCoreCache(ctx context.Context, rec localrefs.Recovered) (bool, error) {
- r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: rec.Repo.Storage, RelativePath: rec.Repo.RelativePath})
+func (s *Syncer) storageCoreCache(ctx context.Context, repo localrefs.RepoRef, path string) (bool, error) {
+ r, err := s.core.Meta.GetRepository(ctx, corestore.RepoKey{Storage: repo.Storage, RelativePath: repo.RelativePath})
if errors.Is(err, corestore.ErrRepositoryNotFound) {
return false, nil
}
@@ -74,12 +172,12 @@ func (s *Syncer) storageCoreCache(ctx context.Context, rec localrefs.Recovered)
return false, nil
}
- if _, err := os.Lstat(rec.Path); errors.Is(err, os.ErrNotExist) {
+ if _, err := os.Lstat(path); errors.Is(err, os.ErrNotExist) {
return false, nil
} else if err != nil {
return false, err
}
- _, ok, err := gitops.ReadWatermark(rec.Path)
+ _, ok, err := gitops.ReadWatermark(path)
if err != nil {
// A watermark that cannot be read is no proof that the storage core wrote
// the repository: it is kept, as untracked() keeps it.
@@ -88,48 +186,46 @@ func (s *Syncer) storageCoreCache(ctx context.Context, rec localrefs.Recovered)
return ok, nil
}
-// RunDurableSteps runs the durable step of every dirty repository each interval
-// until ctx ends (doc/graft/PLAN7.md, LOCAL REFERENCE WRITES RULE).
-func (s *Syncer) RunDurableSteps(ctx context.Context, interval time.Duration) {
- if !s.core.RefDurability.Enabled() {
- return
- }
-
- ticker := time.NewTicker(interval)
- defer ticker.Stop()
- for {
- select {
- case <-ctx.Done():
- return
- case <-ticker.C:
- }
- for _, path := range s.core.RefDurability.Dirty() {
- s.durableStep(ctx, path)
- }
- }
-}
-
// durableStep makes the references of the repository at path durable under the
-// repository lock, and moves the repository aside at once when the kernel
-// reports that a write failed.
-func (s *Syncer) durableStep(ctx context.Context, path string) {
+// repository lock (final: without waiting for the repository to be idle). When
+// the kernel reports that a write failed, the repository is moved aside at
+// once, if it is a cache of the storage core; any other failure is retried at
+// the next step.
+func (s *Syncer) durableStep(ctx context.Context, path string, final bool) {
release, err := s.lock(ctx, path)
if err != nil {
return
}
defer release()
- err = s.core.RefDurability.Settle(path)
+ d := s.core.RefDurability
+ err = d.Settle(path, final)
+ if err == nil {
+ return
+ }
+
+ logger := s.logger.WithError(err).WithField("repository_path", path)
var untrusted *localrefs.UntrustedError
- if errors.As(err, &untrusted) {
- if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {
- s.logger.WithError(err).WithField("repository_path", path).
- Error("storage core: could not move aside a repository whose references may have lost writes")
- }
+ if !errors.As(err, &untrusted) {
+ d.CountStepError()
+ logger.Warn("storage core: durable step of the local references failed, retried at the next step")
return
}
- if err != nil {
- s.logger.WithError(err).WithField("repository_path", path).Error("storage core: durable step of the local references")
+
+ cache, cacheErr := s.storageCoreCache(ctx, untrusted.Repo, path)
+ if cacheErr != nil {
+ d.CountStepError()
+ logger.WithField("cache_check_error", cacheErr.Error()).
+ Error("storage core: the references of a repository may have lost writes; whether it can be hydrated again is not known yet, retried at the next step")
+ return
+ }
+ if !cache {
+ logger.Error("storage core: the references of a repository that is not a cache of the storage core may have lost writes; it is kept in place")
+ return
+ }
+ if err := s.distrustLocked(ctx, untrusted.Repo, path, localrefs.ReasonSyncFailed, untrusted); err != nil {
+ logger.WithField("move_error", err.Error()).
+ Error("storage core: could not move aside a repository whose references may have lost writes")
}
}
@@ -146,8 +242,8 @@ func (s *Syncer) distrust(ctx context.Context, repo localrefs.RepoRef, path, rea
// distrustLocked moves the local repository at path into the temporary
// directory of its storage and removes it there in the background, then drops
// its dirty marker: the next request hydrates it. The move is made durable
-// before the marker goes, so a crash in between moves it again. The caller
-// holds the repository lock.
+// (both directories, and the new directory's entry) before the marker goes, so
+// a crash in between moves it again. The caller holds the repository lock.
func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, path, reason string, cause error) error {
d := s.core.RefDurability
@@ -169,8 +265,10 @@ func (s *Syncer) distrustLocked(ctx context.Context, repo localrefs.RepoRef, pat
_ = os.Remove(stale)
return structerr.NewInternal("storage core: move aside an untrusted repository: %w", err)
}
- if err := syncDir(filepath.Dir(path)); err != nil {
- return structerr.NewInternal("storage core: make the move of an untrusted repository durable: %w", err)
+ for _, dir := range []string{filepath.Dir(path), stale, tmpDir} {
+ if err := syncDir(dir); err != nil {
+ return structerr.NewInternal("storage core: make the move of an untrusted repository durable: %w", err)
+ }
}
s.forgetPending(path)
diff --git a/internal/graft/corestore/syncer/export_test.go b/internal/graft/corestore/syncer/export_test.go
index 162b6c8..a6f05bb 100644
--- a/internal/graft/corestore/syncer/export_test.go
+++ b/internal/graft/corestore/syncer/export_test.go
@@ -8,8 +8,8 @@ func (s *Syncer) SetTestHook(hook func(point string)) {
s.testHook = hook
}
-// DurableStep runs one durable step of the repository at path, as
-// RunDurableSteps does every interval.
-func (s *Syncer) DurableStep(ctx context.Context, path string) {
- s.durableStep(ctx, path)
+// DurableStep runs one durable step of the repository at path, as the
+// durable-step loop does every interval (final: as the stop does).
+func (s *Syncer) DurableStep(ctx context.Context, path string, final bool) {
+ s.durableStep(ctx, path, final)
}
diff --git a/internal/graft/corestore/syncer/syncer.go b/internal/graft/corestore/syncer/syncer.go
index 429f47c..d55a668 100644
--- a/internal/graft/corestore/syncer/syncer.go
+++ b/internal/graft/corestore/syncer/syncer.go
@@ -1379,8 +1379,10 @@ func refTransaction(ctx context.Context, repo *localrepo.Repo, w refWriter, comm
return fmt.Errorf("reference backend: %w", err)
}
+ // Only while this process owns the deferred durability; the syncer's
+ // repository lock is held, and the ownership never goes back.
var durability []gitcmd.CmdOpt
- if w.core.RefDurability.Enabled() {
+ if w.core.RefDurability.Deferring() {
if err := w.core.RefDurability.MarkDirty(ctx, w.repo, w.path); err != nil {
return err
}
diff --git a/internal/graft/corestore/syncer/syncer_localrefs_test.go b/internal/graft/corestore/syncer/syncer_localrefs_test.go
index 0e378ff..38d33f5 100644
--- a/internal/graft/corestore/syncer/syncer_localrefs_test.go
+++ b/internal/graft/corestore/syncer/syncer_localrefs_test.go
@@ -7,6 +7,7 @@ import (
"strings"
"sync/atomic"
"testing"
+ "time"
"github.com/prometheus/client_golang/prometheus"
"github.com/stretchr/testify/require"
@@ -30,7 +31,9 @@ func (e *env) markers(t *testing.T) []string {
require.NoError(t, err)
var names []string
for _, entry := range entries {
- names = append(names, entry.Name())
+ if strings.HasSuffix(entry.Name(), ".json") {
+ names = append(names, entry.Name())
+ }
}
return names
}
@@ -77,7 +80,7 @@ func caughtUpDirty(t *testing.T, opts ...localrefs.DurabilityOption) (*env, core
t.Helper()
e := newEnv(t, envOptions{})
- coretest.EnableLocalRefs(t, e.core, e.locator, opts...)
+ coretest.EnableLocalRefs(t, e.core, e.cfg, opts...)
src := e.newSource(t)
c1 := e.writeCommit(t, src)
repo := e.register(t, e.key(localRelPath), src, c1)
@@ -102,7 +105,7 @@ func TestSyncerCatchUpMarksTheRepositoryDirty(t *testing.T) {
e.verifySynced(t, repo)
require.Equal(t, []string{e.localPath()}, e.core.RefDurability.Dirty())
- e.sy.DurableStep(e.ctx, e.localPath())
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
require.Empty(t, e.markers(t))
require.Empty(t, e.core.RefDurability.Dirty())
e.verifySynced(t, repo)
@@ -125,7 +128,7 @@ func TestSyncerMovesAsideARepositoryDirtyAtAKernelRestart(t *testing.T) {
require.Empty(t, strings.TrimSpace(string(torn)))
// The node restarts after a kernel restart.
- coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-b"))
+ coretest.EnableLocalRefs(t, e.core, e.cfg, localrefs.WithBootID("boot-b"))
e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)
require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))
@@ -145,7 +148,7 @@ func TestSyncerKeepsARepositoryDirtyAtAProcessRestart(t *testing.T) {
e, repo, _ := caughtUpDirty(t, localrefs.WithBootID("boot-a"))
- coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-a"))
+ coretest.EnableLocalRefs(t, e.core, e.cfg, localrefs.WithBootID("boot-a"))
e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)
require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))
@@ -153,7 +156,7 @@ func TestSyncerKeepsARepositoryDirtyAtAProcessRestart(t *testing.T) {
require.Equal(t, []string{e.localPath()}, e.core.RefDurability.Dirty())
e.verifySynced(t, repo)
- e.sy.DurableStep(e.ctx, e.localPath())
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
require.Empty(t, e.markers(t))
}
@@ -164,14 +167,14 @@ func TestSyncerRecoveryKeepsWhatIsNotACache(t *testing.T) {
t.Parallel()
e := newEnv(t, envOptions{})
- coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-a"))
+ coretest.EnableLocalRefs(t, e.core, e.cfg, localrefs.WithBootID("boot-a"))
pb, path := gittest.CreateRepository(t, e.ctx, e.cfg, gittest.CreateRepositoryConfig{
SkipCreationViaService: true,
ReferenceBackend: git.ReferenceBackendReftables,
})
require.NoError(t, e.core.RefDurability.MarkDirty(e.ctx, localrefs.RepoRef{Storage: pb.GetStorageName(), RelativePath: pb.GetRelativePath()}, path))
- coretest.EnableLocalRefs(t, e.core, e.locator, localrefs.WithBootID("boot-b"))
+ coretest.EnableLocalRefs(t, e.core, e.cfg, localrefs.WithBootID("boot-b"))
e.sy = syncer.New(e.core, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)
require.NoError(t, e.sy.RecoverLocalReferences(e.ctx, []string{e.storageName}))
@@ -196,7 +199,7 @@ func TestSyncerDurableStepMovesAsideOnAFailedWrite(t *testing.T) {
}))
failing.Store(true)
- e.sy.DurableStep(e.ctx, e.localPath())
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
failing.Store(false)
require.NoDirExists(t, e.localPath())
@@ -207,3 +210,83 @@ func TestSyncerDurableStepMovesAsideOnAFailedWrite(t *testing.T) {
require.Equal(t, float64(2), syncCount(t, e.sy, "hydrated"), "the first hydration and the one after the move")
e.verifySynced(t, repo)
}
+
+// TestSyncerDurableStepRetriesATransientError verifies that a durable step that
+// fails without a failed fsync (the stack cannot be read for a moment) keeps
+// the repository and its marker and is retried.
+func TestSyncerDurableStepRetriesATransientError(t *testing.T) {
+ t.Parallel()
+
+ e, repo, _ := caughtUpDirty(t)
+ list := filepath.Join(e.localPath(), "reftable", "tables.list")
+ require.NoError(t, os.Rename(list, list+".away"))
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
+ require.NoError(t, os.Rename(list+".away", list))
+
+ require.DirExists(t, e.localPath())
+ require.Len(t, e.markers(t), 1)
+ require.Equal(t, float64(0), e.untrusted(t, localrefs.ReasonSyncFailed))
+
+ e.sy.DurableStep(e.ctx, e.localPath(), false)
+ require.Empty(t, e.markers(t))
+ e.verifySynced(t, repo)
+}
+
+// TestSyncerSecondProcessWaitsForTheOwner is the overlap of a graceful
+// upgrade: while the old process owns the deferred durability, the new one
+// writes with fsync and touches no marker; once the old one is gone, the new
+// one owns, recovers and settles.
+func TestSyncerSecondProcessWaitsForTheOwner(t *testing.T) {
+ t.Parallel()
+
+ e, _, _ := caughtUpDirty(t)
+ storages := []string{e.storageName}
+
+ next := *e.core
+ durability, err := localrefs.NewDurability(e.locator, 0, localrefs.WithBootID(coretest.TestBootID))
+ require.NoError(t, err)
+ next.RefDurability = durability
+ t.Cleanup(func() { _ = durability.Close() })
+ nextSyncer := syncer.New(&next, e.logger, e.locator, e.repoFactory, e.gitCmdFactory)
+
+ stop, err := nextSyncer.StartLocalReferences(e.ctx, storages)
+ require.NoError(t, err)
+ defer stop()
+ require.False(t, durability.Deferring(), "the old process owns the storage")
+ require.Len(t, e.markers(t), 1)
+
+ // The old process exits.
+ require.NoError(t, e.core.RefDurability.Close())
+
+ // The next process tries every durable step interval.
+ require.Eventually(t, func() bool { return durability.Deferring() && len(e.markers(t)) == 0 },
+ 30*time.Second, 10*time.Millisecond, "the next process owns and settles the marker")
+}
+
+// TestSyncerStopSettlesEveryRepository verifies that a clean stop removes the
+// marker of a repository whose writes ended, without waiting for it to be idle.
+func TestSyncerStopSettlesEveryRepository(t *testing.T) {
+ t.Parallel()
+
+ e := newEnv(t, envOptions{})
+ e.core.RefAdmission = localrefs.NewAdmission(10*time.Second, 10*time.Second)
+ durability, err := localrefs.NewDurability(e.locator, time.Hour, localrefs.WithBootID(coretest.TestBootID))
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = durability.Close() })
+ e.core.RefDurability = durability
+
+ stop, err := e.sy.StartLocalReferences(e.ctx, []string{e.storageName})
+ require.NoError(t, err)
+ require.True(t, durability.Deferring())
+
+ src := e.newSource(t)
+ c1 := e.writeCommit(t, src)
+ repo := e.register(t, e.key(localRelPath), src, c1)
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ e.advanceMain(t, repo.ID, src, c1)
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ require.Len(t, e.markers(t), 1, "the repository is not idle for an hour")
+
+ stop()
+ require.Empty(t, e.markers(t))
+}
diff --git a/internal/graft/corestore/voter/voter.go b/internal/graft/corestore/voter/voter.go
index 33d6abf..e3d67bd 100644
--- a/internal/graft/corestore/voter/voter.go
+++ b/internal/graft/corestore/voter/voter.go
@@ -176,6 +176,8 @@ func (v *Voter) Preparing(ctx context.Context, payload gitcmd.HooksPayload, chan
return
}
+ // The transaction is live from now on (it ends in Committed or Aborted).
+ v.begin(ctx, payload, changes)
// Deferred first, so that it runs last: the admission is waited for after
// the pack work, when git is about to take the reference lock.
defer v.admit(ctx, payload, changes)
@@ -797,11 +799,38 @@ func (v *Voter) localRefs() bool {
}
// DeferredReferenceDurability reports whether the reference writes of the
-// transactions this voter commits may run without fsync: authoritative mode
-// with the deferred durability of the node on (doc/graft/PLAN7.md, LOCAL
-// REFERENCE WRITES RULE). The hook manager asks it for UpdateWithHooks.
+// transactions this voter commits may run without fsync: authoritative mode,
+// and this process owns the deferred durability of the node (doc/graft/PLAN7.md,
+// LOCAL REFERENCE WRITES RULE). The hook manager asks it for UpdateWithHooks.
+// It goes from false to true at most once in a process.
func (v *Voter) DeferredReferenceDurability() bool {
- return v != nil && v.core.Authoritative() && v.core.RefDurability.Enabled()
+ return v != nil && v.core.Authoritative() && v.core.RefDurability.Deferring()
+}
+
+// Denied records the start of a transaction that the standalone policy denies:
+// it is live until its Aborted phase, like any other (see begin).
+func (v *Voter) Denied(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {
+ if v == nil || !v.core.Enabled() || payload.Repo == nil {
+ return
+ }
+ v.begin(ctx, payload, changes)
+}
+
+// begin counts the transaction as live for the deferred durability: from
+// before git takes the reference lock until Committed or Aborted, it may write
+// a table or compact the stack, so the durable step keeps the repository's
+// marker meanwhile.
+func (v *Voter) begin(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {
+ if !v.core.Authoritative() || !v.core.RefDurability.Enabled() {
+ return
+ }
+
+ repoPath, err := v.repoPath(ctx, payload)
+ if err != nil {
+ v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path of a starting transaction")
+ return
+ }
+ v.core.RefDurability.Begin(repoPath, localrefs.Key(pendingKey(payload.Repo, changes)))
}
// admit waits for the admission of the transaction to the reference lock of
@@ -822,11 +851,11 @@ func (v *Voter) admit(ctx context.Context, payload gitcmd.HooksPayload, changes
}
// deferDurability lets git write the references of a transaction that the
-// database committed without fsync (authoritative mode with deferred
-// durability): the repository gets its durable dirty marker first, and the
-// transaction counts as in flight until its Committed phase, because git may
-// still write its table and compact the stack until then. It runs in the
-// Prepared phase, under git's reference lock and before git writes anything.
+// database committed without fsync (authoritative mode, and this process owns
+// the deferred durability): the repository gets its durable dirty marker
+// first. The transaction is live since its start (begin), so the durable step
+// keeps the marker until it ends. It runs in the Prepared phase, under git's
+// reference lock and before git writes anything.
func (v *Voter) deferDurability(ctx context.Context, payload gitcmd.HooksPayload, changes []byte, shadow bool) error {
if shadow || !v.DeferredReferenceDurability() {
return nil
@@ -837,11 +866,7 @@ func (v *Voter) deferDurability(ctx context.Context, payload gitcmd.HooksPayload
return err
}
repoKey := corestore.KeyOf(payload.Repo)
- if err := v.core.RefDurability.MarkDirty(ctx, localrefs.RepoRef{Storage: repoKey.Storage, RelativePath: repoKey.RelativePath}, repoPath); err != nil {
- return err
- }
- v.core.RefDurability.Begin(repoPath, localrefs.Key(pendingKey(payload.Repo, changes)))
- return nil
+ return v.core.RefDurability.MarkDirty(ctx, localrefs.RepoRef{Storage: repoKey.Storage, RelativePath: repoKey.RelativePath}, repoPath)
}
// Aborted tells the voter that git aborted the transaction.
@@ -863,14 +888,13 @@ func (v *Voter) Aborted(ctx context.Context, payload gitcmd.HooksPayload, change
key := pendingKey(payload.Repo, changes)
- // A transaction aborted after its Prepared phase stays in flight for the
- // deferred durability (see localrefs.Durability.End): its repository stays
- // dirty.
+ // git wrote nothing more of the transaction once it reports the abort.
if v.localRefs() {
if repoPath, err := v.repoPath(ctx, payload); err != nil {
v.logger.WithError(err).WarnContext(ctx, "storage core: resolving the repository path after an abort, the admission ends at its bound")
} else {
v.core.RefAdmission.Release(repoPath, localrefs.Key(key))
+ v.core.RefDurability.End(repoPath, localrefs.Key(key))
}
}
diff --git a/internal/graft/corestore/voter/voter_localrefs_test.go b/internal/graft/corestore/voter/voter_localrefs_test.go
index 2e1901a..35a9109 100644
--- a/internal/graft/corestore/voter/voter_localrefs_test.go
+++ b/internal/graft/corestore/voter/voter_localrefs_test.go
@@ -4,6 +4,7 @@ import (
"context"
"os"
"path/filepath"
+ "strings"
"testing"
"github.com/stretchr/testify/require"
@@ -46,7 +47,9 @@ func (f *fixture) markers(t *testing.T) []string {
require.NoError(t, err)
var names []string
for _, entry := range entries {
- names = append(names, entry.Name())
+ if strings.HasSuffix(entry.Name(), ".json") {
+ names = append(names, entry.Name())
+ }
}
return names
}
@@ -74,7 +77,7 @@ func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {
// git may still write and compact until Committed: the durable step keeps
// the marker.
- require.NoError(t, f.core.RefDurability.Settle(f.repoPath))
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
require.Len(t, f.markers(t), 1)
gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName(ref), f.a)
@@ -82,14 +85,13 @@ func TestLocalReferenceWritesOfACommittedTransaction(t *testing.T) {
require.False(t, f.admissionHeld(t), "Committed releases the admission")
require.Len(t, f.applier.calls, 1, "the local commit is still noted")
- require.NoError(t, f.core.RefDurability.Settle(f.repoPath))
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
require.Empty(t, f.markers(t))
}
// TestLocalReferenceWritesOfAnAbortedTransaction verifies that an abort after
-// Prepared releases the admission but leaves the repository dirty: the
-// transaction cannot be told from another with the same input, so it is never
-// counted as ended.
+// Prepared releases the admission and ends the transaction: the repository is
+// settled once nothing else is live.
func TestLocalReferenceWritesOfAnAbortedTransaction(t *testing.T) {
t.Parallel()
@@ -101,8 +103,32 @@ func TestLocalReferenceWritesOfAnAbortedTransaction(t *testing.T) {
f.voter.Aborted(f.ctx, f.payload, input)
require.False(t, f.admissionHeld(t))
- require.NoError(t, f.core.RefDurability.Settle(f.repoPath))
- require.Len(t, f.markers(t), 1)
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
+ require.Empty(t, f.markers(t))
+}
+
+// TestLocalReferenceWritesOfTransactionsWithTheSameInput verifies that two
+// live transactions with the same input are both counted: the one denied by
+// the policy and aborted does not end the other one, which may still write.
+func TestLocalReferenceWritesOfTransactionsWithTheSameInput(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
+ input := line(f.zero, f.a.String(), "refs/heads/main")
+
+ f.voter.Preparing(f.ctx, f.payload, input)
+ require.NoError(t, f.prepared(input))
+
+ // The same input from an agent the policy denies.
+ f.voter.Denied(f.ctx, f.payload, input)
+ f.voter.Aborted(f.ctx, f.payload, input)
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, true))
+ require.Len(t, f.markers(t), 1, "the first transaction may still write")
+
+ gittest.WriteRef(t, f.cfg, f.repoPath, git.ReferenceName("refs/heads/main"), f.a)
+ f.voter.Committed(f.ctx, f.payload, input)
+ require.NoError(t, f.core.RefDurability.Settle(f.repoPath, false))
+ require.Empty(t, f.markers(t))
}
// TestLocalReferenceWritesOfARejectedTransaction verifies that a transaction
@@ -131,9 +157,11 @@ func TestLocalReferenceWritesWhenTheMarkerCannotBeWritten(t *testing.T) {
t.Parallel()
f := newFixture(t, fixtureOptions{mode: corestore.ModeAuthoritative, localRefs: true})
+ // The marker directory exists (the ownership created it); no file can be
+ // created in it.
markerDir := filepath.Join(f.cfg.Storages[0].Path, localrefs.MarkerDirName)
- require.NoError(t, os.MkdirAll(filepath.Dir(markerDir), 0o755))
- require.NoError(t, os.WriteFile(markerDir, nil, 0o644))
+ require.NoError(t, os.Chmod(markerDir, 0o500))
+ t.Cleanup(func() { require.NoError(t, os.Chmod(markerDir, 0o700)) })
input := line(f.zero, f.a.String(), "refs/heads/main")
err := f.prepared(input)
diff --git a/internal/graft/corestore/voter/voter_test.go b/internal/graft/corestore/voter/voter_test.go
index 845d120..71e445d 100644
--- a/internal/graft/corestore/voter/voter_test.go
+++ b/internal/graft/corestore/voter/voter_test.go
@@ -134,7 +134,7 @@ func newFixture(t *testing.T, opts fixtureOptions) *fixture {
logger := testhelper.NewLogger(t)
locator := config.NewLocator(cfg)
if opts.localRefs {
- coretest.EnableLocalRefs(t, core, locator)
+ coretest.EnableLocalRefs(t, core, cfg)
}
cache := catfile.NewCache(cfg)
t.Cleanup(cache.Stop)
diff --git a/internal/graft/e2e/storage_localrefs_test.go b/internal/graft/e2e/storage_localrefs_test.go
index 768c93f..48eb1c1 100644
--- a/internal/graft/e2e/storage_localrefs_test.go
+++ b/internal/graft/e2e/storage_localrefs_test.go
@@ -93,6 +93,22 @@ func newRealNode(t *testing.T) *realNode {
}
}
+// sibling returns a second server of the same storage, database and bucket on
+// its own socket, as the new process of a graceful upgrade is.
+func (n *realNode) sibling(t *testing.T) *realNode {
+ t.Helper()
+
+ s := *n
+ s.cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)
+ s.clientCfg = s.cfg
+ s.clientCfg.SocketPath = "unix://" + s.cfg.SocketPath
+ s.cfgPath = testcfg.WriteTemporaryGitalyConfigFile(t, s.cfg)
+ s.logPath = filepath.Join(t.TempDir(), "gitaly.log")
+ s.setupCtx = testhelper.MergeOutgoingMetadata(s.ctx, testcfg.GitalyServersMetadataFromCfg(t, s.clientCfg))
+ s.cmd, s.exited = nil, nil
+ return &s
+}
+
// start starts the server process with the extra environment env and waits
// until it serves.
func (n *realNode) start(t *testing.T, env ...string) {
@@ -205,7 +221,9 @@ func (n *realNode) markers(t *testing.T) []string {
require.NoError(t, err)
var names []string
for _, entry := range entries {
- names = append(names, filepath.Join(n.storagePath(), localrefs.MarkerDirName, entry.Name()))
+ if strings.HasSuffix(entry.Name(), ".json") {
+ names = append(names, filepath.Join(n.storagePath(), localrefs.MarkerDirName, entry.Name()))
+ }
}
return names
}
@@ -363,6 +381,47 @@ func TestStorageLocalReferencesAfterAKernelRestart(t *testing.T) {
require.Contains(t, n.log(t), "moved aside a local repository whose references may have lost writes")
}
+// TestStorageLocalReferencesDuringAGracefulUpgrade runs a second server on
+// the same storage while the first one lives, as a graceful upgrade does: the
+// first one owns the deferred durability, the second writes with fsync and
+// leaves the markers alone. When the first one dies, the second owns, settles
+// and defers. Every write is served throughout.
+func TestStorageLocalReferencesDuringAGracefulUpgrade(t *testing.T) {
+ t.Parallel()
+
+ old := newRealNode(t)
+ old.start(t)
+ rel := old.createRepository(t)
+ base, err := commitFile(old.ctx, old.agent(t, rel), workerName(0), "main", "", "README.md", "base\n")
+ require.NoError(t, err)
+ written := writeBranches(t, old.agent(t, rel), old.ctx, base, "old", 3)
+ written["main"] = base
+ require.Len(t, old.markers(t), 1)
+
+ next := old.sibling(t)
+ next.start(t)
+ require.Contains(t, next.log(t), "another gitaly process owns the deferred durability")
+ for branch, commit := range writeBranches(t, next.agent(t, rel), next.ctx, base, "during", 3) {
+ written[branch] = commit
+ }
+ next.requireServed(t, rel, written)
+
+ // The old process owns the marker and settles it once the repository is
+ // idle; the new process's writes were made with fsync and need none.
+ require.Eventually(t, func() bool { return len(old.markers(t)) == 0 }, 30*time.Second, 50*time.Millisecond)
+
+ old.kill(t)
+ require.Eventually(t, func() bool {
+ return strings.Contains(next.log(t), "this process owns the deferred durability of the storages now")
+ }, 30*time.Second, 50*time.Millisecond)
+
+ for branch, commit := range writeBranches(t, next.agent(t, rel), next.ctx, base, "after", 2) {
+ written[branch] = commit
+ }
+ require.Len(t, next.markers(t), 1, "the new owner writes without fsync")
+ next.requireServed(t, rel, written)
+}
+
// TestStorageLocalReferencesStackDepthIsBounded writes many references
// without fsync from concurrent writers and checks that git's compaction keeps
// the stack shallow: the durable step only makes it durable, compaction stays