Skip to content

Git designed for AI agents.

changelog, hook: fixes from the review of the PLAN9 write path

Federico Elia42d63f01bcd5
diff --git a/doc/graft/PLAN9.md b/doc/graft/PLAN9.md
index a5621fc..52da6e6 100644
--- a/doc/graft/PLAN9.md
+++ b/doc/graft/PLAN9.md
@@ -395,8 +395,12 @@ update hook and update-ref); batch publishing (pack mode) and unbatched publishi
## 4. Invariants (each has a test, section 10)
-* L1 The changeset log is dense and ordered: sequences are consecutive, assigned in
- ticket order, across groups and across processes.
+* L1 The changeset log is dense and ordered: sequences are consecutive across groups
+ and across processes (the log's lock file serializes the appends of every process),
+ and within the process that serves the hooks they are assigned in ticket order. A
+ transaction whose ticket expired before its committed phase is the exception: it is
+ recorded at once, after the transactions that did not wait for it, and counted
+ (`graft_changelog_unordered_records_total`).
* L2 No RPC returns before its entries are durable, and no reader returns an entry
that is not durable.
* L3 A reader with a cursor never skips an entry, including across a crash before
@@ -425,7 +429,15 @@ update hook and update-ref); batch publishing (pack mode) and unbatched publishi
| housekeeping removes a fan-out directory after the link | its descriptor is fsynced; the objects are in a pack |
| crash during pass 1 | any prefix of the links; the quarantine is removed; unreferenced objects at most, as today |
| crash during pass 2 | links durable up to the last committed journal transaction, in order |
-| reader before the first append of the process | loads the state, fsyncs the file, reads up to its valid end |
+| reader before the first append of the process | loads the state, fsyncs the file and the log's directories, reads up to its valid end (amended after review: the directories were not synced, and a writer that crashed between its file and its directory fsync left entries a reader served that a crash could take back) |
+| a committed phase runs after its ticket expired | (amended after review) the ticket is found and made ready in one step under the queue's lock; an expired or released ticket is not found, so the transaction is appended at once, out of order, and counted; never taken for written (the first version could, in the window between finding the ticket and making it ready, and lost the entries silently) |
+| a transaction with the same hook input in another process | the ticket key holds the invocation of the git process, or of the `UserCommitFiles`/land call that drives the hook in process, so an abort or a commit never takes another transaction's ticket |
+| the group write panics | the panic is the group's error: every member reports its updates lost, the queue stops writing and goes on |
+| the log's lock file is not granted (a holder that never releases it) | the append or the read fails after `lockWaitTimeout` (30 s); the append's updates are reported lost |
+| an fsync never returns (hung I/O) | not bounded: its group's members wait with it, and later appends fail at `lockWaitTimeout` |
+| the committed phase cannot resolve the repository path | the ticket is released (queues are kept by storage and relative path) and the updates are reported lost |
+| another process's cache overlaps (graceful restart) | a cache that knows of bytes after its last entry is never trusted by size: the state is loaded again, so an append never cuts off another process's durable entries (amended after review); a cache without such bytes is also checked against the last entry |
+| the storage mode changes between the prepared and the committed phase | cannot: the mode is the storage core's configuration, fixed for the life of the process |
## 6. Config and API changes (exact)
@@ -459,6 +471,9 @@ and Markdown output.
fsync.
* `graft_changeset_ticket_wait_seconds` (histogram): time from the committed phase
to its entry being durable.
+* `graft_changelog_unordered_records_total` (counter, added after review): committed
+ transactions recorded without their place in the order, because their ticket
+ expired before their committed phase or they took none.
## 8. Work packages
diff --git a/internal/git/gitcmd/hooks_payload.go b/internal/git/gitcmd/hooks_payload.go
index 166995b..e414986 100644
--- a/internal/git/gitcmd/hooks_payload.go
+++ b/internal/git/gitcmd/hooks_payload.go
@@ -139,12 +139,13 @@ type ChangelogMeta struct {
// Rewritten is true when the updates rewrite history.
Rewritten bool `json:"rewritten,omitempty"`
// Invocation identifies the git process that runs the reference-transaction
- // hook with this payload. It is a random value set for each git command that
- // git runs the hook for, so the recording hook can tell the transactions of
- // one process apart from those of concurrent processes. Git runs two
- // transactions to delete a packed reference, and the recording hook pairs
- // them by this value. It is empty when the payload drives the hook in process
- // or when the changeset log is disabled.
+ // hook with this payload, or the call of updateref.UpdaterWithHooks that drives
+ // the hook in process. It is a random value set for each git command that git
+ // runs the hook for and for each such call, so the recording hook can tell the
+ // transactions of one process apart from those of concurrent processes, also
+ // when they have the same updates. Git runs two transactions to delete a
+ // packed reference, and the recording hook pairs them by this value. It is
+ // empty when the changeset log is disabled.
Invocation string `json:"invocation,omitempty"`
// MVCCCommit is the commit handle (see mvcc.CommitHandle) of the MVCC cache of
// the request that builds the payload. On a repository that uses the MVCC
diff --git a/internal/gitaly/hook/changeset_order.go b/internal/gitaly/hook/changeset_order.go
index 39f85a9..5daba95 100644
--- a/internal/gitaly/hook/changeset_order.go
+++ b/internal/gitaly/hook/changeset_order.go
@@ -2,9 +2,12 @@ package hook
import (
"crypto/sha256"
+ "fmt"
+ "runtime/debug"
"sync"
"time"
+ "github.com/feder-cr/git-ai/internal/gitaly/storage"
"github.com/feder-cr/git-ai/internal/graft/changelog"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
@@ -44,13 +47,24 @@ var (
Name: "graft_changelog_records_lost_total",
Help: "Committed reference updates that the changeset log does not record, because writing them failed.",
})
+
+ changelogUnorderedRecordsTotal = promauto.NewCounter(prometheus.CounterOpts{
+ Name: "graft_changelog_unordered_records_total",
+ Help: "Committed reference transactions recorded in the changeset log without their place in the order: their ticket expired before their committed phase, or they took none.",
+ })
)
+// appendChangesetGroup writes a group of records to the changeset log. Tests replace it.
+var appendChangesetGroup = changelog.AppendGroup
+
// changesetTicket is the place of a prepared reference transaction in the order of the changeset
// log of its repository.
type changesetTicket struct {
- // key identifies the transaction: the digest of the updates the hook was given. Git gives
- // the same updates to every phase of a transaction.
+ // key identifies the transaction: the digest of the invocation of the git process that runs
+ // it and of the updates the hook was given (changesetTicketKey). Git gives the same updates to
+ // every phase of a transaction, and the invocation tells apart the transactions of different
+ // processes, so that a transaction with the same updates as another never takes, releases or
+ // records the other's ticket.
key [sha256.Size]byte
// done is closed when the entries of the transaction were written (see err), or when the
// transaction was aborted, expired or records nothing.
@@ -98,46 +112,81 @@ func (q *changesetQueue) signal() {
// The tickets live in the process that serves the hook calls, which is the Gitaly process that
// spawned Git. A ticket whose transaction never reports its outcome expires after
// changesetTicketTimeout, after which the later transactions record without waiting for it.
+//
+// A ticket is in one of three states, and every change of state happens under the lock: queued
+// (in the queue, without record), ready (in the queue, with its record: only the group writer
+// removes it, once its entries are written), or gone (out of the queue: written, aborted,
+// released or expired). The committed phase finds its queued ticket and makes it ready in one
+// step (recordChangeset), so a ticket that expired or was released before that step is never
+// mistaken for a written one: the transaction is then recorded at once, out of order, and counted.
+//
+// The queues are kept by repository (changesetQueueKey), not by path, so that a committed phase
+// that cannot resolve the path of its repository can still give up its place.
var changesetTickets = struct {
sync.Mutex
queues map[string]*changesetQueue
}{queues: map[string]*changesetQueue{}}
-// changesetTicketKey returns the key of the transaction with the given hook input.
-func changesetTicketKey(changes []byte) [sha256.Size]byte {
- return sha256.Sum256(changes)
+// changesetQueueKey returns the key of the queue of the repository: its storage and relative
+// path, which every hook payload of the repository carries, whatever its object directories.
+func changesetQueueKey(repo storage.Repository) string {
+ return repo.GetStorageName() + "\x00" + repo.GetRelativePath()
}
-// changesetQueueLocked returns the queue of the repository at repoPath, creating it. The caller
-// must hold changesetTickets.
-func changesetQueueLocked(repoPath string) *changesetQueue {
- q, ok := changesetTickets.queues[repoPath]
+// changesetTicketKey returns the key of the transaction that the git process with the given
+// invocation runs with the given hook input.
+func changesetTicketKey(invocation string, changes []byte) [sha256.Size]byte {
+ digest := sha256.New()
+ digest.Write([]byte(invocation))
+ digest.Write([]byte{0})
+ digest.Write(changes)
+
+ var key [sha256.Size]byte
+ digest.Sum(key[:0])
+ return key
+}
+
+// changesetQueueLocked returns the queue with the given key, creating it. The caller must hold
+// changesetTickets.
+func changesetQueueLocked(queueKey string) *changesetQueue {
+ q, ok := changesetTickets.queues[queueKey]
if !ok {
q = &changesetQueue{changed: make(chan struct{})}
- changesetTickets.queues[repoPath] = q
+ changesetTickets.queues[queueKey] = q
}
return q
}
-// dropIfIdleLocked forgets the queue of repoPath when it holds nothing. The caller must hold
-// changesetTickets.
-func dropIfIdleLocked(repoPath string, q *changesetQueue) {
- if len(q.tickets) == 0 && !q.writing && changesetTickets.queues[repoPath] == q {
- delete(changesetTickets.queues, repoPath)
+// dropIfIdleLocked forgets the queue q with the given key when it holds nothing. The caller must
+// hold changesetTickets.
+func dropIfIdleLocked(queueKey string, q *changesetQueue) {
+ if len(q.tickets) == 0 && !q.writing && changesetTickets.queues[queueKey] == q {
+ delete(changesetTickets.queues, queueKey)
+ }
+}
+
+// queuedTicketLocked returns the position and the queued ticket with the given key in q, the
+// oldest one if there are several, or nil. The caller must hold changesetTickets.
+func queuedTicketLocked(q *changesetQueue, key [sha256.Size]byte) (int, *changesetTicket) {
+ for i, ticket := range q.tickets {
+ if ticket.key == key && ticket.record == nil {
+ return i, ticket
+ }
}
+ return -1, nil
}
-// takeChangesetTicket appends a ticket for the prepared transaction with the given hook input to
-// the queue of the repository at repoPath.
-func takeChangesetTicket(repoPath string, changes []byte) {
+// takeChangesetTicket appends a ticket for the prepared transaction that the git process with the
+// given invocation runs with the given hook input to the queue with the given key.
+func takeChangesetTicket(queueKey, invocation string, changes []byte) {
changesetTickets.Lock()
defer changesetTickets.Unlock()
now := time.Now()
- q := changesetQueueLocked(repoPath)
+ q := changesetQueueLocked(queueKey)
pruneExpiredChangesetTickets(q, now)
q.tickets = append(q.tickets, &changesetTicket{
- key: changesetTicketKey(changes),
+ key: changesetTicketKey(invocation, changes),
done: make(chan struct{}),
deadline: now.Add(changesetTicketTimeout),
})
@@ -161,78 +210,58 @@ func pruneExpiredChangesetTickets(q *changesetQueue, now time.Time) {
}
}
-// findChangesetTicket returns the oldest ticket of the transaction with the given hook input in
-// the queue of the repository at repoPath. It returns nil when the transaction has none, for
-// example because it was prepared before the changeset log was enabled or its ticket expired.
-func findChangesetTicket(repoPath string, changes []byte) *changesetTicket {
- changesetTickets.Lock()
- defer changesetTickets.Unlock()
-
- q, ok := changesetTickets.queues[repoPath]
- if !ok {
- return nil
- }
-
- key := changesetTicketKey(changes)
- for _, ticket := range q.tickets {
- if ticket.key == key && ticket.record == nil {
- return ticket
- }
- }
-
- return nil
-}
-
-// releaseChangesetTicket removes the ticket, whose transaction records nothing, from the queue of
-// the repository at repoPath and wakes the transactions that wait for it.
-func releaseChangesetTicket(repoPath string, ticket *changesetTicket) {
+// releaseChangesetTicket gives up the place of the transaction that the git process with the
+// given invocation runs with the given hook input, in the queue with the given key: its ticket, if
+// it is still queued, is removed, and the transactions that wait for it are woken. It is for a
+// transaction that was aborted or records nothing.
+func releaseChangesetTicket(queueKey, invocation string, changes []byte) {
changesetTickets.Lock()
defer changesetTickets.Unlock()
- q, ok := changesetTickets.queues[repoPath]
+ q, ok := changesetTickets.queues[queueKey]
if !ok {
return
}
- for i, queued := range q.tickets {
- if queued != ticket {
- continue
- }
-
+ if i, ticket := queuedTicketLocked(q, changesetTicketKey(invocation, changes)); ticket != nil {
close(ticket.done)
q.tickets = append(q.tickets[:i:i], q.tickets[i+1:]...)
q.signal()
- break
}
- dropIfIdleLocked(repoPath, q)
+ dropIfIdleLocked(queueKey, q)
}
-// abortChangesetTicket releases the ticket of the aborted transaction with the given hook input,
-// if it has one.
-func abortChangesetTicket(repoPath string, changes []byte) {
- if ticket := findChangesetTicket(repoPath, changes); ticket != nil {
- releaseChangesetTicket(repoPath, ticket)
- }
-}
-
-// recordInOrder records the record of the transaction of ticket in the changeset log of the
-// repository at repoPath, after the transactions whose tickets are before it, and returns once
-// its entries are durable or failed to be written.
+// recordChangeset records record, the committed transaction that the git process with the given
+// invocation runs with the given hook input, in the changeset log of the repository at repoPath,
+// whose queue has the given key. It returns once the entries are durable or failed to be written.
//
-// The ticket becomes ready. While no group is being written and the ticket at the head of the
-// queue is ready, the caller writes every ready ticket at the head with one call of
-// changelog.AppendGroup, one write and one fsync, and wakes their committed phases with the
+// The ticket of the transaction is found and made ready in one step. Without a queued ticket
+// (it expired, or the transaction took none) the record is appended at once, out of order, and
+// counted (graft_changelog_unordered_records_total). Otherwise it is recorded after the
+// transactions whose tickets are before it: while no group is being written and the ticket at
+// the head of the queue is ready, the caller writes every ready ticket at the head with one call
+// of changelog.AppendGroup, one write and one fsync, and wakes their committed phases with the
// outcome. Otherwise it waits for the queue to change or for the ticket at the head to expire.
// The wait does not end with the request: the updates are committed in git already, so their
// entries are always written, in order. It ends at the latest once every ticket before this one
// was written, aborted or expired, which changesetTicketTimeout bounds.
-func recordInOrder(repoPath string, ticket *changesetTicket, record changelog.Record) error {
+func recordChangeset(queueKey, repoPath, invocation string, changes []byte, record changelog.Record) error {
start := time.Now()
defer func() { changesetTicketWaitSeconds.Observe(time.Since(start).Seconds()) }()
changesetTickets.Lock()
- q := changesetQueueLocked(repoPath)
+ var ticket *changesetTicket
+ q, ok := changesetTickets.queues[queueKey]
+ if ok {
+ _, ticket = queuedTicketLocked(q, changesetTicketKey(invocation, changes))
+ }
+ if ticket == nil {
+ changesetTickets.Unlock()
+ changelogUnorderedRecordsTotal.Inc()
+ _, err := appendChangesetGroup(repoPath, []changelog.Record{record})
+ return err
+ }
ticket.record = &record
q.signal()
@@ -247,7 +276,7 @@ func recordInOrder(repoPath string, ticket *changesetTicket, record changelog.Re
pruneExpiredChangesetTickets(q, time.Now())
if !q.writing && len(q.tickets) > 0 && q.tickets[0].record != nil {
- writeChangesetGroupLocked(repoPath, q)
+ writeChangesetGroupLocked(queueKey, repoPath, q)
continue
}
@@ -273,10 +302,10 @@ func recordInOrder(repoPath string, ticket *changesetTicket, record changelog.Re
}
}
-// writeChangesetGroupLocked writes the ready tickets at the head of q, the queue of the repository
-// at repoPath, as one group. The caller must hold changesetTickets, which is released while the
-// group is written.
-func writeChangesetGroupLocked(repoPath string, q *changesetQueue) {
+// writeChangesetGroupLocked writes the ready tickets at the head of q, the queue with the given
+// key of the repository at repoPath, as one group. The caller must hold changesetTickets, which is
+// released while the group is written and held again when the function returns.
+func writeChangesetGroupLocked(queueKey, repoPath string, q *changesetQueue) {
n := 0
for n < len(q.tickets) && q.tickets[n].record != nil {
n++
@@ -291,7 +320,7 @@ func writeChangesetGroupLocked(repoPath string, q *changesetQueue) {
}
start := time.Now()
- _, err := changelog.AppendGroup(repoPath, records)
+ err := appendChangesetGroupRecovering(repoPath, records)
changelogAppendSeconds.Observe(time.Since(start).Seconds())
changelogGroupTransactions.Observe(float64(len(group)))
@@ -314,5 +343,19 @@ func writeChangesetGroupLocked(repoPath string, q *changesetQueue) {
q.tickets = remaining
q.writing = false
q.signal()
- dropIfIdleLocked(repoPath, q)
+ dropIfIdleLocked(queueKey, q)
+}
+
+// appendChangesetGroupRecovering writes the group of records and turns a panic of the write into
+// its error. The group's committed phases, which wait for its outcome, then all see the failure
+// and report their updates as lost, and the queue is not left behind a writer that never ends.
+func appendChangesetGroupRecovering(repoPath string, records []changelog.Record) (returnedErr error) {
+ defer func() {
+ if r := recover(); r != nil {
+ returnedErr = fmt.Errorf("writing the changeset group panicked: %v\n%s", r, debug.Stack())
+ }
+ }()
+
+ _, err := appendChangesetGroup(repoPath, records)
+ return err
}
diff --git a/internal/gitaly/hook/changeset_order_test.go b/internal/gitaly/hook/changeset_order_test.go
index fecc348..5fc4f06 100644
--- a/internal/gitaly/hook/changeset_order_test.go
+++ b/internal/gitaly/hook/changeset_order_test.go
@@ -17,6 +17,7 @@ import (
"github.com/feder-cr/git-ai/internal/graft/changelog"
"github.com/feder-cr/git-ai/internal/testhelper"
"github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
"github.com/prometheus/client_golang/prometheus/testutil"
"github.com/stretchr/testify/require"
)
@@ -25,12 +26,39 @@ import (
// that records its reference updates in the changeset log.
type changesetOrderSetup struct {
repoPath string
+ // queueKey is the key of the repository's queue of tickets.
+ queueKey string
// first moves the branch from base to a, second from a to b, third from b to c. other creates
// another branch at base.
first, second, third, other string
run func(state ReferenceTransactionState, changes string) error
// runWith runs the hook with the context ctx.
runWith func(ctx context.Context, state ReferenceTransactionState, changes string) error
+ // runAs runs the hook as the git process with the given invocation.
+ runAs func(invocation string, state ReferenceTransactionState, changes string) error
+}
+
+// queuedTicket is a ticket of a queue as it was when queuedTickets read it.
+type queuedTicket struct {
+ ready bool
+ deadline time.Time
+}
+
+// queuedTickets returns the tickets of the queue with the given key, read under the queue's
+// lock, nil when there is none.
+func queuedTickets(queueKey string) []queuedTicket {
+ changesetTickets.Lock()
+ defer changesetTickets.Unlock()
+
+ q, ok := changesetTickets.queues[queueKey]
+ if !ok {
+ return nil
+ }
+ tickets := make([]queuedTicket, 0, len(q.tickets))
+ for _, ticket := range q.tickets {
+ tickets = append(tickets, queuedTicket{ready: ticket.record != nil, deadline: ticket.deadline})
+ }
+ return tickets
}
func setupChangesetOrder(t *testing.T, ctx context.Context) changesetOrderSetup {
@@ -50,12 +78,18 @@ func setupChangesetOrder(t *testing.T, ctx context.Context) changesetOrderSetup
t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive,
), NewPostReceiveRegistry())
- payload, err := gitcmd.NewHooksPayload(ctx, cfg, repo, gittest.DefaultObjectHash, nil, &gitcmd.UserDetails{
- UserID: "user-1",
- Username: "user",
- Protocol: "web",
- }, gitcmd.ReferenceTransactionHook, nil).Env()
- require.NoError(t, err)
+ payloadAs := func(invocation string) string {
+ payload := gitcmd.NewHooksPayload(ctx, cfg, repo, gittest.DefaultObjectHash, nil, &gitcmd.UserDetails{
+ UserID: "user-1",
+ Username: "user",
+ Protocol: "web",
+ }, gitcmd.ReferenceTransactionHook, nil)
+ payload.Changelog.Invocation = invocation
+ env, err := payload.Env()
+ require.NoError(t, err)
+ return env
+ }
+ payload := payloadAs("")
runWith := func(ctx context.Context, state ReferenceTransactionState, changes string) error {
return manager.ReferenceTransactionHook(ctx, state, []string{payload}, strings.NewReader(changes))
@@ -63,6 +97,7 @@ func setupChangesetOrder(t *testing.T, ctx context.Context) changesetOrderSetup
return changesetOrderSetup{
repoPath: repoPath,
+ queueKey: changesetQueueKey(repo),
first: fmt.Sprintf("%s %s refs/heads/main\n", base, a),
second: fmt.Sprintf("%s %s refs/heads/main\n", a, b),
third: fmt.Sprintf("%s %s refs/heads/main\n", b, c),
@@ -71,6 +106,9 @@ func setupChangesetOrder(t *testing.T, ctx context.Context) changesetOrderSetup
return runWith(ctx, state, changes)
},
runWith: runWith,
+ runAs: func(invocation string, state ReferenceTransactionState, changes string) error {
+ return manager.ReferenceTransactionHook(ctx, state, []string{payloadAs(invocation)}, strings.NewReader(changes))
+ },
}
}
@@ -214,7 +252,7 @@ func TestReferenceTransactionHook_changesetOrder(t *testing.T) {
require.Eventually(t, func() bool {
changesetTickets.Lock()
defer changesetTickets.Unlock()
- q := changesetTickets.queues[setup.repoPath]
+ q := changesetTickets.queues[setup.queueKey]
return q != nil && len(q.tickets) == 3 && q.tickets[1].record != nil && q.tickets[2].record != nil
}, 10*time.Second, 5*time.Millisecond)
require.Empty(t, recordedChanges(t, setup.repoPath))
@@ -232,13 +270,90 @@ func TestReferenceTransactionHook_changesetOrder(t *testing.T) {
require.Equal(t, entries[0].RecordedAt, entries[1].RecordedAt)
require.Equal(t, entries[0].RecordedAt, entries[2].RecordedAt)
- changesetTickets.Lock()
- _, ok := changesetTickets.queues[setup.repoPath]
- changesetTickets.Unlock()
- require.False(t, ok, "the queue is dropped once it is empty")
+ require.Nil(t, queuedTickets(setup.queueKey), "the queue is dropped once it is empty")
+ })
+
+ t.Run("a transaction with the same updates in another process keeps the other's ticket", func(t *testing.T) {
+ t.Parallel()
+
+ setup := setupChangesetOrder(t, ctx)
+
+ // Process A prepared the update. Process B tried the same update, could not lock the
+ // reference and aborted: its abort must not give up A's place.
+ require.NoError(t, setup.runAs("process-a", ReferenceTransactionPrepared, setup.first))
+ require.NoError(t, setup.runAs("process-b", ReferenceTransactionAborted, setup.first))
+ require.Len(t, queuedTickets(setup.queueKey), 1)
+
+ // B's committed phase, had it committed, would not record with A's ticket either: A's
+ // committed phase finds its own.
+ require.NoError(t, setup.runAs("process-a", ReferenceTransactionPrepared, setup.second))
+ secondDone := make(chan error, 1)
+ go func() { secondDone <- setup.runAs("process-a", ReferenceTransactionCommitted, setup.second) }()
+ require.Eventually(t, func() bool {
+ tickets := queuedTickets(setup.queueKey)
+ return len(tickets) == 2 && tickets[1].ready
+ }, 10*time.Second, 5*time.Millisecond)
+ require.Empty(t, recordedChanges(t, setup.repoPath), "the second waits for the first")
+
+ require.NoError(t, setup.runAs("process-a", ReferenceTransactionCommitted, setup.first))
+ require.NoError(t, <-secondDone)
+ require.Equal(t, []string{setup.first, setup.second}, recordedChanges(t, setup.repoPath))
})
}
+// TestReferenceTransactionHook_changesetCommittedAfterExpiry is not parallel because it lowers the
+// ticket timeout of the package and reads global counters. A committed phase that runs after its
+// ticket expired, and after a later transaction was recorded without it, is recorded too: out of
+// order, counted as such, and not lost.
+func TestReferenceTransactionHook_changesetCommittedAfterExpiry(t *testing.T) {
+ ctx := testhelper.Context(t)
+
+ defaultTimeout := changesetTicketTimeout
+ changesetTicketTimeout = 50 * time.Millisecond
+ t.Cleanup(func() { changesetTicketTimeout = defaultTimeout })
+
+ setup := setupChangesetOrder(t, ctx)
+ unordered := testutil.ToFloat64(changelogUnorderedRecordsTotal)
+ lost := testutil.ToFloat64(changelogRecordsLostTotal)
+
+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.first))
+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.second))
+ // The second waits for the first's ticket to expire, then records.
+ require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.second))
+ require.Equal(t, []string{setup.second}, recordedChanges(t, setup.repoPath))
+
+ // The first's ticket is gone: its committed phase is recorded at once.
+ require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.first))
+ require.Equal(t, []string{setup.second, setup.first}, recordedChanges(t, setup.repoPath))
+ require.InDelta(t, 1, testutil.ToFloat64(changelogUnorderedRecordsTotal)-unordered, 0)
+ require.InDelta(t, 0, testutil.ToFloat64(changelogRecordsLostTotal)-lost, 0)
+ require.Nil(t, queuedTickets(setup.queueKey), "no queue is left behind")
+}
+
+// TestReferenceTransactionHook_changesetGroupPanic is not parallel because it replaces the group
+// writer of the package and reads a global counter. A write that panics fails its group, whose
+// updates are reported lost, and leaves the queue usable.
+func TestReferenceTransactionHook_changesetGroupPanic(t *testing.T) {
+ ctx := testhelper.Context(t)
+ setup := setupChangesetOrder(t, ctx)
+
+ appendChangesetGroup = func(string, []changelog.Record) ([][]changelog.Entry, error) {
+ panic("injected")
+ }
+ t.Cleanup(func() { appendChangesetGroup = changelog.AppendGroup })
+
+ lost := testutil.ToFloat64(changelogRecordsLostTotal)
+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.first))
+ require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.first), "recording never fails the update")
+ require.InDelta(t, 1, testutil.ToFloat64(changelogRecordsLostTotal)-lost, 0)
+ require.Nil(t, queuedTickets(setup.queueKey), "the queue is not left writing")
+
+ appendChangesetGroup = changelog.AppendGroup
+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.second))
+ require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.second))
+ require.Equal(t, []string{setup.second}, recordedChanges(t, setup.repoPath))
+}
+
// TestReferenceTransactionHook_changesetRecordsLost is not parallel because it reads the counter
// of lost records, which is global.
func TestReferenceTransactionHook_changesetRecordsLost(t *testing.T) {
@@ -269,20 +384,43 @@ func TestReferenceTransactionHook_changesetOrderExpiredTicket(t *testing.T) {
// records once the ticket of the first one expired.
require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.first))
require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.second))
+ firstDeadline := queuedTickets(setup.queueKey)[0].deadline
- start := time.Now()
+ // The second records only once the first's ticket expired: not before its deadline.
require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.second))
- require.GreaterOrEqual(t, time.Since(start), 50*time.Millisecond)
+ require.False(t, time.Now().Before(firstDeadline), "recorded before the earlier ticket expired")
require.Equal(t, []string{setup.second}, recordedChanges(t, setup.repoPath))
- // The expired ticket is dropped when the next transaction is prepared.
- require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.first))
- changesetTickets.Lock()
- queue := changesetTickets.queues[setup.repoPath].tickets
- changesetTickets.Unlock()
- require.Len(t, queue, 1)
- require.NoError(t, setup.run(ReferenceTransactionAborted, setup.first))
+ // The expired ticket was dropped: the next transaction finds an empty queue.
+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.third))
+ require.Len(t, queuedTickets(setup.queueKey), 1)
+ require.NoError(t, setup.run(ReferenceTransactionAborted, setup.third))
+}
+
+// TestReferenceTransactionHook_changesetUnresolvedRepository is not parallel because it reads a
+// global counter. A committed phase that cannot resolve its repository path gives up its place, so
+// that later transactions do not wait for it, and reports its updates lost.
+func TestReferenceTransactionHook_changesetUnresolvedRepository(t *testing.T) {
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+
+ manager := NewManager(cfg, config.NewLocator(cfg), testhelper.SharedLogger(t), gittest.NewCommandFactory(t, cfg), nil, &transaction.MockManager{}, gitlab.NewMockClient(
+ t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive,
+ ), NewPostReceiveRegistry())
+
+ // The repository does not exist: its path does not resolve.
+ repo := &gitalypb.Repository{StorageName: cfg.Storages[0].Name, RelativePath: "missing.git"}
+ payload, err := gitcmd.NewHooksPayload(ctx, cfg, repo, gittest.DefaultObjectHash, nil, nil, gitcmd.ReferenceTransactionHook, nil).Env()
+ require.NoError(t, err)
+ changes := fmt.Sprintf("%s %s refs/heads/main\n", gittest.DefaultObjectHash.ZeroOID, gittest.DefaultObjectHash.EmptyTreeOID)
+
+ lost := testutil.ToFloat64(changelogRecordsLostTotal)
+ require.NoError(t, manager.ReferenceTransactionHook(ctx, ReferenceTransactionPrepared, []string{payload}, strings.NewReader(changes)))
+ require.Len(t, queuedTickets(changesetQueueKey(repo)), 1)
+ require.NoError(t, manager.ReferenceTransactionHook(ctx, ReferenceTransactionCommitted, []string{payload}, strings.NewReader(changes)))
+ require.Nil(t, queuedTickets(changesetQueueKey(repo)), "the ticket is released")
+ require.InDelta(t, 1, testutil.ToFloat64(changelogRecordsLostTotal)-lost, 0)
}
func TestReferenceTransactionHook_changesetOrderDisabled(t *testing.T) {
@@ -308,8 +446,5 @@ func TestReferenceTransactionHook_changesetOrderDisabled(t *testing.T) {
require.NoError(t, manager.ReferenceTransactionHook(ctx, ReferenceTransactionPrepared, []string{payload}, strings.NewReader(changes)))
// Without the changeset log, transactions are not ordered.
- changesetTickets.Lock()
- _, ok := changesetTickets.queues[repoPath]
- changesetTickets.Unlock()
- require.False(t, ok)
+ require.Nil(t, queuedTickets(changesetQueueKey(repo)))
}
diff --git a/internal/gitaly/hook/custom.go b/internal/gitaly/hook/custom.go
index 839620c..fd8416e 100644
--- a/internal/gitaly/hook/custom.go
+++ b/internal/gitaly/hook/custom.go
@@ -244,7 +244,7 @@ func (m *GitLabHookManager) customHooksEnv(ctx context.Context, payload gitcmd.H
// configuration, which the command factory applies after them.
func (m *GitLabHookManager) fsyncConfiguration() []gitcmd.ConfigPair {
pairs := []gitcmd.ConfigPair{
- {Key: "core.fsync", Value: "objects,derived-metadata,reference"},
+ {Key: "core.fsync", Value: gitcmd.FsyncComponents},
{Key: "core.fsyncMethod", Value: "fsync"},
}
diff --git a/internal/gitaly/hook/referencetransaction.go b/internal/gitaly/hook/referencetransaction.go
index ebd000c..6f91323 100644
--- a/internal/gitaly/hook/referencetransaction.go
+++ b/internal/gitaly/hook/referencetransaction.go
@@ -74,7 +74,7 @@ func (m *GitLabHookManager) ReferenceTransactionHook(ctx context.Context, state
// An aborted transaction gives up its place in the order of the changeset log, so that the
// transactions prepared after it do not wait for it.
case ReferenceTransactionAborted:
- m.abortChangesets(ctx, payload, changes)
+ m.abortChangesets(payload, env, changes)
if m.storageVoter != nil {
m.storageVoter.Aborted(ctx, payload, changes)
}
@@ -156,42 +156,46 @@ func (m *GitLabHookManager) ReferenceTransactionHook(ctx context.Context, state
// the order of the changeset log now. It does so only once the vote succeeded, because a
// failed hook aborts the transaction.
// In authoritative storage mode the file changelog is not written, so no order is kept for it.
+ // The mode is the storage core's configuration, fixed for the life of the process, so the
+ // prepared and the committed phase of a transaction decide alike.
if state == ReferenceTransactionPrepared && !m.storageAuthoritative() {
- m.prepareChangesets(ctx, payload, changes)
+ m.prepareChangesets(payload, env, changes)
}
return nil
}
-// changesetRepoPath returns the path of the repository whose changeset log records the given
-// hook input, or false when nothing of it is recorded.
-func (m *GitLabHookManager) changesetRepoPath(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) (string, bool) {
- if m.cfg.Graft.Changelog.Disabled || len(changelog.ParseUpdates(changes)) == 0 {
- return "", false
- }
+// ordersChangesets reports whether the transaction with the given hook input takes a place in the
+// order of the changeset log of its repository.
+func (m *GitLabHookManager) ordersChangesets(payload gitcmd.HooksPayload, changes []byte) bool {
+ return !m.cfg.Graft.Changelog.Disabled && payload.Repo != nil && len(changelog.ParseUpdates(changes)) > 0
+}
- repoPath, err := m.locator.GetRepoPath(ctx, payload.Repo)
+// changesetInvocation returns the invocation of the git process that runs the transaction, which
+// tells its ticket apart from those of the transactions of other processes (changesetTicketKey).
+// It is empty when the payload carries none; the prepared and the committed phase of a
+// transaction read the same environment, so they agree.
+func changesetInvocation(env []string) string {
+ meta, err := gitcmd.ChangelogMetaFromEnv(env)
if err != nil {
- m.logger.WithError(err).ErrorContext(ctx, "ordering changesets: resolving repository path")
- return "", false
+ return ""
}
-
- return repoPath, true
+ return meta.Invocation
}
// prepareChangesets gives the prepared transaction its place in the order of the changeset log,
// see changesetTickets.
-func (m *GitLabHookManager) prepareChangesets(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {
- if repoPath, ok := m.changesetRepoPath(ctx, payload, changes); ok {
- takeChangesetTicket(repoPath, changes)
+func (m *GitLabHookManager) prepareChangesets(payload gitcmd.HooksPayload, env []string, changes []byte) {
+ if m.ordersChangesets(payload, changes) {
+ takeChangesetTicket(changesetQueueKey(payload.Repo), changesetInvocation(env), changes)
}
}
// abortChangesets gives up the place of the aborted transaction in the order of the changeset
// log, see changesetTickets.
-func (m *GitLabHookManager) abortChangesets(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {
- if repoPath, ok := m.changesetRepoPath(ctx, payload, changes); ok {
- abortChangesetTicket(repoPath, changes)
+func (m *GitLabHookManager) abortChangesets(payload gitcmd.HooksPayload, env []string, changes []byte) {
+ if m.ordersChangesets(payload, changes) {
+ releaseChangesetTicket(changesetQueueKey(payload.Repo), changesetInvocation(env), changes)
}
}
@@ -233,22 +237,23 @@ func (m *GitLabHookManager) recordChangesets(ctx context.Context, objectHash git
return
}
- repoPath, err := m.locator.GetRepoPath(ctx, payload.Repo)
- if err != nil {
- m.logger.WithError(err).ErrorContext(ctx, "recording changesets: resolving repository path")
- return
- }
-
// The updates are recorded in the order in which their transactions were prepared, which
// is the order in which the references were updated. See changesetTickets. A transaction
// that records nothing gives up its place at once, so that later transactions do not wait
- // for it: a packed-refs cleanup, a transaction whose metadata cannot be read, and one on
- // the MVCC reference backend, whose updates are recorded after its commit.
- ticket := findChangesetTicket(repoPath, changes)
+ // for it: one whose repository path cannot be resolved, a packed-refs cleanup, one whose
+ // metadata cannot be read, and one on the MVCC reference backend, whose updates are
+ // recorded after its commit.
+ queueKey := changesetQueueKey(payload.Repo)
+ invocation := changesetInvocation(env)
release := func() {
- if ticket != nil {
- releaseChangesetTicket(repoPath, ticket)
- }
+ releaseChangesetTicket(queueKey, invocation, changes)
+ }
+
+ repoPath, err := m.locator.GetRepoPath(ctx, payload.Repo)
+ if err != nil {
+ release()
+ m.reportLostChangesets(ctx, updates, fmt.Errorf("resolving repository path: %w", err))
+ return
}
if packedCleanup {
@@ -282,16 +287,10 @@ func (m *GitLabHookManager) recordChangesets(ctx context.Context, objectHash git
return
}
- // A transaction without a ticket, prepared before the changeset log was enabled or whose
- // ticket expired, is recorded at once.
- var recordErr error
- if ticket == nil {
- _, recordErr = changelog.Append(repoPath, record.Meta, record.Updates)
- } else {
- recordErr = recordInOrder(repoPath, ticket, record)
- }
- if recordErr != nil {
- m.reportLostChangesets(ctx, updates, recordErr)
+ // A transaction without a queued ticket, prepared before the changeset log was enabled or
+ // whose ticket expired, is recorded at once (see recordChangeset).
+ if err := recordChangeset(queueKey, repoPath, invocation, changes, record); err != nil {
+ m.reportLostChangesets(ctx, updates, err)
}
}
@@ -340,7 +339,9 @@ func (m *GitLabHookManager) observeCommitted(ctx context.Context, payload gitcmd
// runs the hook for itself are tracked: their hooks payload carries the
// invocation of the git process.
func (m *GitLabHookManager) trackPendingTransaction(state ReferenceTransactionState, env []string, changes []byte) {
- if m.cfg.Graft.Changelog.Disabled || m.pendingTransactions == nil {
+ // In authoritative storage mode recordChangesets does not run, so nothing would ever take a
+ // tracked transaction out again.
+ if m.cfg.Graft.Changelog.Disabled || m.pendingTransactions == nil || m.storageAuthoritative() {
return
}
diff --git a/internal/gitaly/hook/updateref/update_with_hooks.go b/internal/gitaly/hook/updateref/update_with_hooks.go
index 917d6b6..e62a40c 100644
--- a/internal/gitaly/hook/updateref/update_with_hooks.go
+++ b/internal/gitaly/hook/updateref/update_with_hooks.go
@@ -232,7 +232,22 @@ func (u *UpdaterWithHooks) UpdateReference(
quarantinedRepo = quarantineDir.QuarantinedRepo()
}
- hooksPayload, err := gitcmd.NewHooksPayload(ctx, u.cfg, quarantinedRepo, objectHash, transaction, &receiveHooksPayload, gitcmd.ReceivePackHooks, featureflag.FromContext(ctx)).Env()
+ // This function drives the reference-transaction hook of one transaction itself. The
+ // changeset log tells it apart from every other transaction, also from one with the same
+ // updates, by a value unique to this call, as it does for a git process that runs the hook.
+ var invocation string
+ if !u.cfg.Graft.Changelog.Disabled {
+ if invocation, err = gitcmd.NewChangelogInvocation(); err != nil {
+ return err
+ }
+ }
+ newHooksPayload := func(repo *gitalypb.Repository) (string, error) {
+ payload := gitcmd.NewHooksPayload(ctx, u.cfg, repo, objectHash, transaction, &receiveHooksPayload, gitcmd.ReceivePackHooks, featureflag.FromContext(ctx))
+ payload.Changelog.Invocation = invocation
+ return payload.Env()
+ }
+
+ hooksPayload, err := newHooksPayload(quarantinedRepo)
if err != nil {
return fmt.Errorf("constructing hooks payload: %w", err)
}
@@ -254,7 +269,7 @@ func (u *UpdaterWithHooks) UpdateReference(
// We only need to update the hooks payload to the unquarantined repo in case we
// had a quarantine environment. Otherwise, the initial hooks payload is for the
// real repository anyway.
- hooksPayload, err = gitcmd.NewHooksPayload(ctx, u.cfg, repoProto, objectHash, transaction, &receiveHooksPayload, gitcmd.ReceivePackHooks, featureflag.FromContext(ctx)).Env()
+ hooksPayload, err = newHooksPayload(repoProto)
if err != nil {
return fmt.Errorf("constructing quarantined hooks payload: %w", err)
}
diff --git a/internal/graft/changelog/log.go b/internal/graft/changelog/log.go
index a658e70..ddb950e 100644
--- a/internal/graft/changelog/log.go
+++ b/internal/graft/changelog/log.go
@@ -326,8 +326,9 @@ func Append(repoPath string, meta Meta, updates []Update) ([]Entry, error) {
// the repository itself, so entries written there would be lost.
// - Each Gitaly node keeps its own log and its own sequences. Behind Praefect,
// reads may be routed to a replica whose log lags or differs.
-// - The lock is taken without a timeout. A holder that never releases it blocks
-// every append to the repository.
+// - The wait for the lock is bounded by lockWaitTimeout: an append that does
+// not get it fails. An fsync that never returns is not bounded: its append
+// holds the lock, and the appends after it fail at lockWaitTimeout.
func AppendGroup(repoPath string, records []Record) ([][]Entry, error) {
entries := make([][]Entry, len(records))
@@ -415,7 +416,9 @@ func (s *logState) appendLocked(repoPath, path string, records []Record, entries
return fmt.Errorf("stat changelog: %w", err)
}
- if !s.loaded || info.Size() != s.fileSize {
+ if valid, err := s.cacheDescribes(f, info.Size()); err != nil {
+ return err
+ } else if !valid {
if err := s.load(f, info.Size()); err != nil {
return err
}
@@ -515,6 +518,43 @@ func (s *logState) appendLocked(repoPath, path string, records []Record, entries
return nil
}
+// cacheDescribes reports whether the cached state still describes the file f, whose size is
+// size, so that it need not be loaded again. The caller must hold s.mu.
+//
+// The size alone does not tell: while the cache knows of bytes after its last valid entry (a torn
+// tail, or the untrusted bytes of a failed append), another process may have cut them off and
+// appended durable entries that end at the same size, and acting on the cache would cut those
+// off. Such a cache is therefore never trusted. A cache without such bytes can only see the file
+// grow, as no process cuts the file below the end of the durable entries; it is trusted when the
+// size is unchanged and the line before its end still holds its last entry.
+func (s *logState) cacheDescribes(f *os.File, size int64) (bool, error) {
+ if !s.loaded || size != s.fileSize || s.fileSize != s.validEnd {
+ return false, nil
+ }
+ if s.validEnd == 0 {
+ return s.lastSeq == 0, nil
+ }
+
+ nl := s.validEnd - 1
+ prev, err := lastNewlineBefore(f, nl)
+ if err != nil {
+ return false, err
+ }
+ line := make([]byte, nl-(prev+1)+1)
+ if _, err := f.ReadAt(line, prev+1); err != nil {
+ return false, fmt.Errorf("reading changelog: %w", err)
+ }
+ if line[len(line)-1] != '\n' {
+ return false, nil
+ }
+
+ var entry Entry
+ if err := json.Unmarshal(line[:len(line)-1], &entry); err != nil {
+ return false, nil
+ }
+ return entry.Sequence == s.lastSeq, nil
+}
+
// discardFrom cuts off the bytes that a failed append wrote from end on. When
// that fails too, the bytes are remembered as untrusted, so that no reader takes
// them for entries. The caller must hold s.mu.
@@ -716,8 +756,10 @@ var truncateFile = func(f *os.File, size int64) error {
// or the file changed since it was (an append of another process), the state is
// loaded again under the log's lock file, which waits for an append of another
// process, and the file is fsync'd first: bytes that a crashed writer left in the
-// page cache are durable once a reader may see them.
-func (s *logState) durableView(path string, f *os.File) (int64, uint64, error) {
+// page cache are durable once a reader may see them. So are the directory entries
+// of the log, which that writer may have crashed before syncing (see
+// syncLogDirs), unless this process synced them already.
+func (s *logState) durableView(repoPath, path string, f *os.File) (int64, uint64, error) {
s.mu.Lock()
defer s.mu.Unlock()
@@ -725,7 +767,9 @@ func (s *logState) durableView(path string, f *os.File) (int64, uint64, error) {
if err != nil {
return 0, 0, fmt.Errorf("stat changelog: %w", err)
}
- if s.loaded && info.Size() == s.fileSize {
+ if valid, err := s.cacheDescribes(f, info.Size()); err != nil {
+ return 0, 0, err
+ } else if valid && s.dirsSynced {
return s.validEnd, s.lastSeq, nil
}
@@ -738,6 +782,12 @@ func (s *logState) durableView(path string, f *os.File) (int64, uint64, error) {
if err := syncFile(f); err != nil {
return 0, 0, fmt.Errorf("syncing changelog: %w", err)
}
+ if !s.dirsSynced {
+ if err := syncLogDirs(repoPath, path); err != nil {
+ return 0, 0, err
+ }
+ s.dirsSynced = true
+ }
info, err = f.Stat()
if err != nil {
@@ -763,23 +813,57 @@ var syncDir = func(dir string) error {
return d.Sync()
}
+// lockWaitTimeout bounds the wait for the log's lock file, which another process holds for one
+// append. A holder that never releases it would otherwise block every append and every reader of
+// the repository's log: the wait fails instead, and the appends see the error (their updates are
+// reported lost). Each wait that failed keeps a goroutine and a descriptor until the holder
+// releases the lock. Tests lower it.
+var lockWaitTimeout = 30 * time.Second
+
// lockFile takes an exclusive flock on the file at path and returns the
-// function that releases it.
+// function that releases it. It waits at most lockWaitTimeout.
func lockFile(path string) (func(), error) {
f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o644)
if err != nil {
return nil, fmt.Errorf("opening changelog lock: %w", err)
}
+ unlock := func() {
+ _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN)
+ _ = f.Close()
+ }
- if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX); err != nil {
+ err = syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
+ if err == nil {
+ return unlock, nil
+ }
+ if !errors.Is(err, syscall.EWOULDBLOCK) {
f.Close()
return nil, fmt.Errorf("locking changelog: %w", err)
}
- return func() {
- _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN)
- _ = f.Close()
- }, nil
+ // Another process holds the lock: wait for it in a goroutine, which owns the file if the
+ // wait gives up first and releases the lock once it is granted.
+ locked := make(chan error, 1)
+ go func() { locked <- syscall.Flock(int(f.Fd()), syscall.LOCK_EX) }()
+
+ timer := time.NewTimer(lockWaitTimeout)
+ defer timer.Stop()
+ select {
+ case err := <-locked:
+ if err != nil {
+ f.Close()
+ return nil, fmt.Errorf("locking changelog: %w", err)
+ }
+ return unlock, nil
+ case <-timer.C:
+ go func() {
+ if err := <-locked; err == nil {
+ _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN)
+ }
+ _ = f.Close()
+ }()
+ return nil, fmt.Errorf("locking changelog: the lock was not granted within %s", lockWaitTimeout)
+ }
}
// Scan calls fn for every entry of the log of the repository whose git
@@ -804,7 +888,7 @@ func Scan(repoPath string, after uint64, fn func(Entry) (bool, error)) error {
}
defer f.Close()
- end, _, err := state.durableView(path, f)
+ end, _, err := state.durableView(repoPath, path, f)
if err != nil {
return err
}
@@ -868,7 +952,7 @@ func LastSequence(repoPath string) (uint64, error) {
}
defer f.Close()
- _, lastSeq, err := state.durableView(path, f)
+ _, lastSeq, err := state.durableView(repoPath, path, f)
if err != nil {
return 0, err
}
diff --git a/internal/graft/changelog/log_durable_test.go b/internal/graft/changelog/log_durable_test.go
index 6dcb024..6dd48ac 100644
--- a/internal/graft/changelog/log_durable_test.go
+++ b/internal/graft/changelog/log_durable_test.go
@@ -117,20 +117,35 @@ func TestReadersSeeEntriesOnlyOnceDurable(t *testing.T) {
require.NoError(t, err)
require.Contains(t, string(data), "refs/heads/b")
+ // The reader comes without this process's state, as a reader of another process does: it
+ // does not wait for the state's mutex but for the log's lock file, which the append holds.
+ statesMu.Lock()
+ delete(states, Path(repoPath))
+ statesMu.Unlock()
+
type result struct {
refs []string
last uint64
+ err error
}
read := make(chan result, 1)
go func() {
var r result
- r.refs = references(t, repoPath, 0)
- r.last, _ = LastSequence(repoPath)
+ var entries []Entry
+ if entries, r.err = List(repoPath, 0, 10); r.err == nil {
+ for _, entry := range entries {
+ r.refs = append(r.refs, entry.Reference)
+ }
+ r.last, r.err = LastSequence(repoPath)
+ }
read <- r
}()
+ readEarly := false
select {
case r := <-read:
+ readEarly = true
+ require.NoError(t, r.err)
require.Equal(t, []string{"refs/heads/a"}, r.refs, "a reader returned an entry before its fsync")
require.EqualValues(t, 1, r.last)
case <-time.After(200 * time.Millisecond):
@@ -140,8 +155,10 @@ func TestReadersSeeEntriesOnlyOnceDurable(t *testing.T) {
close(release)
require.NoError(t, <-appended)
- r := <-read
- if len(r.refs) == 2 {
+ if !readEarly {
+ // The reader waited for the lock file: it read once the entry was durable.
+ r := <-read
+ require.NoError(t, r.err)
require.Equal(t, []string{"refs/heads/a", "refs/heads/b"}, r.refs)
require.EqualValues(t, 2, r.last)
}
diff --git a/internal/graft/changelog/log_review_test.go b/internal/graft/changelog/log_review_test.go
new file mode 100644
index 0000000..5562e23
--- /dev/null
+++ b/internal/graft/changelog/log_review_test.go
@@ -0,0 +1,142 @@
+package changelog
+
+import (
+ "os"
+ "path/filepath"
+ "sync"
+ "syscall"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+)
+
+// sequencesAndReferences lists the log of repoPath as sequence and reference pairs.
+func sequencesAndReferences(t *testing.T, repoPath string) []string {
+ t.Helper()
+
+ entries, err := List(repoPath, 0, 100)
+ require.NoError(t, err)
+ var listed []string
+ for _, entry := range entries {
+ listed = append(listed, filepath.Base(entry.Reference)+"@"+string(rune('0'+entry.Sequence)))
+ }
+ return listed
+}
+
+// This test replaces syncDir, so it must not run in parallel with other tests. A reader whose
+// process did not append syncs the directories of the log before it serves entries: the writer
+// may have crashed between the fsync of the file and those of the directories.
+func TestReaders_syncDirectoriesOfAnotherProcess(t *testing.T) {
+ var (
+ mu sync.Mutex
+ synced []string
+ )
+ originalSyncDir := syncDir
+ t.Cleanup(func() { syncDir = originalSyncDir })
+ syncDir = func(dir string) error {
+ mu.Lock()
+ synced = append(synced, dir)
+ mu.Unlock()
+ return originalSyncDir(dir)
+ }
+ takeSynced := func() []string {
+ mu.Lock()
+ defer mu.Unlock()
+ result := synced
+ synced = nil
+ return result
+ }
+
+ repoPath := newRepoPath(t)
+ _, err := Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/a", OldOID: zeroOID, NewOID: oidA}})
+ require.NoError(t, err)
+ takeSynced()
+
+ // A reader of another process: this process's state is gone.
+ dropState(repoPath)
+ require.Equal(t, []string{"a@1"}, sequencesAndReferences(t, repoPath))
+ require.ElementsMatch(t, []string{filepath.Dir(Path(repoPath)), repoPath}, takeSynced())
+
+ // Once synced, the next read does not sync them again.
+ require.Equal(t, []string{"a@1"}, sequencesAndReferences(t, repoPath))
+ require.Empty(t, takeSynced())
+}
+
+// This test lowers lockWaitTimeout, so it must not run in parallel with other tests. A lock file
+// that another process never releases fails the appends and the reads after lockWaitTimeout, and
+// the log works again once it is released.
+func TestAppend_lockWaitIsBounded(t *testing.T) {
+ originalTimeout := lockWaitTimeout
+ t.Cleanup(func() { lockWaitTimeout = originalTimeout })
+ lockWaitTimeout = 100 * time.Millisecond
+
+ repoPath := newRepoPath(t)
+ _, err := Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/a", OldOID: zeroOID, NewOID: oidA}})
+ require.NoError(t, err)
+
+ // Another process holds the lock file.
+ holder, err := os.OpenFile(Path(repoPath)+lockSuffix, os.O_RDWR, 0o644)
+ require.NoError(t, err)
+ require.NoError(t, syscall.Flock(int(holder.Fd()), syscall.LOCK_EX))
+
+ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/b", OldOID: zeroOID, NewOID: oidB}})
+ require.ErrorContains(t, err, "the lock was not granted within 100ms")
+
+ // A reader that has to wait for the lock (it comes without the state) fails the same way.
+ dropState(repoPath)
+ _, err = List(repoPath, 0, 10)
+ require.ErrorContains(t, err, "the lock was not granted within 100ms")
+
+ require.NoError(t, syscall.Flock(int(holder.Fd()), syscall.LOCK_UN))
+ require.NoError(t, holder.Close())
+
+ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/c", OldOID: zeroOID, NewOID: oidC}})
+ require.NoError(t, err)
+ require.Equal(t, []string{"a@1", "c@2"}, sequencesAndReferences(t, repoPath))
+}
+
+// TestAppend_doesNotTrustACacheWithATail is the overlap of two processes during a graceful
+// restart. Process A's cache knows of bytes after its last valid entry, which it would cut off on
+// its next append. Process B cut them off itself and appended a durable entry, which ends at the
+// same size. A must not take the size for an unchanged file: it would cut off B's entry.
+func TestAppend_doesNotTrustACacheWithATail(t *testing.T) {
+ t.Parallel()
+
+ repoPath := newRepoPath(t)
+ path := Path(repoPath)
+ _, err := Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/a", OldOID: zeroOID, NewOID: oidA}})
+ require.NoError(t, err)
+
+ statesMu.Lock()
+ processA := states[path]
+ delete(states, path)
+ statesMu.Unlock()
+ afterA := processA.validEnd
+
+ // Process B appends with its own state.
+ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/b", OldOID: zeroOID, NewOID: oidB}})
+ require.NoError(t, err)
+ info, err := os.Stat(path)
+ require.NoError(t, err)
+
+ // Process A is back, and its cache takes everything after its entry for a tail of the size the
+ // file has now.
+ statesMu.Lock()
+ states[path] = processA
+ statesMu.Unlock()
+ processA.mu.Lock()
+ processA.fileSize = info.Size()
+ processA.validEnd = afterA
+ processA.mu.Unlock()
+
+ require.Equal(t, []string{"a@1", "b@2"}, sequencesAndReferences(t, repoPath), "a reader of A sees B's entry")
+
+ processA.mu.Lock()
+ processA.fileSize = info.Size()
+ processA.validEnd = afterA
+ processA.mu.Unlock()
+ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/c", OldOID: zeroOID, NewOID: oidC}})
+ require.NoError(t, err)
+ require.Equal(t, []string{"a@1", "b@2", "c@3"}, sequencesAndReferences(t, repoPath), "A's append keeps B's entry")
+}