Git designed for AI agents.
changelog, hook: fixes from the review of the PLAN9 write path
Federico Elia42d63f01bcd5
diff --git a/doc/graft/PLAN9.md b/doc/graft/PLAN9.mdindex a5621fc..52da6e6 100644--- a/doc/graft/PLAN9.md+++ b/doc/graft/PLAN9.md@@ -395,8 +395,12 @@ update hook and update-ref); batch publishing (pack mode) and unbatched publishi## 4. Invariants (each has a test, section 10)-* L1 The changeset log is dense and ordered: sequences are consecutive, assigned in- ticket order, across groups and across processes.+* L1 The changeset log is dense and ordered: sequences are consecutive across groups+ and across processes (the log's lock file serializes the appends of every process),+ and within the process that serves the hooks they are assigned in ticket order. A+ transaction whose ticket expired before its committed phase is the exception: it is+ recorded at once, after the transactions that did not wait for it, and counted+ (`graft_changelog_unordered_records_total`).* L2 No RPC returns before its entries are durable, and no reader returns an entrythat is not durable.* L3 A reader with a cursor never skips an entry, including across a crash before@@ -425,7 +429,15 @@ update hook and update-ref); batch publishing (pack mode) and unbatched publishi| housekeeping removes a fan-out directory after the link | its descriptor is fsynced; the objects are in a pack || crash during pass 1 | any prefix of the links; the quarantine is removed; unreferenced objects at most, as today || crash during pass 2 | links durable up to the last committed journal transaction, in order |-| reader before the first append of the process | loads the state, fsyncs the file, reads up to its valid end |+| reader before the first append of the process | loads the state, fsyncs the file and the log's directories, reads up to its valid end (amended after review: the directories were not synced, and a writer that crashed between its file and its directory fsync left entries a reader served that a crash could take back) |+| a committed phase runs after its ticket expired | (amended after review) the ticket is found and made ready in one step under the queue's lock; an expired or released ticket is not found, so the transaction is appended at once, out of order, and counted; never taken for written (the first version could, in the window between finding the ticket and making it ready, and lost the entries silently) |+| a transaction with the same hook input in another process | the ticket key holds the invocation of the git process, or of the `UserCommitFiles`/land call that drives the hook in process, so an abort or a commit never takes another transaction's ticket |+| the group write panics | the panic is the group's error: every member reports its updates lost, the queue stops writing and goes on |+| the log's lock file is not granted (a holder that never releases it) | the append or the read fails after `lockWaitTimeout` (30 s); the append's updates are reported lost |+| an fsync never returns (hung I/O) | not bounded: its group's members wait with it, and later appends fail at `lockWaitTimeout` |+| the committed phase cannot resolve the repository path | the ticket is released (queues are kept by storage and relative path) and the updates are reported lost |+| another process's cache overlaps (graceful restart) | a cache that knows of bytes after its last entry is never trusted by size: the state is loaded again, so an append never cuts off another process's durable entries (amended after review); a cache without such bytes is also checked against the last entry |+| the storage mode changes between the prepared and the committed phase | cannot: the mode is the storage core's configuration, fixed for the life of the process |## 6. Config and API changes (exact)@@ -459,6 +471,9 @@ and Markdown output.fsync.* `graft_changeset_ticket_wait_seconds` (histogram): time from the committed phaseto its entry being durable.+* `graft_changelog_unordered_records_total` (counter, added after review): committed+ transactions recorded without their place in the order, because their ticket+ expired before their committed phase or they took none.## 8. Work packagesdiff --git a/internal/git/gitcmd/hooks_payload.go b/internal/git/gitcmd/hooks_payload.goindex 166995b..e414986 100644--- a/internal/git/gitcmd/hooks_payload.go+++ b/internal/git/gitcmd/hooks_payload.go@@ -139,12 +139,13 @@ type ChangelogMeta struct {// Rewritten is true when the updates rewrite history.Rewritten bool `json:"rewritten,omitempty"`// Invocation identifies the git process that runs the reference-transaction- // hook with this payload. It is a random value set for each git command that- // git runs the hook for, so the recording hook can tell the transactions of- // one process apart from those of concurrent processes. Git runs two- // transactions to delete a packed reference, and the recording hook pairs- // them by this value. It is empty when the payload drives the hook in process- // or when the changeset log is disabled.+ // hook with this payload, or the call of updateref.UpdaterWithHooks that drives+ // the hook in process. It is a random value set for each git command that git+ // runs the hook for and for each such call, so the recording hook can tell the+ // transactions of one process apart from those of concurrent processes, also+ // when they have the same updates. Git runs two transactions to delete a+ // packed reference, and the recording hook pairs them by this value. It is+ // empty when the changeset log is disabled.Invocation string `json:"invocation,omitempty"`// MVCCCommit is the commit handle (see mvcc.CommitHandle) of the MVCC cache of// the request that builds the payload. On a repository that uses the MVCCdiff --git a/internal/gitaly/hook/changeset_order.go b/internal/gitaly/hook/changeset_order.goindex 39f85a9..5daba95 100644--- a/internal/gitaly/hook/changeset_order.go+++ b/internal/gitaly/hook/changeset_order.go@@ -2,9 +2,12 @@ package hookimport ("crypto/sha256"+ "fmt"+ "runtime/debug""sync""time"+ "github.com/feder-cr/git-ai/internal/gitaly/storage""github.com/feder-cr/git-ai/internal/graft/changelog""github.com/prometheus/client_golang/prometheus""github.com/prometheus/client_golang/prometheus/promauto"@@ -44,13 +47,24 @@ var (Name: "graft_changelog_records_lost_total",Help: "Committed reference updates that the changeset log does not record, because writing them failed.",})++ changelogUnorderedRecordsTotal = promauto.NewCounter(prometheus.CounterOpts{+ Name: "graft_changelog_unordered_records_total",+ Help: "Committed reference transactions recorded in the changeset log without their place in the order: their ticket expired before their committed phase, or they took none.",+ }))+// appendChangesetGroup writes a group of records to the changeset log. Tests replace it.+var appendChangesetGroup = changelog.AppendGroup+// changesetTicket is the place of a prepared reference transaction in the order of the changeset// log of its repository.type changesetTicket struct {- // key identifies the transaction: the digest of the updates the hook was given. Git gives- // the same updates to every phase of a transaction.+ // key identifies the transaction: the digest of the invocation of the git process that runs+ // it and of the updates the hook was given (changesetTicketKey). Git gives the same updates to+ // every phase of a transaction, and the invocation tells apart the transactions of different+ // processes, so that a transaction with the same updates as another never takes, releases or+ // records the other's ticket.key [sha256.Size]byte// done is closed when the entries of the transaction were written (see err), or when the// transaction was aborted, expired or records nothing.@@ -98,46 +112,81 @@ func (q *changesetQueue) signal() {// The tickets live in the process that serves the hook calls, which is the Gitaly process that// spawned Git. A ticket whose transaction never reports its outcome expires after// changesetTicketTimeout, after which the later transactions record without waiting for it.+//+// A ticket is in one of three states, and every change of state happens under the lock: queued+// (in the queue, without record), ready (in the queue, with its record: only the group writer+// removes it, once its entries are written), or gone (out of the queue: written, aborted,+// released or expired). The committed phase finds its queued ticket and makes it ready in one+// step (recordChangeset), so a ticket that expired or was released before that step is never+// mistaken for a written one: the transaction is then recorded at once, out of order, and counted.+//+// The queues are kept by repository (changesetQueueKey), not by path, so that a committed phase+// that cannot resolve the path of its repository can still give up its place.var changesetTickets = struct {sync.Mutexqueues map[string]*changesetQueue}{queues: map[string]*changesetQueue{}}-// changesetTicketKey returns the key of the transaction with the given hook input.-func changesetTicketKey(changes []byte) [sha256.Size]byte {- return sha256.Sum256(changes)+// changesetQueueKey returns the key of the queue of the repository: its storage and relative+// path, which every hook payload of the repository carries, whatever its object directories.+func changesetQueueKey(repo storage.Repository) string {+ return repo.GetStorageName() + "\x00" + repo.GetRelativePath()}-// changesetQueueLocked returns the queue of the repository at repoPath, creating it. The caller-// must hold changesetTickets.-func changesetQueueLocked(repoPath string) *changesetQueue {- q, ok := changesetTickets.queues[repoPath]+// changesetTicketKey returns the key of the transaction that the git process with the given+// invocation runs with the given hook input.+func changesetTicketKey(invocation string, changes []byte) [sha256.Size]byte {+ digest := sha256.New()+ digest.Write([]byte(invocation))+ digest.Write([]byte{0})+ digest.Write(changes)++ var key [sha256.Size]byte+ digest.Sum(key[:0])+ return key+}++// changesetQueueLocked returns the queue with the given key, creating it. The caller must hold+// changesetTickets.+func changesetQueueLocked(queueKey string) *changesetQueue {+ q, ok := changesetTickets.queues[queueKey]if !ok {q = &changesetQueue{changed: make(chan struct{})}- changesetTickets.queues[repoPath] = q+ changesetTickets.queues[queueKey] = q}return q}-// dropIfIdleLocked forgets the queue of repoPath when it holds nothing. The caller must hold-// changesetTickets.-func dropIfIdleLocked(repoPath string, q *changesetQueue) {- if len(q.tickets) == 0 && !q.writing && changesetTickets.queues[repoPath] == q {- delete(changesetTickets.queues, repoPath)+// dropIfIdleLocked forgets the queue q with the given key when it holds nothing. The caller must+// hold changesetTickets.+func dropIfIdleLocked(queueKey string, q *changesetQueue) {+ if len(q.tickets) == 0 && !q.writing && changesetTickets.queues[queueKey] == q {+ delete(changesetTickets.queues, queueKey)+ }+}++// queuedTicketLocked returns the position and the queued ticket with the given key in q, the+// oldest one if there are several, or nil. The caller must hold changesetTickets.+func queuedTicketLocked(q *changesetQueue, key [sha256.Size]byte) (int, *changesetTicket) {+ for i, ticket := range q.tickets {+ if ticket.key == key && ticket.record == nil {+ return i, ticket+ }}+ return -1, nil}-// takeChangesetTicket appends a ticket for the prepared transaction with the given hook input to-// the queue of the repository at repoPath.-func takeChangesetTicket(repoPath string, changes []byte) {+// takeChangesetTicket appends a ticket for the prepared transaction that the git process with the+// given invocation runs with the given hook input to the queue with the given key.+func takeChangesetTicket(queueKey, invocation string, changes []byte) {changesetTickets.Lock()defer changesetTickets.Unlock()now := time.Now()- q := changesetQueueLocked(repoPath)+ q := changesetQueueLocked(queueKey)pruneExpiredChangesetTickets(q, now)q.tickets = append(q.tickets, &changesetTicket{- key: changesetTicketKey(changes),+ key: changesetTicketKey(invocation, changes),done: make(chan struct{}),deadline: now.Add(changesetTicketTimeout),})@@ -161,78 +210,58 @@ func pruneExpiredChangesetTickets(q *changesetQueue, now time.Time) {}}-// findChangesetTicket returns the oldest ticket of the transaction with the given hook input in-// the queue of the repository at repoPath. It returns nil when the transaction has none, for-// example because it was prepared before the changeset log was enabled or its ticket expired.-func findChangesetTicket(repoPath string, changes []byte) *changesetTicket {- changesetTickets.Lock()- defer changesetTickets.Unlock()-- q, ok := changesetTickets.queues[repoPath]- if !ok {- return nil- }-- key := changesetTicketKey(changes)- for _, ticket := range q.tickets {- if ticket.key == key && ticket.record == nil {- return ticket- }- }-- return nil-}--// releaseChangesetTicket removes the ticket, whose transaction records nothing, from the queue of-// the repository at repoPath and wakes the transactions that wait for it.-func releaseChangesetTicket(repoPath string, ticket *changesetTicket) {+// releaseChangesetTicket gives up the place of the transaction that the git process with the+// given invocation runs with the given hook input, in the queue with the given key: its ticket, if+// it is still queued, is removed, and the transactions that wait for it are woken. It is for a+// transaction that was aborted or records nothing.+func releaseChangesetTicket(queueKey, invocation string, changes []byte) {changesetTickets.Lock()defer changesetTickets.Unlock()- q, ok := changesetTickets.queues[repoPath]+ q, ok := changesetTickets.queues[queueKey]if !ok {return}- for i, queued := range q.tickets {- if queued != ticket {- continue- }-+ if i, ticket := queuedTicketLocked(q, changesetTicketKey(invocation, changes)); ticket != nil {close(ticket.done)q.tickets = append(q.tickets[:i:i], q.tickets[i+1:]...)q.signal()- break}- dropIfIdleLocked(repoPath, q)+ dropIfIdleLocked(queueKey, q)}-// abortChangesetTicket releases the ticket of the aborted transaction with the given hook input,-// if it has one.-func abortChangesetTicket(repoPath string, changes []byte) {- if ticket := findChangesetTicket(repoPath, changes); ticket != nil {- releaseChangesetTicket(repoPath, ticket)- }-}--// recordInOrder records the record of the transaction of ticket in the changeset log of the-// repository at repoPath, after the transactions whose tickets are before it, and returns once-// its entries are durable or failed to be written.+// recordChangeset records record, the committed transaction that the git process with the given+// invocation runs with the given hook input, in the changeset log of the repository at repoPath,+// whose queue has the given key. It returns once the entries are durable or failed to be written.//-// The ticket becomes ready. While no group is being written and the ticket at the head of the-// queue is ready, the caller writes every ready ticket at the head with one call of-// changelog.AppendGroup, one write and one fsync, and wakes their committed phases with the+// The ticket of the transaction is found and made ready in one step. Without a queued ticket+// (it expired, or the transaction took none) the record is appended at once, out of order, and+// counted (graft_changelog_unordered_records_total). Otherwise it is recorded after the+// transactions whose tickets are before it: while no group is being written and the ticket at+// the head of the queue is ready, the caller writes every ready ticket at the head with one call+// of changelog.AppendGroup, one write and one fsync, and wakes their committed phases with the// outcome. Otherwise it waits for the queue to change or for the ticket at the head to expire.// The wait does not end with the request: the updates are committed in git already, so their// entries are always written, in order. It ends at the latest once every ticket before this one// was written, aborted or expired, which changesetTicketTimeout bounds.-func recordInOrder(repoPath string, ticket *changesetTicket, record changelog.Record) error {+func recordChangeset(queueKey, repoPath, invocation string, changes []byte, record changelog.Record) error {start := time.Now()defer func() { changesetTicketWaitSeconds.Observe(time.Since(start).Seconds()) }()changesetTickets.Lock()- q := changesetQueueLocked(repoPath)+ var ticket *changesetTicket+ q, ok := changesetTickets.queues[queueKey]+ if ok {+ _, ticket = queuedTicketLocked(q, changesetTicketKey(invocation, changes))+ }+ if ticket == nil {+ changesetTickets.Unlock()+ changelogUnorderedRecordsTotal.Inc()+ _, err := appendChangesetGroup(repoPath, []changelog.Record{record})+ return err+ }ticket.record = &recordq.signal()@@ -247,7 +276,7 @@ func recordInOrder(repoPath string, ticket *changesetTicket, record changelog.RepruneExpiredChangesetTickets(q, time.Now())if !q.writing && len(q.tickets) > 0 && q.tickets[0].record != nil {- writeChangesetGroupLocked(repoPath, q)+ writeChangesetGroupLocked(queueKey, repoPath, q)continue}@@ -273,10 +302,10 @@ func recordInOrder(repoPath string, ticket *changesetTicket, record changelog.Re}}-// writeChangesetGroupLocked writes the ready tickets at the head of q, the queue of the repository-// at repoPath, as one group. The caller must hold changesetTickets, which is released while the-// group is written.-func writeChangesetGroupLocked(repoPath string, q *changesetQueue) {+// writeChangesetGroupLocked writes the ready tickets at the head of q, the queue with the given+// key of the repository at repoPath, as one group. The caller must hold changesetTickets, which is+// released while the group is written and held again when the function returns.+func writeChangesetGroupLocked(queueKey, repoPath string, q *changesetQueue) {n := 0for n < len(q.tickets) && q.tickets[n].record != nil {n++@@ -291,7 +320,7 @@ func writeChangesetGroupLocked(repoPath string, q *changesetQueue) {}start := time.Now()- _, err := changelog.AppendGroup(repoPath, records)+ err := appendChangesetGroupRecovering(repoPath, records)changelogAppendSeconds.Observe(time.Since(start).Seconds())changelogGroupTransactions.Observe(float64(len(group)))@@ -314,5 +343,19 @@ func writeChangesetGroupLocked(repoPath string, q *changesetQueue) {q.tickets = remainingq.writing = falseq.signal()- dropIfIdleLocked(repoPath, q)+ dropIfIdleLocked(queueKey, q)+}++// appendChangesetGroupRecovering writes the group of records and turns a panic of the write into+// its error. The group's committed phases, which wait for its outcome, then all see the failure+// and report their updates as lost, and the queue is not left behind a writer that never ends.+func appendChangesetGroupRecovering(repoPath string, records []changelog.Record) (returnedErr error) {+ defer func() {+ if r := recover(); r != nil {+ returnedErr = fmt.Errorf("writing the changeset group panicked: %v\n%s", r, debug.Stack())+ }+ }()++ _, err := appendChangesetGroup(repoPath, records)+ return err}diff --git a/internal/gitaly/hook/changeset_order_test.go b/internal/gitaly/hook/changeset_order_test.goindex fecc348..5fc4f06 100644--- a/internal/gitaly/hook/changeset_order_test.go+++ b/internal/gitaly/hook/changeset_order_test.go@@ -17,6 +17,7 @@ import ("github.com/feder-cr/git-ai/internal/graft/changelog""github.com/feder-cr/git-ai/internal/testhelper""github.com/feder-cr/git-ai/internal/testhelper/testcfg"+ "github.com/feder-cr/git-ai/proto/go/gitalypb""github.com/prometheus/client_golang/prometheus/testutil""github.com/stretchr/testify/require")@@ -25,12 +26,39 @@ import (// that records its reference updates in the changeset log.type changesetOrderSetup struct {repoPath string+ // queueKey is the key of the repository's queue of tickets.+ queueKey string// first moves the branch from base to a, second from a to b, third from b to c. other creates// another branch at base.first, second, third, other stringrun func(state ReferenceTransactionState, changes string) error// runWith runs the hook with the context ctx.runWith func(ctx context.Context, state ReferenceTransactionState, changes string) error+ // runAs runs the hook as the git process with the given invocation.+ runAs func(invocation string, state ReferenceTransactionState, changes string) error+}++// queuedTicket is a ticket of a queue as it was when queuedTickets read it.+type queuedTicket struct {+ ready bool+ deadline time.Time+}++// queuedTickets returns the tickets of the queue with the given key, read under the queue's+// lock, nil when there is none.+func queuedTickets(queueKey string) []queuedTicket {+ changesetTickets.Lock()+ defer changesetTickets.Unlock()++ q, ok := changesetTickets.queues[queueKey]+ if !ok {+ return nil+ }+ tickets := make([]queuedTicket, 0, len(q.tickets))+ for _, ticket := range q.tickets {+ tickets = append(tickets, queuedTicket{ready: ticket.record != nil, deadline: ticket.deadline})+ }+ return tickets}func setupChangesetOrder(t *testing.T, ctx context.Context) changesetOrderSetup {@@ -50,12 +78,18 @@ func setupChangesetOrder(t *testing.T, ctx context.Context) changesetOrderSetupt, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive,), NewPostReceiveRegistry())- payload, err := gitcmd.NewHooksPayload(ctx, cfg, repo, gittest.DefaultObjectHash, nil, &gitcmd.UserDetails{- UserID: "user-1",- Username: "user",- Protocol: "web",- }, gitcmd.ReferenceTransactionHook, nil).Env()- require.NoError(t, err)+ payloadAs := func(invocation string) string {+ payload := gitcmd.NewHooksPayload(ctx, cfg, repo, gittest.DefaultObjectHash, nil, &gitcmd.UserDetails{+ UserID: "user-1",+ Username: "user",+ Protocol: "web",+ }, gitcmd.ReferenceTransactionHook, nil)+ payload.Changelog.Invocation = invocation+ env, err := payload.Env()+ require.NoError(t, err)+ return env+ }+ payload := payloadAs("")runWith := func(ctx context.Context, state ReferenceTransactionState, changes string) error {return manager.ReferenceTransactionHook(ctx, state, []string{payload}, strings.NewReader(changes))@@ -63,6 +97,7 @@ func setupChangesetOrder(t *testing.T, ctx context.Context) changesetOrderSetupreturn changesetOrderSetup{repoPath: repoPath,+ queueKey: changesetQueueKey(repo),first: fmt.Sprintf("%s %s refs/heads/main\n", base, a),second: fmt.Sprintf("%s %s refs/heads/main\n", a, b),third: fmt.Sprintf("%s %s refs/heads/main\n", b, c),@@ -71,6 +106,9 @@ func setupChangesetOrder(t *testing.T, ctx context.Context) changesetOrderSetupreturn runWith(ctx, state, changes)},runWith: runWith,+ runAs: func(invocation string, state ReferenceTransactionState, changes string) error {+ return manager.ReferenceTransactionHook(ctx, state, []string{payloadAs(invocation)}, strings.NewReader(changes))+ },}}@@ -214,7 +252,7 @@ func TestReferenceTransactionHook_changesetOrder(t *testing.T) {require.Eventually(t, func() bool {changesetTickets.Lock()defer changesetTickets.Unlock()- q := changesetTickets.queues[setup.repoPath]+ q := changesetTickets.queues[setup.queueKey]return q != nil && len(q.tickets) == 3 && q.tickets[1].record != nil && q.tickets[2].record != nil}, 10*time.Second, 5*time.Millisecond)require.Empty(t, recordedChanges(t, setup.repoPath))@@ -232,13 +270,90 @@ func TestReferenceTransactionHook_changesetOrder(t *testing.T) {require.Equal(t, entries[0].RecordedAt, entries[1].RecordedAt)require.Equal(t, entries[0].RecordedAt, entries[2].RecordedAt)- changesetTickets.Lock()- _, ok := changesetTickets.queues[setup.repoPath]- changesetTickets.Unlock()- require.False(t, ok, "the queue is dropped once it is empty")+ require.Nil(t, queuedTickets(setup.queueKey), "the queue is dropped once it is empty")+ })++ t.Run("a transaction with the same updates in another process keeps the other's ticket", func(t *testing.T) {+ t.Parallel()++ setup := setupChangesetOrder(t, ctx)++ // Process A prepared the update. Process B tried the same update, could not lock the+ // reference and aborted: its abort must not give up A's place.+ require.NoError(t, setup.runAs("process-a", ReferenceTransactionPrepared, setup.first))+ require.NoError(t, setup.runAs("process-b", ReferenceTransactionAborted, setup.first))+ require.Len(t, queuedTickets(setup.queueKey), 1)++ // B's committed phase, had it committed, would not record with A's ticket either: A's+ // committed phase finds its own.+ require.NoError(t, setup.runAs("process-a", ReferenceTransactionPrepared, setup.second))+ secondDone := make(chan error, 1)+ go func() { secondDone <- setup.runAs("process-a", ReferenceTransactionCommitted, setup.second) }()+ require.Eventually(t, func() bool {+ tickets := queuedTickets(setup.queueKey)+ return len(tickets) == 2 && tickets[1].ready+ }, 10*time.Second, 5*time.Millisecond)+ require.Empty(t, recordedChanges(t, setup.repoPath), "the second waits for the first")++ require.NoError(t, setup.runAs("process-a", ReferenceTransactionCommitted, setup.first))+ require.NoError(t, <-secondDone)+ require.Equal(t, []string{setup.first, setup.second}, recordedChanges(t, setup.repoPath))})}+// TestReferenceTransactionHook_changesetCommittedAfterExpiry is not parallel because it lowers the+// ticket timeout of the package and reads global counters. A committed phase that runs after its+// ticket expired, and after a later transaction was recorded without it, is recorded too: out of+// order, counted as such, and not lost.+func TestReferenceTransactionHook_changesetCommittedAfterExpiry(t *testing.T) {+ ctx := testhelper.Context(t)++ defaultTimeout := changesetTicketTimeout+ changesetTicketTimeout = 50 * time.Millisecond+ t.Cleanup(func() { changesetTicketTimeout = defaultTimeout })++ setup := setupChangesetOrder(t, ctx)+ unordered := testutil.ToFloat64(changelogUnorderedRecordsTotal)+ lost := testutil.ToFloat64(changelogRecordsLostTotal)++ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.first))+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.second))+ // The second waits for the first's ticket to expire, then records.+ require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.second))+ require.Equal(t, []string{setup.second}, recordedChanges(t, setup.repoPath))++ // The first's ticket is gone: its committed phase is recorded at once.+ require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.first))+ require.Equal(t, []string{setup.second, setup.first}, recordedChanges(t, setup.repoPath))+ require.InDelta(t, 1, testutil.ToFloat64(changelogUnorderedRecordsTotal)-unordered, 0)+ require.InDelta(t, 0, testutil.ToFloat64(changelogRecordsLostTotal)-lost, 0)+ require.Nil(t, queuedTickets(setup.queueKey), "no queue is left behind")+}++// TestReferenceTransactionHook_changesetGroupPanic is not parallel because it replaces the group+// writer of the package and reads a global counter. A write that panics fails its group, whose+// updates are reported lost, and leaves the queue usable.+func TestReferenceTransactionHook_changesetGroupPanic(t *testing.T) {+ ctx := testhelper.Context(t)+ setup := setupChangesetOrder(t, ctx)++ appendChangesetGroup = func(string, []changelog.Record) ([][]changelog.Entry, error) {+ panic("injected")+ }+ t.Cleanup(func() { appendChangesetGroup = changelog.AppendGroup })++ lost := testutil.ToFloat64(changelogRecordsLostTotal)+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.first))+ require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.first), "recording never fails the update")+ require.InDelta(t, 1, testutil.ToFloat64(changelogRecordsLostTotal)-lost, 0)+ require.Nil(t, queuedTickets(setup.queueKey), "the queue is not left writing")++ appendChangesetGroup = changelog.AppendGroup+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.second))+ require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.second))+ require.Equal(t, []string{setup.second}, recordedChanges(t, setup.repoPath))+}+// TestReferenceTransactionHook_changesetRecordsLost is not parallel because it reads the counter// of lost records, which is global.func TestReferenceTransactionHook_changesetRecordsLost(t *testing.T) {@@ -269,20 +384,43 @@ func TestReferenceTransactionHook_changesetOrderExpiredTicket(t *testing.T) {// records once the ticket of the first one expired.require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.first))require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.second))+ firstDeadline := queuedTickets(setup.queueKey)[0].deadline- start := time.Now()+ // The second records only once the first's ticket expired: not before its deadline.require.NoError(t, setup.run(ReferenceTransactionCommitted, setup.second))- require.GreaterOrEqual(t, time.Since(start), 50*time.Millisecond)+ require.False(t, time.Now().Before(firstDeadline), "recorded before the earlier ticket expired")require.Equal(t, []string{setup.second}, recordedChanges(t, setup.repoPath))- // The expired ticket is dropped when the next transaction is prepared.- require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.first))- changesetTickets.Lock()- queue := changesetTickets.queues[setup.repoPath].tickets- changesetTickets.Unlock()- require.Len(t, queue, 1)- require.NoError(t, setup.run(ReferenceTransactionAborted, setup.first))+ // The expired ticket was dropped: the next transaction finds an empty queue.+ require.NoError(t, setup.run(ReferenceTransactionPrepared, setup.third))+ require.Len(t, queuedTickets(setup.queueKey), 1)+ require.NoError(t, setup.run(ReferenceTransactionAborted, setup.third))+}++// TestReferenceTransactionHook_changesetUnresolvedRepository is not parallel because it reads a+// global counter. A committed phase that cannot resolve its repository path gives up its place, so+// that later transactions do not wait for it, and reports its updates lost.+func TestReferenceTransactionHook_changesetUnresolvedRepository(t *testing.T) {+ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)++ manager := NewManager(cfg, config.NewLocator(cfg), testhelper.SharedLogger(t), gittest.NewCommandFactory(t, cfg), nil, &transaction.MockManager{}, gitlab.NewMockClient(+ t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive,+ ), NewPostReceiveRegistry())++ // The repository does not exist: its path does not resolve.+ repo := &gitalypb.Repository{StorageName: cfg.Storages[0].Name, RelativePath: "missing.git"}+ payload, err := gitcmd.NewHooksPayload(ctx, cfg, repo, gittest.DefaultObjectHash, nil, nil, gitcmd.ReferenceTransactionHook, nil).Env()+ require.NoError(t, err)+ changes := fmt.Sprintf("%s %s refs/heads/main\n", gittest.DefaultObjectHash.ZeroOID, gittest.DefaultObjectHash.EmptyTreeOID)++ lost := testutil.ToFloat64(changelogRecordsLostTotal)+ require.NoError(t, manager.ReferenceTransactionHook(ctx, ReferenceTransactionPrepared, []string{payload}, strings.NewReader(changes)))+ require.Len(t, queuedTickets(changesetQueueKey(repo)), 1)+ require.NoError(t, manager.ReferenceTransactionHook(ctx, ReferenceTransactionCommitted, []string{payload}, strings.NewReader(changes)))+ require.Nil(t, queuedTickets(changesetQueueKey(repo)), "the ticket is released")+ require.InDelta(t, 1, testutil.ToFloat64(changelogRecordsLostTotal)-lost, 0)}func TestReferenceTransactionHook_changesetOrderDisabled(t *testing.T) {@@ -308,8 +446,5 @@ func TestReferenceTransactionHook_changesetOrderDisabled(t *testing.T) {require.NoError(t, manager.ReferenceTransactionHook(ctx, ReferenceTransactionPrepared, []string{payload}, strings.NewReader(changes)))// Without the changeset log, transactions are not ordered.- changesetTickets.Lock()- _, ok := changesetTickets.queues[repoPath]- changesetTickets.Unlock()- require.False(t, ok)+ require.Nil(t, queuedTickets(changesetQueueKey(repo)))}diff --git a/internal/gitaly/hook/custom.go b/internal/gitaly/hook/custom.goindex 839620c..fd8416e 100644--- a/internal/gitaly/hook/custom.go+++ b/internal/gitaly/hook/custom.go@@ -244,7 +244,7 @@ func (m *GitLabHookManager) customHooksEnv(ctx context.Context, payload gitcmd.H// configuration, which the command factory applies after them.func (m *GitLabHookManager) fsyncConfiguration() []gitcmd.ConfigPair {pairs := []gitcmd.ConfigPair{- {Key: "core.fsync", Value: "objects,derived-metadata,reference"},+ {Key: "core.fsync", Value: gitcmd.FsyncComponents},{Key: "core.fsyncMethod", Value: "fsync"},}diff --git a/internal/gitaly/hook/referencetransaction.go b/internal/gitaly/hook/referencetransaction.goindex ebd000c..6f91323 100644--- a/internal/gitaly/hook/referencetransaction.go+++ b/internal/gitaly/hook/referencetransaction.go@@ -74,7 +74,7 @@ func (m *GitLabHookManager) ReferenceTransactionHook(ctx context.Context, state// An aborted transaction gives up its place in the order of the changeset log, so that the// transactions prepared after it do not wait for it.case ReferenceTransactionAborted:- m.abortChangesets(ctx, payload, changes)+ m.abortChangesets(payload, env, changes)if m.storageVoter != nil {m.storageVoter.Aborted(ctx, payload, changes)}@@ -156,42 +156,46 @@ func (m *GitLabHookManager) ReferenceTransactionHook(ctx context.Context, state// the order of the changeset log now. It does so only once the vote succeeded, because a// failed hook aborts the transaction.// In authoritative storage mode the file changelog is not written, so no order is kept for it.+ // The mode is the storage core's configuration, fixed for the life of the process, so the+ // prepared and the committed phase of a transaction decide alike.if state == ReferenceTransactionPrepared && !m.storageAuthoritative() {- m.prepareChangesets(ctx, payload, changes)+ m.prepareChangesets(payload, env, changes)}return nil}-// changesetRepoPath returns the path of the repository whose changeset log records the given-// hook input, or false when nothing of it is recorded.-func (m *GitLabHookManager) changesetRepoPath(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) (string, bool) {- if m.cfg.Graft.Changelog.Disabled || len(changelog.ParseUpdates(changes)) == 0 {- return "", false- }+// ordersChangesets reports whether the transaction with the given hook input takes a place in the+// order of the changeset log of its repository.+func (m *GitLabHookManager) ordersChangesets(payload gitcmd.HooksPayload, changes []byte) bool {+ return !m.cfg.Graft.Changelog.Disabled && payload.Repo != nil && len(changelog.ParseUpdates(changes)) > 0+}- repoPath, err := m.locator.GetRepoPath(ctx, payload.Repo)+// changesetInvocation returns the invocation of the git process that runs the transaction, which+// tells its ticket apart from those of the transactions of other processes (changesetTicketKey).+// It is empty when the payload carries none; the prepared and the committed phase of a+// transaction read the same environment, so they agree.+func changesetInvocation(env []string) string {+ meta, err := gitcmd.ChangelogMetaFromEnv(env)if err != nil {- m.logger.WithError(err).ErrorContext(ctx, "ordering changesets: resolving repository path")- return "", false+ return ""}-- return repoPath, true+ return meta.Invocation}// prepareChangesets gives the prepared transaction its place in the order of the changeset log,// see changesetTickets.-func (m *GitLabHookManager) prepareChangesets(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {- if repoPath, ok := m.changesetRepoPath(ctx, payload, changes); ok {- takeChangesetTicket(repoPath, changes)+func (m *GitLabHookManager) prepareChangesets(payload gitcmd.HooksPayload, env []string, changes []byte) {+ if m.ordersChangesets(payload, changes) {+ takeChangesetTicket(changesetQueueKey(payload.Repo), changesetInvocation(env), changes)}}// abortChangesets gives up the place of the aborted transaction in the order of the changeset// log, see changesetTickets.-func (m *GitLabHookManager) abortChangesets(ctx context.Context, payload gitcmd.HooksPayload, changes []byte) {- if repoPath, ok := m.changesetRepoPath(ctx, payload, changes); ok {- abortChangesetTicket(repoPath, changes)+func (m *GitLabHookManager) abortChangesets(payload gitcmd.HooksPayload, env []string, changes []byte) {+ if m.ordersChangesets(payload, changes) {+ releaseChangesetTicket(changesetQueueKey(payload.Repo), changesetInvocation(env), changes)}}@@ -233,22 +237,23 @@ func (m *GitLabHookManager) recordChangesets(ctx context.Context, objectHash gitreturn}- repoPath, err := m.locator.GetRepoPath(ctx, payload.Repo)- if err != nil {- m.logger.WithError(err).ErrorContext(ctx, "recording changesets: resolving repository path")- return- }-// The updates are recorded in the order in which their transactions were prepared, which// is the order in which the references were updated. See changesetTickets. A transaction// that records nothing gives up its place at once, so that later transactions do not wait- // for it: a packed-refs cleanup, a transaction whose metadata cannot be read, and one on- // the MVCC reference backend, whose updates are recorded after its commit.- ticket := findChangesetTicket(repoPath, changes)+ // for it: one whose repository path cannot be resolved, a packed-refs cleanup, one whose+ // metadata cannot be read, and one on the MVCC reference backend, whose updates are+ // recorded after its commit.+ queueKey := changesetQueueKey(payload.Repo)+ invocation := changesetInvocation(env)release := func() {- if ticket != nil {- releaseChangesetTicket(repoPath, ticket)- }+ releaseChangesetTicket(queueKey, invocation, changes)+ }++ repoPath, err := m.locator.GetRepoPath(ctx, payload.Repo)+ if err != nil {+ release()+ m.reportLostChangesets(ctx, updates, fmt.Errorf("resolving repository path: %w", err))+ return}if packedCleanup {@@ -282,16 +287,10 @@ func (m *GitLabHookManager) recordChangesets(ctx context.Context, objectHash gitreturn}- // A transaction without a ticket, prepared before the changeset log was enabled or whose- // ticket expired, is recorded at once.- var recordErr error- if ticket == nil {- _, recordErr = changelog.Append(repoPath, record.Meta, record.Updates)- } else {- recordErr = recordInOrder(repoPath, ticket, record)- }- if recordErr != nil {- m.reportLostChangesets(ctx, updates, recordErr)+ // A transaction without a queued ticket, prepared before the changeset log was enabled or+ // whose ticket expired, is recorded at once (see recordChangeset).+ if err := recordChangeset(queueKey, repoPath, invocation, changes, record); err != nil {+ m.reportLostChangesets(ctx, updates, err)}}@@ -340,7 +339,9 @@ func (m *GitLabHookManager) observeCommitted(ctx context.Context, payload gitcmd// runs the hook for itself are tracked: their hooks payload carries the// invocation of the git process.func (m *GitLabHookManager) trackPendingTransaction(state ReferenceTransactionState, env []string, changes []byte) {- if m.cfg.Graft.Changelog.Disabled || m.pendingTransactions == nil {+ // In authoritative storage mode recordChangesets does not run, so nothing would ever take a+ // tracked transaction out again.+ if m.cfg.Graft.Changelog.Disabled || m.pendingTransactions == nil || m.storageAuthoritative() {return}diff --git a/internal/gitaly/hook/updateref/update_with_hooks.go b/internal/gitaly/hook/updateref/update_with_hooks.goindex 917d6b6..e62a40c 100644--- a/internal/gitaly/hook/updateref/update_with_hooks.go+++ b/internal/gitaly/hook/updateref/update_with_hooks.go@@ -232,7 +232,22 @@ func (u *UpdaterWithHooks) UpdateReference(quarantinedRepo = quarantineDir.QuarantinedRepo()}- hooksPayload, err := gitcmd.NewHooksPayload(ctx, u.cfg, quarantinedRepo, objectHash, transaction, &receiveHooksPayload, gitcmd.ReceivePackHooks, featureflag.FromContext(ctx)).Env()+ // This function drives the reference-transaction hook of one transaction itself. The+ // changeset log tells it apart from every other transaction, also from one with the same+ // updates, by a value unique to this call, as it does for a git process that runs the hook.+ var invocation string+ if !u.cfg.Graft.Changelog.Disabled {+ if invocation, err = gitcmd.NewChangelogInvocation(); err != nil {+ return err+ }+ }+ newHooksPayload := func(repo *gitalypb.Repository) (string, error) {+ payload := gitcmd.NewHooksPayload(ctx, u.cfg, repo, objectHash, transaction, &receiveHooksPayload, gitcmd.ReceivePackHooks, featureflag.FromContext(ctx))+ payload.Changelog.Invocation = invocation+ return payload.Env()+ }++ hooksPayload, err := newHooksPayload(quarantinedRepo)if err != nil {return fmt.Errorf("constructing hooks payload: %w", err)}@@ -254,7 +269,7 @@ func (u *UpdaterWithHooks) UpdateReference(// We only need to update the hooks payload to the unquarantined repo in case we// had a quarantine environment. Otherwise, the initial hooks payload is for the// real repository anyway.- hooksPayload, err = gitcmd.NewHooksPayload(ctx, u.cfg, repoProto, objectHash, transaction, &receiveHooksPayload, gitcmd.ReceivePackHooks, featureflag.FromContext(ctx)).Env()+ hooksPayload, err = newHooksPayload(repoProto)if err != nil {return fmt.Errorf("constructing quarantined hooks payload: %w", err)}diff --git a/internal/graft/changelog/log.go b/internal/graft/changelog/log.goindex a658e70..ddb950e 100644--- a/internal/graft/changelog/log.go+++ b/internal/graft/changelog/log.go@@ -326,8 +326,9 @@ func Append(repoPath string, meta Meta, updates []Update) ([]Entry, error) {// the repository itself, so entries written there would be lost.// - Each Gitaly node keeps its own log and its own sequences. Behind Praefect,// reads may be routed to a replica whose log lags or differs.-// - The lock is taken without a timeout. A holder that never releases it blocks-// every append to the repository.+// - The wait for the lock is bounded by lockWaitTimeout: an append that does+// not get it fails. An fsync that never returns is not bounded: its append+// holds the lock, and the appends after it fail at lockWaitTimeout.func AppendGroup(repoPath string, records []Record) ([][]Entry, error) {entries := make([][]Entry, len(records))@@ -415,7 +416,9 @@ func (s *logState) appendLocked(repoPath, path string, records []Record, entriesreturn fmt.Errorf("stat changelog: %w", err)}- if !s.loaded || info.Size() != s.fileSize {+ if valid, err := s.cacheDescribes(f, info.Size()); err != nil {+ return err+ } else if !valid {if err := s.load(f, info.Size()); err != nil {return err}@@ -515,6 +518,43 @@ func (s *logState) appendLocked(repoPath, path string, records []Record, entriesreturn nil}+// cacheDescribes reports whether the cached state still describes the file f, whose size is+// size, so that it need not be loaded again. The caller must hold s.mu.+//+// The size alone does not tell: while the cache knows of bytes after its last valid entry (a torn+// tail, or the untrusted bytes of a failed append), another process may have cut them off and+// appended durable entries that end at the same size, and acting on the cache would cut those+// off. Such a cache is therefore never trusted. A cache without such bytes can only see the file+// grow, as no process cuts the file below the end of the durable entries; it is trusted when the+// size is unchanged and the line before its end still holds its last entry.+func (s *logState) cacheDescribes(f *os.File, size int64) (bool, error) {+ if !s.loaded || size != s.fileSize || s.fileSize != s.validEnd {+ return false, nil+ }+ if s.validEnd == 0 {+ return s.lastSeq == 0, nil+ }++ nl := s.validEnd - 1+ prev, err := lastNewlineBefore(f, nl)+ if err != nil {+ return false, err+ }+ line := make([]byte, nl-(prev+1)+1)+ if _, err := f.ReadAt(line, prev+1); err != nil {+ return false, fmt.Errorf("reading changelog: %w", err)+ }+ if line[len(line)-1] != '\n' {+ return false, nil+ }++ var entry Entry+ if err := json.Unmarshal(line[:len(line)-1], &entry); err != nil {+ return false, nil+ }+ return entry.Sequence == s.lastSeq, nil+}+// discardFrom cuts off the bytes that a failed append wrote from end on. When// that fails too, the bytes are remembered as untrusted, so that no reader takes// them for entries. The caller must hold s.mu.@@ -716,8 +756,10 @@ var truncateFile = func(f *os.File, size int64) error {// or the file changed since it was (an append of another process), the state is// loaded again under the log's lock file, which waits for an append of another// process, and the file is fsync'd first: bytes that a crashed writer left in the-// page cache are durable once a reader may see them.-func (s *logState) durableView(path string, f *os.File) (int64, uint64, error) {+// page cache are durable once a reader may see them. So are the directory entries+// of the log, which that writer may have crashed before syncing (see+// syncLogDirs), unless this process synced them already.+func (s *logState) durableView(repoPath, path string, f *os.File) (int64, uint64, error) {s.mu.Lock()defer s.mu.Unlock()@@ -725,7 +767,9 @@ func (s *logState) durableView(path string, f *os.File) (int64, uint64, error) {if err != nil {return 0, 0, fmt.Errorf("stat changelog: %w", err)}- if s.loaded && info.Size() == s.fileSize {+ if valid, err := s.cacheDescribes(f, info.Size()); err != nil {+ return 0, 0, err+ } else if valid && s.dirsSynced {return s.validEnd, s.lastSeq, nil}@@ -738,6 +782,12 @@ func (s *logState) durableView(path string, f *os.File) (int64, uint64, error) {if err := syncFile(f); err != nil {return 0, 0, fmt.Errorf("syncing changelog: %w", err)}+ if !s.dirsSynced {+ if err := syncLogDirs(repoPath, path); err != nil {+ return 0, 0, err+ }+ s.dirsSynced = true+ }info, err = f.Stat()if err != nil {@@ -763,23 +813,57 @@ var syncDir = func(dir string) error {return d.Sync()}+// lockWaitTimeout bounds the wait for the log's lock file, which another process holds for one+// append. A holder that never releases it would otherwise block every append and every reader of+// the repository's log: the wait fails instead, and the appends see the error (their updates are+// reported lost). Each wait that failed keeps a goroutine and a descriptor until the holder+// releases the lock. Tests lower it.+var lockWaitTimeout = 30 * time.Second+// lockFile takes an exclusive flock on the file at path and returns the-// function that releases it.+// function that releases it. It waits at most lockWaitTimeout.func lockFile(path string) (func(), error) {f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o644)if err != nil {return nil, fmt.Errorf("opening changelog lock: %w", err)}+ unlock := func() {+ _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN)+ _ = f.Close()+ }- if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX); err != nil {+ err = syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)+ if err == nil {+ return unlock, nil+ }+ if !errors.Is(err, syscall.EWOULDBLOCK) {f.Close()return nil, fmt.Errorf("locking changelog: %w", err)}- return func() {- _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN)- _ = f.Close()- }, nil+ // Another process holds the lock: wait for it in a goroutine, which owns the file if the+ // wait gives up first and releases the lock once it is granted.+ locked := make(chan error, 1)+ go func() { locked <- syscall.Flock(int(f.Fd()), syscall.LOCK_EX) }()++ timer := time.NewTimer(lockWaitTimeout)+ defer timer.Stop()+ select {+ case err := <-locked:+ if err != nil {+ f.Close()+ return nil, fmt.Errorf("locking changelog: %w", err)+ }+ return unlock, nil+ case <-timer.C:+ go func() {+ if err := <-locked; err == nil {+ _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN)+ }+ _ = f.Close()+ }()+ return nil, fmt.Errorf("locking changelog: the lock was not granted within %s", lockWaitTimeout)+ }}// Scan calls fn for every entry of the log of the repository whose git@@ -804,7 +888,7 @@ func Scan(repoPath string, after uint64, fn func(Entry) (bool, error)) error {}defer f.Close()- end, _, err := state.durableView(path, f)+ end, _, err := state.durableView(repoPath, path, f)if err != nil {return err}@@ -868,7 +952,7 @@ func LastSequence(repoPath string) (uint64, error) {}defer f.Close()- _, lastSeq, err := state.durableView(path, f)+ _, lastSeq, err := state.durableView(repoPath, path, f)if err != nil {return 0, err}diff --git a/internal/graft/changelog/log_durable_test.go b/internal/graft/changelog/log_durable_test.goindex 6dcb024..6dd48ac 100644--- a/internal/graft/changelog/log_durable_test.go+++ b/internal/graft/changelog/log_durable_test.go@@ -117,20 +117,35 @@ func TestReadersSeeEntriesOnlyOnceDurable(t *testing.T) {require.NoError(t, err)require.Contains(t, string(data), "refs/heads/b")+ // The reader comes without this process's state, as a reader of another process does: it+ // does not wait for the state's mutex but for the log's lock file, which the append holds.+ statesMu.Lock()+ delete(states, Path(repoPath))+ statesMu.Unlock()+type result struct {refs []stringlast uint64+ err error}read := make(chan result, 1)go func() {var r result- r.refs = references(t, repoPath, 0)- r.last, _ = LastSequence(repoPath)+ var entries []Entry+ if entries, r.err = List(repoPath, 0, 10); r.err == nil {+ for _, entry := range entries {+ r.refs = append(r.refs, entry.Reference)+ }+ r.last, r.err = LastSequence(repoPath)+ }read <- r}()+ readEarly := falseselect {case r := <-read:+ readEarly = true+ require.NoError(t, r.err)require.Equal(t, []string{"refs/heads/a"}, r.refs, "a reader returned an entry before its fsync")require.EqualValues(t, 1, r.last)case <-time.After(200 * time.Millisecond):@@ -140,8 +155,10 @@ func TestReadersSeeEntriesOnlyOnceDurable(t *testing.T) {close(release)require.NoError(t, <-appended)- r := <-read- if len(r.refs) == 2 {+ if !readEarly {+ // The reader waited for the lock file: it read once the entry was durable.+ r := <-read+ require.NoError(t, r.err)require.Equal(t, []string{"refs/heads/a", "refs/heads/b"}, r.refs)require.EqualValues(t, 2, r.last)}diff --git a/internal/graft/changelog/log_review_test.go b/internal/graft/changelog/log_review_test.gonew file mode 100644index 0000000..5562e23--- /dev/null+++ b/internal/graft/changelog/log_review_test.go@@ -0,0 +1,142 @@+package changelog++import (+ "os"+ "path/filepath"+ "sync"+ "syscall"+ "testing"+ "time"++ "github.com/stretchr/testify/require"+)++// sequencesAndReferences lists the log of repoPath as sequence and reference pairs.+func sequencesAndReferences(t *testing.T, repoPath string) []string {+ t.Helper()++ entries, err := List(repoPath, 0, 100)+ require.NoError(t, err)+ var listed []string+ for _, entry := range entries {+ listed = append(listed, filepath.Base(entry.Reference)+"@"+string(rune('0'+entry.Sequence)))+ }+ return listed+}++// This test replaces syncDir, so it must not run in parallel with other tests. A reader whose+// process did not append syncs the directories of the log before it serves entries: the writer+// may have crashed between the fsync of the file and those of the directories.+func TestReaders_syncDirectoriesOfAnotherProcess(t *testing.T) {+ var (+ mu sync.Mutex+ synced []string+ )+ originalSyncDir := syncDir+ t.Cleanup(func() { syncDir = originalSyncDir })+ syncDir = func(dir string) error {+ mu.Lock()+ synced = append(synced, dir)+ mu.Unlock()+ return originalSyncDir(dir)+ }+ takeSynced := func() []string {+ mu.Lock()+ defer mu.Unlock()+ result := synced+ synced = nil+ return result+ }++ repoPath := newRepoPath(t)+ _, err := Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/a", OldOID: zeroOID, NewOID: oidA}})+ require.NoError(t, err)+ takeSynced()++ // A reader of another process: this process's state is gone.+ dropState(repoPath)+ require.Equal(t, []string{"a@1"}, sequencesAndReferences(t, repoPath))+ require.ElementsMatch(t, []string{filepath.Dir(Path(repoPath)), repoPath}, takeSynced())++ // Once synced, the next read does not sync them again.+ require.Equal(t, []string{"a@1"}, sequencesAndReferences(t, repoPath))+ require.Empty(t, takeSynced())+}++// This test lowers lockWaitTimeout, so it must not run in parallel with other tests. A lock file+// that another process never releases fails the appends and the reads after lockWaitTimeout, and+// the log works again once it is released.+func TestAppend_lockWaitIsBounded(t *testing.T) {+ originalTimeout := lockWaitTimeout+ t.Cleanup(func() { lockWaitTimeout = originalTimeout })+ lockWaitTimeout = 100 * time.Millisecond++ repoPath := newRepoPath(t)+ _, err := Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/a", OldOID: zeroOID, NewOID: oidA}})+ require.NoError(t, err)++ // Another process holds the lock file.+ holder, err := os.OpenFile(Path(repoPath)+lockSuffix, os.O_RDWR, 0o644)+ require.NoError(t, err)+ require.NoError(t, syscall.Flock(int(holder.Fd()), syscall.LOCK_EX))++ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/b", OldOID: zeroOID, NewOID: oidB}})+ require.ErrorContains(t, err, "the lock was not granted within 100ms")++ // A reader that has to wait for the lock (it comes without the state) fails the same way.+ dropState(repoPath)+ _, err = List(repoPath, 0, 10)+ require.ErrorContains(t, err, "the lock was not granted within 100ms")++ require.NoError(t, syscall.Flock(int(holder.Fd()), syscall.LOCK_UN))+ require.NoError(t, holder.Close())++ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/c", OldOID: zeroOID, NewOID: oidC}})+ require.NoError(t, err)+ require.Equal(t, []string{"a@1", "c@2"}, sequencesAndReferences(t, repoPath))+}++// TestAppend_doesNotTrustACacheWithATail is the overlap of two processes during a graceful+// restart. Process A's cache knows of bytes after its last valid entry, which it would cut off on+// its next append. Process B cut them off itself and appended a durable entry, which ends at the+// same size. A must not take the size for an unchanged file: it would cut off B's entry.+func TestAppend_doesNotTrustACacheWithATail(t *testing.T) {+ t.Parallel()++ repoPath := newRepoPath(t)+ path := Path(repoPath)+ _, err := Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/a", OldOID: zeroOID, NewOID: oidA}})+ require.NoError(t, err)++ statesMu.Lock()+ processA := states[path]+ delete(states, path)+ statesMu.Unlock()+ afterA := processA.validEnd++ // Process B appends with its own state.+ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/b", OldOID: zeroOID, NewOID: oidB}})+ require.NoError(t, err)+ info, err := os.Stat(path)+ require.NoError(t, err)++ // Process A is back, and its cache takes everything after its entry for a tail of the size the+ // file has now.+ statesMu.Lock()+ states[path] = processA+ statesMu.Unlock()+ processA.mu.Lock()+ processA.fileSize = info.Size()+ processA.validEnd = afterA+ processA.mu.Unlock()++ require.Equal(t, []string{"a@1", "b@2"}, sequencesAndReferences(t, repoPath), "a reader of A sees B's entry")++ processA.mu.Lock()+ processA.fileSize = info.Size()+ processA.validEnd = afterA+ processA.mu.Unlock()+ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/c", OldOID: zeroOID, NewOID: oidC}})+ require.NoError(t, err)+ require.Equal(t, []string{"a@1", "b@2", "c@3"}, sequencesAndReferences(t, repoPath), "A's append keeps B's entry")+}