Git designed for AI agents.
Final review minors 1-5: checkpoint stamp, seed deadline, seed lifecycle, invariant U, late checkpoints
Federico Eliadac3ea4778ac
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.mdindex 72877b7..54f32bb 100644--- a/doc/graft/PLAN7.md+++ b/doc/graft/PLAN7.md@@ -1414,7 +1414,15 @@ Binding order:(`Internal`), never a partial pack.4. Re-read the record (`t_r`), `Core.UploadPack`. `//! after_upload`.5. `CreateCheckpoint(repoID, {Seq: N, Refs: refs, PackKey, PackBytes})` under the- upload's `CommitContext`. `//! after_commit`.+ upload's `CommitContext`. `//! after_commit`. In its transaction the last+ statements, after the snapshot is written, check the pack's deadline on the+ database clock and stamp `created_at` with the database time (final minor 1):+ the age of a checkpoint (C_gc, C_sync) counts from a time one statement+ before the commit, whatever the size of the snapshot, so the visibility lag+ does not grow with the number of references or lean on `gc_margin`. Tests:+ `TestCheckpointOfManyReferences` (created_at after the snapshot, 100k+ references), `TestCheckpointDeadlineIsCheckedAfterTheSnapshot` (a deadline+ that passes during the snapshot fails the record, nothing recorded).A crash after 3 leaves the pack as an orphan (reclaimed by the orphan protocol); are-run starts over from the head at that time. The build holds no reference lock@@ -1728,6 +1736,30 @@ leaves it:catches up and every reference matches.* Bench with a checkpoint pack of hundreds of MB: requested in a window (seethe final report).+* **Deadline on the database clock (final minor 2).** The job's context ends+ at `readAt + W` on the host's monotonic clock, which stops while the host is+ suspended. A seed that fails is checked against the database clock+ (`checkSeedDeadline`: the record read again, `r.ReadAt + W`): past it, the+ failure is `Unavailable` "seed deadline passed", whatever the seed ran into+ (a checkpoint pack collected after the deadline reads as a missing pack,+ which would otherwise be reported as corruption). A seed that indexed the+ whole pack is installed even past its deadline: it holds every object of the+ checkpoint, verified against its key, and the changeset packs after it are+ covered by the C_sync check right before the install (`errSeedSuperseded`).+* **Removed mid-seed (final minor 3).** The same check finds a repository+ removed, or removed and registered again under a new ID: `Unavailable` "the+ repository was removed during the seed", nothing installed; the next request+ starts from the current state.+* **Lifecycle (final minor 3).** `StopSeeds` (deferred by `gitaly serve` after+ its servers, before the last durable step) cancels the seed jobs, waits for+ them, and refuses new ones ("the server is stopping"). Every build directory+ (`hydrate-*`) holds a flock on its `build.lock` until it is removed;+ `CleanBuilds`, at startup before the listeners, removes those whose lock is+ free (their process died) or that have none and are older than a minute,+ and keeps those of a process that runs (the old one of a graceful upgrade).+* Tests: `TestSeedSupersededByANewerCheckpoint`, `TestSeedPastItsDeadline`,+ `TestSeedOfARepositoryRemovedAndCreatedAgain`, `TestStopSeedsEndsTheSeedJobs`,+ `TestCleanBuildsRemovesOnlyDeadBuilds`.**Tests.** `TestSyncerResumesAHydrationAcrossRounds`: rounds of 4 changesetson a 10-changeset repository give Unavailable at 4 and at 8, with the@@ -1809,10 +1841,14 @@ shutdown.the same `C_gc` rule: no reader uses a checkpoint below `C_gc`, whether thecovered changeset packs are collected or not.* m8: what is due is decided by reads, before the lease: `checkpointDue` and- `C_gc` = `SyncBoundary(gc_grace)`. The runner remembers, per repository, the- `C_gc` it dropped below. A repository with nothing due is skipped- (`nothing_due`) without a write.- * Test: `TestRunOnceDropsSupersededCheckpointsWithGCOff`.+ `C_gc` = `SyncBoundary(gc_grace)`, and the count of checkpoints below it+ (`CheckpointsBelow`). A repository with nothing due is skipped+ (`nothing_due`) without a write. (Final minor 5: the runner first+ remembered the `C_gc` it dropped below; a build of an older sequence that+ committed after a newer one, which `CreateCheckpoint` allows since it does+ not lock the repository, then stayed until `C_gc` moved. The count sees it.)+ * Tests: `TestRunOnceDropsSupersededCheckpointsWithGCOff`,+ `TestRunOnceDropsALateLowerCheckpoint`.* **Database openers (audit).** Only `gitaly serve` opens the storage database(`corestoreopen.Open`, one `sqlstore.Store` per process). The `graft` CLI andevery other binary reach it through gRPC. Two Stores on one SQLite file exist@@ -2277,6 +2313,11 @@ path: the voter is m3's; the land paths call it unchanged.repository row lock.26. **One upload path**: every production upload goes through `Core.UploadPack`, andevery row that references an upload is committed under its `CommitContext`.+ Enforced by `TestUploadPackIsTheOnlyUploader` (`internal/graft/corestore`): in+ the non-test code that can reach a `*corestore.Core`, `Blobs.Put` is called in+ `core.go` only and `Blobs` is never taken as a value outside the core's wiring+ (`core.go`, `open/open.go`). It is invariant U of the reconciliation+ ("Removed repositories").27. **Local references are a cache of the database**: in authoritative mode a localreference write without fsync happens only after its repository has a durabledirty marker; a marker is removed only after the stack is durable, under git'sdiff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.goindex f3359a5..2047bae 100644--- a/internal/cli/gitaly/serve.go+++ b/internal/cli/gitaly/serve.go@@ -641,6 +641,12 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {for _, s := range cfg.Storages {storageNames = append(storageNames, s.Name)}+ // The build directories of hydrations of a process that died are+ // removed; those of a process that runs (the old one of a graceful+ // upgrade) hold their lock and are kept.+ if err := syncer.CleanBuilds(ctx, storageNames); err != nil {+ return err+ }stopLocalReferences, err := syncer.StartLocalReferences(ctx, storageNames)if err != nil {return err@@ -648,6 +654,9 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {// Deferred here, so it runs after the servers stopped: a last durable step// settles every repository whose writes ended.defer stopLocalReferences()+ // Runs before it (deferred later): the seeds in the background end+ // before the last durable step.+ defer syncer.StopSeeds()if cfg.Graft.Storage.MaintenanceAuto() {stopMaintenance, err := startStorageMaintenance(ctx, cfg, storageCore, syncer, locator, localrepoFactory, logger, storageNames)diff --git a/internal/graft/corestore/maintenance/maintenance.go b/internal/graft/corestore/maintenance/maintenance.goindex 986666e..298c644 100644--- a/internal/graft/corestore/maintenance/maintenance.go+++ b/internal/graft/corestore/maintenance/maintenance.go@@ -83,10 +83,6 @@ type Runner struct {// so that two processes of one node (a graceful upgrade) do not both hold a// lease.holder string- // dropped is, per repository ID, the C_gc below which this runner dropped- // the superseded checkpoints. Only the runs read and write it, one at a- // time.- dropped map[int64]uint64runs *prometheus.CounterVeccheckpoints prometheus.Counter@@ -129,7 +125,6 @@ func New(core *corestore.Core, cp *checkpoint.Checkpointer, collector *gc.Colleccfg: cfg,storages: storages,holder: LeaseHolder(core),- dropped: map[int64]uint64{},runs: prometheus.NewCounterVec(prometheus.CounterOpts{Name: "graft_storage_maintenance_runs_total",Help: "Runs of the background maintenance of the storage core, by result (ok, error: at least one step failed).",@@ -282,16 +277,24 @@ func (r *Runner) maintain(ctx context.Context, record corestore.Repository) (strif err != nil {return stepCheckpoint, err}- var boundary uint64+ dropDue := falseif r.cfg.Checkpoints {// C_gc: the newest checkpoint at least gc_grace old on the database- // clock. The checkpoints below it are superseded; a C_gc that this- // runner dropped below already has none left.- if boundary, err = r.core.Meta.SyncBoundary(ctx, record.ID, r.core.GCGrace); err != nil {+ // clock. The checkpoints below it are superseded. They are counted, not+ // remembered: a build of an older sequence that committed after a newer+ // one (CreateCheckpoint does not lock the repository) is below C_gc too.+ boundary, err := r.core.Meta.SyncBoundary(ctx, record.ID, r.core.GCGrace)+ if err != nil {return stepCheckpoint, err}+ if boundary > 0 {+ below, err := r.core.Meta.CheckpointsBelow(ctx, record.ID, boundary)+ if err != nil {+ return stepCheckpoint, err+ }+ dropDue = below > 0+ }}- dropDue := boundary > 0 && boundary != r.dropped[record.ID]if !due && !dropDue && !r.cfg.Collect {r.skipped.WithLabelValues("nothing_due").Inc()return "", nil@@ -340,7 +343,6 @@ func (r *Runner) maintain(ctx context.Context, record corestore.Repository) (strif err != nil {return stepCollect, err}- r.dropped[record.ID] = report.Boundaryr.count(report)return "", nil}diff --git a/internal/graft/corestore/maintenance/maintenance_test.go b/internal/graft/corestore/maintenance/maintenance_test.goindex 6ffce63..f24b47a 100644--- a/internal/graft/corestore/maintenance/maintenance_test.go+++ b/internal/graft/corestore/maintenance/maintenance_test.go@@ -268,6 +268,47 @@ func TestRunOnceDropsSupersededCheckpointsWithGCOff(t *testing.T) {require.Equal(t, before+1, e.metric("graft_storage_maintenance_skipped_total", "nothing_due"))}+// TestRunOnceDropsALateLowerCheckpoint: a checkpoint of an older sequence+// recorded after the superseded ones were dropped (a build that committed+// after a newer one: CreateCheckpoint does not lock the repository) is dropped+// by the next run, without C_gc moving (final minor 5).+func TestRunOnceDropsALateLowerCheckpoint(t *testing.T) {+ t.Parallel()+ ctx := testhelper.Context(t)+ cfg := config3(1, 0)+ cfg.Collect = false+ e := newEnv(t, coretest.Options{}, cfg)++ var built []corestore.Checkpoint+ for i := 0; i < 2; i++ {+ e.advance(1)+ e.fresh()+ require.NoError(t, e.runner.RunOnce(ctx))+ cp, ok := e.newestCheckpoint()+ require.True(t, ok)+ built = append(built, cp)+ }+ e.ageCheckpoints()+ require.NoError(t, e.runner.RunOnce(ctx))+ require.Len(t, e.checkpoints(), 1)++ // The late build: a checkpoint below C_gc, recorded now.+ late := built[0]+ content := []byte("the pack of a late build")+ lateKey := corestore.NewPackKey(e.repo.ID, sha256.Sum256(content))+ require.NoError(t, e.core.Blobs.Put(ctx, lateKey, bytes.NewReader(content)))+ e.exec("INSERT INTO graft_checkpoints (repo_id, seq, pack_key, pack_bytes, created_at) VALUES (?, ?, ?, ?, ?)",+ e.repo.ID, int64(late.Seq), lateKey, len(content), e.past(0))+ require.Len(t, e.checkpoints(), 2)++ require.NoError(t, e.runner.RunOnce(ctx))+ require.Len(t, e.checkpoints(), 1, "the late checkpoint below C_gc is dropped")+ require.Equal(t, built[1].Seq, e.checkpoints()[0].Seq)+ exists, err := e.core.Blobs.Exists(ctx, lateKey)+ require.NoError(t, err)+ require.False(t, exists, "and its pack")+}+// TestRunOnceReconcilesLeaks: with the collection on, a run reconciles the// bucket before it collects the removed repositories, and its metrics count// the leaks found and tombstoned and the IDs ahead of the allocator.diff --git a/internal/graft/corestore/metastore.go b/internal/graft/corestore/metastore.goindex f085511..73aa78e 100644--- a/internal/graft/corestore/metastore.go+++ b/internal/graft/corestore/metastore.go@@ -250,6 +250,10 @@ type MetaStore interface {// DeleteRemovedRepository removes a tombstone; a missing one is success.DeleteRemovedRepository(ctx context.Context, repoID int64) error+ // CheckpointsBelow returns the number of checkpoints of the repository+ // below seq.+ CheckpointsBelow(ctx context.Context, repoID int64, seq uint64) (int, error)+// ObserveAllocator reads the repository ID allocator and the// reconciliation cursor and, unless dryRun, records the observation// (AllocatorView.Now, Position) and drops the observations older than thediff --git a/internal/graft/corestore/sqlstore/checkpoint.go b/internal/graft/corestore/sqlstore/checkpoint.goindex 99b90ed..d1226e1 100644--- a/internal/graft/corestore/sqlstore/checkpoint.go+++ b/internal/graft/corestore/sqlstore/checkpoint.go@@ -69,22 +69,24 @@ func (s *Store) CreateCheckpoint(ctx context.Context, repoID int64, in corestorereturn nil}+ // A deadline already passed fails before the snapshot is written; it is+ // checked again as the last statement.if err := s.checkPackDeadline(ctx, tx, in.PackDeadline); err != nil {return err}- now, err := s.wallTime(ctx, tx)+ provisional, err := s.wallTime(ctx, tx)if err != nil {return err}// Another build of the same sequence may commit first (the repository// is not locked): its checkpoint is kept and returned, as one found- // above.+ // above. created_at is set again at the end.res, err := tx.ExecContext(ctx, s.bind(`INSERT INTO graft_checkpoints (repo_id, seq, pack_key, pack_bytes, created_at)VALUES (?, ?, ?, ?, ?)ON CONFLICT (repo_id, seq) DO NOTHING- `), repoID, int64(in.Seq), in.PackKey, in.PackBytes, s.timeArg(now))+ `), repoID, int64(in.Seq), in.PackKey, in.PackBytes, s.timeArg(provisional))if err != nil {return fmt.Errorf("insert checkpoint: %w", err)}@@ -125,6 +127,24 @@ func (s *Store) CreateCheckpoint(ctx context.Context, repoID int64, in corestore}}+ // The last statements (integrated review, final minor 1): the age of a+ // checkpoint (C_gc, C_sync) counts from created_at, and the pack's+ // deadline bounds the commit, so both are read on the database clock+ // after the snapshot, however long it took to write: the time between+ // created_at and the commit is that of one statement, not of the+ // snapshot, and does not lean on gc_margin.+ if err := s.checkPackDeadline(ctx, tx, in.PackDeadline); err != nil {+ return err+ }+ now, err := s.wallTime(ctx, tx)+ if err != nil {+ return err+ }+ if _, err := tx.ExecContext(ctx, s.bind("UPDATE graft_checkpoints SET created_at = ? WHERE repo_id = ? AND seq = ?"),+ s.timeArg(now), repoID, int64(in.Seq)); err != nil {+ return fmt.Errorf("stamp checkpoint: %w", err)+ }+cp = corestore.Checkpoint{RepoID: repoID,Seq: in.Seq,@@ -142,6 +162,19 @@ func (s *Store) CreateCheckpoint(ctx context.Context, repoID int64, in corestorereturn cp, created, nil}+// CheckpointsBelow implements corestore.MetaStore.+func (s *Store) CheckpointsBelow(ctx context.Context, repoID int64, seq uint64) (int, error) {+ if seq > math.MaxInt64 {+ seq = math.MaxInt64+ }+ var n int+ if err := s.readDB.QueryRowContext(ctx, s.bind("SELECT COUNT(*) FROM graft_checkpoints WHERE repo_id = ? AND seq < ?"),+ repoID, int64(seq)).Scan(&n); err != nil {+ return 0, fmt.Errorf("count checkpoints: %w", err)+ }+ return n, nil+}+// validateCheckpoint checks a checkpoint before anything is read.func validateCheckpoint(repoID int64, in corestore.CheckpointInput) error {if in.Seq == 0 || in.Seq > math.MaxInt64 {diff --git a/internal/graft/corestore/sqlstore/checkpoint_large_test.go b/internal/graft/corestore/sqlstore/checkpoint_large_test.goindex 184f7d1..dc7276d 100644--- a/internal/graft/corestore/sqlstore/checkpoint_large_test.go+++ b/internal/graft/corestore/sqlstore/checkpoint_large_test.go@@ -74,5 +74,48 @@ func TestCheckpointOfManyReferences(t *testing.T) {got, err := s.CheckpointRefs(ctx, repo.ID, 1)require.NoError(t, err)require.Equal(t, refs, got)++ // created_at is stamped after the snapshot (final minor 1): it is not+ // the time the record began. The database runs on this host, so its+ // clock is the test's.+ cp, ok, err := s.NewestCheckpoint(ctx, repo.ID)+ require.NoError(t, err)+ require.True(t, ok)+ require.True(t, cp.CreatedAt.After(began.Add(took/2)),+ "created_at %s, record from %s for %s", cp.CreatedAt, began, took)+ })+}++// TestCheckpointDeadlineIsCheckedAfterTheSnapshot: a deadline that passes+// while a large snapshot is written fails the record, and nothing is recorded+// (final minor 1).+func TestCheckpointDeadlineIsCheckedAfterTheSnapshot(t *testing.T) {+ const references = 100_000++ forEachDialect(t, func(t *testing.T, ctx context.Context, s *sqlstore.Store) {+ repo, _ := headRepo(t, ctx, s, "deadline during the snapshot", 1)+ refs := make(map[string]corestore.Target, references)+ for i := 0; i < references; i++ {+ refs[fmt.Sprintf("refs/heads/branch-%06d", i)] = oid(i + 1)+ }++ // How long the snapshot takes here, then a deadline at a tenth of it:+ // open at the first check, passed at the last.+ began := time.Now()+ _, created, err := s.CreateCheckpoint(ctx, repo.ID, corestore.CheckpointInput{+ Seq: 1, Refs: refs, PackKey: ownedKey(repo.ID, "measure"), PackDeadline: time.Now().Add(time.Hour), PackBytes: 1,+ })+ took := time.Since(began)+ require.NoError(t, err)+ require.True(t, created)++ next, _ := headRepo(t, ctx, s, "deadline during the snapshot, again", 1)+ _, _, err = s.CreateCheckpoint(ctx, next.ID, corestore.CheckpointInput{+ Seq: 1, Refs: refs, PackKey: ownedKey(next.ID, "late"), PackDeadline: time.Now().Add(took / 10), PackBytes: 1,+ })+ require.ErrorIs(t, err, corestore.ErrPackDeadline, "the record took %s", took)+ _, ok, err := s.NewestCheckpoint(ctx, next.ID)+ require.NoError(t, err)+ require.False(t, ok, "nothing is recorded")})}diff --git a/internal/graft/corestore/syncer/builds.go b/internal/graft/corestore/syncer/builds.gonew file mode 100644index 0000000..3c40337--- /dev/null+++ b/internal/graft/corestore/syncer/builds.go@@ -0,0 +1,119 @@+package syncer++import (+ "context"+ "errors"+ "fmt"+ "io/fs"+ "os"+ "path/filepath"+ "strings"+ "syscall"+ "time"+)++// buildLockName is the file in a build directory on which the process that+// runs the build holds an exclusive flock until it removed the directory. The+// kernel releases it when the process dies, so a process cleaning up tells the+// builds of a process that died from those of one that runs (the old process+// of a graceful upgrade).+const buildLockName = "build.lock"++// unlockedBuildAge is how old a build directory without its lock file must be+// before CleanBuilds removes it: one whose process died between creating the+// directory and its lock file, which takes microseconds.+const unlockedBuildAge = time.Minute++// lockBuild creates the lock file of the build directory dir and takes its+// flock.+func lockBuild(dir string) (*os.File, error) {+ f, err := os.OpenFile(filepath.Join(dir, buildLockName), os.O_CREATE|os.O_RDWR, 0o600)+ if err != nil {+ return nil, err+ }+ if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {+ _ = f.Close()+ return nil, fmt.Errorf("lock: %w", err)+ }+ return f, nil+}++// CleanBuilds removes the build directories (hydrate-*) of hydrations and+// seeds that no process runs any more: their process died (a crash, a kill)+// before it removed them, and nothing else does. A directory whose lock file+// another process holds is in use and kept. gitaly serve calls it at startup,+// before the listeners open (final minor 3).+func (s *Syncer) CleanBuilds(ctx context.Context, storages []string) error {+ removed := 0+ for _, storage := range storages {+ tmpDir, err := s.locator.TempDir(storage)+ if err != nil {+ return err+ }+ entries, err := os.ReadDir(tmpDir)+ if errors.Is(err, fs.ErrNotExist) {+ continue+ }+ if err != nil {+ return fmt.Errorf("storage core: list the temporary directory of %s: %w", storage, err)+ }+ for _, entry := range entries {+ if !entry.IsDir() || !strings.HasPrefix(entry.Name(), strings.TrimSuffix(tmpHydratePatt, "*")) {+ continue+ }+ dir := filepath.Join(tmpDir, entry.Name())+ stale, err := staleBuild(dir)+ if err != nil {+ return fmt.Errorf("storage core: check the build directory %s: %w", dir, err)+ }+ if !stale {+ continue+ }+ if err := os.RemoveAll(dir); err != nil {+ return fmt.Errorf("storage core: remove the build directory %s: %w", dir, err)+ }+ removed+++ }+ }+ if removed > 0 {+ s.logger.WithField("directories", removed).InfoContext(ctx, "storage core: removed the build directories of hydrations that no process runs")+ }+ return nil+}++// staleBuild reports whether no process runs the build in dir: its lock file+// can be locked, or it has none and dir is older than unlockedBuildAge. A+// lock taken here is released when the directory is gone or kept.+func staleBuild(dir string) (bool, error) {+ f, err := os.OpenFile(filepath.Join(dir, buildLockName), os.O_RDWR, 0)+ if errors.Is(err, fs.ErrNotExist) {+ info, err := os.Stat(dir)+ if err != nil {+ return false, err+ }+ return time.Since(info.ModTime()) >= unlockedBuildAge, nil+ }+ if err != nil {+ return false, err+ }+ defer func() { _ = f.Close() }()++ switch err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); {+ case errors.Is(err, syscall.EWOULDBLOCK):+ return false, nil+ case err != nil:+ return false, err+ }+ return true, nil+}++// StopSeeds cancels the running seed jobs and waits for them to end; no seed+// starts afterwards. gitaly serve calls it after its servers stopped, before+// the last durable step of the local references (final minor 3).+func (s *Syncer) StopSeeds() {+ s.mu.Lock()+ s.seedsStopped = true+ s.mu.Unlock()+ s.stopSeeds()+ s.seedsRunning.Wait()+}diff --git a/internal/graft/corestore/syncer/syncer.go b/internal/graft/corestore/syncer/syncer.goindex 7f10f15..e1bad94 100644--- a/internal/graft/corestore/syncer/syncer.go+++ b/internal/graft/corestore/syncer/syncer.go@@ -146,6 +146,13 @@ type Syncer struct {reported map[int64]time.Time// seeds are the running seed jobs, keyed by corestore.RepoKey.String().seeds map[string]*seedJob+ // seedsStopped is set by StopSeeds: no seed starts after it.+ seedsStopped bool+ // seedsCtx ends the seed jobs when StopSeeds calls stopSeeds;+ // seedsRunning counts them.+ seedsCtx context.Context+ stopSeeds context.CancelFunc+ seedsRunning sync.WaitGroupsyncTotal *prometheus.CounterVecsyncSeconds prometheus.Histogram@@ -176,7 +183,10 @@ var (// New returns the syncer of core.func New(core *corestore.Core, logger log.Logger, locator storage.Locator, repoFactory localrepo.Factory, gitCmdFactory gitcmd.CommandFactory) *Syncer {+ seedsCtx, stopSeeds := context.WithCancel(context.Background())s := &Syncer{+ seedsCtx: seedsCtx,+ stopSeeds: stopSeeds,core: core,logger: logger,locator: locator,@@ -1535,6 +1545,9 @@ type build struct {path stringtmpDir stringrepo *localrepo.Repo+ // lock is the flock of the build directory (buildLockName), held until it+ // is removed.+ lock *os.File}// newBuild creates an empty bare repository to hydrate in the temporary@@ -1556,6 +1569,10 @@ func (s *Syncer) newBuild(ctx context.Context, repo *gitalypb.Repository, r corereturn nil, structerr.NewInternal("storage core: create hydration directory: %w", err)}b := &build{dir: dir, path: filepath.Join(dir, "repo.git"), tmpDir: tmpDir}+ if b.lock, err = lockBuild(dir); err != nil {+ b.remove()+ return nil, structerr.NewInternal("storage core: lock the hydration directory: %w", err)+ }if err := gitops.InitBare(ctx, s.gitCmdFactory, b.path, r.ObjectFormat); err != nil {b.remove()return nil, structerr.NewInternal("storage core: init repository: %w", err)@@ -1570,7 +1587,12 @@ func (s *Syncer) newBuild(ctx context.Context, repo *gitalypb.Repository, r core}// remove deletes what is left of the build (nothing once it was installed).-func (b *build) remove() { _ = os.RemoveAll(b.dir) }+func (b *build) remove() {+ _ = os.RemoveAll(b.dir)+ if b.lock != nil {+ _ = b.lock.Close()+ }+}// finishBuild sets the references of the build, writes its watermark at seq// and installs it at path (install, with boundary).@@ -1659,9 +1681,19 @@ func (s *Syncer) startSeed(ctx context.Context, key corestore.RepoKey, repo *git}j := &seedJob{cp: cp, done: make(chan struct{})}+ if s.seedsStopped {+ j.err = structerr.NewUnavailable("storage core: the server is stopping: %s is not seeded", key)+ close(j.done)+ return j+ }s.seeds[key.String()] = j+ // Detached from the request, ended at the deadline or by StopSeeds.jobCtx, cancel := context.WithDeadline(context.WithoutCancel(ctx), readAt.Add(s.core.WriteDeadline()))+ stopOnShutdown := context.AfterFunc(s.seedsCtx, cancel)+ s.seedsRunning.Add(1)go func() {+ defer s.seedsRunning.Done()+ defer stopOnShutdown()defer cancel()err := s.runSeed(jobCtx, repo, r, hash, path, cp)if err != nil && !errors.Is(err, errSeedSuperseded) {@@ -1677,9 +1709,65 @@ func (s *Syncer) startSeed(ctx context.Context, key corestore.RepoKey, repo *gitreturn j}+// errSeedDeadline ends a seed that ran past its deadline on the database+// clock: the checkpoint it read may be collected from then on.+var errSeedDeadline = errors.New("seed deadline passed")++// errSeedRepositoryGone ends a seed whose repository was removed, or removed+// and created again under a new ID, while it ran.+var errSeedRepositoryGone = errors.New("the repository was removed during the seed")+// runSeed seeds a build from the checkpoint cp and installs it at path with-// the watermark cp.Seq.+// the watermark cp.Seq. A seed that failed after its deadline on the database+// clock, or whose repository went meanwhile, fails as that: what it ran into+// (a pack collected after the deadline looks like a missing pack, a removed+// repository like a missing record) is a consequence.func (s *Syncer) runSeed(ctx context.Context, repo *gitalypb.Repository, r corestore.Repository, hash git.ObjectHash, path string, cp corestore.Checkpoint) error {+ err := s.seedAndInstall(ctx, repo, r, hash, path, cp)+ if err == nil || errors.Is(err, errSeedSuperseded) || errors.Is(err, errSeedDeadline) || errors.Is(err, errSeedRepositoryGone) {+ return err+ }+ if s.seedsCtx.Err() != nil {+ // StopSeeds cancelled it: the server is stopping.+ return structerr.NewUnavailable("storage core: the server is stopping: the seed of %s from checkpoint %d is cancelled (%v)", r.Key, cp.Seq, err)+ }+ if checkErr := s.checkSeedDeadline(context.WithoutCancel(ctx), r, cp); checkErr != nil {+ // The cause is reported as text: the code is that of checkErr.+ return fmt.Errorf("%w (the seed failed with: %v)", checkErr, err)+ }+ return err+}++// checkSeedDeadline reads the repository record again: on the database clock+// the seed must be before r.ReadAt + W (Core.WriteDeadline), the bound under+// which the checkpoint, the newest when r was read, cannot be collected; and+// the repository must still be r. The job's context ends at the same bound on+// this host's monotonic clock, which stops while the host is suspended; the+// database clock does not (as for the writers, final minor 2).+func (s *Syncer) checkSeedDeadline(ctx context.Context, r corestore.Repository, cp corestore.Checkpoint) error {+ if r.ReadAt.IsZero() {+ return structerr.NewInternal("storage core: seed of %s: the record carries no database read time", r.Key)+ }+ ctx, cancel := context.WithTimeout(ctx, s.syncTimeout())+ defer cancel()++ current, err := s.core.Meta.GetRepository(ctx, r.Key)+ switch {+ case errors.Is(err, corestore.ErrRepositoryNotFound) || (err == nil && current.ID != r.ID):+ return structerr.NewUnavailable("storage core: seeding %s from checkpoint %d: %w; the next request starts from its current state", r.Key, cp.Seq, errSeedRepositoryGone)+ case err != nil:+ return structerr.NewUnavailable("storage core: seeding %s: read the repository record: %w", r.Key, err)+ }+ deadline := r.ReadAt.Add(s.core.WriteDeadline())+ if !current.ReadAt.Before(deadline) {+ return structerr.NewUnavailable("storage core: seeding %s from checkpoint %d: %w at %s on the database clock (%s): the checkpoint may be collected; the next request seeds from the newest",+ r.Key, cp.Seq, errSeedDeadline, deadline.UTC().Format(time.RFC3339Nano), current.ReadAt.UTC().Format(time.RFC3339Nano))+ }+ return nil+}++// seedAndInstall is runSeed without the mapping of its failures.+func (s *Syncer) seedAndInstall(ctx context.Context, repo *gitalypb.Repository, r corestore.Repository, hash git.ObjectHash, path string, cp corestore.Checkpoint) error {b, err := s.newBuild(ctx, repo, r)if err != nil {return err@@ -1690,6 +1778,10 @@ func (s *Syncer) runSeed(ctx context.Context, repo *gitalypb.Repository, r coresif err != nil {return err}+ // A seed that indexed the whole pack holds every object of the+ // checkpoint, verified against its key, so it may be installed after its+ // deadline; what the deadline protects is the changeset packs after the+ // checkpoint, which C_sync below covers.boundary, err := s.syncBoundary(ctx, r.ID)if err != nil {return errdiff --git a/internal/graft/corestore/syncer/syncer_seed_test.go b/internal/graft/corestore/syncer/syncer_seed_test.gonew file mode 100644index 0000000..d22c4e5--- /dev/null+++ b/internal/graft/corestore/syncer/syncer_seed_test.go@@ -0,0 +1,216 @@+package syncer_test++import (+ "os"+ "path/filepath"+ "sync"+ "syscall"+ "testing"+ "time"++ "github.com/stretchr/testify/require"+ "google.golang.org/grpc/codes"+ "google.golang.org/grpc/status"++ "github.com/feder-cr/git-ai/internal/graft/corestore"+ "github.com/feder-cr/git-ai/internal/graft/corestore/sqlstore"+)++// seedFixture is a repository written on node a with a checkpoint followed by+// changesets, and a fresh node b whose first seed the test holds at the "seed"+// point (after the checkpoint's references are read, before its pack is+// indexed) until release is closed. b's rounds are long, so a round waits for+// the seed and returns its result.+type seedFixture struct {+ a, b *env+ src source+ repo corestore.Repository+ cp corestore.Checkpoint+ seeding chan struct{}+ release chan struct{}+}++func newSeedFixture(t *testing.T) *seedFixture {+ t.Helper()++ f := &seedFixture{a: newEnv(t, envOptions{}), seeding: make(chan struct{}), release: make(chan struct{})}+ f.src = f.a.newSource(t)+ c := f.a.writeCommit(t, f.src)+ f.repo = f.a.register(t, f.a.key(localRelPath), f.src, c)+ c = f.a.advanceMain(t, f.repo.ID, f.src, c)+ f.cp = f.a.checkpoint(t, f.repo.ID, f.src)+ f.a.advanceMain(t, f.repo.ID, f.src, c)++ f.b = f.a.secondNode(t, sqlstore.SQLite)+ f.b.core.SyncTimeout = time.Minute+ var once sync.Once+ f.b.sy.SetTestHook(func(point string) {+ if point != "seed" {+ return+ }+ first := false+ once.Do(func() { first = true; close(f.seeding) })+ if first {+ <-f.release+ }+ })+ return f+}++// ensureFresh runs EnsureFresh on b in the background.+func (f *seedFixture) ensureFresh() <-chan error {+ result := make(chan error, 1)+ go func() { result <- f.b.sy.EnsureFresh(f.b.ctx, f.b.pbRepo(localRelPath)) }()+ return result+}++// TestSeedSupersededByANewerCheckpoint: a checkpoint that becomes C_sync while+// a seed from an older one runs makes that seed install nothing; the round+// seeds from the newest and catches up (final minor 3).+func TestSeedSupersededByANewerCheckpoint(t *testing.T) {+ t.Parallel()+ f := newSeedFixture(t)++ result := f.ensureFresh()+ <-f.seeding+ newer := f.a.checkpoint(t, f.repo.ID, f.src)+ require.Greater(t, newer.Seq, f.cp.Seq)+ f.a.ageCheckpoint(t, f.repo.ID, newer.Seq, f.b.core.GCGrace+time.Minute)+ close(f.release)++ require.NoError(t, <-result)+ require.Equal(t, float64(2), counter(t, f.b.sy, "graft_storage_sync_checkpoint_seeds_total", ""), "the superseded seed, then one from the newest")+ repo, err := f.b.core.Meta.GetRepository(f.b.ctx, f.repo.Key)+ require.NoError(t, err)+ f.b.verifySynced(t, repo)+}++// TestSeedPastItsDeadline: a seed held past its deadline (readAt + W) while+// its checkpoint pack is collected, as garbage collection may then do, fails+// as a seed past its deadline, Unavailable, not as a missing pack; the next+// request starts again (final minor 2).+func TestSeedPastItsDeadline(t *testing.T) {+ t.Parallel()+ f := newSeedFixture(t)+ // W = gc_grace - gc_margin = 1 s on node b.+ f.b.core.GCGrace = 2 * time.Second+ f.b.core.GCMargin = time.Second++ result := f.ensureFresh()+ <-f.seeding+ require.NoError(t, f.a.core.Blobs.Delete(f.a.ctx, f.cp.PackKey), "collected after the deadline")+ // The deadline is a time: the seed is held until it passed.+ <-time.After(f.b.core.WriteDeadline() + 200*time.Millisecond)+ close(f.release)++ err := <-result+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)+ require.ErrorContains(t, err, "seed deadline passed")+ _, ok, appliedErr := f.b.sy.Applied(f.b.ctx, f.b.pbRepo(localRelPath))+ require.NoError(t, appliedErr)+ require.False(t, ok, "nothing is installed")+}++// TestSeedOfARepositoryRemovedAndCreatedAgain: the repository is removed and+// registered again under a new ID while a seed of the old one runs; the seed+// installs nothing and fails as such, and the next request hydrates the new+// repository (final minor 3).+func TestSeedOfARepositoryRemovedAndCreatedAgain(t *testing.T) {+ t.Parallel()+ f := newSeedFixture(t)++ result := f.ensureFresh()+ <-f.seeding+ removed, err := f.a.core.Meta.DeleteRepository(f.a.ctx, f.repo.Key)+ require.NoError(t, err)+ require.True(t, removed)+ again := f.a.register(t, f.repo.Key, f.src, f.a.writeCommit(t, f.src))+ require.NotEqual(t, f.repo.ID, again.ID)+ close(f.release)++ err = <-result+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)+ require.ErrorContains(t, err, "the repository was removed during the seed")++ require.NoError(t, f.b.sy.EnsureFresh(f.b.ctx, f.b.pbRepo(localRelPath)))+ again, err = f.b.core.Meta.GetRepository(f.b.ctx, again.Key)+ require.NoError(t, err)+ f.b.verifySynced(t, again)+}++// TestStopSeedsEndsTheSeedJobs: StopSeeds cancels a running seed, waits for it+// and its build directory to be gone, and no seed starts after it (final+// minor 3).+func TestStopSeedsEndsTheSeedJobs(t *testing.T) {+ t.Parallel()+ f := newSeedFixture(t)++ result := f.ensureFresh()+ <-f.seeding+ stopped := make(chan struct{})+ go func() {+ f.b.sy.StopSeeds()+ close(stopped)+ }()+ select {+ case <-stopped:+ t.Fatal("StopSeeds returned while the seed runs")+ default:+ }+ close(f.release)+ <-stopped++ err := <-result+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)+ tmpDir, tmpErr := f.b.locator.TempDir(f.b.storageName)+ require.NoError(t, tmpErr)+ builds, globErr := filepath.Glob(filepath.Join(tmpDir, "hydrate-*"))+ require.NoError(t, globErr)+ require.Empty(t, builds, "the build directory is removed")++ err = f.b.sy.EnsureFresh(f.b.ctx, f.b.pbRepo(localRelPath))+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)+ require.ErrorContains(t, err, "the server is stopping")+}++// TestCleanBuildsRemovesOnlyDeadBuilds: at startup the build directories of+// a process that died are removed (their lock is free, or they never got one+// and are old), those of a running process are kept (final minor 3).+func TestCleanBuildsRemovesOnlyDeadBuilds(t *testing.T) {+ t.Parallel()+ e := newEnv(t, envOptions{})+ tmpDir, err := e.locator.TempDir(e.storageName)+ require.NoError(t, err)+ require.NoError(t, os.MkdirAll(tmpDir, 0o700))++ mkdir := func(name string) string {+ dir := filepath.Join(tmpDir, name)+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "repo.git"), 0o700))+ return dir+ }+ lockFile := func(dir string) *os.File {+ f, err := os.OpenFile(filepath.Join(dir, "build.lock"), os.O_CREATE|os.O_RDWR, 0o600)+ require.NoError(t, err)+ t.Cleanup(func() { _ = f.Close() })+ return f+ }++ running := mkdir("hydrate-running")+ require.NoError(t, syscall.Flock(int(lockFile(running).Fd()), syscall.LOCK_EX|syscall.LOCK_NB))+ dead := mkdir("hydrate-dead")+ require.NoError(t, lockFile(dead).Close(), "a lock file nobody holds: its process died")+ oldUnlocked := mkdir("hydrate-old-unlocked")+ require.NoError(t, os.Chtimes(oldUnlocked, time.Now().Add(-2*time.Minute), time.Now().Add(-2*time.Minute)))+ newUnlocked := mkdir("hydrate-new-unlocked")+ other := mkdir("stale-other")++ require.NoError(t, e.sy.CleanBuilds(e.ctx, []string{e.storageName}))+ for dir, kept := range map[string]bool{running: true, dead: false, oldUnlocked: false, newUnlocked: true, other: true} {+ _, err := os.Stat(dir)+ if kept {+ require.NoError(t, err, "%s is kept", dir)+ } else {+ require.ErrorIs(t, err, os.ErrNotExist, "%s is removed", dir)+ }+ }+}diff --git a/internal/graft/corestore/uploadpack_only_test.go b/internal/graft/corestore/uploadpack_only_test.gonew file mode 100644index 0000000..1405f67--- /dev/null+++ b/internal/graft/corestore/uploadpack_only_test.go@@ -0,0 +1,129 @@+package corestore_test++import (+ "go/ast"+ "go/parser"+ "go/token"+ "io/fs"+ "os"+ "path/filepath"+ "strings"+ "testing"++ "github.com/stretchr/testify/require"+)++// TestUploadPackIsTheOnlyUploader enforces invariant U (doc/graft/PLAN7.md,+// RULE 26 and "Removed repositories"): every object of the bucket is written+// by Core.UploadPack, which needs a repository record that a read found. The+// bucket is reached through the field Core.Blobs only, so in the module's+// non-test code:+// - Blobs.Put is called in core.go only (UploadPack);+// - Blobs is not taken as a value (assigned, passed, returned), which would+// let a Put happen under another name, except where the core is built.+//+// Tests may write the bucket directly to set up states. The files checked are+// those that can reach a *corestore.Core: they import corestore or one of its+// packages, call the dependencies' GetGraftStorage, or are in internal/graft+// or the storagecore service (other fields named Blobs, in protobuf messages+// and git packfile code, are unrelated).+func TestUploadPackIsTheOnlyUploader(t *testing.T) {+ root := moduleRoot(t)+ // Where Blobs may be used as a value: the core is built and closed there.+ wiring := map[string]bool{+ "internal/graft/corestore/core.go": true,+ "internal/graft/corestore/open/open.go": true,+ }++ var violations []string+ fset := token.NewFileSet()+ err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {+ if err != nil {+ return err+ }+ rel, err := filepath.Rel(root, path)+ if err != nil {+ return err+ }+ if d.IsDir() {+ if rel != "." && (strings.HasPrefix(d.Name(), ".") || d.Name() == "_build" || d.Name() == "_support" || d.Name() == "testdata") {+ return filepath.SkipDir+ }+ return nil+ }+ if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {+ return nil+ }+ src, err := os.ReadFile(path)+ if err != nil {+ return err+ }+ file, err := parser.ParseFile(fset, path, src, parser.SkipObjectResolution)+ if err != nil {+ return err+ }+ rel = filepath.ToSlash(rel)+ if !reachesCore(rel, file, src) {+ return nil+ }++ var stack []ast.Node+ ast.Inspect(file, func(n ast.Node) bool {+ if n == nil {+ stack = stack[:len(stack)-1]+ return true+ }+ stack = append(stack, n)+ sel, ok := n.(*ast.SelectorExpr)+ if !ok || sel.Sel.Name != "Blobs" {+ return true+ }+ pos := fset.Position(sel.Pos())+ // The parent of x.Blobs: a method selection (x.Blobs.M) or a use as+ // a value.+ if len(stack) >= 2 {+ if method, ok := stack[len(stack)-2].(*ast.SelectorExpr); ok && method.X == sel {+ if method.Sel.Name == "Put" && rel != "internal/graft/corestore/core.go" {+ violations = append(violations, pos.String()+": Blobs.Put outside Core.UploadPack")+ }+ return true+ }+ }+ if !wiring[rel] {+ violations = append(violations, pos.String()+": Blobs used as a value outside the core's wiring")+ }+ return true+ })+ return nil+ })+ require.NoError(t, err)+ require.Empty(t, violations, "every bucket write goes through Core.UploadPack (RULE 26, invariant U)")+}++// reachesCore reports whether the file at rel can reach a *corestore.Core.+func reachesCore(rel string, file *ast.File, src []byte) bool {+ if strings.HasPrefix(rel, "internal/graft/") || strings.HasPrefix(rel, "internal/gitaly/service/storagecore/") {+ return true+ }+ for _, imp := range file.Imports {+ if strings.HasPrefix(strings.Trim(imp.Path.Value, `"`), "github.com/feder-cr/git-ai/internal/graft/corestore") {+ return true+ }+ }+ return strings.Contains(string(src), "GetGraftStorage")+}++// moduleRoot returns the directory of the module's go.mod.+func moduleRoot(t *testing.T) string {+ t.Helper()+ dir, err := os.Getwd()+ require.NoError(t, err)+ for {+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {+ return dir+ }+ parent := filepath.Dir(dir)+ require.NotEqual(t, dir, parent, "no go.mod above the test")+ dir = parent+ }+}