Skip to content

Git designed for AI agents.

Final review minors 1-5: checkpoint stamp, seed deadline, seed lifecycle, invariant U, late checkpoints

Federico Eliadac3ea4778ac
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.md
index 72877b7..54f32bb 100644
--- a/doc/graft/PLAN7.md
+++ b/doc/graft/PLAN7.md
@@ -1414,7 +1414,15 @@ Binding order:
(`Internal`), never a partial pack.
4. Re-read the record (`t_r`), `Core.UploadPack`. `//! after_upload`.
5. `CreateCheckpoint(repoID, {Seq: N, Refs: refs, PackKey, PackBytes})` under the
- upload's `CommitContext`. `//! after_commit`.
+ upload's `CommitContext`. `//! after_commit`. In its transaction the last
+ statements, after the snapshot is written, check the pack's deadline on the
+ database clock and stamp `created_at` with the database time (final minor 1):
+ the age of a checkpoint (C_gc, C_sync) counts from a time one statement
+ before the commit, whatever the size of the snapshot, so the visibility lag
+ does not grow with the number of references or lean on `gc_margin`. Tests:
+ `TestCheckpointOfManyReferences` (created_at after the snapshot, 100k
+ references), `TestCheckpointDeadlineIsCheckedAfterTheSnapshot` (a deadline
+ that passes during the snapshot fails the record, nothing recorded).
A crash after 3 leaves the pack as an orphan (reclaimed by the orphan protocol); a
re-run starts over from the head at that time. The build holds no reference lock
@@ -1728,6 +1736,30 @@ leaves it:
catches up and every reference matches.
* Bench with a checkpoint pack of hundreds of MB: requested in a window (see
the final report).
+* **Deadline on the database clock (final minor 2).** The job's context ends
+ at `readAt + W` on the host's monotonic clock, which stops while the host is
+ suspended. A seed that fails is checked against the database clock
+ (`checkSeedDeadline`: the record read again, `r.ReadAt + W`): past it, the
+ failure is `Unavailable` "seed deadline passed", whatever the seed ran into
+ (a checkpoint pack collected after the deadline reads as a missing pack,
+ which would otherwise be reported as corruption). A seed that indexed the
+ whole pack is installed even past its deadline: it holds every object of the
+ checkpoint, verified against its key, and the changeset packs after it are
+ covered by the C_sync check right before the install (`errSeedSuperseded`).
+* **Removed mid-seed (final minor 3).** The same check finds a repository
+ removed, or removed and registered again under a new ID: `Unavailable` "the
+ repository was removed during the seed", nothing installed; the next request
+ starts from the current state.
+* **Lifecycle (final minor 3).** `StopSeeds` (deferred by `gitaly serve` after
+ its servers, before the last durable step) cancels the seed jobs, waits for
+ them, and refuses new ones ("the server is stopping"). Every build directory
+ (`hydrate-*`) holds a flock on its `build.lock` until it is removed;
+ `CleanBuilds`, at startup before the listeners, removes those whose lock is
+ free (their process died) or that have none and are older than a minute,
+ and keeps those of a process that runs (the old one of a graceful upgrade).
+* Tests: `TestSeedSupersededByANewerCheckpoint`, `TestSeedPastItsDeadline`,
+ `TestSeedOfARepositoryRemovedAndCreatedAgain`, `TestStopSeedsEndsTheSeedJobs`,
+ `TestCleanBuildsRemovesOnlyDeadBuilds`.
**Tests.** `TestSyncerResumesAHydrationAcrossRounds`: rounds of 4 changesets
on a 10-changeset repository give Unavailable at 4 and at 8, with the
@@ -1809,10 +1841,14 @@ shutdown.
the same `C_gc` rule: no reader uses a checkpoint below `C_gc`, whether the
covered changeset packs are collected or not.
* m8: what is due is decided by reads, before the lease: `checkpointDue` and
- `C_gc` = `SyncBoundary(gc_grace)`. The runner remembers, per repository, the
- `C_gc` it dropped below. A repository with nothing due is skipped
- (`nothing_due`) without a write.
- * Test: `TestRunOnceDropsSupersededCheckpointsWithGCOff`.
+ `C_gc` = `SyncBoundary(gc_grace)`, and the count of checkpoints below it
+ (`CheckpointsBelow`). A repository with nothing due is skipped
+ (`nothing_due`) without a write. (Final minor 5: the runner first
+ remembered the `C_gc` it dropped below; a build of an older sequence that
+ committed after a newer one, which `CreateCheckpoint` allows since it does
+ not lock the repository, then stayed until `C_gc` moved. The count sees it.)
+ * Tests: `TestRunOnceDropsSupersededCheckpointsWithGCOff`,
+ `TestRunOnceDropsALateLowerCheckpoint`.
* **Database openers (audit).** Only `gitaly serve` opens the storage database
(`corestoreopen.Open`, one `sqlstore.Store` per process). The `graft` CLI and
every other binary reach it through gRPC. Two Stores on one SQLite file exist
@@ -2277,6 +2313,11 @@ path: the voter is m3's; the land paths call it unchanged.
repository row lock.
26. **One upload path**: every production upload goes through `Core.UploadPack`, and
every row that references an upload is committed under its `CommitContext`.
+ Enforced by `TestUploadPackIsTheOnlyUploader` (`internal/graft/corestore`): in
+ the non-test code that can reach a `*corestore.Core`, `Blobs.Put` is called in
+ `core.go` only and `Blobs` is never taken as a value outside the core's wiring
+ (`core.go`, `open/open.go`). It is invariant U of the reconciliation
+ ("Removed repositories").
27. **Local references are a cache of the database**: in authoritative mode a local
reference write without fsync happens only after its repository has a durable
dirty marker; a marker is removed only after the stack is durable, under git's
diff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.go
index f3359a5..2047bae 100644
--- a/internal/cli/gitaly/serve.go
+++ b/internal/cli/gitaly/serve.go
@@ -641,6 +641,12 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
for _, s := range cfg.Storages {
storageNames = append(storageNames, s.Name)
}
+ // The build directories of hydrations of a process that died are
+ // removed; those of a process that runs (the old one of a graceful
+ // upgrade) hold their lock and are kept.
+ if err := syncer.CleanBuilds(ctx, storageNames); err != nil {
+ return err
+ }
stopLocalReferences, err := syncer.StartLocalReferences(ctx, storageNames)
if err != nil {
return err
@@ -648,6 +654,9 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
// Deferred here, so it runs after the servers stopped: a last durable step
// settles every repository whose writes ended.
defer stopLocalReferences()
+ // Runs before it (deferred later): the seeds in the background end
+ // before the last durable step.
+ defer syncer.StopSeeds()
if cfg.Graft.Storage.MaintenanceAuto() {
stopMaintenance, err := startStorageMaintenance(ctx, cfg, storageCore, syncer, locator, localrepoFactory, logger, storageNames)
diff --git a/internal/graft/corestore/maintenance/maintenance.go b/internal/graft/corestore/maintenance/maintenance.go
index 986666e..298c644 100644
--- a/internal/graft/corestore/maintenance/maintenance.go
+++ b/internal/graft/corestore/maintenance/maintenance.go
@@ -83,10 +83,6 @@ type Runner struct {
// so that two processes of one node (a graceful upgrade) do not both hold a
// lease.
holder string
- // dropped is, per repository ID, the C_gc below which this runner dropped
- // the superseded checkpoints. Only the runs read and write it, one at a
- // time.
- dropped map[int64]uint64
runs *prometheus.CounterVec
checkpoints prometheus.Counter
@@ -129,7 +125,6 @@ func New(core *corestore.Core, cp *checkpoint.Checkpointer, collector *gc.Collec
cfg: cfg,
storages: storages,
holder: LeaseHolder(core),
- dropped: map[int64]uint64{},
runs: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "graft_storage_maintenance_runs_total",
Help: "Runs of the background maintenance of the storage core, by result (ok, error: at least one step failed).",
@@ -282,16 +277,24 @@ func (r *Runner) maintain(ctx context.Context, record corestore.Repository) (str
if err != nil {
return stepCheckpoint, err
}
- var boundary uint64
+ dropDue := false
if r.cfg.Checkpoints {
// C_gc: the newest checkpoint at least gc_grace old on the database
- // clock. The checkpoints below it are superseded; a C_gc that this
- // runner dropped below already has none left.
- if boundary, err = r.core.Meta.SyncBoundary(ctx, record.ID, r.core.GCGrace); err != nil {
+ // clock. The checkpoints below it are superseded. They are counted, not
+ // remembered: a build of an older sequence that committed after a newer
+ // one (CreateCheckpoint does not lock the repository) is below C_gc too.
+ boundary, err := r.core.Meta.SyncBoundary(ctx, record.ID, r.core.GCGrace)
+ if err != nil {
return stepCheckpoint, err
}
+ if boundary > 0 {
+ below, err := r.core.Meta.CheckpointsBelow(ctx, record.ID, boundary)
+ if err != nil {
+ return stepCheckpoint, err
+ }
+ dropDue = below > 0
+ }
}
- dropDue := boundary > 0 && boundary != r.dropped[record.ID]
if !due && !dropDue && !r.cfg.Collect {
r.skipped.WithLabelValues("nothing_due").Inc()
return "", nil
@@ -340,7 +343,6 @@ func (r *Runner) maintain(ctx context.Context, record corestore.Repository) (str
if err != nil {
return stepCollect, err
}
- r.dropped[record.ID] = report.Boundary
r.count(report)
return "", nil
}
diff --git a/internal/graft/corestore/maintenance/maintenance_test.go b/internal/graft/corestore/maintenance/maintenance_test.go
index 6ffce63..f24b47a 100644
--- a/internal/graft/corestore/maintenance/maintenance_test.go
+++ b/internal/graft/corestore/maintenance/maintenance_test.go
@@ -268,6 +268,47 @@ func TestRunOnceDropsSupersededCheckpointsWithGCOff(t *testing.T) {
require.Equal(t, before+1, e.metric("graft_storage_maintenance_skipped_total", "nothing_due"))
}
+// TestRunOnceDropsALateLowerCheckpoint: a checkpoint of an older sequence
+// recorded after the superseded ones were dropped (a build that committed
+// after a newer one: CreateCheckpoint does not lock the repository) is dropped
+// by the next run, without C_gc moving (final minor 5).
+func TestRunOnceDropsALateLowerCheckpoint(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ cfg := config3(1, 0)
+ cfg.Collect = false
+ e := newEnv(t, coretest.Options{}, cfg)
+
+ var built []corestore.Checkpoint
+ for i := 0; i < 2; i++ {
+ e.advance(1)
+ e.fresh()
+ require.NoError(t, e.runner.RunOnce(ctx))
+ cp, ok := e.newestCheckpoint()
+ require.True(t, ok)
+ built = append(built, cp)
+ }
+ e.ageCheckpoints()
+ require.NoError(t, e.runner.RunOnce(ctx))
+ require.Len(t, e.checkpoints(), 1)
+
+ // The late build: a checkpoint below C_gc, recorded now.
+ late := built[0]
+ content := []byte("the pack of a late build")
+ lateKey := corestore.NewPackKey(e.repo.ID, sha256.Sum256(content))
+ require.NoError(t, e.core.Blobs.Put(ctx, lateKey, bytes.NewReader(content)))
+ e.exec("INSERT INTO graft_checkpoints (repo_id, seq, pack_key, pack_bytes, created_at) VALUES (?, ?, ?, ?, ?)",
+ e.repo.ID, int64(late.Seq), lateKey, len(content), e.past(0))
+ require.Len(t, e.checkpoints(), 2)
+
+ require.NoError(t, e.runner.RunOnce(ctx))
+ require.Len(t, e.checkpoints(), 1, "the late checkpoint below C_gc is dropped")
+ require.Equal(t, built[1].Seq, e.checkpoints()[0].Seq)
+ exists, err := e.core.Blobs.Exists(ctx, lateKey)
+ require.NoError(t, err)
+ require.False(t, exists, "and its pack")
+}
+
// TestRunOnceReconcilesLeaks: with the collection on, a run reconciles the
// bucket before it collects the removed repositories, and its metrics count
// the leaks found and tombstoned and the IDs ahead of the allocator.
diff --git a/internal/graft/corestore/metastore.go b/internal/graft/corestore/metastore.go
index f085511..73aa78e 100644
--- a/internal/graft/corestore/metastore.go
+++ b/internal/graft/corestore/metastore.go
@@ -250,6 +250,10 @@ type MetaStore interface {
// DeleteRemovedRepository removes a tombstone; a missing one is success.
DeleteRemovedRepository(ctx context.Context, repoID int64) error
+ // CheckpointsBelow returns the number of checkpoints of the repository
+ // below seq.
+ CheckpointsBelow(ctx context.Context, repoID int64, seq uint64) (int, error)
+
// ObserveAllocator reads the repository ID allocator and the
// reconciliation cursor and, unless dryRun, records the observation
// (AllocatorView.Now, Position) and drops the observations older than the
diff --git a/internal/graft/corestore/sqlstore/checkpoint.go b/internal/graft/corestore/sqlstore/checkpoint.go
index 99b90ed..d1226e1 100644
--- a/internal/graft/corestore/sqlstore/checkpoint.go
+++ b/internal/graft/corestore/sqlstore/checkpoint.go
@@ -69,22 +69,24 @@ func (s *Store) CreateCheckpoint(ctx context.Context, repoID int64, in corestore
return nil
}
+ // A deadline already passed fails before the snapshot is written; it is
+ // checked again as the last statement.
if err := s.checkPackDeadline(ctx, tx, in.PackDeadline); err != nil {
return err
}
- now, err := s.wallTime(ctx, tx)
+ provisional, err := s.wallTime(ctx, tx)
if err != nil {
return err
}
// Another build of the same sequence may commit first (the repository
// is not locked): its checkpoint is kept and returned, as one found
- // above.
+ // above. created_at is set again at the end.
res, err := tx.ExecContext(ctx, s.bind(`
INSERT INTO graft_checkpoints (repo_id, seq, pack_key, pack_bytes, created_at)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT (repo_id, seq) DO NOTHING
- `), repoID, int64(in.Seq), in.PackKey, in.PackBytes, s.timeArg(now))
+ `), repoID, int64(in.Seq), in.PackKey, in.PackBytes, s.timeArg(provisional))
if err != nil {
return fmt.Errorf("insert checkpoint: %w", err)
}
@@ -125,6 +127,24 @@ func (s *Store) CreateCheckpoint(ctx context.Context, repoID int64, in corestore
}
}
+ // The last statements (integrated review, final minor 1): the age of a
+ // checkpoint (C_gc, C_sync) counts from created_at, and the pack's
+ // deadline bounds the commit, so both are read on the database clock
+ // after the snapshot, however long it took to write: the time between
+ // created_at and the commit is that of one statement, not of the
+ // snapshot, and does not lean on gc_margin.
+ if err := s.checkPackDeadline(ctx, tx, in.PackDeadline); err != nil {
+ return err
+ }
+ now, err := s.wallTime(ctx, tx)
+ if err != nil {
+ return err
+ }
+ if _, err := tx.ExecContext(ctx, s.bind("UPDATE graft_checkpoints SET created_at = ? WHERE repo_id = ? AND seq = ?"),
+ s.timeArg(now), repoID, int64(in.Seq)); err != nil {
+ return fmt.Errorf("stamp checkpoint: %w", err)
+ }
+
cp = corestore.Checkpoint{
RepoID: repoID,
Seq: in.Seq,
@@ -142,6 +162,19 @@ func (s *Store) CreateCheckpoint(ctx context.Context, repoID int64, in corestore
return cp, created, nil
}
+// CheckpointsBelow implements corestore.MetaStore.
+func (s *Store) CheckpointsBelow(ctx context.Context, repoID int64, seq uint64) (int, error) {
+ if seq > math.MaxInt64 {
+ seq = math.MaxInt64
+ }
+ var n int
+ if err := s.readDB.QueryRowContext(ctx, s.bind("SELECT COUNT(*) FROM graft_checkpoints WHERE repo_id = ? AND seq < ?"),
+ repoID, int64(seq)).Scan(&n); err != nil {
+ return 0, fmt.Errorf("count checkpoints: %w", err)
+ }
+ return n, nil
+}
+
// validateCheckpoint checks a checkpoint before anything is read.
func validateCheckpoint(repoID int64, in corestore.CheckpointInput) error {
if in.Seq == 0 || in.Seq > math.MaxInt64 {
diff --git a/internal/graft/corestore/sqlstore/checkpoint_large_test.go b/internal/graft/corestore/sqlstore/checkpoint_large_test.go
index 184f7d1..dc7276d 100644
--- a/internal/graft/corestore/sqlstore/checkpoint_large_test.go
+++ b/internal/graft/corestore/sqlstore/checkpoint_large_test.go
@@ -74,5 +74,48 @@ func TestCheckpointOfManyReferences(t *testing.T) {
got, err := s.CheckpointRefs(ctx, repo.ID, 1)
require.NoError(t, err)
require.Equal(t, refs, got)
+
+ // created_at is stamped after the snapshot (final minor 1): it is not
+ // the time the record began. The database runs on this host, so its
+ // clock is the test's.
+ cp, ok, err := s.NewestCheckpoint(ctx, repo.ID)
+ require.NoError(t, err)
+ require.True(t, ok)
+ require.True(t, cp.CreatedAt.After(began.Add(took/2)),
+ "created_at %s, record from %s for %s", cp.CreatedAt, began, took)
+ })
+}
+
+// TestCheckpointDeadlineIsCheckedAfterTheSnapshot: a deadline that passes
+// while a large snapshot is written fails the record, and nothing is recorded
+// (final minor 1).
+func TestCheckpointDeadlineIsCheckedAfterTheSnapshot(t *testing.T) {
+ const references = 100_000
+
+ forEachDialect(t, func(t *testing.T, ctx context.Context, s *sqlstore.Store) {
+ repo, _ := headRepo(t, ctx, s, "deadline during the snapshot", 1)
+ refs := make(map[string]corestore.Target, references)
+ for i := 0; i < references; i++ {
+ refs[fmt.Sprintf("refs/heads/branch-%06d", i)] = oid(i + 1)
+ }
+
+ // How long the snapshot takes here, then a deadline at a tenth of it:
+ // open at the first check, passed at the last.
+ began := time.Now()
+ _, created, err := s.CreateCheckpoint(ctx, repo.ID, corestore.CheckpointInput{
+ Seq: 1, Refs: refs, PackKey: ownedKey(repo.ID, "measure"), PackDeadline: time.Now().Add(time.Hour), PackBytes: 1,
+ })
+ took := time.Since(began)
+ require.NoError(t, err)
+ require.True(t, created)
+
+ next, _ := headRepo(t, ctx, s, "deadline during the snapshot, again", 1)
+ _, _, err = s.CreateCheckpoint(ctx, next.ID, corestore.CheckpointInput{
+ Seq: 1, Refs: refs, PackKey: ownedKey(next.ID, "late"), PackDeadline: time.Now().Add(took / 10), PackBytes: 1,
+ })
+ require.ErrorIs(t, err, corestore.ErrPackDeadline, "the record took %s", took)
+ _, ok, err := s.NewestCheckpoint(ctx, next.ID)
+ require.NoError(t, err)
+ require.False(t, ok, "nothing is recorded")
})
}
diff --git a/internal/graft/corestore/syncer/builds.go b/internal/graft/corestore/syncer/builds.go
new file mode 100644
index 0000000..3c40337
--- /dev/null
+++ b/internal/graft/corestore/syncer/builds.go
@@ -0,0 +1,119 @@
+package syncer
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+ "syscall"
+ "time"
+)
+
+// buildLockName is the file in a build directory on which the process that
+// runs the build holds an exclusive flock until it removed the directory. The
+// kernel releases it when the process dies, so a process cleaning up tells the
+// builds of a process that died from those of one that runs (the old process
+// of a graceful upgrade).
+const buildLockName = "build.lock"
+
+// unlockedBuildAge is how old a build directory without its lock file must be
+// before CleanBuilds removes it: one whose process died between creating the
+// directory and its lock file, which takes microseconds.
+const unlockedBuildAge = time.Minute
+
+// lockBuild creates the lock file of the build directory dir and takes its
+// flock.
+func lockBuild(dir string) (*os.File, error) {
+ f, err := os.OpenFile(filepath.Join(dir, buildLockName), os.O_CREATE|os.O_RDWR, 0o600)
+ if err != nil {
+ return nil, err
+ }
+ if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
+ _ = f.Close()
+ return nil, fmt.Errorf("lock: %w", err)
+ }
+ return f, nil
+}
+
+// CleanBuilds removes the build directories (hydrate-*) of hydrations and
+// seeds that no process runs any more: their process died (a crash, a kill)
+// before it removed them, and nothing else does. A directory whose lock file
+// another process holds is in use and kept. gitaly serve calls it at startup,
+// before the listeners open (final minor 3).
+func (s *Syncer) CleanBuilds(ctx context.Context, storages []string) error {
+ removed := 0
+ for _, storage := range storages {
+ tmpDir, err := s.locator.TempDir(storage)
+ if err != nil {
+ return err
+ }
+ entries, err := os.ReadDir(tmpDir)
+ if errors.Is(err, fs.ErrNotExist) {
+ continue
+ }
+ if err != nil {
+ return fmt.Errorf("storage core: list the temporary directory of %s: %w", storage, err)
+ }
+ for _, entry := range entries {
+ if !entry.IsDir() || !strings.HasPrefix(entry.Name(), strings.TrimSuffix(tmpHydratePatt, "*")) {
+ continue
+ }
+ dir := filepath.Join(tmpDir, entry.Name())
+ stale, err := staleBuild(dir)
+ if err != nil {
+ return fmt.Errorf("storage core: check the build directory %s: %w", dir, err)
+ }
+ if !stale {
+ continue
+ }
+ if err := os.RemoveAll(dir); err != nil {
+ return fmt.Errorf("storage core: remove the build directory %s: %w", dir, err)
+ }
+ removed++
+ }
+ }
+ if removed > 0 {
+ s.logger.WithField("directories", removed).InfoContext(ctx, "storage core: removed the build directories of hydrations that no process runs")
+ }
+ return nil
+}
+
+// staleBuild reports whether no process runs the build in dir: its lock file
+// can be locked, or it has none and dir is older than unlockedBuildAge. A
+// lock taken here is released when the directory is gone or kept.
+func staleBuild(dir string) (bool, error) {
+ f, err := os.OpenFile(filepath.Join(dir, buildLockName), os.O_RDWR, 0)
+ if errors.Is(err, fs.ErrNotExist) {
+ info, err := os.Stat(dir)
+ if err != nil {
+ return false, err
+ }
+ return time.Since(info.ModTime()) >= unlockedBuildAge, nil
+ }
+ if err != nil {
+ return false, err
+ }
+ defer func() { _ = f.Close() }()
+
+ switch err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); {
+ case errors.Is(err, syscall.EWOULDBLOCK):
+ return false, nil
+ case err != nil:
+ return false, err
+ }
+ return true, nil
+}
+
+// StopSeeds cancels the running seed jobs and waits for them to end; no seed
+// starts afterwards. gitaly serve calls it after its servers stopped, before
+// the last durable step of the local references (final minor 3).
+func (s *Syncer) StopSeeds() {
+ s.mu.Lock()
+ s.seedsStopped = true
+ s.mu.Unlock()
+ s.stopSeeds()
+ s.seedsRunning.Wait()
+}
diff --git a/internal/graft/corestore/syncer/syncer.go b/internal/graft/corestore/syncer/syncer.go
index 7f10f15..e1bad94 100644
--- a/internal/graft/corestore/syncer/syncer.go
+++ b/internal/graft/corestore/syncer/syncer.go
@@ -146,6 +146,13 @@ type Syncer struct {
reported map[int64]time.Time
// seeds are the running seed jobs, keyed by corestore.RepoKey.String().
seeds map[string]*seedJob
+ // seedsStopped is set by StopSeeds: no seed starts after it.
+ seedsStopped bool
+ // seedsCtx ends the seed jobs when StopSeeds calls stopSeeds;
+ // seedsRunning counts them.
+ seedsCtx context.Context
+ stopSeeds context.CancelFunc
+ seedsRunning sync.WaitGroup
syncTotal *prometheus.CounterVec
syncSeconds prometheus.Histogram
@@ -176,7 +183,10 @@ var (
// New returns the syncer of core.
func New(core *corestore.Core, logger log.Logger, locator storage.Locator, repoFactory localrepo.Factory, gitCmdFactory gitcmd.CommandFactory) *Syncer {
+ seedsCtx, stopSeeds := context.WithCancel(context.Background())
s := &Syncer{
+ seedsCtx: seedsCtx,
+ stopSeeds: stopSeeds,
core: core,
logger: logger,
locator: locator,
@@ -1535,6 +1545,9 @@ type build struct {
path string
tmpDir string
repo *localrepo.Repo
+ // lock is the flock of the build directory (buildLockName), held until it
+ // is removed.
+ lock *os.File
}
// newBuild creates an empty bare repository to hydrate in the temporary
@@ -1556,6 +1569,10 @@ func (s *Syncer) newBuild(ctx context.Context, repo *gitalypb.Repository, r core
return nil, structerr.NewInternal("storage core: create hydration directory: %w", err)
}
b := &build{dir: dir, path: filepath.Join(dir, "repo.git"), tmpDir: tmpDir}
+ if b.lock, err = lockBuild(dir); err != nil {
+ b.remove()
+ return nil, structerr.NewInternal("storage core: lock the hydration directory: %w", err)
+ }
if err := gitops.InitBare(ctx, s.gitCmdFactory, b.path, r.ObjectFormat); err != nil {
b.remove()
return nil, structerr.NewInternal("storage core: init repository: %w", err)
@@ -1570,7 +1587,12 @@ func (s *Syncer) newBuild(ctx context.Context, repo *gitalypb.Repository, r core
}
// remove deletes what is left of the build (nothing once it was installed).
-func (b *build) remove() { _ = os.RemoveAll(b.dir) }
+func (b *build) remove() {
+ _ = os.RemoveAll(b.dir)
+ if b.lock != nil {
+ _ = b.lock.Close()
+ }
+}
// finishBuild sets the references of the build, writes its watermark at seq
// and installs it at path (install, with boundary).
@@ -1659,9 +1681,19 @@ func (s *Syncer) startSeed(ctx context.Context, key corestore.RepoKey, repo *git
}
j := &seedJob{cp: cp, done: make(chan struct{})}
+ if s.seedsStopped {
+ j.err = structerr.NewUnavailable("storage core: the server is stopping: %s is not seeded", key)
+ close(j.done)
+ return j
+ }
s.seeds[key.String()] = j
+ // Detached from the request, ended at the deadline or by StopSeeds.
jobCtx, cancel := context.WithDeadline(context.WithoutCancel(ctx), readAt.Add(s.core.WriteDeadline()))
+ stopOnShutdown := context.AfterFunc(s.seedsCtx, cancel)
+ s.seedsRunning.Add(1)
go func() {
+ defer s.seedsRunning.Done()
+ defer stopOnShutdown()
defer cancel()
err := s.runSeed(jobCtx, repo, r, hash, path, cp)
if err != nil && !errors.Is(err, errSeedSuperseded) {
@@ -1677,9 +1709,65 @@ func (s *Syncer) startSeed(ctx context.Context, key corestore.RepoKey, repo *git
return j
}
+// errSeedDeadline ends a seed that ran past its deadline on the database
+// clock: the checkpoint it read may be collected from then on.
+var errSeedDeadline = errors.New("seed deadline passed")
+
+// errSeedRepositoryGone ends a seed whose repository was removed, or removed
+// and created again under a new ID, while it ran.
+var errSeedRepositoryGone = errors.New("the repository was removed during the seed")
+
// runSeed seeds a build from the checkpoint cp and installs it at path with
-// the watermark cp.Seq.
+// the watermark cp.Seq. A seed that failed after its deadline on the database
+// clock, or whose repository went meanwhile, fails as that: what it ran into
+// (a pack collected after the deadline looks like a missing pack, a removed
+// repository like a missing record) is a consequence.
func (s *Syncer) runSeed(ctx context.Context, repo *gitalypb.Repository, r corestore.Repository, hash git.ObjectHash, path string, cp corestore.Checkpoint) error {
+ err := s.seedAndInstall(ctx, repo, r, hash, path, cp)
+ if err == nil || errors.Is(err, errSeedSuperseded) || errors.Is(err, errSeedDeadline) || errors.Is(err, errSeedRepositoryGone) {
+ return err
+ }
+ if s.seedsCtx.Err() != nil {
+ // StopSeeds cancelled it: the server is stopping.
+ return structerr.NewUnavailable("storage core: the server is stopping: the seed of %s from checkpoint %d is cancelled (%v)", r.Key, cp.Seq, err)
+ }
+ if checkErr := s.checkSeedDeadline(context.WithoutCancel(ctx), r, cp); checkErr != nil {
+ // The cause is reported as text: the code is that of checkErr.
+ return fmt.Errorf("%w (the seed failed with: %v)", checkErr, err)
+ }
+ return err
+}
+
+// checkSeedDeadline reads the repository record again: on the database clock
+// the seed must be before r.ReadAt + W (Core.WriteDeadline), the bound under
+// which the checkpoint, the newest when r was read, cannot be collected; and
+// the repository must still be r. The job's context ends at the same bound on
+// this host's monotonic clock, which stops while the host is suspended; the
+// database clock does not (as for the writers, final minor 2).
+func (s *Syncer) checkSeedDeadline(ctx context.Context, r corestore.Repository, cp corestore.Checkpoint) error {
+ if r.ReadAt.IsZero() {
+ return structerr.NewInternal("storage core: seed of %s: the record carries no database read time", r.Key)
+ }
+ ctx, cancel := context.WithTimeout(ctx, s.syncTimeout())
+ defer cancel()
+
+ current, err := s.core.Meta.GetRepository(ctx, r.Key)
+ switch {
+ case errors.Is(err, corestore.ErrRepositoryNotFound) || (err == nil && current.ID != r.ID):
+ return structerr.NewUnavailable("storage core: seeding %s from checkpoint %d: %w; the next request starts from its current state", r.Key, cp.Seq, errSeedRepositoryGone)
+ case err != nil:
+ return structerr.NewUnavailable("storage core: seeding %s: read the repository record: %w", r.Key, err)
+ }
+ deadline := r.ReadAt.Add(s.core.WriteDeadline())
+ if !current.ReadAt.Before(deadline) {
+ return structerr.NewUnavailable("storage core: seeding %s from checkpoint %d: %w at %s on the database clock (%s): the checkpoint may be collected; the next request seeds from the newest",
+ r.Key, cp.Seq, errSeedDeadline, deadline.UTC().Format(time.RFC3339Nano), current.ReadAt.UTC().Format(time.RFC3339Nano))
+ }
+ return nil
+}
+
+// seedAndInstall is runSeed without the mapping of its failures.
+func (s *Syncer) seedAndInstall(ctx context.Context, repo *gitalypb.Repository, r corestore.Repository, hash git.ObjectHash, path string, cp corestore.Checkpoint) error {
b, err := s.newBuild(ctx, repo, r)
if err != nil {
return err
@@ -1690,6 +1778,10 @@ func (s *Syncer) runSeed(ctx context.Context, repo *gitalypb.Repository, r cores
if err != nil {
return err
}
+ // A seed that indexed the whole pack holds every object of the
+ // checkpoint, verified against its key, so it may be installed after its
+ // deadline; what the deadline protects is the changeset packs after the
+ // checkpoint, which C_sync below covers.
boundary, err := s.syncBoundary(ctx, r.ID)
if err != nil {
return err
diff --git a/internal/graft/corestore/syncer/syncer_seed_test.go b/internal/graft/corestore/syncer/syncer_seed_test.go
new file mode 100644
index 0000000..d22c4e5
--- /dev/null
+++ b/internal/graft/corestore/syncer/syncer_seed_test.go
@@ -0,0 +1,216 @@
+package syncer_test
+
+import (
+ "os"
+ "path/filepath"
+ "sync"
+ "syscall"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+
+ "github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/sqlstore"
+)
+
+// seedFixture is a repository written on node a with a checkpoint followed by
+// changesets, and a fresh node b whose first seed the test holds at the "seed"
+// point (after the checkpoint's references are read, before its pack is
+// indexed) until release is closed. b's rounds are long, so a round waits for
+// the seed and returns its result.
+type seedFixture struct {
+ a, b *env
+ src source
+ repo corestore.Repository
+ cp corestore.Checkpoint
+ seeding chan struct{}
+ release chan struct{}
+}
+
+func newSeedFixture(t *testing.T) *seedFixture {
+ t.Helper()
+
+ f := &seedFixture{a: newEnv(t, envOptions{}), seeding: make(chan struct{}), release: make(chan struct{})}
+ f.src = f.a.newSource(t)
+ c := f.a.writeCommit(t, f.src)
+ f.repo = f.a.register(t, f.a.key(localRelPath), f.src, c)
+ c = f.a.advanceMain(t, f.repo.ID, f.src, c)
+ f.cp = f.a.checkpoint(t, f.repo.ID, f.src)
+ f.a.advanceMain(t, f.repo.ID, f.src, c)
+
+ f.b = f.a.secondNode(t, sqlstore.SQLite)
+ f.b.core.SyncTimeout = time.Minute
+ var once sync.Once
+ f.b.sy.SetTestHook(func(point string) {
+ if point != "seed" {
+ return
+ }
+ first := false
+ once.Do(func() { first = true; close(f.seeding) })
+ if first {
+ <-f.release
+ }
+ })
+ return f
+}
+
+// ensureFresh runs EnsureFresh on b in the background.
+func (f *seedFixture) ensureFresh() <-chan error {
+ result := make(chan error, 1)
+ go func() { result <- f.b.sy.EnsureFresh(f.b.ctx, f.b.pbRepo(localRelPath)) }()
+ return result
+}
+
+// TestSeedSupersededByANewerCheckpoint: a checkpoint that becomes C_sync while
+// a seed from an older one runs makes that seed install nothing; the round
+// seeds from the newest and catches up (final minor 3).
+func TestSeedSupersededByANewerCheckpoint(t *testing.T) {
+ t.Parallel()
+ f := newSeedFixture(t)
+
+ result := f.ensureFresh()
+ <-f.seeding
+ newer := f.a.checkpoint(t, f.repo.ID, f.src)
+ require.Greater(t, newer.Seq, f.cp.Seq)
+ f.a.ageCheckpoint(t, f.repo.ID, newer.Seq, f.b.core.GCGrace+time.Minute)
+ close(f.release)
+
+ require.NoError(t, <-result)
+ require.Equal(t, float64(2), counter(t, f.b.sy, "graft_storage_sync_checkpoint_seeds_total", ""), "the superseded seed, then one from the newest")
+ repo, err := f.b.core.Meta.GetRepository(f.b.ctx, f.repo.Key)
+ require.NoError(t, err)
+ f.b.verifySynced(t, repo)
+}
+
+// TestSeedPastItsDeadline: a seed held past its deadline (readAt + W) while
+// its checkpoint pack is collected, as garbage collection may then do, fails
+// as a seed past its deadline, Unavailable, not as a missing pack; the next
+// request starts again (final minor 2).
+func TestSeedPastItsDeadline(t *testing.T) {
+ t.Parallel()
+ f := newSeedFixture(t)
+ // W = gc_grace - gc_margin = 1 s on node b.
+ f.b.core.GCGrace = 2 * time.Second
+ f.b.core.GCMargin = time.Second
+
+ result := f.ensureFresh()
+ <-f.seeding
+ require.NoError(t, f.a.core.Blobs.Delete(f.a.ctx, f.cp.PackKey), "collected after the deadline")
+ // The deadline is a time: the seed is held until it passed.
+ <-time.After(f.b.core.WriteDeadline() + 200*time.Millisecond)
+ close(f.release)
+
+ err := <-result
+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)
+ require.ErrorContains(t, err, "seed deadline passed")
+ _, ok, appliedErr := f.b.sy.Applied(f.b.ctx, f.b.pbRepo(localRelPath))
+ require.NoError(t, appliedErr)
+ require.False(t, ok, "nothing is installed")
+}
+
+// TestSeedOfARepositoryRemovedAndCreatedAgain: the repository is removed and
+// registered again under a new ID while a seed of the old one runs; the seed
+// installs nothing and fails as such, and the next request hydrates the new
+// repository (final minor 3).
+func TestSeedOfARepositoryRemovedAndCreatedAgain(t *testing.T) {
+ t.Parallel()
+ f := newSeedFixture(t)
+
+ result := f.ensureFresh()
+ <-f.seeding
+ removed, err := f.a.core.Meta.DeleteRepository(f.a.ctx, f.repo.Key)
+ require.NoError(t, err)
+ require.True(t, removed)
+ again := f.a.register(t, f.repo.Key, f.src, f.a.writeCommit(t, f.src))
+ require.NotEqual(t, f.repo.ID, again.ID)
+ close(f.release)
+
+ err = <-result
+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)
+ require.ErrorContains(t, err, "the repository was removed during the seed")
+
+ require.NoError(t, f.b.sy.EnsureFresh(f.b.ctx, f.b.pbRepo(localRelPath)))
+ again, err = f.b.core.Meta.GetRepository(f.b.ctx, again.Key)
+ require.NoError(t, err)
+ f.b.verifySynced(t, again)
+}
+
+// TestStopSeedsEndsTheSeedJobs: StopSeeds cancels a running seed, waits for it
+// and its build directory to be gone, and no seed starts after it (final
+// minor 3).
+func TestStopSeedsEndsTheSeedJobs(t *testing.T) {
+ t.Parallel()
+ f := newSeedFixture(t)
+
+ result := f.ensureFresh()
+ <-f.seeding
+ stopped := make(chan struct{})
+ go func() {
+ f.b.sy.StopSeeds()
+ close(stopped)
+ }()
+ select {
+ case <-stopped:
+ t.Fatal("StopSeeds returned while the seed runs")
+ default:
+ }
+ close(f.release)
+ <-stopped
+
+ err := <-result
+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)
+ tmpDir, tmpErr := f.b.locator.TempDir(f.b.storageName)
+ require.NoError(t, tmpErr)
+ builds, globErr := filepath.Glob(filepath.Join(tmpDir, "hydrate-*"))
+ require.NoError(t, globErr)
+ require.Empty(t, builds, "the build directory is removed")
+
+ err = f.b.sy.EnsureFresh(f.b.ctx, f.b.pbRepo(localRelPath))
+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)
+ require.ErrorContains(t, err, "the server is stopping")
+}
+
+// TestCleanBuildsRemovesOnlyDeadBuilds: at startup the build directories of
+// a process that died are removed (their lock is free, or they never got one
+// and are old), those of a running process are kept (final minor 3).
+func TestCleanBuildsRemovesOnlyDeadBuilds(t *testing.T) {
+ t.Parallel()
+ e := newEnv(t, envOptions{})
+ tmpDir, err := e.locator.TempDir(e.storageName)
+ require.NoError(t, err)
+ require.NoError(t, os.MkdirAll(tmpDir, 0o700))
+
+ mkdir := func(name string) string {
+ dir := filepath.Join(tmpDir, name)
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "repo.git"), 0o700))
+ return dir
+ }
+ lockFile := func(dir string) *os.File {
+ f, err := os.OpenFile(filepath.Join(dir, "build.lock"), os.O_CREATE|os.O_RDWR, 0o600)
+ require.NoError(t, err)
+ t.Cleanup(func() { _ = f.Close() })
+ return f
+ }
+
+ running := mkdir("hydrate-running")
+ require.NoError(t, syscall.Flock(int(lockFile(running).Fd()), syscall.LOCK_EX|syscall.LOCK_NB))
+ dead := mkdir("hydrate-dead")
+ require.NoError(t, lockFile(dead).Close(), "a lock file nobody holds: its process died")
+ oldUnlocked := mkdir("hydrate-old-unlocked")
+ require.NoError(t, os.Chtimes(oldUnlocked, time.Now().Add(-2*time.Minute), time.Now().Add(-2*time.Minute)))
+ newUnlocked := mkdir("hydrate-new-unlocked")
+ other := mkdir("stale-other")
+
+ require.NoError(t, e.sy.CleanBuilds(e.ctx, []string{e.storageName}))
+ for dir, kept := range map[string]bool{running: true, dead: false, oldUnlocked: false, newUnlocked: true, other: true} {
+ _, err := os.Stat(dir)
+ if kept {
+ require.NoError(t, err, "%s is kept", dir)
+ } else {
+ require.ErrorIs(t, err, os.ErrNotExist, "%s is removed", dir)
+ }
+ }
+}
diff --git a/internal/graft/corestore/uploadpack_only_test.go b/internal/graft/corestore/uploadpack_only_test.go
new file mode 100644
index 0000000..1405f67
--- /dev/null
+++ b/internal/graft/corestore/uploadpack_only_test.go
@@ -0,0 +1,129 @@
+package corestore_test
+
+import (
+ "go/ast"
+ "go/parser"
+ "go/token"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+)
+
+// TestUploadPackIsTheOnlyUploader enforces invariant U (doc/graft/PLAN7.md,
+// RULE 26 and "Removed repositories"): every object of the bucket is written
+// by Core.UploadPack, which needs a repository record that a read found. The
+// bucket is reached through the field Core.Blobs only, so in the module's
+// non-test code:
+// - Blobs.Put is called in core.go only (UploadPack);
+// - Blobs is not taken as a value (assigned, passed, returned), which would
+// let a Put happen under another name, except where the core is built.
+//
+// Tests may write the bucket directly to set up states. The files checked are
+// those that can reach a *corestore.Core: they import corestore or one of its
+// packages, call the dependencies' GetGraftStorage, or are in internal/graft
+// or the storagecore service (other fields named Blobs, in protobuf messages
+// and git packfile code, are unrelated).
+func TestUploadPackIsTheOnlyUploader(t *testing.T) {
+ root := moduleRoot(t)
+ // Where Blobs may be used as a value: the core is built and closed there.
+ wiring := map[string]bool{
+ "internal/graft/corestore/core.go": true,
+ "internal/graft/corestore/open/open.go": true,
+ }
+
+ var violations []string
+ fset := token.NewFileSet()
+ err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ rel, err := filepath.Rel(root, path)
+ if err != nil {
+ return err
+ }
+ if d.IsDir() {
+ if rel != "." && (strings.HasPrefix(d.Name(), ".") || d.Name() == "_build" || d.Name() == "_support" || d.Name() == "testdata") {
+ return filepath.SkipDir
+ }
+ return nil
+ }
+ if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
+ return nil
+ }
+ src, err := os.ReadFile(path)
+ if err != nil {
+ return err
+ }
+ file, err := parser.ParseFile(fset, path, src, parser.SkipObjectResolution)
+ if err != nil {
+ return err
+ }
+ rel = filepath.ToSlash(rel)
+ if !reachesCore(rel, file, src) {
+ return nil
+ }
+
+ var stack []ast.Node
+ ast.Inspect(file, func(n ast.Node) bool {
+ if n == nil {
+ stack = stack[:len(stack)-1]
+ return true
+ }
+ stack = append(stack, n)
+ sel, ok := n.(*ast.SelectorExpr)
+ if !ok || sel.Sel.Name != "Blobs" {
+ return true
+ }
+ pos := fset.Position(sel.Pos())
+ // The parent of x.Blobs: a method selection (x.Blobs.M) or a use as
+ // a value.
+ if len(stack) >= 2 {
+ if method, ok := stack[len(stack)-2].(*ast.SelectorExpr); ok && method.X == sel {
+ if method.Sel.Name == "Put" && rel != "internal/graft/corestore/core.go" {
+ violations = append(violations, pos.String()+": Blobs.Put outside Core.UploadPack")
+ }
+ return true
+ }
+ }
+ if !wiring[rel] {
+ violations = append(violations, pos.String()+": Blobs used as a value outside the core's wiring")
+ }
+ return true
+ })
+ return nil
+ })
+ require.NoError(t, err)
+ require.Empty(t, violations, "every bucket write goes through Core.UploadPack (RULE 26, invariant U)")
+}
+
+// reachesCore reports whether the file at rel can reach a *corestore.Core.
+func reachesCore(rel string, file *ast.File, src []byte) bool {
+ if strings.HasPrefix(rel, "internal/graft/") || strings.HasPrefix(rel, "internal/gitaly/service/storagecore/") {
+ return true
+ }
+ for _, imp := range file.Imports {
+ if strings.HasPrefix(strings.Trim(imp.Path.Value, `"`), "github.com/feder-cr/git-ai/internal/graft/corestore") {
+ return true
+ }
+ }
+ return strings.Contains(string(src), "GetGraftStorage")
+}
+
+// moduleRoot returns the directory of the module's go.mod.
+func moduleRoot(t *testing.T) string {
+ t.Helper()
+ dir, err := os.Getwd()
+ require.NoError(t, err)
+ for {
+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {
+ return dir
+ }
+ parent := filepath.Dir(dir)
+ require.NotEqual(t, dir, parent, "no go.mod above the test")
+ dir = parent
+ }
+}