Skip to content

Git designed for AI agents.

Hydration that always progresses, checkpoints by default, fifth-review minors

Federico Elia0592c763caa7
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.md
index 2f56b21..4c7b6d9 100644
--- a/doc/graft/CONFIGURATION.md
+++ b/doc/graft/CONFIGURATION.md
@@ -210,12 +210,13 @@ are in [STORAGE.md](STORAGE.md).
| `database` | string | required when on | `sqlite:///absolute/path.db`, or `postgres://user@host:port/db?sslmode=...` (`postgresql://` too). All nodes use the same database. Credentials are not written here: Postgres reads the `PG*` environment variables and `.pgpass`. The schema is created at startup. |
| `bucket` | string | required when on | gocloud URL of the object storage: `file:///absolute/dir`, `s3://bucket?region=...`, `gs://bucket` or `azblob://container`. All nodes use the same bucket. Credentials come from the cloud SDK's environment (`AWS_*`, `GOOGLE_APPLICATION_CREDENTIALS`, `AZURE_*`). |
| `node_id` | string | the host name | Name of this node in the database (changesets record it, and `graft storage state` shows it). Must not contain spaces or slashes. Give every node its own. |
-| `sync_timeout` | string (Go duration) | `"10s"` | Budget of the catch-up before a request in authoritative mode. A catch-up that cannot finish in time (for example because the database is unreachable or a local writer holds a reference lock) fails the request with `Unavailable`. Must be positive. |
+| `sync_timeout` | string (Go duration) | `"10s"` | Budget of one sync round before a request in authoritative mode. A round that cannot reach the head in time keeps what it did: the local repository stays at the changeset it reached, the request fails with `Unavailable`, and the next request's round goes on from there. A round that cannot finish at all (for example because the database is unreachable or a local writer holds a reference lock) fails the request with `Unavailable` too. Must be positive. |
| `shadow_timeout` | string (Go duration) | `"5s"` | Budget of the shadow-mode work of one reference transaction, for its Preparing phase (pack build and upload, before git locks the references) and its Prepared phase together. When it runs out, the work is abandoned, logged and counted, and the git write proceeds. Must be positive. |
| `gc_grace` | string (Go duration) | `"1h"` | How long garbage collection waits before it deletes a pack it observed unreferenced, and how old a checkpoint must be before the packs it covers are deleted. Must be greater than `sync_timeout + gc_margin`. |
| `gc_margin` | string (Go duration) | `"5m"` | Safety margin of the writers' deadline: a write uploads its pack and commits its changeset within `gc_grace - gc_margin`, else it fails. It also covers the difference between a node's clock and the database's. Must be positive. |
-| `maintenance` | string | `"off"` | `"auto"` runs the background checkpoints and garbage collection: every `maintenance_interval`, the packs of removed repositories are deleted, and every repository this node has a current copy of is checkpointed when the thresholds below say so and its dead packs are collected, under a per-repository lease so that two nodes do not do the same work. Authoritative mode only. `graft storage` commands do the same on demand in either case. |
-| `maintenance_interval` | string (Go duration) | `"1h"` | Time between two background maintenance runs. Must be positive. |
+| `checkpoints` | string | `"auto"` in authoritative mode, `"off"` otherwise | `"auto"` runs the background checkpoints: every `maintenance_interval`, every repository this node has a current copy of is checkpointed when the thresholds below say so, under a per-repository lease so that two nodes do not do the same work. On by default: a checkpoint deletes nothing (it adds a pack and database rows), and without checkpoints a node that joins replays every changeset of a repository. `"off"` turns them off. `"auto"` requires authoritative mode. |
+| `gc` | string | `"off"` | `"auto"` runs the background garbage collection: every `maintenance_interval`, the packs of removed repositories are deleted, and so are the dead packs of every repository this node has a current copy of, under the same lease. Off by default because it is the operation that deletes data. Authoritative mode only. `graft storage gc` does the same on demand in either case. |
+| `maintenance_interval` | string (Go duration) | `"1h"` | Time between two runs of the background checkpoints and garbage collection. Must be positive. |
| `checkpoint_changesets` | integer | `1000` | Checkpoint a repository once this many changesets follow its newest checkpoint. `0` never checkpoints by count. |
| `checkpoint_pack_bytes` | integer | `268435456` (256 MiB) | Checkpoint a repository once the packs of the changesets after its newest checkpoint hold this many bytes. `0` never checkpoints by size. |
| `lease_ttl` | string (Go duration) | `"10m"` | How long a node holds a repository's maintenance lease; renewed every third of it while the work runs. Must be positive. |
@@ -226,16 +227,16 @@ mode = "authoritative"
database = "sqlite:///var/opt/graft/meta.db"
bucket = "file:///var/opt/graft/objects"
node_id = "node-a"
-maintenance = "auto"
+gc = "auto" # checkpoints are on by default in authoritative mode
```
Startup fails when the mode is unknown, authoritative mode is set without
`standalone = true`, the database URL is missing or has another scheme, the
bucket is missing or not a URL, `node_id` contains spaces or slashes, a
duration is not a positive Go duration, `gc_grace` is not greater than
-`sync_timeout + gc_margin`, `maintenance` is neither `off` nor `auto`,
-`maintenance = "auto"` is set outside authoritative mode, or
-`checkpoint_pack_bytes` is negative.
+`sync_timeout + gc_margin`, `checkpoints` or `gc` is neither `off` nor `auto`,
+either is `"auto"` outside authoritative mode, or `checkpoint_pack_bytes` is
+negative.
## `graft-cache` flags
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.md
index 67b3154..d4cc5bc 100644
--- a/doc/graft/PLAN7.md
+++ b/doc/graft/PLAN7.md
@@ -804,6 +804,33 @@ repository-wide lock.
`TestDropMarkerGuards`. The conditions: live processes, git's lock, git
activity, untrusted, the plain-marker rewrite, the missing-path removal,
the recovered untrusted record, the reboot distrust.
+ * Known limits of the model, each covered elsewhere:
+ * the grace floor is 0 in the model, so time does not decide
+ (`TestGitActivityPushesTheGraceForward`, and the recovery tests);
+ * the syncer's own writer, which marks the repository under the
+ repository lock, is not modelled (the syncer tests);
+ * auto-compaction is not a separate step: it holds git's lock and
+ temporary tables, which `gitActive` sees (`TestRecoveredMarkerWaitsForGit`);
+ * the steps are sequential: what git's reference lock excludes is the
+ `ownLock` guard (`TestDropMarkerGuards`).
+* Fifth review (of 70d4a2b): no blocker, no major. Minors:
+ * A repository without a reftable stack (gone, or a files-backend copy being
+ imported) has nothing written without fsync (`hasStack`). Settle, Forget and
+ `dropMarker` treat it so, and the recovery never fails on one
+ (`TestRepositoryWithoutAStackHasNothingToMakeDurable`).
+ * Lock and temporary files of a stack older than an hour, the grace of
+ Gitaly's housekeeping (`ReftableLockfileGracePeriod`), are left by a git
+ process that died. They are not activity, and the durable step removes such
+ a list lock as housekeeping would, so a crash cannot pin a marker
+ (`TestStaleGitLockDoesNotPinTheMarker`).
+ * `ImportRepository` (passthrough) refuses an untrusted repository
+ (`TestImportRefusesAnUntrustedRepository`). The other passthrough RPCs:
+ * `VerifyRepository` and `CreateCheckpoint` run `EnsureFresh`, which
+ refuses;
+ * `GetStorageState`, `CollectGarbage` and `CollectRemovedRepositories`
+ read no local references;
+ * the hook RPCs run inside a write that the gate admitted.
+ * STORAGE.md: "never served" means from the next request on.
**Tests.**
* `localrefs`:
@@ -1423,6 +1450,57 @@ wm.Seq < C_sync`; otherwise it catches up as in phase 6 (the WP55 RULE unchanged
* `TestSyncerRoundBoundedBySyncTimeout` shows that a bucket read is cancelled by
the round's deadline and nothing is installed.
+### Resumable rounds and the per-pack unit (after WP79's benchmark, approved by the PM)
+
+The WP79 benchmark found that a hydration was one round, all or nothing,
+within `sync_timeout`. At the default 10 s, a fresh node could not hydrate a
+repository of about 1100 changesets with no checkpoint: every round started
+from zero. Two root causes, both fixed. The bound itself stays: the GC BOUNDARY
+rule needs every round to end within `sync_timeout`.
+
+1. *Rounds keep what they did.*
+ * `round` (`newRound`) stops the fetch from starting packs once only a
+ quarter of the budget is left (`roundReserve`). The fetch then ends at the
+ last changeset whose pack, and those of every changeset before it, are
+ indexed.
+ * A hydration installs its build at that changeset, with that watermark. It
+ is at or above `C_sync`, since the seed is the newest checkpoint.
+ * A catch-up applies the range it fetched; its extensions (a newer local
+ write, an unexplained local value) are not bounded, because they are what
+ makes the range explain the local repository.
+ * A round that leaves the copy below the head answers `Unavailable` ("at
+ changeset p of H"), counted as `graft_storage_sync_total{result="partial"}`.
+ The next request's round goes on from p.
+ * A round that fetched no changeset at all installs nothing.
+2. *The unit of work is cheaper.*
+ * `indexPacks` downloads, verifies and indexes up to `fetchConcurrency` (8)
+ packs at once: a bucket read, a SHA-256 and a `git index-pack` each.
+ * The packs are self-contained (index-pack without `--fix-thin`), so their
+ order does not matter, and git writes each under its own name.
+ * The reference changes were already folded in memory and applied once
+ (`applyBuild` for a hydration, `applyMoves` for a catch-up).
+
+**Who reads a partial copy outside `EnsureFresh`.** A partial copy is a valid
+local repository at changeset p ≥ `C_sync`: what a node that was offline has.
+It is read by the same readers as such a copy, and none of them serves it as
+current:
+* Every repository-scoped RPC goes through the gate, whose `EnsureFresh`
+ answers `Unavailable` while the copy is below the head.
+* The storage core RPCs: `VerifyRepository` runs `EnsureFresh` first;
+ `GetStorageState` reports p as `applied_sequence`, which is what it is.
+* The background workers:
+ * the maintenance reads only the watermark, and its checkpoint runs
+ `EnsureFresh`, so it waits for the head;
+ * the review-roots sweeper runs `EnsureFresh`;
+ * Gitaly's housekeeping optimizes the objects and serves nothing.
+* Offline tools (`gitaly git`, `gitaly-debug`) read whatever is on disk, as for
+ any lagging copy (STORAGE.md).
+
+**Tests.** `TestSyncerResumesAHydrationAcrossRounds`: rounds of 4 changesets
+on a 10-changeset repository give Unavailable at 4 and at 8, with the
+watermark installed each time, and then the head. All the existing syncer
+tests run through the concurrent fetch.
+
### Verify after compaction (`verify_objects.go`, WP76)
`checkStoredObjects` indexes the newest checkpoint's pack and the changeset packs
@@ -1482,6 +1560,15 @@ shutdown.
an orphan pack deleted gc_grace after a run observed it; Start/stop.
* Not tested: a lease lost in the middle of a run (it needs a build slower than
the renewal).
+* **Changed after WP79 (PM): checkpoints are on by default, GC is opt-in.**
+ With `maintenance = "off"` nothing created checkpoints, so every large
+ repository eventually hit the hydration wall above. A checkpoint deletes
+ nothing (it adds a pack and rows). So the setting is split:
+ * `checkpoints` defaults to `"auto"` in authoritative mode;
+ * `gc`, which deletes, defaults to `"off"`.
+ `maintenance_interval` and the lease serve both, and `Config.Checkpoints` and
+ `Config.Collect` gate the steps. Test: `TestRunOnceDoesOnlyWhatIsOn`. The e2e
+ test of the background checkpoints runs with the defaults.
* **Database openers (audit).** Only `gitaly serve` opens the storage database
(`corestoreopen.Open`, one `sqlstore.Store` per process). The `graft` CLI and
every other binary reach it through gRPC. Two Stores on one SQLite file exist
@@ -1493,7 +1580,8 @@ shutdown.
```toml
[graft.storage]
# ... mode, database, bucket, node_id, sync_timeout, shadow_timeout ...
-maintenance = "off" # off | auto: background checkpoints and GC (authoritative only). Default off.
+checkpoints = "auto" # auto | off: background checkpoints (authoritative only). Default auto there (changed after WP79).
+gc = "off" # off | auto: background GC (authoritative only). Default off.
maintenance_interval = "1h" # how often the background task runs
checkpoint_changesets = 1000 # checkpoint after this many changesets since the newest one (0 = never)
checkpoint_pack_bytes = 268435456 # ...or after this many pack bytes since the newest one (0 = never)
@@ -1503,7 +1591,7 @@ lease_ttl = "10m" # maintenance lease duration, renewed during
```
Defaults are `Default*` constants with the values above. Validation (extends
-`GraftStorage.validate`): `maintenance in {"", "off", "auto"}`; durations positive;
+`GraftStorage.validate`): `checkpoints` and `gc` in `{"", "off", "auto"}`, `"auto"` in authoritative mode only; durations positive;
thresholds `>= 0`; `gc_margin < gc_grace`; **`gc_grace > sync_timeout + gc_margin`**
(GC BOUNDARY rule). `checkpoint_retain` of the first version does not exist.
@@ -1887,7 +1975,9 @@ numbers). Work files live in `/home/feder/m3-work/`, never in `/tmp` (PM.md).
swept by `graft storage gc --removed` and by the background task.
2. The checkpoint closure includes the head and base commits of open reviews
(through their keep-around references since WP74b).
-3. `maintenance` defaults to `off` in this release.
+3. `maintenance` defaults to `off` in this release. Superseded after WP79's
+ benchmark (PM): `checkpoints` defaults to `auto` in authoritative mode, and
+ `gc` stays `off` (see "Background maintenance").
4. The benchmark window is requested at WP79.
5. (Second review) Change A approved; the B/C refinement confirmed; the review
diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.md
index d0bebb4..7049f5d 100644
--- a/doc/graft/STORAGE.md
+++ b/doc/graft/STORAGE.md
@@ -197,11 +197,12 @@ mode = "authoritative" # off | shadow | authoritative
database = "sqlite:///var/opt/graft/meta.db" # or postgres://[email protected]:5432/graft?sslmode=verify-full
bucket = "file:///var/opt/graft/objects" # gocloud URL: file://, s3://, gs://, azblob://
node_id = "node-a" # default: the host name
-sync_timeout = "10s" # catch-up budget per request
+sync_timeout = "10s" # budget of one sync round per request
shadow_timeout = "5s" # shadow work budget per transaction
gc_grace = "1h" # age before a dead pack is deleted
gc_margin = "5m" # writers' deadline is gc_grace - gc_margin
-maintenance = "auto" # off (default) | auto: background checkpoints and GC
+checkpoints = "auto" # auto (default in authoritative mode) | off
+gc = "auto" # off (default) | auto: background garbage collection
maintenance_interval = "1h"
checkpoint_changesets = 1000 # 0 = never by count
checkpoint_pack_bytes = 268435456 # 0 = never by size
@@ -223,10 +224,19 @@ Startup fails when the mode is unknown, authoritative mode is set without
`standalone = true`, the database URL is missing or neither `sqlite:///` nor
`postgres://` (`postgresql://`), the bucket is missing or not a URL, `node_id`
contains spaces or slashes, a duration is not a positive Go duration,
-`gc_grace` is not greater than `sync_timeout + gc_margin`, or `maintenance =
-"auto"` is set outside authoritative mode. The background maintenance is off by
-default; its metrics are `graft_storage_maintenance_*` (runs, checkpoints, packs
-and bytes deleted, skipped repositories, failed steps).
+`gc_grace` is not greater than `sync_timeout + gc_margin`, or `checkpoints` or
+`gc` is `"auto"` outside authoritative mode. In authoritative mode the
+background checkpoints run by default and the background garbage collection
+does not:
+- A checkpoint deletes nothing (it adds one pack and its rows), and it is what
+ keeps a hydration short. Without one, a node that joins replays every
+ changeset of the repository.
+- Garbage collection deletes packs, so it is turned on deliberately
+ (`gc = "auto"`), or run with `graft storage gc`. Until it runs, the bucket
+ only grows.
+
+The metrics are `graft_storage_maintenance_*` (runs, checkpoints, packs and
+bytes deleted, skipped repositories, failed steps).
## Operations
@@ -525,7 +535,11 @@ or re-import the repository from scratch (see
durable step's fsync fails, the repository is moved aside at once and logged,
and the next request hydrates it. The kernel reports such a failure once, so
from then on the repository is untrusted. Every request for it fails with
- `Unavailable`, and its marker is never removed, until it is moved aside. If
+ `Unavailable`, and its marker is never removed, until it is moved aside.
+ "Every request" means from the next one on. Each request checks once, in
+ its sync round, so a request already past its round when the failure was
+ found completes on what it read. `graft storage import` refuses an
+ untrusted repository too. If
the move fails, it is retried every second. A repository that is not a
cache of the storage core (the database cannot rebuild it) stays in place,
untrusted and not served, for an operator. Other errors of the step (a file
@@ -731,12 +745,17 @@ show:
and the bucket, not that nodes keep no local state. Every request runs git
on a local copy, a node's first request for a repository downloads all of its
packs, and there is no virtual filesystem.
-- **Checkpoints and pack garbage collection run in the background only when
- `maintenance = "auto"`** (authoritative mode; off by default). Without it,
- they run when an operator runs `graft storage checkpoint` and `graft storage
- gc` (see [Checkpoints and garbage collection](#checkpoints-and-garbage-collection));
- otherwise packs are never deleted and hydration replays every changeset after
- the newest checkpoint, from 1 when there is none.
+- **Pack garbage collection runs in the background only with `gc = "auto"`**
+ (authoritative mode; off by default). Otherwise it runs when an operator runs
+ `graft storage gc` (see [Checkpoints and garbage collection](#checkpoints-and-garbage-collection)),
+ and packs are never deleted. The background checkpoints are on by default.
+ With `checkpoints = "off"` and no `graft storage checkpoint`, a hydration
+ replays every changeset of the repository from 1. Its rounds resume, but that
+ takes many requests on a long history.
+- **A sync round is bounded by `sync_timeout`.** A hydration or catch-up that
+ needs more than one round answers `Unavailable` until it reaches the head;
+ each round keeps what it did. A single pack that cannot be downloaded and
+ indexed within one round never completes.
- **Praefect is not supported in authoritative mode.** Configuration validation
requires `[graft] standalone = true` for authoritative mode. Shadow mode
works with or without Praefect.
diff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.go
index 1a515e4..050359b 100644
--- a/internal/cli/gitaly/serve.go
+++ b/internal/cli/gitaly/serve.go
@@ -975,8 +975,9 @@ func openLocalReferenceWrites(core *corestore.Core, locator storage.Locator, log
}
// startStorageMaintenance starts the background checkpoints and garbage
-// collection of the storage core ([graft.storage] maintenance = "auto",
-// authoritative mode only; doc/graft/PLAN7.md, "Background maintenance").
+// collection of the storage core ([graft.storage] checkpoints, on by default,
+// and gc, opt-in; authoritative mode only; doc/graft/PLAN7.md, "Background
+// maintenance").
func startStorageMaintenance(ctx context.Context, cfg config.Cfg, core *corestore.Core, fresh corestore.Freshener, locator storage.Locator, repoFactory localrepo.Factory, logger log.Logger, storageNames []string) (func(), error) {
interval, err := cfg.Graft.Storage.MaintenanceIntervalOrDefault()
if err != nil {
@@ -993,14 +994,19 @@ func startStorageMaintenance(ctx context.Context, cfg config.Cfg, core *corestor
fresh, logger,
storagemaintenance.Config{
Interval: interval,
+ Checkpoints: cfg.Graft.Storage.CheckpointsAuto(),
CheckpointChangesets: cfg.Graft.Storage.CheckpointChangesetsOrDefault(),
CheckpointPackBytes: cfg.Graft.Storage.CheckpointPackBytesOrDefault(),
+ Collect: cfg.Graft.Storage.GCAuto(),
LeaseTTL: leaseTTL,
},
storageNames,
)
prometheus.MustRegister(runner)
- logger.WithField("interval", interval.String()).Info("graft storage core: background maintenance enabled")
+ logger.WithField("interval", interval.String()).
+ WithField("checkpoints", cfg.Graft.Storage.CheckpointsAuto()).
+ WithField("gc", cfg.Graft.Storage.GCAuto()).
+ Info("graft storage core: background maintenance enabled")
return runner.Start(ctx), nil
}
diff --git a/internal/gitaly/config/config_graft_storage.go b/internal/gitaly/config/config_graft_storage.go
index eecd19e..3db9f67 100644
--- a/internal/gitaly/config/config_graft_storage.go
+++ b/internal/gitaly/config/config_graft_storage.go
@@ -40,13 +40,20 @@ type GraftStorage struct {
// Empty means DefaultGraftStorageGCMargin. See doc/graft/STORAGE.md for the
// clock assumption it covers.
GCMargin string `json:"gc_margin" toml:"gc_margin,omitempty"`
- // Maintenance is "off" (the default, also when empty) or "auto": the
- // background checkpoints and garbage collection of the storage core, in
- // authoritative mode only (doc/graft/PLAN7.md, "Background maintenance").
+ // Checkpoints is "auto" or "off": the background checkpoints of the storage
+ // core (doc/graft/PLAN7.md, "Background maintenance"). Empty means "auto" in
+ // authoritative mode, where checkpoints exist, and "off" otherwise. A
+ // checkpoint deletes nothing (it adds a pack and rows), and without one a
+ // fresh node replays every changeset of a repository.
+ Checkpoints string `json:"checkpoints" toml:"checkpoints,omitempty"`
+ // GC is "off" (the default, also when empty) or "auto": the background
+ // garbage collection, which deletes packs of the bucket. Opt-in, since it is
+ // the one operation that destroys data. "auto" requires authoritative mode.
// The graft storage commands work in either case.
- Maintenance string `json:"maintenance" toml:"maintenance,omitempty"`
- // MaintenanceInterval is how often the background maintenance runs (Go
- // duration). Empty means DefaultGraftStorageMaintenanceInterval.
+ GC string `json:"gc" toml:"gc,omitempty"`
+ // MaintenanceInterval is how often the background checkpoints and garbage
+ // collection run (Go duration). Empty means
+ // DefaultGraftStorageMaintenanceInterval.
MaintenanceInterval string `json:"maintenance_interval" toml:"maintenance_interval,omitempty"`
// CheckpointChangesets makes the maintenance checkpoint a repository once
// this many changesets follow its newest checkpoint; 0 never checkpoints by
@@ -63,7 +70,7 @@ type GraftStorage struct {
LeaseTTL string `json:"lease_ttl" toml:"lease_ttl,omitempty"`
}
-// Values of GraftStorage.Maintenance.
+// Values of GraftStorage.Checkpoints and GraftStorage.GC.
const (
GraftStorageMaintenanceOff = "off"
GraftStorageMaintenanceAuto = "auto"
@@ -83,8 +90,24 @@ const (
DefaultGraftStorageLeaseTTL = 10 * time.Minute
)
-// MaintenanceAuto reports whether the background maintenance runs.
-func (s GraftStorage) MaintenanceAuto() bool { return s.Maintenance == GraftStorageMaintenanceAuto }
+// CheckpointsAuto reports whether the background checkpoints run: "auto", or
+// unset in authoritative mode.
+func (s GraftStorage) CheckpointsAuto() bool {
+ switch s.Checkpoints {
+ case GraftStorageMaintenanceAuto:
+ return true
+ case "":
+ return s.Mode == "authoritative"
+ }
+ return false
+}
+
+// GCAuto reports whether the background garbage collection runs.
+func (s GraftStorage) GCAuto() bool { return s.GC == GraftStorageMaintenanceAuto }
+
+// MaintenanceAuto reports whether the background maintenance runs: its
+// checkpoints, its garbage collection, or both.
+func (s GraftStorage) MaintenanceAuto() bool { return s.CheckpointsAuto() || s.GCAuto() }
// MaintenanceIntervalOrDefault parses MaintenanceInterval, applying the default.
func (s GraftStorage) MaintenanceIntervalOrDefault() (time.Duration, error) {
@@ -116,14 +139,16 @@ func (s GraftStorage) CheckpointPackBytesOrDefault() int64 {
// validateMaintenance checks the background maintenance settings.
func (s GraftStorage) validateMaintenance() error {
- switch s.Maintenance {
- case "", GraftStorageMaintenanceOff:
- case GraftStorageMaintenanceAuto:
- if s.Mode != "authoritative" {
- return fmt.Errorf("graft storage: maintenance = %q requires mode = \"authoritative\" (checkpoints exist in authoritative mode only)", s.Maintenance)
+ for _, setting := range []struct{ name, value string }{{"checkpoints", s.Checkpoints}, {"gc", s.GC}} {
+ switch setting.value {
+ case "", GraftStorageMaintenanceOff:
+ case GraftStorageMaintenanceAuto:
+ if s.Mode != "authoritative" {
+ return fmt.Errorf("graft storage: %s = %q requires mode = \"authoritative\" (the background maintenance runs in authoritative mode only)", setting.name, setting.value)
+ }
+ default:
+ return fmt.Errorf("graft storage: unknown %s %q (want off or auto)", setting.name, setting.value)
}
- default:
- return fmt.Errorf("graft storage: unknown maintenance %q (want off or auto)", s.Maintenance)
}
if _, err := s.MaintenanceIntervalOrDefault(); err != nil {
diff --git a/internal/gitaly/config/config_graft_storage_test.go b/internal/gitaly/config/config_graft_storage_test.go
index d911632..ba1973d 100644
--- a/internal/gitaly/config/config_graft_storage_test.go
+++ b/internal/gitaly/config/config_graft_storage_test.go
@@ -40,11 +40,13 @@ func TestGraftStorageValidate(t *testing.T) {
{desc: "bad gc_margin", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, GCMargin: "0"}, err: "gc_margin"},
{desc: "gc_grace not above sync_timeout + gc_margin", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, GCGrace: "6m", GCMargin: "5m", SyncTimeout: "1m"}, err: "gc_grace 6m0s must be greater than sync_timeout 1m0s + gc_margin 5m0s"},
{desc: "gc_grace just above sync_timeout + gc_margin", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, GCGrace: "361s", GCMargin: "5m", SyncTimeout: "1m"}},
- {desc: "maintenance auto in authoritative mode", storage: GraftStorage{Mode: "authoritative", Database: valid.Database, Bucket: valid.Bucket, Maintenance: "auto", MaintenanceInterval: "15m", LeaseTTL: "2m", CheckpointChangesets: ptr[uint64](0), CheckpointPackBytes: ptr[int64](0)}, standalone: true},
- {desc: "maintenance off", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Maintenance: "off"}},
- {desc: "maintenance auto in shadow mode", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Maintenance: "auto"}, err: `maintenance = "auto" requires mode = "authoritative"`},
- {desc: "maintenance auto with the storage core off", storage: GraftStorage{Maintenance: "auto"}, err: `requires mode = "authoritative"`},
- {desc: "unknown maintenance", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Maintenance: "nightly"}, err: "unknown maintenance"},
+ {desc: "checkpoints and gc auto in authoritative mode", storage: GraftStorage{Mode: "authoritative", Database: valid.Database, Bucket: valid.Bucket, Checkpoints: "auto", GC: "auto", MaintenanceInterval: "15m", LeaseTTL: "2m", CheckpointChangesets: ptr[uint64](0), CheckpointPackBytes: ptr[int64](0)}, standalone: true},
+ {desc: "checkpoints and gc off", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Checkpoints: "off", GC: "off"}},
+ {desc: "checkpoints auto in shadow mode", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Checkpoints: "auto"}, err: `checkpoints = "auto" requires mode = "authoritative"`},
+ {desc: "gc auto in shadow mode", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, GC: "auto"}, err: `gc = "auto" requires mode = "authoritative"`},
+ {desc: "gc auto with the storage core off", storage: GraftStorage{GC: "auto"}, err: `requires mode = "authoritative"`},
+ {desc: "unknown checkpoints", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Checkpoints: "nightly"}, err: "unknown checkpoints"},
+ {desc: "unknown gc", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, GC: "yes"}, err: "unknown gc"},
{desc: "bad maintenance interval", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, MaintenanceInterval: "0s"}, err: "maintenance_interval"},
{desc: "bad lease ttl", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, LeaseTTL: "never"}, err: "lease_ttl"},
{desc: "negative checkpoint pack bytes", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, CheckpointPackBytes: ptr[int64](-1)}, err: "checkpoint_pack_bytes"},
@@ -85,7 +87,7 @@ func TestGraftStorageDefaults(t *testing.T) {
require.Equal(t, DefaultGraftStorageGCMargin, margin)
require.NoError(t, s.validateGC(), "the defaults are consistent")
- require.False(t, s.MaintenanceAuto(), "the background maintenance is off by default")
+ require.False(t, s.MaintenanceAuto(), "the background maintenance is off with the storage core off")
interval, err := s.MaintenanceIntervalOrDefault()
require.NoError(t, err)
require.Equal(t, DefaultGraftStorageMaintenanceInterval, interval)
@@ -100,6 +102,19 @@ func TestGraftStorageDefaults(t *testing.T) {
sync, err = s.SyncTimeoutOrDefault()
require.NoError(t, err)
require.Equal(t, 250*time.Millisecond, sync)
+ require.False(t, s.MaintenanceAuto(), "no checkpoints outside authoritative mode")
+
+ // Authoritative mode checkpoints by default (checkpoints delete nothing);
+ // the garbage collection, which deletes, is opt-in.
+ s = GraftStorage{Mode: "authoritative"}
+ require.True(t, s.CheckpointsAuto())
+ require.False(t, s.GCAuto())
+ require.True(t, s.MaintenanceAuto())
+ s.Checkpoints = "off"
+ require.False(t, s.MaintenanceAuto())
+ s.GC = "auto"
+ require.True(t, s.GCAuto())
+ require.True(t, s.MaintenanceAuto())
}
// TestGraftStorageMaintenanceFromTOML checks that a threshold set to 0 in the
@@ -110,7 +125,7 @@ func TestGraftStorageMaintenanceFromTOML(t *testing.T) {
cfg, err := Load(strings.NewReader(`
[graft.storage]
mode = "authoritative"
-maintenance = "auto"
+gc = "auto"
maintenance_interval = "30m"
checkpoint_changesets = 0
checkpoint_pack_bytes = 1048576
@@ -119,7 +134,8 @@ lease_ttl = "3m"
require.NoError(t, err)
s := cfg.Graft.Storage
- require.True(t, s.MaintenanceAuto())
+ require.True(t, s.CheckpointsAuto())
+ require.True(t, s.GCAuto())
require.Equal(t, uint64(0), s.CheckpointChangesetsOrDefault())
require.Equal(t, int64(1<<20), s.CheckpointPackBytesOrDefault())
interval, err := s.MaintenanceIntervalOrDefault()
diff --git a/internal/gitaly/service/storagecore/import.go b/internal/gitaly/service/storagecore/import.go
index 491c9a0..14a0546 100644
--- a/internal/gitaly/service/storagecore/import.go
+++ b/internal/gitaly/service/storagecore/import.go
@@ -64,6 +64,15 @@ func (s *server) ImportRepository(ctx context.Context, req *gitalypb.ImportRepos
if err := s.validateRepository(ctx, repo); err != nil {
return nil, err
}
+ // An import makes the local copy the database's truth. One whose references
+ // may have lost writes is never imported (the gate does not see this RPC).
+ path, err := s.deps.GetLocator().GetRepoPath(ctx, repo)
+ if err != nil {
+ return nil, err
+ }
+ if core.RefDurability.Untrusted(path) {
+ return nil, structerr.NewUnavailable("storage core: the local references of the repository may have lost writes; it cannot be imported until it is rebuilt")
+ }
local := s.localRepo(repo)
backend, err := local.ReferenceBackend(ctx)
diff --git a/internal/gitaly/service/storagecore/maintenance_test.go b/internal/gitaly/service/storagecore/maintenance_test.go
index 08cdf18..40607dd 100644
--- a/internal/gitaly/service/storagecore/maintenance_test.go
+++ b/internal/gitaly/service/storagecore/maintenance_test.go
@@ -4,6 +4,9 @@ import (
"bytes"
"crypto/sha256"
"fmt"
+ "os"
+ "path/filepath"
+ "strings"
"testing"
"time"
@@ -12,6 +15,8 @@ import (
"google.golang.org/grpc/status"
"github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/coretest"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
"github.com/feder-cr/git-ai/internal/graft/corestore/sqlstore"
"github.com/feder-cr/git-ai/proto/go/gitalypb"
)
@@ -211,6 +216,38 @@ func TestStorageCoreRejectsReservedRelativePaths(t *testing.T) {
}
}
+// TestImportRefusesAnUntrustedRepository verifies that ImportRepository, which
+// the gate does not see, never makes a local copy whose references may have
+// lost writes the database's truth.
+func TestImportRefusesAnUntrustedRepository(t *testing.T) {
+ t.Parallel()
+
+ h := setup(t, corestore.ModeAuthoritative, sqlstore.SQLite)
+ ref := localrefs.RepoRef{Storage: h.repo.GetStorageName(), RelativePath: h.repo.GetRelativePath()}
+
+ // A marker that records a failed fsync, as a durable step of the previous
+ // process left it; this process recovers it.
+ coretest.EnableLocalRefs(t, h.core, h.cfg)
+ require.NoError(t, h.core.RefDurability.MarkDirty(h.ctx, ref, h.repoPath))
+ markers, err := filepath.Glob(filepath.Join(h.cfg.Storages[0].Path, localrefs.MarkerDirName, "*.json"))
+ require.NoError(t, err)
+ require.Len(t, markers, 1)
+ content, err := os.ReadFile(markers[0])
+ require.NoError(t, err)
+ content = []byte(strings.Replace(string(content), "}", `,"untrusted":"fsync: input/output error"}`, 1))
+ require.NoError(t, os.WriteFile(markers[0], content, 0o600))
+
+ coretest.EnableLocalRefs(t, h.core, h.cfg)
+ _, err = h.core.RefDurability.Recover(h.ctx, []string{h.repo.GetStorageName()})
+ require.NoError(t, err)
+ require.True(t, h.core.RefDurability.Untrusted(h.repoPath))
+
+ _, err = h.client.ImportRepository(h.ctx, &gitalypb.ImportRepositoryRequest{Repository: h.repo})
+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)
+ _, err = h.core.Meta.GetRepository(h.ctx, corestore.KeyOf(h.repo))
+ require.ErrorIs(t, err, corestore.ErrRepositoryNotFound, "nothing was imported")
+}
+
// packKeys returns the objects of the repository in the bucket.
func (h *harness) packKeys(repoID int64) map[string]bool {
h.t.Helper()
diff --git a/internal/graft/corestore/localrefs/durability.go b/internal/graft/corestore/localrefs/durability.go
index 2a81b48..37eb1b1 100644
--- a/internal/graft/corestore/localrefs/durability.go
+++ b/internal/graft/corestore/localrefs/durability.go
@@ -78,6 +78,14 @@ const (
crashExitCode = 75
)
+// staleGitFileAge is the age past which a git lock or temporary file of a
+// reftable stack belongs to a git process that died: Gitaly's housekeeping
+// removes such a lock (housekeeping.ReftableLockfileGracePeriod, the same
+// value), and no git process holds a stack's lock for that long. Such a file is
+// not git activity, and the durable step removes a stale lock of the list as
+// housekeeping would, so that it never keeps a marker forever.
+const staleGitFileAge = time.Hour
+
// stackLockTag starts the content of the reference lock the durable step
// takes, followed by a random nonce of that lock. git never writes it (a lock
// of git holds table names), so a lock with this tag found by a new owner was
@@ -645,17 +653,15 @@ func (d *Durability) Settle(path string, final bool) error {
due := st.nLive == 0 && (final || time.Since(st.lastWrite) >= d.idle)
d.mu.Unlock()
- exists := true
- if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {
- exists = false
- } else if err != nil {
- return fmt.Errorf("stat the repository: %w", err)
+ stack, err := hasStack(path)
+ if err != nil {
+ return err
}
- if !dirty || !exists {
- // A repository that is gone has no stack; a clean state only leaves the
- // map. A transaction that marked an existing repository meanwhile leaves
- // it to the next step, under git's reference lock (dropMarker refuses
- // otherwise).
+ if !dirty || !stack {
+ // A repository without a stack (gone, or not reftable) has nothing to
+ // make durable; a clean state only leaves the map. A transaction that
+ // marked a repository with a stack meanwhile leaves it to the next step,
+ // under git's reference lock (dropMarker refuses otherwise).
st := d.lockState(path)
defer st.file.Unlock()
_, err := d.dropMarker(path, st, false)
@@ -710,12 +716,13 @@ func (d *Durability) settleLocked(path string, repo RepoRef) error {
// - no git process of this process that writes the references without fsync
// is live (nLive);
// - the repository is not untrusted;
-// - when the repository exists, the caller holds git's reference lock
-// (ownLock) and made the stack durable under it (settleLocked): no git
-// process can write meanwhile, and the ones that exited before are in what
-// was made durable. A repository that does not exist has no stack, and a
-// git process cannot write it (the caller holds the syncer's repository
-// lock, so it is not hydrated meanwhile);
+// - when the repository has a reftable stack, the caller holds git's
+// reference lock (ownLock) and made the stack durable under it
+// (settleLocked): no git process can write meanwhile, and the ones that
+// exited before are in what was made durable. A repository without one
+// (gone, or not reftable: hasStack) has nothing written without fsync, and
+// none is created meanwhile (the caller holds the syncer's repository lock,
+// so it is not hydrated meanwhile);
// - no git process writes the stack now (gitActive: a git lock or a table
// being written; seeing one also pushes settleAfter a recovery grace
// forward), and settleAfter has passed.
@@ -727,18 +734,13 @@ func (d *Durability) settleLocked(path string, repo RepoRef) error {
// trusted leaves the map, so that the states do not grow with every repository
// ever written.
func (d *Durability) dropMarker(path string, st *refState, ownLock bool) (bool, error) {
- exists := true
- if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {
- exists = false
- } else if err != nil {
- return false, fmt.Errorf("stat the repository: %w", err)
+ stack, err := hasStack(path)
+ if err != nil {
+ return false, err
}
- active := false
- if exists {
- var err error
- if active, err = gitActive(path, ownLock); err != nil {
- return false, err
- }
+ active, err := gitActive(path, ownLock)
+ if err != nil {
+ return false, err
}
d.mu.Lock()
@@ -748,7 +750,7 @@ func (d *Durability) dropMarker(path string, st *refState, ownLock bool) (bool,
st.settleAfter = until
}
}
- safe := st.dirty && st.nLive == 0 && st.untrusted == nil && (ownLock || !exists) && !active && !now.Before(st.settleAfter)
+ safe := st.dirty && st.nLive == 0 && st.untrusted == nil && (ownLock || !stack) && !active && !now.Before(st.settleAfter)
if safe {
st.dirty = false
d.dirty.Dec()
@@ -818,7 +820,8 @@ func (d *Durability) stepErrorLocked(st *refState, repo RepoRef, path string, er
// the repository at path: git's lock of the list (unless ownLock: the caller
// holds it), a lock of a table being compacted, or a table being written is
// there (the names of git's reftable/stack.c). A repository without a reftable
-// directory has no stack to write.
+// directory has no stack to write. A file older than staleGitFileAge is left by
+// a git process that died, not activity.
func gitActive(path string, ownLock bool) (bool, error) {
entries, err := os.ReadDir(filepath.Join(path, "reftable"))
if errors.Is(err, fs.ErrNotExist) {
@@ -832,13 +835,40 @@ func gitActive(path string, ownLock bool) (bool, error) {
if ownLock && name == "tables.list.lock" {
continue
}
- if strings.HasSuffix(name, ".lock") || strings.Contains(name, ".temp.") {
+ if !strings.HasSuffix(name, ".lock") && !strings.Contains(name, ".temp.") {
+ continue
+ }
+ info, err := entry.Info()
+ if errors.Is(err, fs.ErrNotExist) {
+ // Gone since the listing: the git process finished with it.
+ continue
+ }
+ if err != nil {
+ return false, fmt.Errorf("stat %q: %w", name, err)
+ }
+ if time.Since(info.ModTime()) < staleGitFileAge {
return true, nil
}
}
return false, nil
}
+// hasStack reports whether the repository at path has a reftable stack: its
+// reftable directory. A repository that is gone, or one without the reftable
+// backend (a files-backend copy being imported), has none, so nothing of it is
+// written without fsync and nothing needs to be made durable. A reftable
+// directory whose table list cannot be read is a stack all the same (git
+// replaces the list by a rename, so it never lacks one): making it durable
+// fails, and the marker stays.
+func hasStack(path string) (bool, error) {
+ if _, err := os.Lstat(filepath.Join(path, "reftable")); errors.Is(err, fs.ErrNotExist) {
+ return false, nil
+ } else if err != nil {
+ return false, fmt.Errorf("stat the reftable directory: %w", err)
+ }
+ return true, nil
+}
+
// stackLock is git's reference lock of a repository, taken by the durable
// step.
type stackLock struct {
@@ -881,13 +911,45 @@ func lockStack(path string) (*stackLock, bool, error) {
return nil, false, fmt.Errorf("take the reference lock: %w", err)
}
- if err := os.Link(tmp.Name(), lockPath); err != nil {
- if errors.Is(err, fs.ErrExist) {
+ for attempt := 0; ; attempt++ {
+ err := os.Link(tmp.Name(), lockPath)
+ if err == nil {
+ return &stackLock{path: lockPath, content: content, info: info}, true, nil
+ }
+ if !errors.Is(err, fs.ErrExist) {
+ return nil, false, fmt.Errorf("take the reference lock: %w", err)
+ }
+ if attempt > 0 {
+ return nil, false, nil
+ }
+ stale, err := removeStaleGitLock(lockPath)
+ if err != nil {
+ return nil, false, err
+ }
+ if !stale {
return nil, false, nil
}
- return nil, false, fmt.Errorf("take the reference lock: %w", err)
}
- return &stackLock{path: lockPath, content: content, info: info}, true, nil
+}
+
+// removeStaleGitLock removes the lock of a stack's table list when it is older
+// than staleGitFileAge: a git process that died left it, and Gitaly's
+// housekeeping would remove it by the same rule. It reports whether it did.
+func removeStaleGitLock(lockPath string) (bool, error) {
+ info, err := os.Lstat(lockPath)
+ if errors.Is(err, fs.ErrNotExist) {
+ return true, nil
+ }
+ if err != nil {
+ return false, fmt.Errorf("stat the reference lock: %w", err)
+ }
+ if time.Since(info.ModTime()) < staleGitFileAge {
+ return false, nil
+ }
+ if err := os.Remove(lockPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return false, fmt.Errorf("remove a stale reference lock: %w", err)
+ }
+ return true, nil
}
// release removes the lock if it is still the one the step created (same file,
@@ -1089,10 +1151,12 @@ func (d *Durability) Forget(ctx context.Context, repo RepoRef, path string) erro
// durable step does; when the lock is busy the marker stays for the next step.
// The caller holds the state's file lock.
func (d *Durability) forgetLocked(path string, st *refState, repo RepoRef) (bool, error) {
- if _, err := os.Lstat(path); errors.Is(err, fs.ErrNotExist) {
+ stack, err := hasStack(path)
+ if err != nil {
+ return false, err
+ }
+ if !stack {
return d.dropMarker(path, st, false)
- } else if err != nil {
- return false, fmt.Errorf("stat the repository: %w", err)
}
lock, ok, err := lockStack(path)
diff --git a/internal/graft/corestore/localrefs/durability_model_test.go b/internal/graft/corestore/localrefs/durability_model_test.go
index 85c732d..9d78070 100644
--- a/internal/graft/corestore/localrefs/durability_model_test.go
+++ b/internal/graft/corestore/localrefs/durability_model_test.go
@@ -12,6 +12,10 @@ import (
"time"
"github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/git/housekeeping"
)
// TestMarkerLifecycleModel drives random interleavings of everything that
@@ -167,6 +171,72 @@ func TestGitActivityPushesTheGraceForward(t *testing.T) {
next.mu.Unlock()
}
+// TestStaleGitLockDoesNotPinTheMarker verifies git's lock files left by a git
+// process that died: a lock of the table list or a temporary table older than
+// housekeeping's grace is not activity, and the durable step removes such a
+// list lock as housekeeping would, takes its own and settles. A younger one
+// keeps the marker.
+func TestStaleGitLockDoesNotPinTheMarker(t *testing.T) {
+ t.Parallel()
+ require.Equal(t, housekeeping.ReftableLockfileGracePeriod, staleGitFileAge, "the rule of Gitaly's housekeeping")
+
+ f := setupDurability(t)
+ d := f.durability(t, "boot-a", 0)
+ require.NoError(t, d.MarkDirty(f.ctx, f.repo, f.path))
+
+ lock := filepath.Join(f.path, "reftable", "tables.list.lock")
+ temp := filepath.Join(f.path, "reftable", "0x000000000009-0x000000000009-abcdef01.ref.temp.XyZ123")
+ for _, file := range []string{lock, temp} {
+ require.NoError(t, os.WriteFile(file, nil, 0o666))
+ }
+ require.NoError(t, d.Settle(f.path, true))
+ require.FileExists(t, f.markerPath(t, d), "a git process may hold them")
+
+ old := time.Now().Add(-staleGitFileAge - time.Minute)
+ for _, file := range []string{lock, temp} {
+ require.NoError(t, os.Chtimes(file, old, old))
+ }
+ require.NoError(t, d.Settle(f.path, true))
+ require.NoFileExists(t, f.markerPath(t, d))
+ require.NoFileExists(t, lock, "removed as housekeeping would, and the step's own released")
+}
+
+// TestRepositoryWithoutAStackHasNothingToMakeDurable verifies a repository
+// that is not reftable (a files-backend copy being imported at the path): it
+// has no stack, so its marker is settled and handed over without one, and the
+// recovery of a kernel restart does not fail on it.
+func TestRepositoryWithoutAStackHasNothingToMakeDurable(t *testing.T) {
+ t.Parallel()
+ f := setupDurability(t)
+ ctx := f.ctx
+ pb, path := gittest.CreateRepository(t, ctx, f.cfg, gittest.CreateRepositoryConfig{
+ SkipCreationViaService: true,
+ ReferenceBackend: git.ReferenceBackendFiles,
+ })
+ repo := RepoRef{Storage: pb.GetStorageName(), RelativePath: pb.GetRelativePath()}
+ markerPath := func(d *Durability) string {
+ p, err := d.markerPath(ctx, repo, path)
+ require.NoError(t, err)
+ return p
+ }
+
+ d := f.durability(t, "boot-a", 0)
+ require.NoError(t, d.MarkDirty(ctx, repo, path))
+ require.NoError(t, d.Settle(path, false))
+ require.NoFileExists(t, markerPath(d))
+
+ require.NoError(t, d.MarkDirty(ctx, repo, path))
+ require.NoError(t, d.Close())
+ next := f.durability(t, "boot-b", 0)
+ found, err := next.Recover(ctx, []string{repo.Storage})
+ require.NoError(t, err)
+ require.Len(t, found, 1)
+ require.True(t, found[0].Reboot)
+ require.NoError(t, next.Forget(ctx, repo, path), "kept in place: not a cache of the storage core")
+ require.NoFileExists(t, markerPath(next))
+ require.Empty(t, next.Tracked())
+}
+
// modelProc is a git process of the model.
type modelProc struct {
end func()
diff --git a/internal/graft/corestore/maintenance/maintenance.go b/internal/graft/corestore/maintenance/maintenance.go
index 3f79212..6fc8ef5 100644
--- a/internal/graft/corestore/maintenance/maintenance.go
+++ b/internal/graft/corestore/maintenance/maintenance.go
@@ -1,15 +1,17 @@
// Package maintenance runs the background checkpoints and garbage collection
-// of the storage core (doc/graft/PLAN7.md, "Background maintenance"). It is
-// off by default and runs in authoritative mode only; the graft storage
-// commands do the same work on demand.
+// of the storage core (doc/graft/PLAN7.md, "Background maintenance"), in
+// authoritative mode only. The checkpoints run by default, since they delete
+// nothing and keep the hydrations short; the garbage collection, which deletes,
+// is opt-in. The graft storage commands do the same work on demand.
//
-// Every interval it collects the removed repositories of the node's storages,
-// then, for every repository of those storages that this node has a current
-// local copy of, it takes the repository's maintenance lease (skipping a
-// repository whose lease another process holds), checkpoints the repository
-// when enough changesets or pack bytes follow its newest checkpoint, and
-// collects its dead packs. The lease only keeps two nodes from doing the same
-// work at once: the checkpoint and the collector are correct without it.
+// Every interval it collects the removed repositories of the node's storages
+// (with Collect), then, for every repository of those storages that this node
+// has a current local copy of, it takes the repository's maintenance lease
+// (skipping a repository whose lease another process holds), checkpoints the
+// repository when enough changesets or pack bytes follow its newest checkpoint
+// (with Checkpoints), and collects its dead packs (with Collect). The lease
+// only keeps two nodes from doing the same work at once: the checkpoint and the
+// collector are correct without it.
package maintenance
import (
@@ -49,6 +51,13 @@ const (
type Config struct {
// Interval is the time between two runs.
Interval time.Duration
+ // Checkpoints makes the runs checkpoint the repositories (by the thresholds
+ // below). Checkpoints delete nothing.
+ Checkpoints bool
+ // Collect makes the runs collect garbage: the dead packs of the
+ // repositories and the objects of the removed ones. It deletes data, so it
+ // is configured on its own.
+ Collect bool
// CheckpointChangesets checkpoints a repository once this many changesets
// follow its newest checkpoint; 0 never checkpoints by count.
CheckpointChangesets uint64
@@ -186,6 +195,9 @@ func (r *Runner) RunOnce(ctx context.Context) error {
}
for _, storageName := range r.storages {
+ if !r.cfg.Collect {
+ break
+ }
report, err := r.collector.CollectRemoved(ctx, storageName, false)
if err != nil {
fail(stepCollectRemoved, err, log.Fields{"storage": storageName})
@@ -264,6 +276,9 @@ func (r *Runner) maintain(ctx context.Context, record corestore.Repository) (str
}
}
+ if !r.cfg.Collect {
+ return "", nil
+ }
report, err := r.collector.CollectRepository(leaseCtx, record.ID, false)
if errors.Is(err, corestore.ErrRepositoryNotFound) {
return "", nil
@@ -323,7 +338,7 @@ func HoldLease(ctx context.Context, core *corestore.Core, logger log.Logger, rep
// checkpointDue reports whether the repository needs a checkpoint: enough
// changesets, or enough pack bytes, follow its newest checkpoint.
func (r *Runner) checkpointDue(ctx context.Context, record corestore.Repository) (bool, error) {
- if r.cfg.CheckpointChangesets == 0 && r.cfg.CheckpointPackBytes == 0 {
+ if !r.cfg.Checkpoints || (r.cfg.CheckpointChangesets == 0 && r.cfg.CheckpointPackBytes == 0) {
return false, nil
}
diff --git a/internal/graft/corestore/maintenance/maintenance_test.go b/internal/graft/corestore/maintenance/maintenance_test.go
index 5f4b418..3f8dba9 100644
--- a/internal/graft/corestore/maintenance/maintenance_test.go
+++ b/internal/graft/corestore/maintenance/maintenance_test.go
@@ -169,8 +169,53 @@ func (e *env) metric(name, label string) float64 {
return 0
}
+// config3 is a configuration with the checkpoints and the garbage collection
+// on, and the given checkpoint thresholds.
func config3(changesets uint64, packBytes int64) maintenance.Config {
- return maintenance.Config{Interval: time.Hour, CheckpointChangesets: changesets, CheckpointPackBytes: packBytes, LeaseTTL: time.Minute}
+ return maintenance.Config{Interval: time.Hour, Checkpoints: true, Collect: true, CheckpointChangesets: changesets, CheckpointPackBytes: packBytes, LeaseTTL: time.Minute}
+}
+
+// TestRunOnceDoesOnlyWhatIsOn verifies that the checkpoints and the garbage
+// collection are configured apart: checkpoints alone delete nothing, and the
+// collection alone creates no checkpoint.
+func TestRunOnceDoesOnlyWhatIsOn(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ orphan := func(e *env) string {
+ content := []byte("an orphan pack")
+ key := corestore.NewPackKey(e.repo.ID, sha256.Sum256(content))
+ require.NoError(t, e.core.Blobs.Put(ctx, key, bytes.NewReader(content)))
+ return key
+ }
+ opts := coretest.Options{GCGrace: 200 * time.Millisecond, GCMargin: 50 * time.Millisecond}
+
+ checkpointsOnly := config3(1, 0)
+ checkpointsOnly.Collect = false
+ e := newEnv(t, opts, checkpointsOnly)
+ e.fresh()
+ key := orphan(e)
+ require.Never(t, func() bool {
+ require.NoError(t, e.runner.RunOnce(ctx))
+ exists, err := e.core.Blobs.Exists(ctx, key)
+ require.NoError(t, err)
+ return !exists
+ }, time.Second, 100*time.Millisecond, "checkpoints alone delete nothing")
+ _, ok := e.newestCheckpoint()
+ require.True(t, ok)
+
+ collectOnly := config3(1, 0)
+ collectOnly.Checkpoints = false
+ e = newEnv(t, opts, collectOnly)
+ e.fresh()
+ key = orphan(e)
+ require.Eventually(t, func() bool {
+ require.NoError(t, e.runner.RunOnce(ctx))
+ exists, err := e.core.Blobs.Exists(ctx, key)
+ require.NoError(t, err)
+ return !exists
+ }, 30*time.Second, 100*time.Millisecond)
+ _, ok = e.newestCheckpoint()
+ require.False(t, ok, "the collection alone creates no checkpoint")
}
func TestRunOnceCheckpointsAfterEnoughChangesets(t *testing.T) {
@@ -338,7 +383,13 @@ func TestHoldLeaseRenewsUntilReleased(t *testing.T) {
t.Parallel()
ctx := testhelper.Context(t)
e := newEnv(t, coretest.Options{}, config3(0, 0))
- const ttl = 300 * time.Millisecond
+ // The renewal runs every third of the time to live, so a renewal delayed by
+ // up to two thirds of it (2 s) keeps the lease. Every renewal and every
+ // probe below is a database write transaction (SQLite: the write admission
+ // and an fsync), whose latency on a loaded host is what a shorter time to
+ // live would measure instead of the renewal. The probes are spaced for the
+ // same reason.
+ const ttl = 3 * time.Second
const holder = "node-a/1/request-1"
acquired, err := e.core.Meta.AcquireMaintenanceLease(ctx, e.repo.ID, holder, ttl)
@@ -351,7 +402,7 @@ func TestHoldLeaseRenewsUntilReleased(t *testing.T) {
acquired, err := e.core.Meta.AcquireMaintenanceLease(ctx, e.repo.ID, "node-b/1", ttl)
require.NoError(t, err)
return acquired
- }, 5*ttl, ttl/10, "the lease is renewed")
+ }, 5*ttl/2, ttl/6, "the lease is renewed")
require.NoError(t, leaseCtx.Err())
require.False(t, lost.Load())
diff --git a/internal/graft/corestore/syncer/export_test.go b/internal/graft/corestore/syncer/export_test.go
index a6f05bb..7203704 100644
--- a/internal/graft/corestore/syncer/export_test.go
+++ b/internal/graft/corestore/syncer/export_test.go
@@ -8,6 +8,12 @@ func (s *Syncer) SetTestHook(hook func(point string)) {
s.testHook = hook
}
+// SetRoundLimit ends every sync round after n changesets, so that tests get
+// partial rounds deterministically. Set it before the first EnsureFresh.
+func (s *Syncer) SetRoundLimit(n int) {
+ s.roundLimit = n
+}
+
// DurableStep runs one durable step of the repository at path, as the
// durable-step loop does every interval (final: as the stop does).
func (s *Syncer) DurableStep(ctx context.Context, path string, final bool) {
diff --git a/internal/graft/corestore/syncer/syncer.go b/internal/graft/corestore/syncer/syncer.go
index 13936fc..4486e67 100644
--- a/internal/graft/corestore/syncer/syncer.go
+++ b/internal/graft/corestore/syncer/syncer.go
@@ -57,6 +57,7 @@ import (
"time"
"github.com/prometheus/client_golang/prometheus"
+ "golang.org/x/sync/errgroup"
"golang.org/x/sync/singleflight"
"github.com/feder-cr/git-ai/internal/git"
@@ -77,14 +78,25 @@ const (
defaultSyncTimeout = 10 * time.Second
// changesetBatch is how many changesets are listed from the database at once.
changesetBatch = 500
+ // fetchConcurrency bounds the packs a fetch downloads, verifies and indexes
+ // at once. Each is a bucket read, a SHA-256 and a git index-pack process:
+ // done one at a time, their latencies add up per pack.
+ fetchConcurrency = 8
+ // roundReserve is the share of the sync budget a round keeps for its local
+ // work after the fetch (the references, the watermark, the install): the
+ // fetch stops starting packs when only that much of the budget is left.
+ roundReserve = 4
// reportInterval is the minimum time between two ReportApplied calls for one repository.
reportInterval = time.Second
initialBackoff = 10 * time.Millisecond
maxBackoff = 500 * time.Millisecond
- resultFresh = "fresh"
- resultCaughtUp = "caught_up"
- resultHydrated = "hydrated"
+ resultFresh = "fresh"
+ resultCaughtUp = "caught_up"
+ resultHydrated = "hydrated"
+ // resultPartial is a round that ran out of its budget and left the local
+ // repository at a changeset before the head: the next round goes on from it.
+ resultPartial = "partial"
resultError = "error"
tmpPackPattern = "graft-sync-pack-*"
tmpHydratePatt = "hydrate-*"
@@ -113,6 +125,9 @@ type Syncer struct {
// ("fetched": the range is downloaded, nothing checked yet; "checked": the
// local references are checked, the apply follows). Nil in production.
testHook func(point string)
+ // roundLimit, when set by a test, ends every sync round after that many
+ // changesets (round.limit). Zero in production.
+ roundLimit int
// mu protects the maps below. It is never held while waiting for a
// repository lock.
@@ -167,7 +182,7 @@ func New(core *corestore.Core, logger log.Logger, locator storage.Locator, repoF
reported: map[int64]time.Time{},
syncTotal: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "graft_storage_sync_total",
- Help: "Number of storage core syncs before a request, by result (fresh, caught_up, hydrated, error).",
+ Help: "Number of storage core syncs before a request, by result (fresh, caught_up, hydrated, partial: the round's budget ran out before the head and the next round goes on, error).",
}, []string{"result"}),
syncSeconds: prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "graft_storage_sync_seconds",
@@ -187,7 +202,7 @@ func New(core *corestore.Core, logger log.Logger, locator storage.Locator, repoF
Help: "Number of hydrations that started from a checkpoint instead of changeset 1.",
}),
}
- for _, result := range []string{resultFresh, resultCaughtUp, resultHydrated, resultError} {
+ for _, result := range []string{resultFresh, resultCaughtUp, resultHydrated, resultPartial, resultError} {
s.syncTotal.WithLabelValues(result)
}
return s
@@ -352,6 +367,7 @@ func (s *Syncer) sync(ctx context.Context, key corestore.RepoKey) error {
return err
}
+ rd := s.newRound(ctx)
for {
if err := ctx.Err(); err != nil {
return structerr.NewUnavailable("storage core: sync: %w", err)
@@ -405,33 +421,49 @@ func (s *Syncer) sync(ctx context.Context, key corestore.RepoKey) error {
if err := s.checkReplaceable(ctx, r, path, wm, ok); err != nil {
return err
}
- err := s.hydrate(ctx, repo, r, hash, path, boundary)
+ reached, err := s.hydrate(ctx, repo, r, hash, path, boundary, rd)
if errors.Is(err, errRestart) {
continue
}
if err != nil {
return err
}
+ if reached < r.HeadSeq {
+ result = resultPartial
+ return partialError(key, reached, r.HeadSeq)
+ }
result = resultHydrated
return nil
}
- fresh, err := s.catchUp(ctx, repo, r, hash, path, wm.Seq)
+ fresh, reached, err := s.catchUp(ctx, repo, r, hash, path, wm.Seq, rd)
if errors.Is(err, errRestart) {
continue
}
if err != nil {
return err
}
- if fresh {
+ switch {
+ case reached < r.HeadSeq:
+ result = resultPartial
+ return partialError(key, reached, r.HeadSeq)
+ case fresh:
result = resultFresh
- } else {
+ default:
result = resultCaughtUp
}
return nil
}
}
+// partialError is the answer of a round that left the local repository behind
+// the head (doc/graft/PLAN7.md, "Resumable rounds"): the repository is not
+// served as current, and the next request's round goes on from reached.
+func partialError(key corestore.RepoKey, reached, head uint64) error {
+ return structerr.NewUnavailable("storage core: %s is at changeset %d of %d after one sync round; the next request goes on from there", key, reached, head).
+ WithMetadata("reached", reached).WithMetadata("head", head)
+}
+
// syncBoundary returns C_sync of the repository: the sequence of the newest
// checkpoint created at least gc_grace − sync_timeout − gc_margin ago on the
// database clock, 0 when there is none (doc/graft/PLAN7.md, GC BOUNDARY rule).
@@ -885,16 +917,22 @@ func (f *fetched) names() []string {
// local write committed in between fails the transaction rather than being
// reverted, which matters because its NoteCommitted sequence would otherwise
// move the watermark over a reference that no longer has its value.
-func (s *Syncer) catchUp(ctx context.Context, repo *gitalypb.Repository, r corestore.Repository, hash git.ObjectHash, path string, after uint64) (fresh bool, _ error) {
+//
+// The first fetch is bounded by the round (rd): when the round is over before
+// the head, the range fetched so far is applied, and the next round goes on
+// from it. The extensions of the range (a newer local write, an unexplained
+// local value) are not bounded: they are what makes the range explain the
+// local repository. It returns the changeset the watermark reached.
+func (s *Syncer) catchUp(ctx context.Context, repo *gitalypb.Repository, r corestore.Repository, hash git.ObjectHash, path string, after uint64, rd *round) (fresh bool, reached uint64, _ error) {
tmpDir, err := s.tmpDir(repo.GetStorageName())
if err != nil {
- return false, err
+ return false, 0, err
}
repoObj := s.repoFactory.Build(repo)
f := newFetched(after)
- if err := s.fetch(ctx, repoObj, tmpDir, r.ID, hash, f, r.HeadSeq); err != nil {
- return false, err
+ if err := s.fetch(ctx, repoObj, tmpDir, r.ID, hash, f, r.HeadSeq, rd); err != nil {
+ return false, 0, err
}
s.hook("fetched")
@@ -907,10 +945,10 @@ func (s *Syncer) catchUp(ctx context.Context, repo *gitalypb.Repository, r cores
for {
chk, err := s.checkLocked(ctx, repoObj, path, r, f, repair)
if err != nil {
- return false, err
+ return false, 0, err
}
if chk.done {
- return s.caughtUp(ctx, r, f, false), nil
+ return s.caughtUp(ctx, r, f, false), chk.seq, nil
}
if chk.mismatch == "" {
@@ -921,11 +959,11 @@ func (s *Syncer) catchUp(ctx context.Context, repo *gitalypb.Repository, r cores
// changesets are pulled into the range first.
upTo, err := s.newerLocalWrites(ctx, r.ID, f.head, chk.moves)
if err != nil {
- return false, err
+ return false, 0, err
}
if upTo > f.head {
- if err := s.fetch(ctx, repoObj, tmpDir, r.ID, hash, f, upTo); err != nil {
- return false, err
+ if err := s.fetch(ctx, repoObj, tmpDir, r.ID, hash, f, upTo, nil); err != nil {
+ return false, 0, err
}
r.HeadSeq = max(r.HeadSeq, upTo)
continue
@@ -935,11 +973,11 @@ func (s *Syncer) catchUp(ctx context.Context, repo *gitalypb.Repository, r cores
out, err := s.applyLocked(ctx, repoObj, hash, path, r, f, chk.moves)
if err != nil {
- return false, err
+ return false, 0, err
}
switch {
case out.done:
- return s.caughtUp(ctx, r, f, out.applied), nil
+ return s.caughtUp(ctx, r, f, out.applied), out.seq, nil
case out.changed:
// A local write changed a reference since the check: the
// transaction failed on its expected old value. Check again.
@@ -952,7 +990,7 @@ func (s *Syncer) catchUp(ctx context.Context, repo *gitalypb.Repository, r cores
select {
case <-ctx.Done():
timer.Stop()
- return false, structerr.NewUnavailable("storage core: reference locked by a concurrent writer: %w", out.lockErr)
+ return false, 0, structerr.NewUnavailable("storage core: reference locked by a concurrent writer: %w", out.lockErr)
case <-timer.C:
}
backoff = min(backoff*2, maxBackoff)
@@ -966,16 +1004,16 @@ func (s *Syncer) catchUp(ctx context.Context, repo *gitalypb.Repository, r cores
latest, err := s.core.Meta.GetRepository(ctx, r.Key)
if err != nil {
if errors.Is(err, corestore.ErrRepositoryNotFound) {
- return false, structerr.NewNotFound("repository not found")
+ return false, 0, structerr.NewNotFound("repository not found")
}
- return false, structerr.NewUnavailable("storage core: get repository: %w", err)
+ return false, 0, structerr.NewUnavailable("storage core: get repository: %w", err)
}
if latest.ID != r.ID {
- return false, errRestart
+ return false, 0, errRestart
}
if latest.HeadSeq > f.head {
- if err := s.fetch(ctx, repoObj, tmpDir, r.ID, hash, f, latest.HeadSeq); err != nil {
- return false, err
+ if err := s.fetch(ctx, repoObj, tmpDir, r.ID, hash, f, latest.HeadSeq, nil); err != nil {
+ return false, 0, err
}
r.HeadSeq = latest.HeadSeq
continue
@@ -1020,8 +1058,9 @@ func (m move) verify() bool {
// checkOutcome is the result of one checkLocked attempt.
type checkOutcome struct {
// done reports that the watermark is at the head of the range already (the
- // node committed the changesets itself meanwhile).
+ // node committed the changesets itself meanwhile), at seq.
done bool
+ seq uint64
// mismatch names a local reference whose value the range does not explain.
mismatch string
// moves are the references of the range, from the local values read (see
@@ -1060,7 +1099,7 @@ func (s *Syncer) checkLocked(ctx context.Context, repo *localrepo.Repo, path str
}
wm = s.advanceLocked(path, wm)
if wm.Seq >= f.head {
- return checkOutcome{done: true}, nil
+ return checkOutcome{done: true, seq: wm.Seq}, nil
}
local, err := gitops.ReadRefValues(ctx, repo, f.names())
@@ -1139,10 +1178,11 @@ func (s *Syncer) newerLocalWrites(ctx context.Context, repoID int64, after uint6
// applyOutcome is the result of one applyLocked attempt.
type applyOutcome struct {
- // done reports that the local repository is at the head of the range;
- // applied that this attempt moved the watermark there (false: it was there
- // already).
+ // done reports that the local repository is at the head of the range, with
+ // its watermark at seq; applied that this attempt moved the watermark there
+ // (false: it was there already).
done, applied bool
+ seq uint64
// changed reports that a local reference no longer has the value the check
// read: the check must run again.
changed bool
@@ -1172,7 +1212,7 @@ func (s *Syncer) applyLocked(ctx context.Context, repo *localrepo.Repo, hash git
}
wm = s.advanceLocked(path, wm)
if wm.Seq >= f.head {
- return applyOutcome{done: true}, nil
+ return applyOutcome{done: true, seq: wm.Seq}, nil
}
w := refWriter{core: s.core, repo: localrefs.RepoRef{Storage: r.Key.Storage, RelativePath: r.Key.RelativePath}, path: path}
@@ -1192,23 +1232,23 @@ func (s *Syncer) applyLocked(ctx context.Context, repo *localrepo.Repo, hash git
return applyOutcome{}, structerr.NewInternal("storage core: apply references: %w", err)
}
- if err := s.commitWatermarkLocked(path, r.ID, f.head); err != nil {
+ wm, err = s.commitWatermarkLocked(path, r.ID, f.head)
+ if err != nil {
return applyOutcome{}, err
}
- return applyOutcome{done: true, applied: true}, nil
+ return applyOutcome{done: true, applied: true, seq: wm.Seq}, nil
}
// commitWatermarkLocked writes the watermark (repoID, seq) and advances it
// through the sequences this node committed after seq, contiguous with it:
-// their effects are in the local repository already. The caller holds the
-// repository lock.
-func (s *Syncer) commitWatermarkLocked(path string, repoID int64, seq uint64) error {
+// their effects are in the local repository already. It returns the watermark
+// reached. The caller holds the repository lock.
+func (s *Syncer) commitWatermarkLocked(path string, repoID int64, seq uint64) (corestore.Watermark, error) {
wm := corestore.Watermark{RepoID: repoID, Seq: seq}
if err := gitops.WriteWatermark(path, wm); err != nil {
- return structerr.NewInternal("storage core: write watermark: %w", err)
+ return corestore.Watermark{}, structerr.NewInternal("storage core: write watermark: %w", err)
}
- s.advanceLocked(path, wm)
- return nil
+ return s.advanceLocked(path, wm), nil
}
// movedSince reports whether a reference of moves no longer has its old value.
@@ -1454,41 +1494,51 @@ func refTransaction(ctx context.Context, repo *localrepo.Repo, w refWriter, comm
// checkpoint"). A local repository at path (another repository ID, invalid, or
// below boundary, C_sync) is moved away first and removed in the background.
// Only the final moves are done under the repository lock.
-func (s *Syncer) hydrate(ctx context.Context, repo *gitalypb.Repository, r corestore.Repository, hash git.ObjectHash, path string, boundary uint64) error {
+//
+// When the round is over before the head (rd), the build is installed at the
+// changeset it reached, with that watermark: a valid local repository behind
+// the head, like that of a node that was offline, which the next round catches
+// up. It is at or above boundary, since the seed is the newest checkpoint. It
+// returns the changeset reached. A round that reached nothing installs nothing
+// and fails as a round out of budget.
+func (s *Syncer) hydrate(ctx context.Context, repo *gitalypb.Repository, r corestore.Repository, hash git.ObjectHash, path string, boundary uint64, rd *round) (uint64, error) {
storageName := repo.GetStorageName()
storagePath, err := s.locator.GetStorageByName(ctx, storageName)
if err != nil {
- return err
+ return 0, err
}
tmpDir, err := s.tmpDir(storageName)
if err != nil {
- return err
+ return 0, err
}
build, err := os.MkdirTemp(tmpDir, tmpHydratePatt)
if err != nil {
- return structerr.NewInternal("storage core: create hydration directory: %w", err)
+ return 0, structerr.NewInternal("storage core: create hydration directory: %w", err)
}
defer func() { _ = os.RemoveAll(build) }()
buildPath := filepath.Join(build, "repo.git")
if err := gitops.InitBare(ctx, s.gitCmdFactory, buildPath, r.ObjectFormat); err != nil {
- return structerr.NewInternal("storage core: init repository: %w", err)
+ return 0, structerr.NewInternal("storage core: init repository: %w", err)
}
rel, err := filepath.Rel(storagePath, buildPath)
if err != nil {
- return structerr.NewInternal("storage core: relative hydration path: %w", err)
+ return 0, structerr.NewInternal("storage core: relative hydration path: %w", err)
}
buildRepo := s.repoFactory.Build(&gitalypb.Repository{StorageName: storageName, RelativePath: rel})
refs, f, err := s.seed(ctx, buildRepo, tmpDir, r, hash)
if err != nil {
- return err
+ return 0, err
}
- if err := s.fetch(ctx, buildRepo, tmpDir, r.ID, hash, f, r.HeadSeq); err != nil {
- return err
+ if err := s.fetch(ctx, buildRepo, tmpDir, r.ID, hash, f, r.HeadSeq, rd); err != nil {
+ return 0, err
+ }
+ if f.head == 0 {
+ return 0, structerr.NewUnavailable("storage core: the hydration of %s fetched no changeset within %s", r.Key, s.syncTimeout())
}
for name, target := range f.finalRefs() {
if target.Absent() {
@@ -1498,20 +1548,20 @@ func (s *Syncer) hydrate(ctx context.Context, repo *gitalypb.Repository, r cores
}
}
if err := s.applyBuild(ctx, buildRepo, refs); err != nil {
- return err
+ return 0, err
}
if err := gitops.WriteWatermark(buildPath, corestore.Watermark{RepoID: r.ID, Seq: f.head}); err != nil {
- return structerr.NewInternal("storage core: write watermark: %w", err)
+ return 0, structerr.NewInternal("storage core: write watermark: %w", err)
}
if err := s.install(ctx, repo, r, path, buildPath, tmpDir, boundary); err != nil {
- return err
+ return 0, err
}
s.syncChangesets.Add(float64(f.changesets))
s.syncBytes.Add(float64(f.bytes))
s.report(ctx, r, f.head)
- return nil
+ return f.head, nil
}
// seed starts a hydration: from the newest checkpoint when there is one (its
@@ -1631,8 +1681,16 @@ func (s *Syncer) tmpDir(storageName string) (string, error) {
// what they did to every reference. Nothing is applied to the references
// here. Each batch is validated before its packs are downloaded. The
// repository lock is not held.
-func (s *Syncer) fetch(ctx context.Context, repo *localrepo.Repo, tmpDir string, repoID int64, hash git.ObjectHash, f *fetched, upTo uint64) error {
+//
+// With a round (rd), the fetch stops starting packs once the round is over
+// (round.over) and leaves f at the last changeset whose pack and those of every
+// changeset before it are indexed: f.head is then below upTo, and the caller
+// applies what was fetched. Without one (nil) it fetches through upTo.
+func (s *Syncer) fetch(ctx context.Context, repo *localrepo.Repo, tmpDir string, repoID int64, hash git.ObjectHash, f *fetched, upTo uint64, rd *round) error {
for f.head < upTo {
+ if rd.over() {
+ return nil
+ }
cur := f.head
end := cur + changesetBatch
if end > upTo || end < cur {
@@ -1653,14 +1711,12 @@ func (s *Syncer) fetch(ctx context.Context, repo *localrepo.Repo, tmpDir string,
}
}
- for _, cs := range changesets {
- if cs.PackKey != "" {
- n, err := s.indexPack(ctx, repo, tmpDir, cs.PackKey)
- if err != nil {
- return err
- }
- f.bytes += n
- }
+ indexed, n, err := s.indexPacks(ctx, repo, tmpDir, changesets, rd)
+ if err != nil {
+ return err
+ }
+ f.bytes += n
+ for _, cs := range changesets[:indexed] {
for _, u := range cs.Updates {
h := f.refs[u.Name]
if h == nil {
@@ -1672,13 +1728,95 @@ func (s *Syncer) fetch(ctx context.Context, repo *localrepo.Repo, tmpDir string,
}
}
- f.changesets += len(changesets)
- f.head = end
+ f.changesets += indexed
+ f.head = cur + uint64(indexed)
+ if indexed < len(changesets) {
+ return nil
+ }
}
return nil
}
+// indexPacks downloads, verifies and indexes the packs of changesets, at most
+// fetchConcurrency at once: the packs are self-contained (index-pack without
+// --fix-thin), so their order does not matter, and git writes each one under
+// its own name. It starts them in order and stops starting them once the round
+// is over: it returns how many leading changesets have their packs indexed,
+// and the bytes of those packs. The first failure cancels the others and is
+// returned.
+func (s *Syncer) indexPacks(ctx context.Context, repo *localrepo.Repo, tmpDir string, changesets []corestore.Changeset, rd *round) (int, int64, error) {
+ g, gctx := errgroup.WithContext(ctx)
+ g.SetLimit(fetchConcurrency)
+ sizes := make([]int64, len(changesets))
+ started := len(changesets)
+ for i, cs := range changesets {
+ if rd.over() {
+ started = i
+ break
+ }
+ rd.count()
+ if cs.PackKey == "" {
+ continue
+ }
+ g.Go(func() error {
+ n, err := s.indexPack(gctx, repo, tmpDir, cs.PackKey)
+ sizes[i] = n
+ return err
+ })
+ }
+ if err := g.Wait(); err != nil {
+ return 0, 0, err
+ }
+
+ var total int64
+ for _, n := range sizes[:started] {
+ total += n
+ }
+ return started, total, nil
+}
+
+// round bounds the fetches of one sync round (doc/graft/PLAN7.md, "Resumable
+// rounds"): a round stops starting packs when only a roundReserve share of the
+// sync budget is left, so that it can apply what it fetched within the budget,
+// and the next round goes on from there. A nil round never ends.
+type round struct {
+ // stopBy is when the round stops starting packs; zero when the context has
+ // no deadline.
+ stopBy time.Time
+ // limit, when positive, ends the round after that many changesets (tests,
+ // through export_test.go: a deterministic round).
+ limit int
+ fetched int
+}
+
+// newRound returns the round of a sync running under ctx.
+func (s *Syncer) newRound(ctx context.Context) *round {
+ rd := &round{limit: s.roundLimit}
+ if deadline, ok := ctx.Deadline(); ok {
+ rd.stopBy = deadline.Add(-s.syncTimeout() / roundReserve)
+ }
+ return rd
+}
+
+// over reports whether the round must stop starting packs.
+func (rd *round) over() bool {
+ if rd == nil {
+ return false
+ }
+ if rd.limit > 0 && rd.fetched >= rd.limit {
+ return true
+ }
+ return !rd.stopBy.IsZero() && !time.Now().Before(rd.stopBy)
+}
+
+// count records one more changeset fetched in the round.
+func (rd *round) count() {
+ if rd != nil {
+ rd.fetched++
+ }
+}
+
// validateChangeset checks a changeset read from the database (RULE 10): its
// sequence, its pack key and every reference name and new target.
func validateChangeset(cs corestore.Changeset, repoID int64, wantSeq uint64, hash git.ObjectHash) error {
diff --git a/internal/graft/corestore/syncer/syncer_rounds_test.go b/internal/graft/corestore/syncer/syncer_rounds_test.go
new file mode 100644
index 0000000..d763e0f
--- /dev/null
+++ b/internal/graft/corestore/syncer/syncer_rounds_test.go
@@ -0,0 +1,53 @@
+package syncer_test
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+)
+
+// TestSyncerResumesAHydrationAcrossRounds verifies resumable rounds
+// (doc/graft/PLAN7.md, "Resumable rounds"): a round that runs out of its budget
+// keeps what it did. A hydration that does not reach the head installs the
+// repository at the changeset it reached; the next round catches up from there.
+// Each round answers Unavailable until the head is reached, so a partial copy
+// is never served as current.
+func TestSyncerResumesAHydrationAcrossRounds(t *testing.T) {
+ t.Parallel()
+
+ e := newEnv(t, envOptions{})
+ src := e.newSource(t)
+ c := e.writeCommit(t, src)
+ repo := e.register(t, e.key(localRelPath), src, c)
+ for i := 0; i < 9; i++ {
+ c = e.advanceMain(t, repo.ID, src, c)
+ }
+ repo, err := e.core.Meta.GetRepository(e.ctx, repo.Key)
+ require.NoError(t, err)
+ require.EqualValues(t, 10, repo.HeadSeq)
+
+ // The changesets were written on node-a; this node joins with an empty
+ // disk. (Changesets of the node itself after a range are pulled into it:
+ // the local repository may hold their effects, see newerLocalWrites.)
+ e.core.NodeID = "node-b"
+ e.sy.SetRoundLimit(4)
+ for _, reached := range []uint64{4, 8} {
+ err := e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath))
+ require.Equal(t, codes.Unavailable, status.Code(err), "%v", err)
+ require.ErrorContains(t, err, "at changeset")
+
+ wm, ok, err := e.sy.Applied(e.ctx, e.pbRepo(localRelPath))
+ require.NoError(t, err)
+ require.True(t, ok, "the round installed what it reached")
+ require.Equal(t, repo.ID, wm.RepoID)
+ require.Equal(t, reached, wm.Seq)
+ }
+
+ require.NoError(t, e.sy.EnsureFresh(e.ctx, e.pbRepo(localRelPath)))
+ require.Equal(t, float64(2), syncCount(t, e.sy, "partial"))
+ require.Equal(t, float64(0), syncCount(t, e.sy, "hydrated"), "the first round's hydration was partial")
+ require.Equal(t, float64(1), syncCount(t, e.sy, "caught_up"))
+ e.verifySynced(t, repo)
+}
diff --git a/internal/graft/e2e/storage_localrefs_test.go b/internal/graft/e2e/storage_localrefs_test.go
index cfe84b8..30399f0 100644
--- a/internal/graft/e2e/storage_localrefs_test.go
+++ b/internal/graft/e2e/storage_localrefs_test.go
@@ -472,7 +472,8 @@ func TestStorageLocalReferencesDuringAGracefulUpgrade(t *testing.T) {
}
// TestStorageBackgroundMaintenanceCheckpoints runs the real server with the
-// background maintenance on (doc/graft/PLAN7.md, "Background maintenance"): a
+// default background maintenance of authoritative mode (doc/graft/PLAN7.md,
+// "Background maintenance": checkpoints on, garbage collection off): a
// repository that gets enough changesets is checkpointed without a request,
// and keeps serving every write.
func TestStorageBackgroundMaintenanceCheckpoints(t *testing.T) {
@@ -480,7 +481,6 @@ func TestStorageBackgroundMaintenanceCheckpoints(t *testing.T) {
changesets := uint64(2)
n := newRealNode(t, func(s *config.GraftStorage) {
- s.Maintenance = config.GraftStorageMaintenanceAuto
s.MaintenanceInterval = "1s"
s.CheckpointChangesets = &changesets
})