Skip to content

Git designed for AI agents.

WP77: background maintenance of the storage core, its configuration

Federico Eliab36c18612c4e
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.md
index d5d155d..2f56b21 100644
--- a/doc/graft/CONFIGURATION.md
+++ b/doc/graft/CONFIGURATION.md
@@ -212,6 +212,13 @@ are in [STORAGE.md](STORAGE.md).
| `node_id` | string | the host name | Name of this node in the database (changesets record it, and `graft storage state` shows it). Must not contain spaces or slashes. Give every node its own. |
| `sync_timeout` | string (Go duration) | `"10s"` | Budget of the catch-up before a request in authoritative mode. A catch-up that cannot finish in time (for example because the database is unreachable or a local writer holds a reference lock) fails the request with `Unavailable`. Must be positive. |
| `shadow_timeout` | string (Go duration) | `"5s"` | Budget of the shadow-mode work of one reference transaction, for its Preparing phase (pack build and upload, before git locks the references) and its Prepared phase together. When it runs out, the work is abandoned, logged and counted, and the git write proceeds. Must be positive. |
+| `gc_grace` | string (Go duration) | `"1h"` | How long garbage collection waits before it deletes a pack it observed unreferenced, and how old a checkpoint must be before the packs it covers are deleted. Must be greater than `sync_timeout + gc_margin`. |
+| `gc_margin` | string (Go duration) | `"5m"` | Safety margin of the writers' deadline: a write uploads its pack and commits its changeset within `gc_grace - gc_margin`, else it fails. It also covers the difference between a node's clock and the database's. Must be positive. |
+| `maintenance` | string | `"off"` | `"auto"` runs the background checkpoints and garbage collection: every `maintenance_interval`, the packs of removed repositories are deleted, and every repository this node has a current copy of is checkpointed when the thresholds below say so and its dead packs are collected, under a per-repository lease so that two nodes do not do the same work. Authoritative mode only. `graft storage` commands do the same on demand in either case. |
+| `maintenance_interval` | string (Go duration) | `"1h"` | Time between two background maintenance runs. Must be positive. |
+| `checkpoint_changesets` | integer | `1000` | Checkpoint a repository once this many changesets follow its newest checkpoint. `0` never checkpoints by count. |
+| `checkpoint_pack_bytes` | integer | `268435456` (256 MiB) | Checkpoint a repository once the packs of the changesets after its newest checkpoint hold this many bytes. `0` never checkpoints by size. |
+| `lease_ttl` | string (Go duration) | `"10m"` | How long a node holds a repository's maintenance lease; renewed every third of it while the work runs. Must be positive. |
```toml
[graft.storage]
@@ -219,12 +226,16 @@ mode = "authoritative"
database = "sqlite:///var/opt/graft/meta.db"
bucket = "file:///var/opt/graft/objects"
node_id = "node-a"
+maintenance = "auto"
```
Startup fails when the mode is unknown, authoritative mode is set without
`standalone = true`, the database URL is missing or has another scheme, the
-bucket is missing or not a URL, `node_id` contains spaces or slashes, or a
-timeout is not a positive Go duration.
+bucket is missing or not a URL, `node_id` contains spaces or slashes, a
+duration is not a positive Go duration, `gc_grace` is not greater than
+`sync_timeout + gc_margin`, `maintenance` is neither `off` nor `auto`,
+`maintenance = "auto"` is set outside authoritative mode, or
+`checkpoint_pack_bytes` is negative.
## `graft-cache` flags
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.md
index 89c2705..cb98074 100644
--- a/doc/graft/PLAN7.md
+++ b/doc/graft/PLAN7.md
@@ -1325,6 +1325,33 @@ the changesets after the newest checkpoint `>= checkpoint_pack_bytes`, then
`serve.go` (the storage core block only), started with the server and stopped on
shutdown.
+**As implemented (WP77).**
+* Signature: `New(core, cp, collector, fresh, logger, cfg, storages []string)`.
+ The node's storage names replace the locator, which has no list of storages,
+ and `RunOnce` is exported for the tests.
+* The lease holder is `node_id/PID`, so the old and the new process of a
+ graceful upgrade never both hold a lease.
+* A lost lease (a renewal that fails or finds another holder) cancels the run
+ of that repository.
+* The pack-byte threshold reads the changesets after the newest checkpoint a
+ page at a time, only until the threshold is reached, and only when the count
+ threshold did not decide already.
+* A failure of one repository or one step does not stop the others; the
+ failures are counted by step.
+* `maintenance = "auto"` outside authoritative mode is a startup error, also
+ when the mode is off.
+* Metrics: `graft_storage_maintenance_{runs,checkpoints,packs_deleted,bytes_reclaimed,skipped,failures}_total`.
+* Tests: checkpoint by count and by bytes; both thresholds off; skip without a
+ local copy; skip when another holder has the lease, and the run's own release;
+ an orphan pack deleted gc_grace after a run observed it; Start/stop.
+* Not tested: a lease lost in the middle of a run (it needs a build slower than
+ the renewal).
+* **Database openers (audit).** Only `gitaly serve` opens the storage database
+ (`corestoreopen.Open`, one `sqlstore.Store` per process). The `graft` CLI and
+ every other binary reach it through gRPC. Two Stores on one SQLite file exist
+ only when two `gitaly serve` processes run on it: several nodes on one host,
+ or the overlap of a graceful upgrade. STORAGE.md (Limits) states it.
+
### Configuration (`config_graft_storage.go`, my section)
```toml
diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.md
index 29cb166..1276228 100644
--- a/doc/graft/STORAGE.md
+++ b/doc/graft/STORAGE.md
@@ -199,6 +199,13 @@ bucket = "file:///var/opt/graft/objects" # gocloud URL: file://, s3://, g
node_id = "node-a" # default: the host name
sync_timeout = "10s" # catch-up budget per request
shadow_timeout = "5s" # shadow work budget per transaction
+gc_grace = "1h" # age before a dead pack is deleted
+gc_margin = "5m" # writers' deadline is gc_grace - gc_margin
+maintenance = "auto" # off (default) | auto: background checkpoints and GC
+maintenance_interval = "1h"
+checkpoint_changesets = 1000 # 0 = never by count
+checkpoint_pack_bytes = 268435456 # 0 = never by size
+lease_ttl = "10m"
```
Every key is described in [CONFIGURATION.md](CONFIGURATION.md#graftstorage).
@@ -215,7 +222,11 @@ created and migrated at startup.
Startup fails when the mode is unknown, authoritative mode is set without
`standalone = true`, the database URL is missing or neither `sqlite:///` nor
`postgres://` (`postgresql://`), the bucket is missing or not a URL, `node_id`
-contains spaces or slashes, or a timeout is not a positive Go duration.
+contains spaces or slashes, a duration is not a positive Go duration,
+`gc_grace` is not greater than `sync_timeout + gc_margin`, or `maintenance =
+"auto"` is set outside authoritative mode. The background maintenance is off by
+default; its metrics are `graft_storage_maintenance_*` (runs, checkpoints, packs
+and bytes deleted, skipped repositories, failed steps).
## Operations
@@ -601,10 +612,11 @@ show:
and the bucket, not that nodes keep no local state. Every request runs git
on a local copy, a node's first request for a repository downloads all of its
packs, and there is no virtual filesystem.
-- **No garbage collection of packs yet.** Packs are never deleted: neither the
- orphans of writes that died before the database commit nor the packs of
- removed repositories. There is no checkpoint compaction either: hydration
- replays every changeset from 1. Both are planned after the first release.
+- **Checkpoints and pack garbage collection run in the background only when
+ `maintenance = "auto"`** (authoritative mode; off by default). Without it,
+ packs are never deleted and hydration replays every changeset after the
+ newest checkpoint, from 1 when there is none. The on-demand `graft storage`
+ commands for both are not available yet.
- **Praefect is not supported in authoritative mode.** Configuration validation
requires `[graft] standalone = true` for authoritative mode. Shadow mode
works with or without Praefect.
diff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.go
index 116a080..1a515e4 100644
--- a/internal/cli/gitaly/serve.go
+++ b/internal/cli/gitaly/serve.go
@@ -36,8 +36,11 @@ import (
"github.com/feder-cr/git-ai/internal/gitaly/transaction"
"github.com/feder-cr/git-ai/internal/gitlab"
"github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/checkpoint"
"github.com/feder-cr/git-ai/internal/graft/corestore/gate"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/gc"
"github.com/feder-cr/git-ai/internal/graft/corestore/localrefs"
+ storagemaintenance "github.com/feder-cr/git-ai/internal/graft/corestore/maintenance"
corestoreopen "github.com/feder-cr/git-ai/internal/graft/corestore/open"
"github.com/feder-cr/git-ai/internal/graft/corestore/reviewroots"
corestoresyncer "github.com/feder-cr/git-ai/internal/graft/corestore/syncer"
@@ -646,6 +649,14 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
// settles every repository whose writes ended.
defer stopLocalReferences()
+ if cfg.Graft.Storage.MaintenanceAuto() {
+ stopMaintenance, err := startStorageMaintenance(ctx, cfg, storageCore, syncer, locator, localrepoFactory, logger, storageNames)
+ if err != nil {
+ return err
+ }
+ defer stopMaintenance()
+ }
+
storageGate = gate.New(storageCore, syncer, logger, locator, localrepoFactory, protoregistry.GitalyProtoPreregistered)
}
@@ -963,6 +974,36 @@ func openLocalReferenceWrites(core *corestore.Core, locator storage.Locator, log
return nil
}
+// startStorageMaintenance starts the background checkpoints and garbage
+// collection of the storage core ([graft.storage] maintenance = "auto",
+// authoritative mode only; doc/graft/PLAN7.md, "Background maintenance").
+func startStorageMaintenance(ctx context.Context, cfg config.Cfg, core *corestore.Core, fresh corestore.Freshener, locator storage.Locator, repoFactory localrepo.Factory, logger log.Logger, storageNames []string) (func(), error) {
+ interval, err := cfg.Graft.Storage.MaintenanceIntervalOrDefault()
+ if err != nil {
+ return nil, err
+ }
+ leaseTTL, err := cfg.Graft.Storage.LeaseTTLOrDefault()
+ if err != nil {
+ return nil, err
+ }
+
+ runner := storagemaintenance.New(core,
+ checkpoint.New(core, logger, locator, repoFactory, fresh),
+ gc.New(core, logger),
+ fresh, logger,
+ storagemaintenance.Config{
+ Interval: interval,
+ CheckpointChangesets: cfg.Graft.Storage.CheckpointChangesetsOrDefault(),
+ CheckpointPackBytes: cfg.Graft.Storage.CheckpointPackBytesOrDefault(),
+ LeaseTTL: leaseTTL,
+ },
+ storageNames,
+ )
+ prometheus.MustRegister(runner)
+ logger.WithField("interval", interval.String()).Info("graft storage core: background maintenance enabled")
+ return runner.Start(ctx), nil
+}
+
// newPackObjectsAdaptiveLimit builds the adaptive limit backing a pack objects limiter. The limit
// only moves when the configuration asks for an adaptive one, and is otherwise pinned to the
// configured concurrency.
diff --git a/internal/gitaly/config/config_graft_storage.go b/internal/gitaly/config/config_graft_storage.go
index 8e69ebb..eecd19e 100644
--- a/internal/gitaly/config/config_graft_storage.go
+++ b/internal/gitaly/config/config_graft_storage.go
@@ -40,6 +40,102 @@ type GraftStorage struct {
// Empty means DefaultGraftStorageGCMargin. See doc/graft/STORAGE.md for the
// clock assumption it covers.
GCMargin string `json:"gc_margin" toml:"gc_margin,omitempty"`
+ // Maintenance is "off" (the default, also when empty) or "auto": the
+ // background checkpoints and garbage collection of the storage core, in
+ // authoritative mode only (doc/graft/PLAN7.md, "Background maintenance").
+ // The graft storage commands work in either case.
+ Maintenance string `json:"maintenance" toml:"maintenance,omitempty"`
+ // MaintenanceInterval is how often the background maintenance runs (Go
+ // duration). Empty means DefaultGraftStorageMaintenanceInterval.
+ MaintenanceInterval string `json:"maintenance_interval" toml:"maintenance_interval,omitempty"`
+ // CheckpointChangesets makes the maintenance checkpoint a repository once
+ // this many changesets follow its newest checkpoint; 0 never checkpoints by
+ // count. Unset means DefaultGraftStorageCheckpointChangesets.
+ CheckpointChangesets *uint64 `json:"checkpoint_changesets" toml:"checkpoint_changesets,omitempty"`
+ // CheckpointPackBytes makes the maintenance checkpoint a repository once the
+ // packs of the changesets after its newest checkpoint hold this many bytes;
+ // 0 never checkpoints by size. Unset means
+ // DefaultGraftStorageCheckpointPackBytes.
+ CheckpointPackBytes *int64 `json:"checkpoint_pack_bytes" toml:"checkpoint_pack_bytes,omitempty"`
+ // LeaseTTL is how long a node holds the maintenance lease of a repository
+ // before it must renew it (Go duration). Empty means
+ // DefaultGraftStorageLeaseTTL.
+ LeaseTTL string `json:"lease_ttl" toml:"lease_ttl,omitempty"`
+}
+
+// Values of GraftStorage.Maintenance.
+const (
+ GraftStorageMaintenanceOff = "off"
+ GraftStorageMaintenanceAuto = "auto"
+)
+
+const (
+ // DefaultGraftStorageMaintenanceInterval is the default
+ // GraftStorage.MaintenanceInterval.
+ DefaultGraftStorageMaintenanceInterval = time.Hour
+ // DefaultGraftStorageCheckpointChangesets is the default
+ // GraftStorage.CheckpointChangesets.
+ DefaultGraftStorageCheckpointChangesets uint64 = 1000
+ // DefaultGraftStorageCheckpointPackBytes is the default
+ // GraftStorage.CheckpointPackBytes (256 MiB).
+ DefaultGraftStorageCheckpointPackBytes int64 = 256 << 20
+ // DefaultGraftStorageLeaseTTL is the default GraftStorage.LeaseTTL.
+ DefaultGraftStorageLeaseTTL = 10 * time.Minute
+)
+
+// MaintenanceAuto reports whether the background maintenance runs.
+func (s GraftStorage) MaintenanceAuto() bool { return s.Maintenance == GraftStorageMaintenanceAuto }
+
+// MaintenanceIntervalOrDefault parses MaintenanceInterval, applying the default.
+func (s GraftStorage) MaintenanceIntervalOrDefault() (time.Duration, error) {
+ return positiveDurationOrDefault("maintenance_interval", s.MaintenanceInterval, DefaultGraftStorageMaintenanceInterval)
+}
+
+// LeaseTTLOrDefault parses LeaseTTL, applying the default.
+func (s GraftStorage) LeaseTTLOrDefault() (time.Duration, error) {
+ return positiveDurationOrDefault("lease_ttl", s.LeaseTTL, DefaultGraftStorageLeaseTTL)
+}
+
+// CheckpointChangesetsOrDefault returns CheckpointChangesets, applying the
+// default.
+func (s GraftStorage) CheckpointChangesetsOrDefault() uint64 {
+ if s.CheckpointChangesets == nil {
+ return DefaultGraftStorageCheckpointChangesets
+ }
+ return *s.CheckpointChangesets
+}
+
+// CheckpointPackBytesOrDefault returns CheckpointPackBytes, applying the
+// default.
+func (s GraftStorage) CheckpointPackBytesOrDefault() int64 {
+ if s.CheckpointPackBytes == nil {
+ return DefaultGraftStorageCheckpointPackBytes
+ }
+ return *s.CheckpointPackBytes
+}
+
+// validateMaintenance checks the background maintenance settings.
+func (s GraftStorage) validateMaintenance() error {
+ switch s.Maintenance {
+ case "", GraftStorageMaintenanceOff:
+ case GraftStorageMaintenanceAuto:
+ if s.Mode != "authoritative" {
+ return fmt.Errorf("graft storage: maintenance = %q requires mode = \"authoritative\" (checkpoints exist in authoritative mode only)", s.Maintenance)
+ }
+ default:
+ return fmt.Errorf("graft storage: unknown maintenance %q (want off or auto)", s.Maintenance)
+ }
+
+ if _, err := s.MaintenanceIntervalOrDefault(); err != nil {
+ return err
+ }
+ if _, err := s.LeaseTTLOrDefault(); err != nil {
+ return err
+ }
+ if s.CheckpointPackBytes != nil && *s.CheckpointPackBytes < 0 {
+ return fmt.Errorf("graft storage: checkpoint_pack_bytes %d must not be negative", *s.CheckpointPackBytes)
+ }
+ return nil
}
const (
@@ -116,7 +212,9 @@ func positiveDurationOrDefault(name, value string, def time.Duration) (time.Dura
func (s GraftStorage) validate(standalone bool) error {
switch s.Mode {
case "", "off":
- return nil
+ // The other settings are not read when the storage core is off, but a
+ // maintenance that cannot run is an error, not silence.
+ return s.validateMaintenance()
case "shadow", "authoritative":
default:
return fmt.Errorf("graft storage: unknown mode %q (want off, shadow or authoritative)", s.Mode)
@@ -153,6 +251,9 @@ func (s GraftStorage) validate(standalone bool) error {
if _, err := s.ShadowTimeoutOrDefault(); err != nil {
return err
}
+ if err := s.validateMaintenance(); err != nil {
+ return err
+ }
return s.validateGC()
}
diff --git a/internal/gitaly/config/config_graft_storage_test.go b/internal/gitaly/config/config_graft_storage_test.go
index 04a7b8f..d911632 100644
--- a/internal/gitaly/config/config_graft_storage_test.go
+++ b/internal/gitaly/config/config_graft_storage_test.go
@@ -1,6 +1,7 @@
package config
import (
+ "strings"
"testing"
"time"
@@ -39,6 +40,14 @@ func TestGraftStorageValidate(t *testing.T) {
{desc: "bad gc_margin", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, GCMargin: "0"}, err: "gc_margin"},
{desc: "gc_grace not above sync_timeout + gc_margin", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, GCGrace: "6m", GCMargin: "5m", SyncTimeout: "1m"}, err: "gc_grace 6m0s must be greater than sync_timeout 1m0s + gc_margin 5m0s"},
{desc: "gc_grace just above sync_timeout + gc_margin", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, GCGrace: "361s", GCMargin: "5m", SyncTimeout: "1m"}},
+ {desc: "maintenance auto in authoritative mode", storage: GraftStorage{Mode: "authoritative", Database: valid.Database, Bucket: valid.Bucket, Maintenance: "auto", MaintenanceInterval: "15m", LeaseTTL: "2m", CheckpointChangesets: ptr[uint64](0), CheckpointPackBytes: ptr[int64](0)}, standalone: true},
+ {desc: "maintenance off", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Maintenance: "off"}},
+ {desc: "maintenance auto in shadow mode", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Maintenance: "auto"}, err: `maintenance = "auto" requires mode = "authoritative"`},
+ {desc: "maintenance auto with the storage core off", storage: GraftStorage{Maintenance: "auto"}, err: `requires mode = "authoritative"`},
+ {desc: "unknown maintenance", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, Maintenance: "nightly"}, err: "unknown maintenance"},
+ {desc: "bad maintenance interval", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, MaintenanceInterval: "0s"}, err: "maintenance_interval"},
+ {desc: "bad lease ttl", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, LeaseTTL: "never"}, err: "lease_ttl"},
+ {desc: "negative checkpoint pack bytes", storage: GraftStorage{Mode: "shadow", Database: valid.Database, Bucket: valid.Bucket, CheckpointPackBytes: ptr[int64](-1)}, err: "checkpoint_pack_bytes"},
} {
t.Run(tc.desc, func(t *testing.T) {
t.Parallel()
@@ -76,6 +85,16 @@ func TestGraftStorageDefaults(t *testing.T) {
require.Equal(t, DefaultGraftStorageGCMargin, margin)
require.NoError(t, s.validateGC(), "the defaults are consistent")
+ require.False(t, s.MaintenanceAuto(), "the background maintenance is off by default")
+ interval, err := s.MaintenanceIntervalOrDefault()
+ require.NoError(t, err)
+ require.Equal(t, DefaultGraftStorageMaintenanceInterval, interval)
+ ttl, err := s.LeaseTTLOrDefault()
+ require.NoError(t, err)
+ require.Equal(t, DefaultGraftStorageLeaseTTL, ttl)
+ require.Equal(t, DefaultGraftStorageCheckpointChangesets, s.CheckpointChangesetsOrDefault())
+ require.Equal(t, DefaultGraftStorageCheckpointPackBytes, s.CheckpointPackBytesOrDefault())
+
s = GraftStorage{Mode: "shadow", SyncTimeout: "250ms"}
require.True(t, s.Enabled())
sync, err = s.SyncTimeoutOrDefault()
@@ -83,6 +102,36 @@ func TestGraftStorageDefaults(t *testing.T) {
require.Equal(t, 250*time.Millisecond, sync)
}
+// TestGraftStorageMaintenanceFromTOML checks that a threshold set to 0 in the
+// file means never, not the default.
+func TestGraftStorageMaintenanceFromTOML(t *testing.T) {
+ t.Parallel()
+
+ cfg, err := Load(strings.NewReader(`
+[graft.storage]
+mode = "authoritative"
+maintenance = "auto"
+maintenance_interval = "30m"
+checkpoint_changesets = 0
+checkpoint_pack_bytes = 1048576
+lease_ttl = "3m"
+`))
+ require.NoError(t, err)
+
+ s := cfg.Graft.Storage
+ require.True(t, s.MaintenanceAuto())
+ require.Equal(t, uint64(0), s.CheckpointChangesetsOrDefault())
+ require.Equal(t, int64(1<<20), s.CheckpointPackBytesOrDefault())
+ interval, err := s.MaintenanceIntervalOrDefault()
+ require.NoError(t, err)
+ require.Equal(t, 30*time.Minute, interval)
+ ttl, err := s.LeaseTTLOrDefault()
+ require.NoError(t, err)
+ require.Equal(t, 3*time.Minute, ttl)
+}
+
+func ptr[T any](v T) *T { return &v }
+
func TestGraftValidateChecksStorage(t *testing.T) {
t.Parallel()
diff --git a/internal/graft/corestore/maintenance/maintenance.go b/internal/graft/corestore/maintenance/maintenance.go
new file mode 100644
index 0000000..ef2c7a7
--- /dev/null
+++ b/internal/graft/corestore/maintenance/maintenance.go
@@ -0,0 +1,369 @@
+// Package maintenance runs the background checkpoints and garbage collection
+// of the storage core (doc/graft/PLAN7.md, "Background maintenance"). It is
+// off by default and runs in authoritative mode only; the graft storage
+// commands do the same work on demand.
+//
+// Every interval it collects the removed repositories of the node's storages,
+// then, for every repository of those storages that this node has a current
+// local copy of, it takes the repository's maintenance lease (skipping a
+// repository whose lease another process holds), checkpoints the repository
+// when enough changesets or pack bytes follow its newest checkpoint, and
+// collects its dead packs. The lease only keeps two nodes from doing the same
+// work at once: the checkpoint and the collector are correct without it.
+package maintenance
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "os"
+ "sync"
+ "time"
+
+ "github.com/prometheus/client_golang/prometheus"
+
+ "github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/checkpoint"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/gc"
+ "github.com/feder-cr/git-ai/internal/log"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
+)
+
+// changesetPage is how many changesets are read at once to sum pack bytes.
+const changesetPage = 500
+
+// releaseTimeout bounds the release of a lease after a run ended.
+const releaseTimeout = 10 * time.Second
+
+// Steps, the label of graft_storage_maintenance_failures_total.
+const (
+ stepCollectRemoved = "collect_removed"
+ stepList = "list"
+ stepLease = "lease"
+ stepCheckpoint = "checkpoint"
+ stepCollect = "collect"
+)
+
+// Config is the configuration of the background maintenance.
+type Config struct {
+ // Interval is the time between two runs.
+ Interval time.Duration
+ // CheckpointChangesets checkpoints a repository once this many changesets
+ // follow its newest checkpoint; 0 never checkpoints by count.
+ CheckpointChangesets uint64
+ // CheckpointPackBytes checkpoints a repository once the packs of the
+ // changesets after its newest checkpoint hold this many bytes; 0 never
+ // checkpoints by size.
+ CheckpointPackBytes int64
+ // LeaseTTL is how long a run holds a repository's lease before renewing it.
+ LeaseTTL time.Duration
+}
+
+// Runner runs the background maintenance of one node.
+type Runner struct {
+ core *corestore.Core
+ cp *checkpoint.Checkpointer
+ collector *gc.Collector
+ fresh corestore.Freshener
+ logger log.Logger
+ cfg Config
+ storages []string
+ // holder names this process in the leases: the node ID and the process ID,
+ // so that two processes of one node (a graceful upgrade) do not both hold a
+ // lease.
+ holder string
+
+ runs *prometheus.CounterVec
+ checkpoints prometheus.Counter
+ packs prometheus.Counter
+ bytes prometheus.Counter
+ skipped *prometheus.CounterVec
+ failures *prometheus.CounterVec
+}
+
+var _ prometheus.Collector = (*Runner)(nil)
+
+// New returns the runner of core for the node's storages.
+func New(core *corestore.Core, cp *checkpoint.Checkpointer, collector *gc.Collector, fresh corestore.Freshener, logger log.Logger, cfg Config, storages []string) *Runner {
+ r := &Runner{
+ core: core,
+ cp: cp,
+ collector: collector,
+ fresh: fresh,
+ logger: logger,
+ cfg: cfg,
+ storages: storages,
+ holder: fmt.Sprintf("%s/%d", core.NodeID, os.Getpid()),
+ runs: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "graft_storage_maintenance_runs_total",
+ Help: "Runs of the background maintenance of the storage core, by result (ok, error: at least one step failed).",
+ }, []string{"result"}),
+ checkpoints: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_storage_maintenance_checkpoints_total",
+ Help: "Checkpoints created by the background maintenance.",
+ }),
+ packs: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_storage_maintenance_packs_deleted_total",
+ Help: "Bucket objects deleted by the background garbage collection.",
+ }),
+ bytes: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_storage_maintenance_bytes_reclaimed_total",
+ Help: "Bytes of the bucket objects deleted by the background garbage collection.",
+ }),
+ skipped: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "graft_storage_maintenance_skipped_total",
+ Help: "Repositories the background maintenance skipped, by reason (no_local_copy, lease_held).",
+ }, []string{"reason"}),
+ failures: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "graft_storage_maintenance_failures_total",
+ Help: "Failed steps of the background maintenance, by step (collect_removed, list, lease, checkpoint, collect).",
+ }, []string{"step"}),
+ }
+ for _, result := range []string{"ok", "error"} {
+ r.runs.WithLabelValues(result)
+ }
+ for _, reason := range []string{"no_local_copy", "lease_held"} {
+ r.skipped.WithLabelValues(reason)
+ }
+ for _, step := range []string{stepCollectRemoved, stepList, stepLease, stepCheckpoint, stepCollect} {
+ r.failures.WithLabelValues(step)
+ }
+ return r
+}
+
+// Start runs the maintenance every interval, the first time one interval after
+// the start, until stop is called; stop waits for a run in progress to end
+// (its context is cancelled).
+func (r *Runner) Start(ctx context.Context) (stop func()) {
+ ctx, cancel := context.WithCancel(ctx)
+ var wg sync.WaitGroup
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ ticker := time.NewTicker(r.cfg.Interval)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ }
+ _ = r.RunOnce(ctx)
+ }
+ }()
+ return func() {
+ cancel()
+ wg.Wait()
+ }
+}
+
+// RunOnce runs the maintenance once. A failure of one repository or one step
+// does not stop the others; the returned error joins them.
+func (r *Runner) RunOnce(ctx context.Context) error {
+ var errs []error
+ fail := func(step string, err error, fields log.Fields) {
+ r.failures.WithLabelValues(step).Inc()
+ r.logger.WithError(err).WithFields(fields).WithField("step", step).Error("storage core: background maintenance step failed")
+ errs = append(errs, err)
+ }
+
+ for _, storageName := range r.storages {
+ report, err := r.collector.CollectRemoved(ctx, storageName, false)
+ if err != nil {
+ fail(stepCollectRemoved, err, log.Fields{"storage": storageName})
+ continue
+ }
+ r.count(report)
+ }
+
+ for _, storageName := range r.storages {
+ repos, err := r.core.Meta.ListRepositories(ctx, storageName)
+ if err != nil {
+ fail(stepList, err, log.Fields{"storage": storageName})
+ continue
+ }
+ for _, record := range repos {
+ if ctx.Err() != nil {
+ errs = append(errs, ctx.Err())
+ break
+ }
+ if step, err := r.maintain(ctx, record); err != nil {
+ fail(step, err, log.Fields{"repository": record.Key.String(), "repository_id": record.ID})
+ }
+ }
+ }
+
+ result := "ok"
+ if len(errs) > 0 {
+ result = "error"
+ }
+ r.runs.WithLabelValues(result).Inc()
+ return errors.Join(errs...)
+}
+
+// maintain checkpoints and collects one repository under its lease. It returns
+// the step that failed with the error.
+func (r *Runner) maintain(ctx context.Context, record corestore.Repository) (string, error) {
+ repo := &gitalypb.Repository{StorageName: record.Key.Storage, RelativePath: record.Key.RelativePath}
+
+ wm, ok, err := r.fresh.Applied(ctx, repo)
+ if err != nil {
+ return stepList, fmt.Errorf("read the local watermark: %w", err)
+ }
+ if !ok || wm.RepoID != record.ID {
+ r.skipped.WithLabelValues("no_local_copy").Inc()
+ return "", nil
+ }
+
+ acquired, err := r.core.Meta.AcquireMaintenanceLease(ctx, record.ID, r.holder, r.cfg.LeaseTTL)
+ if errors.Is(err, corestore.ErrRepositoryNotFound) {
+ return "", nil
+ }
+ if err != nil {
+ return stepLease, err
+ }
+ if !acquired {
+ r.skipped.WithLabelValues("lease_held").Inc()
+ return "", nil
+ }
+
+ leaseCtx, cancel := context.WithCancel(ctx)
+ renewed := r.renew(leaseCtx, cancel, record.ID)
+ defer func() {
+ cancel()
+ <-renewed
+ releaseCtx, cancelRelease := context.WithTimeout(context.WithoutCancel(ctx), releaseTimeout)
+ defer cancelRelease()
+ if err := r.core.Meta.ReleaseMaintenanceLease(releaseCtx, record.ID, r.holder); err != nil {
+ r.logger.WithError(err).WithField("repository_id", record.ID).Warn("storage core: release the maintenance lease; it expires by itself")
+ }
+ }()
+
+ due, err := r.checkpointDue(leaseCtx, record)
+ if err != nil {
+ return stepCheckpoint, err
+ }
+ if due {
+ _, created, err := r.cp.Create(leaseCtx, repo)
+ if err != nil {
+ return stepCheckpoint, err
+ }
+ if created {
+ r.checkpoints.Inc()
+ }
+ }
+
+ report, err := r.collector.CollectRepository(leaseCtx, record.ID, false)
+ if errors.Is(err, corestore.ErrRepositoryNotFound) {
+ return "", nil
+ }
+ if err != nil {
+ return stepCollect, err
+ }
+ r.count(report)
+ return "", nil
+}
+
+// renew renews the lease of repoID every third of its time to live until ctx
+// ends, and cancels the run (cancel) when the lease is lost: another process
+// may then do the same work, which is wasted but correct. The returned channel
+// is closed when it stops.
+func (r *Runner) renew(ctx context.Context, cancel context.CancelFunc, repoID int64) <-chan struct{} {
+ done := make(chan struct{})
+ go func() {
+ defer close(done)
+ ticker := time.NewTicker(r.cfg.LeaseTTL / 3)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-ctx.Done():
+ return
+ case <-ticker.C:
+ }
+ held, err := r.core.Meta.RenewMaintenanceLease(ctx, repoID, r.holder, r.cfg.LeaseTTL)
+ if ctx.Err() != nil {
+ return
+ }
+ if err != nil || !held {
+ r.logger.WithError(err).WithField("repository_id", repoID).WithField("held", held).
+ Warn("storage core: the maintenance lease was lost, the run of the repository stops")
+ r.failures.WithLabelValues(stepLease).Inc()
+ cancel()
+ return
+ }
+ }
+ }()
+ return done
+}
+
+// checkpointDue reports whether the repository needs a checkpoint: enough
+// changesets, or enough pack bytes, follow its newest checkpoint.
+func (r *Runner) checkpointDue(ctx context.Context, record corestore.Repository) (bool, error) {
+ if r.cfg.CheckpointChangesets == 0 && r.cfg.CheckpointPackBytes == 0 {
+ return false, nil
+ }
+
+ var base uint64
+ newest, ok, err := r.core.Meta.NewestCheckpoint(ctx, record.ID)
+ if err != nil {
+ return false, err
+ }
+ if ok {
+ base = newest.Seq
+ }
+ if record.HeadSeq <= base {
+ return false, nil
+ }
+ if r.cfg.CheckpointChangesets > 0 && record.HeadSeq-base >= r.cfg.CheckpointChangesets {
+ return true, nil
+ }
+ if r.cfg.CheckpointPackBytes == 0 {
+ return false, nil
+ }
+
+ // Fewer changesets than the count threshold follow the checkpoint (or the
+ // count is off): their pack bytes decide, read a page at a time and only
+ // until the threshold is reached.
+ var total int64
+ for after := base; after < record.HeadSeq; {
+ upTo := min(after+changesetPage, record.HeadSeq)
+ changesets, err := r.core.Meta.ListChangesets(ctx, record.ID, after, upTo)
+ if err != nil {
+ return false, err
+ }
+ for _, cs := range changesets {
+ total += cs.PackBytes
+ if total >= r.cfg.CheckpointPackBytes {
+ return true, nil
+ }
+ }
+ after = upTo
+ }
+ return false, nil
+}
+
+// count adds a collection report to the metrics.
+func (r *Runner) count(report gc.Report) {
+ r.packs.Add(float64(report.PacksDeleted))
+ r.bytes.Add(float64(report.BytesReclaimed))
+}
+
+// Describe implements prometheus.Collector.
+func (r *Runner) Describe(descs chan<- *prometheus.Desc) {
+ r.runs.Describe(descs)
+ r.checkpoints.Describe(descs)
+ r.packs.Describe(descs)
+ r.bytes.Describe(descs)
+ r.skipped.Describe(descs)
+ r.failures.Describe(descs)
+}
+
+// Collect implements prometheus.Collector.
+func (r *Runner) Collect(metrics chan<- prometheus.Metric) {
+ r.runs.Collect(metrics)
+ r.checkpoints.Collect(metrics)
+ r.packs.Collect(metrics)
+ r.bytes.Collect(metrics)
+ r.skipped.Collect(metrics)
+ r.failures.Collect(metrics)
+}
diff --git a/internal/graft/corestore/maintenance/maintenance_test.go b/internal/graft/corestore/maintenance/maintenance_test.go
new file mode 100644
index 0000000..897a251
--- /dev/null
+++ b/internal/graft/corestore/maintenance/maintenance_test.go
@@ -0,0 +1,294 @@
+package maintenance_test
+
+import (
+ "bytes"
+ "crypto/sha256"
+ "testing"
+ "time"
+
+ "github.com/prometheus/client_golang/prometheus"
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/catfile"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/git/localrepo"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/checkpoint"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/coretest"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/gc"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/gitops"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/maintenance"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/syncer"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
+)
+
+func TestMain(m *testing.M) {
+ testhelper.Run(m)
+}
+
+const relPath = "repo.git"
+
+// env is one node with a repository of the storage core: its database record,
+// a source repository its packs are built from, and the node's syncer.
+type env struct {
+ t *testing.T
+ cfg config.Cfg
+ core *corestore.Core
+ sy *syncer.Syncer
+ runner *maintenance.Runner
+ repo corestore.Repository
+ src *localrepo.Repo
+ srcDir string
+ head git.ObjectID
+}
+
+func newEnv(t *testing.T, opts coretest.Options, cfg maintenance.Config) *env {
+ t.Helper()
+
+ gcfg := testcfg.Build(t)
+ ctx := testhelper.Context(t)
+ logger := testhelper.NewLogger(t)
+ locator := config.NewLocator(gcfg)
+ gitCmdFactory := gittest.NewCommandFactory(t, gcfg)
+ cache := catfile.NewCache(gcfg)
+ t.Cleanup(cache.Stop)
+ repoFactory := localrepo.NewFactory(logger, locator, gitCmdFactory, cache)
+
+ core := coretest.Open(t, opts)
+ sy := syncer.New(core, logger, locator, repoFactory, gitCmdFactory)
+ runner := maintenance.New(core, checkpoint.New(core, logger, locator, repoFactory, sy), gc.New(core, logger), sy, logger, cfg,
+ []string{gcfg.Storages[0].Name})
+
+ srcPB, srcDir := gittest.CreateRepository(t, ctx, gcfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})
+ e := &env{t: t, cfg: gcfg, core: core, sy: sy, runner: runner, src: localrepo.NewTestRepo(t, gcfg, srcPB), srcDir: srcDir}
+
+ repo, _, err := core.Meta.RegisterRepository(ctx, e.key(), gittest.DefaultObjectHash.Format, corestore.InitialState{})
+ require.NoError(t, err)
+ e.repo = repo
+ e.head = e.commit("c0")
+ e.changeset([]corestore.RefUpdate{
+ {Name: "HEAD", Expect: corestore.ExpectValue, New: corestore.SymbolicTarget("refs/heads/main")},
+ {Name: "refs/heads/main", Expect: corestore.ExpectValue, New: corestore.Target(e.head)},
+ }, e.head)
+ return e
+}
+
+func (e *env) key() corestore.RepoKey { return corestore.KeyOf(e.pbRepo()) }
+
+func (e *env) pbRepo() *gitalypb.Repository {
+ return &gitalypb.Repository{StorageName: e.cfg.Storages[0].Name, RelativePath: relPath}
+}
+
+func (e *env) commit(message string, parents ...git.ObjectID) git.ObjectID {
+ return gittest.WriteCommit(e.t, e.cfg, e.srcDir, gittest.WithMessage(message), gittest.WithParents(parents...))
+}
+
+func (e *env) changeset(updates []corestore.RefUpdate, tips ...git.ObjectID) {
+ e.t.Helper()
+ ctx := testhelper.Context(e.t)
+
+ pack, err := gitops.BuildPack(ctx, e.src, tips, e.t.TempDir())
+ require.NoError(e.t, err)
+ defer func() { require.NoError(e.t, pack.Remove()) }()
+
+ in := corestore.ChangesetInput{RepoID: e.repo.ID, Updates: updates, Operation: "test", Actor: "tester", NodeID: "node-a"}
+ if !pack.Empty() {
+ in.PackKey = corestore.NewPackKey(e.repo.ID, pack.SHA256)
+ rc, err := pack.Open()
+ require.NoError(e.t, err)
+ require.NoError(e.t, e.core.Blobs.Put(ctx, in.PackKey, rc))
+ require.NoError(e.t, rc.Close())
+ in.PackBytes = pack.Size
+ }
+ _, err = e.core.Meta.CommitChangeset(ctx, in)
+ require.NoError(e.t, err)
+}
+
+// advance moves main by n commits, one changeset each.
+func (e *env) advance(n int) {
+ e.t.Helper()
+ for i := 0; i < n; i++ {
+ next := e.commit("next", e.head)
+ e.changeset([]corestore.RefUpdate{{Name: "refs/heads/main", Expect: corestore.ExpectValue, Old: corestore.Target(e.head), New: corestore.Target(next)}}, next)
+ e.head = next
+ }
+}
+
+// fresh catches the node's local copy up.
+func (e *env) fresh() {
+ e.t.Helper()
+ require.NoError(e.t, e.sy.EnsureFresh(testhelper.Context(e.t), e.pbRepo()))
+}
+
+func (e *env) newestCheckpoint() (corestore.Checkpoint, bool) {
+ e.t.Helper()
+ cp, ok, err := e.core.Meta.NewestCheckpoint(testhelper.Context(e.t), e.repo.ID)
+ require.NoError(e.t, err)
+ return cp, ok
+}
+
+func (e *env) headSeq() uint64 {
+ e.t.Helper()
+ r, err := e.core.Meta.GetRepository(testhelper.Context(e.t), e.key())
+ require.NoError(e.t, err)
+ return r.HeadSeq
+}
+
+// metric returns the value of the counter name with the label value (when
+// not empty).
+func (e *env) metric(name, label string) float64 {
+ e.t.Helper()
+ reg := prometheus.NewPedanticRegistry()
+ require.NoError(e.t, reg.Register(e.runner))
+ families, err := reg.Gather()
+ require.NoError(e.t, err)
+ for _, family := range families {
+ if family.GetName() != name {
+ continue
+ }
+ for _, m := range family.GetMetric() {
+ if label == "" {
+ return m.GetCounter().GetValue()
+ }
+ for _, l := range m.GetLabel() {
+ if l.GetValue() == label {
+ return m.GetCounter().GetValue()
+ }
+ }
+ }
+ }
+ return 0
+}
+
+func config3(changesets uint64, packBytes int64) maintenance.Config {
+ return maintenance.Config{Interval: time.Hour, CheckpointChangesets: changesets, CheckpointPackBytes: packBytes, LeaseTTL: time.Minute}
+}
+
+func TestRunOnceCheckpointsAfterEnoughChangesets(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ e := newEnv(t, coretest.Options{}, config3(3, 0))
+ e.fresh()
+
+ require.NoError(t, e.runner.RunOnce(ctx))
+ _, ok := e.newestCheckpoint()
+ require.False(t, ok, "one changeset is below the threshold")
+
+ e.advance(2)
+ e.fresh()
+ require.NoError(t, e.runner.RunOnce(ctx))
+ cp, ok := e.newestCheckpoint()
+ require.True(t, ok)
+ require.Equal(t, e.headSeq(), cp.Seq)
+ require.Equal(t, float64(1), e.metric("graft_storage_maintenance_checkpoints_total", ""))
+
+ // Nothing follows the checkpoint: no new one.
+ require.NoError(t, e.runner.RunOnce(ctx))
+ require.Equal(t, float64(1), e.metric("graft_storage_maintenance_checkpoints_total", ""))
+ require.Equal(t, float64(3), e.metric("graft_storage_maintenance_runs_total", "ok"))
+}
+
+func TestRunOnceCheckpointsAfterEnoughPackBytes(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ e := newEnv(t, coretest.Options{}, config3(0, 1))
+ e.fresh()
+
+ require.NoError(t, e.runner.RunOnce(ctx))
+ cp, ok := e.newestCheckpoint()
+ require.True(t, ok, "the first changeset's pack is over one byte")
+ require.Equal(t, e.headSeq(), cp.Seq)
+
+ off := newEnv(t, coretest.Options{}, config3(0, 0))
+ off.fresh()
+ require.NoError(t, off.runner.RunOnce(ctx))
+ _, ok = off.newestCheckpoint()
+ require.False(t, ok, "both thresholds 0 never checkpoint")
+}
+
+func TestRunOnceSkipsARepositoryWithoutALocalCopy(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ e := newEnv(t, coretest.Options{}, config3(1, 0))
+
+ require.NoError(t, e.runner.RunOnce(ctx))
+ _, ok := e.newestCheckpoint()
+ require.False(t, ok)
+ require.Equal(t, float64(1), e.metric("graft_storage_maintenance_skipped_total", "no_local_copy"))
+}
+
+func TestRunOnceSkipsARepositoryWhoseLeaseIsHeld(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ e := newEnv(t, coretest.Options{}, config3(1, 0))
+ e.fresh()
+
+ acquired, err := e.core.Meta.AcquireMaintenanceLease(ctx, e.repo.ID, "node-b/1", time.Hour)
+ require.NoError(t, err)
+ require.True(t, acquired)
+
+ require.NoError(t, e.runner.RunOnce(ctx))
+ _, ok := e.newestCheckpoint()
+ require.False(t, ok)
+ require.Equal(t, float64(1), e.metric("graft_storage_maintenance_skipped_total", "lease_held"))
+
+ require.NoError(t, e.core.Meta.ReleaseMaintenanceLease(ctx, e.repo.ID, "node-b/1"))
+ require.NoError(t, e.runner.RunOnce(ctx))
+ _, ok = e.newestCheckpoint()
+ require.True(t, ok)
+
+ // The run released its lease: another holder takes it at once.
+ acquired, err = e.core.Meta.AcquireMaintenanceLease(ctx, e.repo.ID, "node-b/1", time.Hour)
+ require.NoError(t, err)
+ require.True(t, acquired)
+}
+
+// TestRunOnceCollectsAnOrphanPack verifies that the runs collect the
+// repository: an object under its pack prefix that no row references is
+// deleted once a run observed it gc_grace earlier.
+func TestRunOnceCollectsAnOrphanPack(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ e := newEnv(t, coretest.Options{GCGrace: 200 * time.Millisecond, GCMargin: 50 * time.Millisecond}, config3(0, 0))
+ e.fresh()
+
+ content := []byte("an orphan pack")
+ key := corestore.NewPackKey(e.repo.ID, sha256.Sum256(content))
+ require.NoError(t, e.core.Blobs.Put(ctx, key, bytes.NewReader(content)))
+
+ // The first run observes it; a run gc_grace later deletes it.
+ require.NoError(t, e.runner.RunOnce(ctx))
+ exists, err := e.core.Blobs.Exists(ctx, key)
+ require.NoError(t, err)
+ require.True(t, exists, "observed, not yet deleted")
+
+ require.Eventually(t, func() bool {
+ require.NoError(t, e.runner.RunOnce(ctx))
+ exists, err := e.core.Blobs.Exists(ctx, key)
+ require.NoError(t, err)
+ return !exists
+ }, 30*time.Second, 100*time.Millisecond)
+ require.Equal(t, float64(1), e.metric("graft_storage_maintenance_packs_deleted_total", ""))
+ require.Equal(t, float64(len(content)), e.metric("graft_storage_maintenance_bytes_reclaimed_total", ""))
+}
+
+func TestStartRunsEveryIntervalUntilStopped(t *testing.T) {
+ t.Parallel()
+ ctx := testhelper.Context(t)
+ cfg := config3(1, 0)
+ cfg.Interval = 20 * time.Millisecond
+ e := newEnv(t, coretest.Options{}, cfg)
+ e.fresh()
+
+ stop := e.runner.Start(ctx)
+ require.Eventually(t, func() bool {
+ _, ok := e.newestCheckpoint()
+ return ok
+ }, 30*time.Second, 10*time.Millisecond)
+ // stop returns once the loop has ended (a run in progress is cancelled).
+ stop()
+}
diff --git a/internal/graft/e2e/storage_localrefs_test.go b/internal/graft/e2e/storage_localrefs_test.go
index 48eb1c1..8b5af52 100644
--- a/internal/graft/e2e/storage_localrefs_test.go
+++ b/internal/graft/e2e/storage_localrefs_test.go
@@ -53,7 +53,7 @@ type realNode struct {
exited chan struct{}
}
-func newRealNode(t *testing.T) *realNode {
+func newRealNode(t *testing.T, configure ...func(*config.GraftStorage)) *realNode {
t.Helper()
graft, tokens := workerGraft(1)
@@ -67,6 +67,9 @@ func newRealNode(t *testing.T) *realNode {
Bucket: coretest.BucketURL(t, false),
NodeID: "node-a",
}
+ for _, c := range configure {
+ c(&cfg.Graft.Storage)
+ }
cfg.SocketPath = testhelper.GetTemporaryGitalySocketFileName(t)
testcfg.BuildGitalySSH(t, cfg)
testcfg.BuildGitalyHooks(t, cfg)
@@ -422,6 +425,34 @@ func TestStorageLocalReferencesDuringAGracefulUpgrade(t *testing.T) {
next.requireServed(t, rel, written)
}
+// TestStorageBackgroundMaintenanceCheckpoints runs the real server with the
+// background maintenance on (doc/graft/PLAN7.md, "Background maintenance"): a
+// repository that gets enough changesets is checkpointed without a request,
+// and keeps serving every write.
+func TestStorageBackgroundMaintenanceCheckpoints(t *testing.T) {
+ t.Parallel()
+
+ changesets := uint64(2)
+ n := newRealNode(t, func(s *config.GraftStorage) {
+ s.Maintenance = config.GraftStorageMaintenanceAuto
+ s.MaintenanceInterval = "1s"
+ s.CheckpointChangesets = &changesets
+ })
+ n.start(t)
+ require.Contains(t, n.log(t), "background maintenance enabled")
+
+ rel := n.createRepository(t)
+ onA := n.agent(t, rel)
+ base, err := commitFile(n.ctx, onA, workerName(0), "main", "", "README.md", "base\n")
+ require.NoError(t, err)
+ written := writeBranches(t, onA, n.ctx, base, "maintained", 3)
+ written["main"] = base
+
+ require.Eventually(t, func() bool { return strings.Contains(n.log(t), "storage core: checkpoint created") },
+ 60*time.Second, 100*time.Millisecond, "the maintenance checkpoints the repository")
+ n.requireServed(t, rel, written)
+}
+
// TestStorageLocalReferencesStackDepthIsBounded writes many references
// without fsync from concurrent writers and checks that git's compaction keeps
// the stack shallow: the durable step only makes it durable, compaction stays