Skip to content

Git designed for AI agents.

review roots: own namespace, lost commits, housekeeping backstop

Federico Elia86ca0e1b119d
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.md
index 1f0dae5..439e926 100644
--- a/doc/graft/PLAN7.md
+++ b/doc/graft/PLAN7.md
@@ -347,7 +347,7 @@ prune, expiry now) removed it on today's code, on both dialects
(`TestOpenReviewCommitsSurviveHousekeeping`, run before the change).
**Rule: review roots are references.** The head and base commits of a review
-are protected by `refs/keep-around/<oid>` (target `<oid>`), written through
+are protected by `refs/keep-around/review/<oid>` (target `<oid>`), written through
the ordinary write path: the reference-transaction hook, so the voter and the
database CAS in shadow and authoritative mode. Then:
* every node keeps them through housekeeping (they are local references after
@@ -359,7 +359,12 @@ database CAS in shadow and authoritative mode. Then:
`refs/keep-around/` is a hidden internal prefix in Gitaly: it is neither
advertised nor writable over the git protocols
(`uploadpack.hideRefs`/`receive.hideRefs`). The Graft changelog does not
-record it, because it records only branches and tags.
+record it, because it records only branches and tags. The review roots have a
+namespace of their own, `refs/keep-around/review/`, because GitLab writes
+`refs/keep-around/<oid>` for its own purposes (pipelines, merge requests). No
+sweep and no database check ever touches a reference outside that namespace
+(tested: a GitLab keep-around reference of an open review's commit is neither
+guarded nor swept).
* **Writes (review service, every mode).** `CreateReview` writes the references
of head and base (`reviewroots.Protect`: one reference transaction that
@@ -386,27 +391,54 @@ record it, because it records only branches and tags.
the bound only stops a pathological loop.
* **Deletion.**
* `CloseReview` deletes the references of the closed review's commits that no
- other open review uses (`reviewroots.Sweep` with those candidates). Each
- deletion is its own transaction expecting the value read. A deletion that a
- concurrent review write makes the database refuse is kept. A deletion that
- fails otherwise is logged at warning level: the close is already recorded,
- and the reference only keeps objects alive longer.
- * The maintenance task (WP77) runs the full sweep, which deletes leftovers
- such as those of a crash, and the backfill (`reviewroots.Backfill`), which
- writes the references of open reviews written before this rule or imported
- from shadow-mode files when their objects are present, and reports the
- others (counts and review IDs). A review whose objects are already lost
- blocks nothing.
+ other open review uses (`reviewroots.DefaultSweeper.Sweep` with those
+ candidates, trigger `close`). Each deletion is its own transaction expecting
+ the value read. A deletion that a concurrent review write makes the database
+ refuse is kept. A deletion that fails otherwise is not the request's error,
+ because the close is already recorded: it is logged at warning level and
+ counted (`graft_review_roots_sweep_failures_total{trigger="close"}`), and
+ the backstop below deletes the reference. Not returning it is a design
+ decision with a backstop, not a swallowed error.
+ * **Backstop, every mode.** After Gitaly's daily housekeeping of a repository
+ (`DailyOptimizationWorker`, on by default, the same job that prunes
+ unreferenced objects), `reviewroots.Sweeper.Housekeep` runs the backfill
+ and then the full sweep of the repository's review references. The
+ backfill writes the references that open reviews lack: reviews written
+ before this rule, or imported from shadow-mode files. The full sweep
+ deletes the references no open review uses: leftovers of a failed close
+ sweep or of a crash. In authoritative mode it first catches the repository
+ up and uses the database's reviews; it skips an untracked repository or an
+ incomplete import. In off and shadow mode it uses the review files. Metrics
+ (`reviewroots.DefaultSweeper`, registered by `gitaly serve`):
+ `graft_review_roots_deleted_total`, `graft_review_roots_backfilled_total`,
+ `graft_review_roots_missing_total`, and
+ `graft_review_roots_sweep_failures_total{trigger}`, so a persistent failure
+ shows as a growing counter. Lost commits are logged with their review ID
+ and commit. Limit: an operator who disables the daily maintenance also
+ disables this backstop (and the scheduled object pruning it guards).
* A sweep may skip references younger than `gc_grace` to make the
creation/sweep race rare. That is an efficiency measure only; correctness
comes from the invariant and the bounded retry.
+* **A review whose commit is lost (decision).** A review written before this rule
+ whose head or base commit is gone from the repository cannot be kept any
+ more. `UpdateReview` refuses it with `FailedPrecondition` naming the review
+ (metadata `review_id`) and the commit (`commit_id`). The base is protected
+ before the descent check, so the error is never a misleading "head does not
+ descend". The backfill reports the same pair (`MissingRoot{ReviewID,
+ Commit}`). The review can still be read and closed.
+ Why not fetch the object from the storage core as hydration does: nothing
+ kept it live there either. Only references keep objects live, and this commit
+ had none, so after GC no pack holds it. Before GC some changeset pack may
+ still hold it, but no index maps an object to its pack, so finding it means
+ downloading packs until it turns up. That would be a fallback that works only
+ until the next collection (RULE 24 forbids such paths). The state is known
+ and documented, so a precise `FailedPrecondition` is the answer.
* **Off and shadow mode.** The review store is the files store and the same
references are written, so housekeeping keeps the commits there too; in
shadow mode the voter records them in the database. The database invariant
is not enforced there: the files are the source of truth, and shadow
- changesets are forced, so `CommitChangeset` does not refuse them. In off
- mode the references of a closed review are deleted by its close; there is no
- maintenance sweep.
+ changesets are forced, so `CommitChangeset` does not refuse them. The close
+ sweep and the housekeeping backstop run there as in authoritative mode.
* **What it costs.** One reference transaction per review creation and per
head move (one changeset, plus a pack only when the commit was not yet in the
storage core), one per released commit at a close, and no changelog entry.
@@ -596,7 +628,7 @@ var (
)
// KeepAroundPrefix and KeepAroundRef name the references of the REVIEW ROOTS RULE.
-const KeepAroundPrefix = "refs/keep-around/"
+const KeepAroundPrefix = "refs/keep-around/review/"
func KeepAroundRef(oid string) string
// NewPackKey returns a new owned pack key for a pack of the repository with the
diff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.md
index 1f09ba9..4eb2f42 100644
--- a/doc/graft/STORAGE.md
+++ b/doc/graft/STORAGE.md
@@ -91,7 +91,7 @@ design is [PLAN6.md](PLAN6.md).
database transaction as the reference change, and reviews are stored in the
database. A watcher on another node sees new entries by polling every 250 ms.
- **Review commits are kept by references.** `CreateReview` and `UpdateReview`
- write a hidden reference `refs/keep-around/<commit>` for the head and base of
+ write a hidden reference `refs/keep-around/review/<commit>` for the head and base of
the review before they record it, so an open review keeps its commits alive
after their branch is deleted: on every node (housekeeping keeps what
references reach), in checkpoints and through garbage collection. A review of
@@ -105,7 +105,13 @@ design is [PLAN6.md](PLAN6.md).
written as well (housekeeping keeps the commits), but nothing enforces them:
the review files are the source of truth there. Each review creation and each
head move costs one reference transaction, so one changeset and no changelog
- entry.
+ entry. GitLab's own `refs/keep-around/<commit>` references are never touched.
+ The daily housekeeping of every repository (on by default) also writes the
+ references that open reviews lack and deletes those no open review uses;
+ watch `graft_review_roots_sweep_failures_total` and
+ `graft_review_roots_missing_total`. A review written by an older version
+ whose commit was lost since can still be read and closed, but `UpdateReview`
+ refuses it with `FailedPrecondition` naming the review and the commit.
- **Repository IDs.** A repository is identified by its storage name and
relative path, and gets an ID that is never reused: a repository removed and
created again is a new repository, and the old caches are recognized as
diff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.go
index 6e2599c..0d6abde 100644
--- a/internal/cli/gitaly/serve.go
+++ b/internal/cli/gitaly/serve.go
@@ -38,6 +38,7 @@ import (
"github.com/feder-cr/git-ai/internal/graft/corestore"
"github.com/feder-cr/git-ai/internal/graft/corestore/gate"
corestoreopen "github.com/feder-cr/git-ai/internal/graft/corestore/open"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/reviewroots"
corestoresyncer "github.com/feder-cr/git-ai/internal/graft/corestore/syncer"
"github.com/feder-cr/git-ai/internal/graft/corestore/voter"
"github.com/feder-cr/git-ai/internal/graft/derived"
@@ -604,6 +605,9 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
return fmt.Errorf("graft storage core: %w", err)
}
defer func() { _ = storageCore.Close() }()
+ // The review roots are written and swept in every mode, also without the
+ // storage core.
+ prometheus.MustRegister(reviewroots.DefaultSweeper)
hookManagerOpts := []hook.ManagerOption{hook.WithEagerObserver(eagerObserver)}
var (
@@ -839,7 +843,14 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
ctx,
logger,
maintenance.DailyOptimizationWorker(cfg, maintenance.OptimizerFunc(func(ctx context.Context, logger log.Logger, repo storage.Repository) error {
- return housekeepingManager.OptimizeRepository(ctx, localrepo.New(logger, locator, gitCmdFactory, catfileCache, repo))
+ local := localrepo.New(logger, locator, gitCmdFactory, catfileCache, repo)
+ if err := housekeepingManager.OptimizeRepository(ctx, local); err != nil {
+ return err
+ }
+ // Graft: the backstop of the review roots, in every mode, where the
+ // housekeeping that prunes unreferenced objects runs
+ // (doc/graft/PLAN7.md, REVIEW ROOTS RULE).
+ return housekeepReviewRoots(ctx, logger, storageCore, storageSyncer, local, repo)
})),
)
if err != nil {
@@ -884,6 +895,32 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {
// newPackObjectsAdaptiveLimit builds the adaptive limit backing a pack objects limiter. The limit
// only moves when the configuration asks for an adaptive one, and is otherwise pinned to the
// configured concurrency.
+// housekeepReviewRoots runs the backstop of the review roots after the daily
+// housekeeping of a repository: it writes the keep-around references that
+// open reviews lack and deletes those no open review uses (in authoritative
+// mode after catching the repository up). Open reviews whose commits are lost
+// are logged with their IDs and commits; the counts are metrics of
+// reviewroots.DefaultSweeper.
+func housekeepReviewRoots(ctx context.Context, logger log.Logger, core *corestore.Core, fresh corestore.Freshener, local *localrepo.Repo, repo storage.Repository) error {
+ backfill, sweep, err := reviewroots.DefaultSweeper.Housekeep(ctx, core, fresh, local, repo)
+ for _, missing := range backfill.Missing {
+ logger.WithFields(log.Fields{
+ "review_id": missing.ReviewID,
+ "commit_id": missing.Commit,
+ }).Warn("review roots: a commit of an open review is not in the repository; the review can only be closed")
+ }
+ if backfill.Protected > 0 || sweep.Deleted > 0 {
+ logger.WithFields(log.Fields{
+ "backfilled": backfill.Protected,
+ "deleted": sweep.Deleted,
+ }).Info("review roots: housekeeping")
+ }
+ if err != nil {
+ return fmt.Errorf("review roots: %w", err)
+ }
+ return nil
+}
+
func newPackObjectsAdaptiveLimit(name string, limits config.ConcurrencyLimits) *limiter.AdaptiveLimit {
if !limits.Adaptive {
return limiter.NewAdaptiveLimit(name, limiter.AdaptiveSetting{
diff --git a/internal/gitaly/service/review/review.go b/internal/gitaly/service/review/review.go
index 7d82505..65e99ec 100644
--- a/internal/gitaly/service/review/review.go
+++ b/internal/gitaly/service/review/review.go
@@ -82,7 +82,7 @@ func (s *server) CreateReview(ctx context.Context, req *gitalypb.CreateReviewReq
var stored *gitalypb.Review
err = s.writeRoots(ctx, repo, st, func(st store.ReviewStore, protect rootsProtect) error {
- if err := protect(head, base); err != nil {
+ if err := protect("", head, base); err != nil {
return err
}
@@ -231,7 +231,14 @@ func moveHead(ctx context.Context, repo *localrepo.Repo, req *gitalypb.UpdateRev
base := git.ObjectID(r.GetBaseCommitId())
if head.String() == r.GetHeadCommitId() {
- return protect(reviewCommits(head, base)...)
+ return protect(r.GetId(), reviewCommits(head, base)...)
+ }
+ // The base is a root of the open review whatever this update does; a base
+ // that is gone is reported as such, not as a head that does not descend.
+ if base != "" {
+ if err := protect(r.GetId(), base); err != nil {
+ return err
+ }
}
descends, err := repo.IsAncestor(ctx, base.Revision(), head.Revision())
@@ -268,7 +275,7 @@ func moveHead(ctx context.Context, repo *localrepo.Repo, req *gitalypb.UpdateRev
v.Current = survives
}
- if err := protect(reviewCommits(head, base)...); err != nil {
+ if err := protect(r.GetId(), reviewCommits(head, base)...); err != nil {
return err
}
diff --git a/internal/gitaly/service/review/roots.go b/internal/gitaly/service/review/roots.go
index 9e9bcca..a6edf5f 100644
--- a/internal/gitaly/service/review/roots.go
+++ b/internal/gitaly/service/review/roots.go
@@ -23,8 +23,9 @@ type requiredRootsStore interface {
}
// rootsProtect writes the keep-around references of commits that a review
-// write is about to record.
-type rootsProtect func(commits ...git.ObjectID) error
+// write is about to record; reviewID names the review in an error, empty for a
+// review that is not created yet.
+type rootsProtect func(reviewID string, commits ...git.ObjectID) error
// writeRoots runs write, a review write that may record new head or base
// commits, under the REVIEW ROOTS RULE (doc/graft/PLAN7.md). write calls
@@ -38,8 +39,18 @@ type rootsProtect func(commits ...git.ObjectID) error
// writes the reference again), at most maxRootsAttempts times in all, after
// which the request is Aborted.
func (s *server) writeRoots(ctx context.Context, repo *localrepo.Repo, st store.ReviewStore, write func(store.ReviewStore, rootsProtect) error) error {
- protect := func(commits ...git.ObjectID) error {
+ protect := func(reviewID string, commits ...git.ObjectID) error {
if err := reviewroots.Protect(ctx, repo, commits); err != nil {
+ var missing *reviewroots.MissingCommitsError
+ if errors.As(err, &missing) {
+ // A review written before its commits were kept by references,
+ // whose commit was lost since: nothing in the storage core kept it
+ // live either, so it cannot be brought back (doc/graft/PLAN7.md,
+ // REVIEW ROOTS RULE). The backfill reports the same commit.
+ return structerr.NewFailedPrecondition("commits %v of review %q are not in the repository any more; the review can only be closed", missing.Commits, reviewID).
+ WithMetadata("review_id", reviewID).
+ WithMetadata("commit_id", missing.Commits)
+ }
return structerr.NewInternal("protect the commits of the review: %w", err)
}
if s.rootsProtected != nil {
@@ -65,12 +76,13 @@ func (s *server) writeRoots(ctx context.Context, repo *localrepo.Repo, st store.
// releaseRoots deletes the keep-around references of the commits of a review
// that was just closed, unless another open review uses them. The close is
-// already recorded: a reference that cannot be deleted now only keeps its
-// objects alive longer, until the next close of a review of the same commit or
-// the maintenance sweep of the storage core.
+// already recorded, so a failure is not the request's: it is logged and counted
+// (graft_review_roots_sweep_failures_total{trigger="close"}), and the reference
+// only keeps its objects alive until the housekeeping backstop deletes it
+// (reviewroots.Sweeper.Housekeep, which runs in every mode).
func (s *server) releaseRoots(ctx context.Context, repo *localrepo.Repo, st store.ReviewStore, commits []git.ObjectID) {
- if _, err := reviewroots.Sweep(ctx, repo, st, commits); err != nil {
+ if _, err := reviewroots.DefaultSweeper.Sweep(ctx, repo, st, commits, reviewroots.TriggerClose); err != nil {
s.deps.GetLogger().WithError(err).WithField("commits", commits).WarnContext(ctx,
- "review: the keep-around references of a closed review were not deleted; they keep their objects until a later sweep")
+ "review: the keep-around references of a closed review were not deleted; the housekeeping backstop deletes them")
}
}
diff --git a/internal/gitaly/service/review/roots_test.go b/internal/gitaly/service/review/roots_test.go
index 23c4e6b..95fbf04 100644
--- a/internal/gitaly/service/review/roots_test.go
+++ b/internal/gitaly/service/review/roots_test.go
@@ -232,6 +232,46 @@ func TestAuthoritativeReviewWriteSurvivesASweep(t *testing.T) {
}
}
+// TestUpdateOfAReviewWhoseCommitIsLost: a review written before the REVIEW
+// ROOTS RULE whose base commit is gone cannot be kept any more. UpdateReview
+// refuses it with FailedPrecondition naming the review and the commit (the
+// same pair the housekeeping backfill reports), never Internal; the review can
+// still be read and closed.
+func TestUpdateOfAReviewWhoseCommitIsLost(t *testing.T) {
+ ctx, cfg, core, client := setupAuthoritative(t)
+
+ repo, repoPath := createLocalRepository(t, ctx, cfg)
+ base := gittest.WriteCommit(t, cfg, repoPath, gittest.WithMessage("base"), gittest.WithBranch("main"))
+ head := gittest.WriteCommit(t, cfg, repoPath, gittest.WithMessage("head"), gittest.WithParents(base), gittest.WithBranch("feature"))
+ lost := strings.Repeat("c", gittest.DefaultObjectHash.EncodedLen())
+ importRepository(t, cfg, core, repo, repoPath, coretest.ImportOptions{
+ Reviews: []*gitalypb.Review{{
+ Id: "7",
+ TargetBranch: []byte("main"),
+ HeadCommitId: head.String(),
+ BaseCommitId: lost,
+ Author: anonymousActor,
+ }},
+ })
+ // The first request lets the gate replace the imported copy by a cache built
+ // from the storage core.
+ require.Len(t, listStoredReviews(t, ctx, client, repo, true), 1)
+ next := gittest.WriteCommit(t, cfg, repoPath, gittest.WithMessage("next"), gittest.WithParents(head))
+
+ for _, moveTo := range []git.ObjectID{head, next} {
+ _, err := client.UpdateReview(ctx, &gitalypb.UpdateReviewRequest{Repository: repo, Id: "7", HeadCommitId: moveTo.String()})
+ require.Equal(t, codes.FailedPrecondition, status.Code(err), "%v", err)
+ require.ErrorContains(t, err, lost)
+ require.ErrorContains(t, err, `review "7"`)
+ }
+
+ got, err := client.GetReview(ctx, &gitalypb.GetReviewRequest{Repository: repo, Id: "7"})
+ require.NoError(t, err)
+ require.Equal(t, head.String(), got.GetReview().GetHeadCommitId())
+ _, err = client.CloseReview(ctx, &gitalypb.CloseReviewRequest{Repository: repo, Id: "7"})
+ require.NoError(t, err)
+}
+
// TestReviewRootsOutsideAuthoritativeMode: with the storage core off or in
// shadow mode the reviews are files, and the service still writes the
// keep-around references: housekeeping keeps an open review's commit after its
diff --git a/internal/graft/corestore/reviewroots/reviewroots.go b/internal/graft/corestore/reviewroots/reviewroots.go
index d0519a1..4bb17eb 100644
--- a/internal/graft/corestore/reviewroots/reviewroots.go
+++ b/internal/graft/corestore/reviewroots/reviewroots.go
@@ -29,10 +29,23 @@ import (
"github.com/feder-cr/git-ai/internal/graft/review/store"
)
+// MissingCommitsError is the error of Protect for commits that are not in the
+// repository: a commit can only be kept while it exists. It happens for reviews
+// written before the REVIEW ROOTS RULE whose commit was lost since; the storage
+// core cannot bring it back either (nothing kept it live).
+type MissingCommitsError struct {
+ Commits []git.ObjectID
+}
+
+func (e *MissingCommitsError) Error() string {
+ return fmt.Sprintf("commits %v are not in the repository", e.Commits)
+}
+
// Protect writes the keep-around reference of every commit that does not have
-// it, in one reference transaction. The commits must be in the repository. A
-// commit whose reference exists with its value is left alone, so Protect of
-// protected commits writes nothing.
+// it, in one reference transaction. A commit that is not in the repository is a
+// *MissingCommitsError and nothing is written. A commit whose reference exists
+// with its value is left alone, so Protect of protected commits writes
+// nothing.
func Protect(ctx context.Context, repo *localrepo.Repo, commits []git.ObjectID) error {
names := make([]string, 0, len(commits))
byName := make(map[string]git.ObjectID, len(commits))
@@ -63,6 +76,18 @@ func Protect(ctx context.Context, repo *localrepo.Repo, commits []git.ObjectID)
return nil
}
+ unprotected := make([]git.ObjectID, 0, len(missing))
+ for _, name := range missing {
+ unprotected = append(unprotected, byName[name])
+ }
+ absent, err := gitops.MissingObjects(ctx, repo, unprotected)
+ if err != nil {
+ return fmt.Errorf("look up the commits: %w", err)
+ }
+ if len(absent) > 0 {
+ return &MissingCommitsError{Commits: absent}
+ }
+
updater, err := updateref.New(ctx, repo)
if err != nil {
return fmt.Errorf("start the keep-around update: %w", err)
@@ -212,9 +237,17 @@ type BackfillReport struct {
// Protected is the number of commits whose keep-around reference was
// written.
Protected int
- // Missing are the IDs of the open reviews with a commit that is not in the
- // repository (or not an object ID): it cannot be protected any more.
- Missing []string
+ // Missing are the commits of open reviews that are not in the repository
+ // (or not object IDs): they cannot be protected any more. Such a review
+ // can still be read and closed; UpdateReview refuses it with
+ // FailedPrecondition naming the same review and commit.
+ Missing []MissingRoot
+}
+
+// MissingRoot is a commit of an open review that the repository does not have.
+type MissingRoot struct {
+ ReviewID string
+ Commit string
}
// Backfill writes the keep-around references of the open reviews that do not
@@ -240,7 +273,7 @@ func Backfill(ctx context.Context, repo *localrepo.Repo, reviews store.ReviewSto
continue
}
if hash.ValidateHex(commit) != nil {
- report.Missing = appendOnce(report.Missing, r.GetId())
+ report.Missing = append(report.Missing, MissingRoot{ReviewID: r.GetId(), Commit: commit})
continue
}
oid := git.ObjectID(commit)
@@ -271,7 +304,7 @@ func Backfill(ctx context.Context, repo *localrepo.Repo, reviews store.ReviewSto
for _, oid := range missing {
absent[oid] = struct{}{}
for _, id := range byCommit[oid] {
- report.Missing = appendOnce(report.Missing, id)
+ report.Missing = append(report.Missing, MissingRoot{ReviewID: id, Commit: oid.String()})
}
}
@@ -285,7 +318,10 @@ func Backfill(ctx context.Context, repo *localrepo.Repo, reviews store.ReviewSto
return report, err
}
report.Protected = len(present)
- sort.Strings(report.Missing)
+ sort.Slice(report.Missing, func(i, j int) bool {
+ a, b := report.Missing[i], report.Missing[j]
+ return a.ReviewID < b.ReviewID || (a.ReviewID == b.ReviewID && a.Commit < b.Commit)
+ })
return report, nil
}
@@ -305,12 +341,3 @@ func openRoots(reviews store.ReviewStore) (map[string]struct{}, error) {
}
return used, nil
}
-
-func appendOnce(ids []string, id string) []string {
- for _, existing := range ids {
- if existing == id {
- return ids
- }
- }
- return append(ids, id)
-}
diff --git a/internal/graft/corestore/reviewroots/reviewroots_test.go b/internal/graft/corestore/reviewroots/reviewroots_test.go
index 4dd6a97..6126d64 100644
--- a/internal/graft/corestore/reviewroots/reviewroots_test.go
+++ b/internal/graft/corestore/reviewroots/reviewroots_test.go
@@ -109,9 +109,14 @@ func TestProtect(t *testing.T) {
require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c3}))
require.Equal(t, keepAroundOf(c1, c2, c3), f.keepArounds())
- // A commit that is not in the repository cannot be protected.
+ // A commit that is not in the repository cannot be protected; nothing is
+ // written, also for the commits that are present.
absent := git.ObjectID(strings.Repeat("a", gittest.DefaultObjectHash.EncodedLen()))
- require.Error(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{absent}))
+ c4 := f.write("c4", c3)
+ err := reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c4, absent})
+ var missing *reviewroots.MissingCommitsError
+ require.ErrorAs(t, err, &missing)
+ require.Equal(t, []git.ObjectID{absent}, missing.Commits)
require.Equal(t, keepAroundOf(c1, c2, c3), f.keepArounds())
}
@@ -127,6 +132,16 @@ func TestSweep(t *testing.T) {
f.review(c2.String(), c0.String(), gitalypb.Review_STATE_CLOSED)
require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c0, c1, c2, c3}))
+ // GitLab's own keep-around references are not review roots: no sweep
+ // touches them.
+ gitlab := git.ReferenceName("refs/keep-around/" + c2.String())
+ gittest.WriteRef(t, testcfg.Build(t), f.path, gitlab, c2)
+ defer func() {
+ refs, err := gitops.ReadRefs(f.ctx, f.repo)
+ require.NoError(t, err)
+ require.Equal(t, corestore.Target(c2), refs[gitlab.String()], "GitLab's keep-around reference is kept")
+ }()
+
// Candidates only: the open review's head is kept, the closed review's goes.
report, err := reviewroots.Sweep(f.ctx, f.repo, f.reviews, []git.ObjectID{c1, c2})
require.NoError(t, err)
@@ -163,7 +178,10 @@ func TestBackfill(t *testing.T) {
report, err := reviewroots.Backfill(f.ctx, f.repo, f.reviews)
require.NoError(t, err)
require.Equal(t, 1, report.Protected, "the open review's head; its base had its reference")
- require.Equal(t, []string{lost.GetId(), garbled.GetId()}, report.Missing)
+ require.Equal(t, []reviewroots.MissingRoot{
+ {ReviewID: lost.GetId(), Commit: absent},
+ {ReviewID: garbled.GetId(), Commit: "not an object ID"},
+ }, report.Missing)
require.Equal(t, keepAroundOf(c0, c1), f.keepArounds(), "closed reviews are not backfilled")
// A second backfill has nothing left to protect.
diff --git a/internal/graft/corestore/reviewroots/sweeper.go b/internal/graft/corestore/reviewroots/sweeper.go
new file mode 100644
index 0000000..dd974b1
--- /dev/null
+++ b/internal/graft/corestore/reviewroots/sweeper.go
@@ -0,0 +1,155 @@
+package reviewroots
+
+import (
+ "context"
+ "errors"
+ "fmt"
+
+ "github.com/prometheus/client_golang/prometheus"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/localrepo"
+ "github.com/feder-cr/git-ai/internal/gitaly/storage"
+ "github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/dbreview"
+ "github.com/feder-cr/git-ai/internal/graft/review/store"
+)
+
+// Triggers of a sweep, the label of the sweep metrics.
+const (
+ // TriggerClose is the sweep of the commits of a review that was just closed.
+ TriggerClose = "close"
+ // TriggerHousekeeping is the backstop that runs with the object
+ // housekeeping of a repository (Housekeep).
+ TriggerHousekeeping = "housekeeping"
+)
+
+// Sweeper runs the sweeps and the backfill of the review roots and counts
+// them, so that a failure that its caller does not return (the sweep of a
+// close, which is recorded already) is visible, and a persistent one shows as
+// a growing counter.
+type Sweeper struct {
+ deleted prometheus.Counter
+ protected prometheus.Counter
+ missing prometheus.Counter
+ failures *prometheus.CounterVec
+}
+
+var _ prometheus.Collector = (*Sweeper)(nil)
+
+// DefaultSweeper is the sweeper of the process; gitaly serve registers it.
+var DefaultSweeper = NewSweeper()
+
+// NewSweeper returns a sweeper with its own counters.
+func NewSweeper() *Sweeper {
+ s := &Sweeper{
+ deleted: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_review_roots_deleted_total",
+ Help: "Number of keep-around references of review commits deleted because no open review uses them.",
+ }),
+ protected: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_review_roots_backfilled_total",
+ Help: "Number of keep-around references of open-review commits written by the housekeeping backfill.",
+ }),
+ missing: prometheus.NewCounter(prometheus.CounterOpts{
+ Name: "graft_review_roots_missing_total",
+ Help: "Number of open-review commits found missing from a repository by the housekeeping backfill.",
+ }),
+ failures: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "graft_review_roots_sweep_failures_total",
+ Help: "Number of sweeps of review keep-around references that failed, by trigger (close, housekeeping).",
+ }, []string{"trigger"}),
+ }
+ for _, trigger := range []string{TriggerClose, TriggerHousekeeping} {
+ s.failures.WithLabelValues(trigger)
+ }
+ return s
+}
+
+// Describe implements prometheus.Collector.
+func (s *Sweeper) Describe(descs chan<- *prometheus.Desc) {
+ s.deleted.Describe(descs)
+ s.protected.Describe(descs)
+ s.missing.Describe(descs)
+ s.failures.Describe(descs)
+}
+
+// Collect implements prometheus.Collector.
+func (s *Sweeper) Collect(metrics chan<- prometheus.Metric) {
+ s.deleted.Collect(metrics)
+ s.protected.Collect(metrics)
+ s.missing.Collect(metrics)
+ s.failures.Collect(metrics)
+}
+
+// Sweep runs Sweep and counts it under trigger.
+func (s *Sweeper) Sweep(ctx context.Context, repo *localrepo.Repo, reviews store.ReviewStore, candidates []git.ObjectID, trigger string) (SweepReport, error) {
+ report, err := Sweep(ctx, repo, reviews, candidates)
+ s.deleted.Add(float64(report.Deleted))
+ if err != nil {
+ s.failures.WithLabelValues(trigger).Inc()
+ }
+ return report, err
+}
+
+// Housekeep is the backstop of the review roots, run with the object
+// housekeeping of the repository (gitaly's daily maintenance) in every mode:
+// it writes the keep-around references that open reviews lack (Backfill) and
+// deletes those that no open review uses (Sweep of every keep-around
+// reference), so that a reference a close could not delete, or one written
+// before a crash, does not keep its objects forever. In authoritative mode the
+// reviews are the database's and the local repository is caught up first (fresh
+// is the node's Freshener); an untracked repository, or one whose import is
+// not complete, has no reviews in the database and is skipped. In off and
+// shadow mode (core nil or not authoritative) the reviews are the files of the
+// repository.
+func (s *Sweeper) Housekeep(ctx context.Context, core *corestore.Core, fresh corestore.Freshener, repo *localrepo.Repo, repoProto storage.Repository) (BackfillReport, SweepReport, error) {
+ reviews, ok, err := housekeepingStore(ctx, core, fresh, repo, repoProto)
+ if err != nil {
+ s.failures.WithLabelValues(TriggerHousekeeping).Inc()
+ return BackfillReport{}, SweepReport{}, err
+ }
+ if !ok {
+ return BackfillReport{}, SweepReport{}, nil
+ }
+
+ backfill, err := Backfill(ctx, repo, reviews)
+ s.protected.Add(float64(backfill.Protected))
+ s.missing.Add(float64(len(backfill.Missing)))
+ if err != nil {
+ s.failures.WithLabelValues(TriggerHousekeeping).Inc()
+ return backfill, SweepReport{}, fmt.Errorf("backfill: %w", err)
+ }
+
+ sweep, err := s.Sweep(ctx, repo, reviews, nil, TriggerHousekeeping)
+ if err != nil {
+ return backfill, sweep, fmt.Errorf("sweep: %w", err)
+ }
+ return backfill, sweep, nil
+}
+
+// housekeepingStore returns the review store of the repository for Housekeep;
+// ok is false when the repository has none to keep.
+func housekeepingStore(ctx context.Context, core *corestore.Core, fresh corestore.Freshener, repo *localrepo.Repo, repoProto storage.Repository) (store.ReviewStore, bool, error) {
+ if !core.Authoritative() {
+ path, err := repo.Path(ctx)
+ if err != nil {
+ return nil, false, fmt.Errorf("repository path: %w", err)
+ }
+ return store.Open(path), true, nil
+ }
+
+ record, err := core.Meta.GetRepository(ctx, corestore.KeyOf(repoProto))
+ switch {
+ case errors.Is(err, corestore.ErrRepositoryNotFound):
+ return nil, false, nil
+ case err != nil:
+ return nil, false, fmt.Errorf("storage core: %w", err)
+ case record.HeadSeq == 0:
+ return nil, false, nil
+ }
+ if err := fresh.EnsureFresh(ctx, repoProto); err != nil {
+ return nil, false, fmt.Errorf("catch up: %w", err)
+ }
+ return dbreview.New(ctx, core.Meta, record.ID), true, nil
+}
diff --git a/internal/graft/corestore/reviewroots/sweeper_test.go b/internal/graft/corestore/reviewroots/sweeper_test.go
new file mode 100644
index 0000000..93f0975
--- /dev/null
+++ b/internal/graft/corestore/reviewroots/sweeper_test.go
@@ -0,0 +1,126 @@
+package reviewroots_test
+
+import (
+ "errors"
+ "testing"
+
+ "github.com/prometheus/client_golang/prometheus"
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/graft/corestore"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/coretest"
+ "github.com/feder-cr/git-ai/internal/graft/corestore/reviewroots"
+ "github.com/feder-cr/git-ai/internal/graft/review/store"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
+)
+
+// counter returns the value of the counter family name of the sweeper (with
+// the trigger label when trigger is not empty).
+func counter(t *testing.T, s *reviewroots.Sweeper, name, trigger string) float64 {
+ t.Helper()
+
+ reg := prometheus.NewPedanticRegistry()
+ require.NoError(t, reg.Register(s))
+ families, err := reg.Gather()
+ require.NoError(t, err)
+ for _, family := range families {
+ if family.GetName() != name {
+ continue
+ }
+ for _, metric := range family.GetMetric() {
+ if trigger == "" {
+ return metric.GetCounter().GetValue()
+ }
+ for _, label := range metric.GetLabel() {
+ if label.GetName() == "trigger" && label.GetValue() == trigger {
+ return metric.GetCounter().GetValue()
+ }
+ }
+ }
+ }
+ return 0
+}
+
+// brokenStore is a review store whose listing fails.
+type brokenStore struct{ store.ReviewStore }
+
+func (brokenStore) List(string, bool, int) ([]*gitalypb.Review, error) {
+ return nil, errors.New("the review files cannot be read")
+}
+
+func TestSweeperCounts(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t)
+ c0 := f.write("c0")
+ require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c0}))
+
+ s := reviewroots.NewSweeper()
+ _, err := s.Sweep(f.ctx, f.repo, brokenStore{}, nil, reviewroots.TriggerClose)
+ require.Error(t, err)
+ require.Equal(t, float64(1), counter(t, s, "graft_review_roots_sweep_failures_total", reviewroots.TriggerClose))
+ require.Zero(t, counter(t, s, "graft_review_roots_sweep_failures_total", reviewroots.TriggerHousekeeping))
+
+ report, err := s.Sweep(f.ctx, f.repo, f.reviews, nil, reviewroots.TriggerClose)
+ require.NoError(t, err)
+ require.Equal(t, 1, report.Deleted)
+ require.Equal(t, float64(1), counter(t, s, "graft_review_roots_deleted_total", ""))
+ require.Equal(t, float64(1), counter(t, s, "graft_review_roots_sweep_failures_total", reviewroots.TriggerClose))
+}
+
+// TestHousekeepWithoutTheStorageCore is the backstop in off and shadow mode:
+// the reviews are the files of the repository; the references open reviews
+// lack are written, and a leaked one (a close whose sweep failed) is deleted.
+func TestHousekeepWithoutTheStorageCore(t *testing.T) {
+ t.Parallel()
+
+ for _, mode := range []corestore.Mode{corestore.ModeOff, corestore.ModeShadow} {
+ t.Run(string(mode), func(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t)
+ c0 := f.write("c0")
+ c1 := f.write("c1", c0)
+ leaked := f.write("leaked", c0)
+ f.review(c1.String(), c0.String(), gitalypb.Review_STATE_OPEN_UNSPECIFIED)
+ require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{leaked}))
+
+ var core *corestore.Core
+ if mode == corestore.ModeShadow {
+ core = coretest.Open(t, coretest.Options{Mode: corestore.ModeShadow})
+ }
+
+ s := reviewroots.NewSweeper()
+ backfill, sweep, err := s.Housekeep(f.ctx, core, corestore.NopFreshener{}, f.repo, f.repo)
+ require.NoError(t, err)
+ require.Equal(t, 2, backfill.Protected)
+ require.Empty(t, backfill.Missing)
+ require.Equal(t, reviewroots.SweepReport{Deleted: 1, Kept: 2}, sweep)
+ require.Equal(t, keepAroundOf(c0, c1), f.keepArounds())
+
+ require.Equal(t, float64(2), counter(t, s, "graft_review_roots_backfilled_total", ""))
+ require.Equal(t, float64(1), counter(t, s, "graft_review_roots_deleted_total", ""))
+ require.Zero(t, counter(t, s, "graft_review_roots_sweep_failures_total", reviewroots.TriggerHousekeeping))
+ })
+ }
+}
+
+// TestHousekeepSkipsUntrackedRepositories: in authoritative mode the reviews
+// are the database's; a repository the database does not track has none, and
+// its local references are left alone.
+func TestHousekeepSkipsUntrackedRepositories(t *testing.T) {
+ t.Parallel()
+
+ f := newFixture(t)
+ c0 := f.write("c0")
+ require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c0}))
+
+ core := coretest.Open(t, coretest.Options{Mode: corestore.ModeAuthoritative})
+ s := reviewroots.NewSweeper()
+ backfill, sweep, err := s.Housekeep(f.ctx, core, corestore.NopFreshener{}, f.repo, f.repo)
+ require.NoError(t, err)
+ require.Equal(t, reviewroots.BackfillReport{}, backfill)
+ require.Equal(t, reviewroots.SweepReport{}, sweep)
+ require.Equal(t, keepAroundOf(c0), f.keepArounds())
+}
diff --git a/internal/graft/corestore/sqlstore/reviewroots_test.go b/internal/graft/corestore/sqlstore/reviewroots_test.go
index fcc590e..9d44bc3 100644
--- a/internal/graft/corestore/sqlstore/reviewroots_test.go
+++ b/internal/graft/corestore/sqlstore/reviewroots_test.go
@@ -129,6 +129,14 @@ func TestOpenReviewRootsCannotBeReleased(t *testing.T) {
_, err = commit(ctx, s, repo.ID, update(corestore.KeepAroundRef(string(closed)), closed, ""))
require.NoError(t, err)
+ // GitLab's own keep-around reference of the same commit is not a review
+ // root: it is not guarded.
+ gitlab := "refs/keep-around/" + string(head)
+ _, err = commit(ctx, s, repo.ID, create(gitlab, head))
+ require.NoError(t, err)
+ _, err = commit(ctx, s, repo.ID, update(gitlab, head, ""))
+ require.NoError(t, err)
+
// Shadow mode records what git did: forced changesets are not refused.
_, err = s.CommitChangeset(ctx, corestore.ChangesetInput{
RepoID: repo.ID,
diff --git a/internal/graft/corestore/types.go b/internal/graft/corestore/types.go
index 8951076..5723064 100644
--- a/internal/graft/corestore/types.go
+++ b/internal/graft/corestore/types.go
@@ -328,10 +328,13 @@ func parsePackKey(key string) (repoID int64, sum [32]byte, owned, ok bool) {
}
// KeepAroundPrefix is the prefix of the references that keep the commits of
-// open reviews alive: Gitaly hides it from clients (it is neither advertised
-// nor writable over the git protocols), housekeeping and garbage collection
-// respect it like every reference (doc/graft/PLAN7.md, REVIEW ROOTS RULE).
-const KeepAroundPrefix = "refs/keep-around/"
+// open reviews alive (doc/graft/PLAN7.md, REVIEW ROOTS RULE). It is under
+// refs/keep-around/, which Gitaly hides from clients (neither advertised nor
+// writable over the git protocols) and which housekeeping and garbage
+// collection respect like every reference. It is a namespace of its own:
+// GitLab writes refs/keep-around/<oid> for its own purposes, and the sweeps of
+// the review roots must never delete those.
+const KeepAroundPrefix = "refs/keep-around/review/"
// KeepAroundRef returns the keep-around reference of a commit; its target is
// the commit itself.