Git designed for AI agents.
review roots: own namespace, lost commits, housekeeping backstop
Federico Elia86ca0e1b119d
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.mdindex 1f0dae5..439e926 100644--- a/doc/graft/PLAN7.md+++ b/doc/graft/PLAN7.md@@ -347,7 +347,7 @@ prune, expiry now) removed it on today's code, on both dialects(`TestOpenReviewCommitsSurviveHousekeeping`, run before the change).**Rule: review roots are references.** The head and base commits of a review-are protected by `refs/keep-around/<oid>` (target `<oid>`), written through+are protected by `refs/keep-around/review/<oid>` (target `<oid>`), written throughthe ordinary write path: the reference-transaction hook, so the voter and thedatabase CAS in shadow and authoritative mode. Then:* every node keeps them through housekeeping (they are local references after@@ -359,7 +359,12 @@ database CAS in shadow and authoritative mode. Then:`refs/keep-around/` is a hidden internal prefix in Gitaly: it is neitheradvertised nor writable over the git protocols(`uploadpack.hideRefs`/`receive.hideRefs`). The Graft changelog does not-record it, because it records only branches and tags.+record it, because it records only branches and tags. The review roots have a+namespace of their own, `refs/keep-around/review/`, because GitLab writes+`refs/keep-around/<oid>` for its own purposes (pipelines, merge requests). No+sweep and no database check ever touches a reference outside that namespace+(tested: a GitLab keep-around reference of an open review's commit is neither+guarded nor swept).* **Writes (review service, every mode).** `CreateReview` writes the referencesof head and base (`reviewroots.Protect`: one reference transaction that@@ -386,27 +391,54 @@ record it, because it records only branches and tags.the bound only stops a pathological loop.* **Deletion.*** `CloseReview` deletes the references of the closed review's commits that no- other open review uses (`reviewroots.Sweep` with those candidates). Each- deletion is its own transaction expecting the value read. A deletion that a- concurrent review write makes the database refuse is kept. A deletion that- fails otherwise is logged at warning level: the close is already recorded,- and the reference only keeps objects alive longer.- * The maintenance task (WP77) runs the full sweep, which deletes leftovers- such as those of a crash, and the backfill (`reviewroots.Backfill`), which- writes the references of open reviews written before this rule or imported- from shadow-mode files when their objects are present, and reports the- others (counts and review IDs). A review whose objects are already lost- blocks nothing.+ other open review uses (`reviewroots.DefaultSweeper.Sweep` with those+ candidates, trigger `close`). Each deletion is its own transaction expecting+ the value read. A deletion that a concurrent review write makes the database+ refuse is kept. A deletion that fails otherwise is not the request's error,+ because the close is already recorded: it is logged at warning level and+ counted (`graft_review_roots_sweep_failures_total{trigger="close"}`), and+ the backstop below deletes the reference. Not returning it is a design+ decision with a backstop, not a swallowed error.+ * **Backstop, every mode.** After Gitaly's daily housekeeping of a repository+ (`DailyOptimizationWorker`, on by default, the same job that prunes+ unreferenced objects), `reviewroots.Sweeper.Housekeep` runs the backfill+ and then the full sweep of the repository's review references. The+ backfill writes the references that open reviews lack: reviews written+ before this rule, or imported from shadow-mode files. The full sweep+ deletes the references no open review uses: leftovers of a failed close+ sweep or of a crash. In authoritative mode it first catches the repository+ up and uses the database's reviews; it skips an untracked repository or an+ incomplete import. In off and shadow mode it uses the review files. Metrics+ (`reviewroots.DefaultSweeper`, registered by `gitaly serve`):+ `graft_review_roots_deleted_total`, `graft_review_roots_backfilled_total`,+ `graft_review_roots_missing_total`, and+ `graft_review_roots_sweep_failures_total{trigger}`, so a persistent failure+ shows as a growing counter. Lost commits are logged with their review ID+ and commit. Limit: an operator who disables the daily maintenance also+ disables this backstop (and the scheduled object pruning it guards).* A sweep may skip references younger than `gc_grace` to make thecreation/sweep race rare. That is an efficiency measure only; correctnesscomes from the invariant and the bounded retry.+* **A review whose commit is lost (decision).** A review written before this rule+ whose head or base commit is gone from the repository cannot be kept any+ more. `UpdateReview` refuses it with `FailedPrecondition` naming the review+ (metadata `review_id`) and the commit (`commit_id`). The base is protected+ before the descent check, so the error is never a misleading "head does not+ descend". The backfill reports the same pair (`MissingRoot{ReviewID,+ Commit}`). The review can still be read and closed.+ Why not fetch the object from the storage core as hydration does: nothing+ kept it live there either. Only references keep objects live, and this commit+ had none, so after GC no pack holds it. Before GC some changeset pack may+ still hold it, but no index maps an object to its pack, so finding it means+ downloading packs until it turns up. That would be a fallback that works only+ until the next collection (RULE 24 forbids such paths). The state is known+ and documented, so a precise `FailedPrecondition` is the answer.* **Off and shadow mode.** The review store is the files store and the samereferences are written, so housekeeping keeps the commits there too; inshadow mode the voter records them in the database. The database invariantis not enforced there: the files are the source of truth, and shadow- changesets are forced, so `CommitChangeset` does not refuse them. In off- mode the references of a closed review are deleted by its close; there is no- maintenance sweep.+ changesets are forced, so `CommitChangeset` does not refuse them. The close+ sweep and the housekeeping backstop run there as in authoritative mode.* **What it costs.** One reference transaction per review creation and perhead move (one changeset, plus a pack only when the commit was not yet in thestorage core), one per released commit at a close, and no changelog entry.@@ -596,7 +628,7 @@ var ()// KeepAroundPrefix and KeepAroundRef name the references of the REVIEW ROOTS RULE.-const KeepAroundPrefix = "refs/keep-around/"+const KeepAroundPrefix = "refs/keep-around/review/"func KeepAroundRef(oid string) string// NewPackKey returns a new owned pack key for a pack of the repository with thediff --git a/doc/graft/STORAGE.md b/doc/graft/STORAGE.mdindex 1f09ba9..4eb2f42 100644--- a/doc/graft/STORAGE.md+++ b/doc/graft/STORAGE.md@@ -91,7 +91,7 @@ design is [PLAN6.md](PLAN6.md).database transaction as the reference change, and reviews are stored in thedatabase. A watcher on another node sees new entries by polling every 250 ms.- **Review commits are kept by references.** `CreateReview` and `UpdateReview`- write a hidden reference `refs/keep-around/<commit>` for the head and base of+ write a hidden reference `refs/keep-around/review/<commit>` for the head and base ofthe review before they record it, so an open review keeps its commits aliveafter their branch is deleted: on every node (housekeeping keeps whatreferences reach), in checkpoints and through garbage collection. A review of@@ -105,7 +105,13 @@ design is [PLAN6.md](PLAN6.md).written as well (housekeeping keeps the commits), but nothing enforces them:the review files are the source of truth there. Each review creation and eachhead move costs one reference transaction, so one changeset and no changelog- entry.+ entry. GitLab's own `refs/keep-around/<commit>` references are never touched.+ The daily housekeeping of every repository (on by default) also writes the+ references that open reviews lack and deletes those no open review uses;+ watch `graft_review_roots_sweep_failures_total` and+ `graft_review_roots_missing_total`. A review written by an older version+ whose commit was lost since can still be read and closed, but `UpdateReview`+ refuses it with `FailedPrecondition` naming the review and the commit.- **Repository IDs.** A repository is identified by its storage name andrelative path, and gets an ID that is never reused: a repository removed andcreated again is a new repository, and the old caches are recognized asdiff --git a/internal/cli/gitaly/serve.go b/internal/cli/gitaly/serve.goindex 6e2599c..0d6abde 100644--- a/internal/cli/gitaly/serve.go+++ b/internal/cli/gitaly/serve.go@@ -38,6 +38,7 @@ import ("github.com/feder-cr/git-ai/internal/graft/corestore""github.com/feder-cr/git-ai/internal/graft/corestore/gate"corestoreopen "github.com/feder-cr/git-ai/internal/graft/corestore/open"+ "github.com/feder-cr/git-ai/internal/graft/corestore/reviewroots"corestoresyncer "github.com/feder-cr/git-ai/internal/graft/corestore/syncer""github.com/feder-cr/git-ai/internal/graft/corestore/voter""github.com/feder-cr/git-ai/internal/graft/derived"@@ -604,6 +605,9 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {return fmt.Errorf("graft storage core: %w", err)}defer func() { _ = storageCore.Close() }()+ // The review roots are written and swept in every mode, also without the+ // storage core.+ prometheus.MustRegister(reviewroots.DefaultSweeper)hookManagerOpts := []hook.ManagerOption{hook.WithEagerObserver(eagerObserver)}var (@@ -839,7 +843,14 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {ctx,logger,maintenance.DailyOptimizationWorker(cfg, maintenance.OptimizerFunc(func(ctx context.Context, logger log.Logger, repo storage.Repository) error {- return housekeepingManager.OptimizeRepository(ctx, localrepo.New(logger, locator, gitCmdFactory, catfileCache, repo))+ local := localrepo.New(logger, locator, gitCmdFactory, catfileCache, repo)+ if err := housekeepingManager.OptimizeRepository(ctx, local); err != nil {+ return err+ }+ // Graft: the backstop of the review roots, in every mode, where the+ // housekeeping that prunes unreferenced objects runs+ // (doc/graft/PLAN7.md, REVIEW ROOTS RULE).+ return housekeepReviewRoots(ctx, logger, storageCore, storageSyncer, local, repo)})),)if err != nil {@@ -884,6 +895,32 @@ func run(appCtx *cli.Command, cfg config.Cfg, logger log.Logger) error {// newPackObjectsAdaptiveLimit builds the adaptive limit backing a pack objects limiter. The limit// only moves when the configuration asks for an adaptive one, and is otherwise pinned to the// configured concurrency.+// housekeepReviewRoots runs the backstop of the review roots after the daily+// housekeeping of a repository: it writes the keep-around references that+// open reviews lack and deletes those no open review uses (in authoritative+// mode after catching the repository up). Open reviews whose commits are lost+// are logged with their IDs and commits; the counts are metrics of+// reviewroots.DefaultSweeper.+func housekeepReviewRoots(ctx context.Context, logger log.Logger, core *corestore.Core, fresh corestore.Freshener, local *localrepo.Repo, repo storage.Repository) error {+ backfill, sweep, err := reviewroots.DefaultSweeper.Housekeep(ctx, core, fresh, local, repo)+ for _, missing := range backfill.Missing {+ logger.WithFields(log.Fields{+ "review_id": missing.ReviewID,+ "commit_id": missing.Commit,+ }).Warn("review roots: a commit of an open review is not in the repository; the review can only be closed")+ }+ if backfill.Protected > 0 || sweep.Deleted > 0 {+ logger.WithFields(log.Fields{+ "backfilled": backfill.Protected,+ "deleted": sweep.Deleted,+ }).Info("review roots: housekeeping")+ }+ if err != nil {+ return fmt.Errorf("review roots: %w", err)+ }+ return nil+}+func newPackObjectsAdaptiveLimit(name string, limits config.ConcurrencyLimits) *limiter.AdaptiveLimit {if !limits.Adaptive {return limiter.NewAdaptiveLimit(name, limiter.AdaptiveSetting{diff --git a/internal/gitaly/service/review/review.go b/internal/gitaly/service/review/review.goindex 7d82505..65e99ec 100644--- a/internal/gitaly/service/review/review.go+++ b/internal/gitaly/service/review/review.go@@ -82,7 +82,7 @@ func (s *server) CreateReview(ctx context.Context, req *gitalypb.CreateReviewReqvar stored *gitalypb.Reviewerr = s.writeRoots(ctx, repo, st, func(st store.ReviewStore, protect rootsProtect) error {- if err := protect(head, base); err != nil {+ if err := protect("", head, base); err != nil {return err}@@ -231,7 +231,14 @@ func moveHead(ctx context.Context, repo *localrepo.Repo, req *gitalypb.UpdateRevbase := git.ObjectID(r.GetBaseCommitId())if head.String() == r.GetHeadCommitId() {- return protect(reviewCommits(head, base)...)+ return protect(r.GetId(), reviewCommits(head, base)...)+ }+ // The base is a root of the open review whatever this update does; a base+ // that is gone is reported as such, not as a head that does not descend.+ if base != "" {+ if err := protect(r.GetId(), base); err != nil {+ return err+ }}descends, err := repo.IsAncestor(ctx, base.Revision(), head.Revision())@@ -268,7 +275,7 @@ func moveHead(ctx context.Context, repo *localrepo.Repo, req *gitalypb.UpdateRevv.Current = survives}- if err := protect(reviewCommits(head, base)...); err != nil {+ if err := protect(r.GetId(), reviewCommits(head, base)...); err != nil {return err}diff --git a/internal/gitaly/service/review/roots.go b/internal/gitaly/service/review/roots.goindex 9e9bcca..a6edf5f 100644--- a/internal/gitaly/service/review/roots.go+++ b/internal/gitaly/service/review/roots.go@@ -23,8 +23,9 @@ type requiredRootsStore interface {}// rootsProtect writes the keep-around references of commits that a review-// write is about to record.-type rootsProtect func(commits ...git.ObjectID) error+// write is about to record; reviewID names the review in an error, empty for a+// review that is not created yet.+type rootsProtect func(reviewID string, commits ...git.ObjectID) error// writeRoots runs write, a review write that may record new head or base// commits, under the REVIEW ROOTS RULE (doc/graft/PLAN7.md). write calls@@ -38,8 +39,18 @@ type rootsProtect func(commits ...git.ObjectID) error// writes the reference again), at most maxRootsAttempts times in all, after// which the request is Aborted.func (s *server) writeRoots(ctx context.Context, repo *localrepo.Repo, st store.ReviewStore, write func(store.ReviewStore, rootsProtect) error) error {- protect := func(commits ...git.ObjectID) error {+ protect := func(reviewID string, commits ...git.ObjectID) error {if err := reviewroots.Protect(ctx, repo, commits); err != nil {+ var missing *reviewroots.MissingCommitsError+ if errors.As(err, &missing) {+ // A review written before its commits were kept by references,+ // whose commit was lost since: nothing in the storage core kept it+ // live either, so it cannot be brought back (doc/graft/PLAN7.md,+ // REVIEW ROOTS RULE). The backfill reports the same commit.+ return structerr.NewFailedPrecondition("commits %v of review %q are not in the repository any more; the review can only be closed", missing.Commits, reviewID).+ WithMetadata("review_id", reviewID).+ WithMetadata("commit_id", missing.Commits)+ }return structerr.NewInternal("protect the commits of the review: %w", err)}if s.rootsProtected != nil {@@ -65,12 +76,13 @@ func (s *server) writeRoots(ctx context.Context, repo *localrepo.Repo, st store.// releaseRoots deletes the keep-around references of the commits of a review// that was just closed, unless another open review uses them. The close is-// already recorded: a reference that cannot be deleted now only keeps its-// objects alive longer, until the next close of a review of the same commit or-// the maintenance sweep of the storage core.+// already recorded, so a failure is not the request's: it is logged and counted+// (graft_review_roots_sweep_failures_total{trigger="close"}), and the reference+// only keeps its objects alive until the housekeeping backstop deletes it+// (reviewroots.Sweeper.Housekeep, which runs in every mode).func (s *server) releaseRoots(ctx context.Context, repo *localrepo.Repo, st store.ReviewStore, commits []git.ObjectID) {- if _, err := reviewroots.Sweep(ctx, repo, st, commits); err != nil {+ if _, err := reviewroots.DefaultSweeper.Sweep(ctx, repo, st, commits, reviewroots.TriggerClose); err != nil {s.deps.GetLogger().WithError(err).WithField("commits", commits).WarnContext(ctx,- "review: the keep-around references of a closed review were not deleted; they keep their objects until a later sweep")+ "review: the keep-around references of a closed review were not deleted; the housekeeping backstop deletes them")}}diff --git a/internal/gitaly/service/review/roots_test.go b/internal/gitaly/service/review/roots_test.goindex 23c4e6b..95fbf04 100644--- a/internal/gitaly/service/review/roots_test.go+++ b/internal/gitaly/service/review/roots_test.go@@ -232,6 +232,46 @@ func TestAuthoritativeReviewWriteSurvivesASweep(t *testing.T) {}}+// TestUpdateOfAReviewWhoseCommitIsLost: a review written before the REVIEW+// ROOTS RULE whose base commit is gone cannot be kept any more. UpdateReview+// refuses it with FailedPrecondition naming the review and the commit (the+// same pair the housekeeping backfill reports), never Internal; the review can+// still be read and closed.+func TestUpdateOfAReviewWhoseCommitIsLost(t *testing.T) {+ ctx, cfg, core, client := setupAuthoritative(t)++ repo, repoPath := createLocalRepository(t, ctx, cfg)+ base := gittest.WriteCommit(t, cfg, repoPath, gittest.WithMessage("base"), gittest.WithBranch("main"))+ head := gittest.WriteCommit(t, cfg, repoPath, gittest.WithMessage("head"), gittest.WithParents(base), gittest.WithBranch("feature"))+ lost := strings.Repeat("c", gittest.DefaultObjectHash.EncodedLen())+ importRepository(t, cfg, core, repo, repoPath, coretest.ImportOptions{+ Reviews: []*gitalypb.Review{{+ Id: "7",+ TargetBranch: []byte("main"),+ HeadCommitId: head.String(),+ BaseCommitId: lost,+ Author: anonymousActor,+ }},+ })+ // The first request lets the gate replace the imported copy by a cache built+ // from the storage core.+ require.Len(t, listStoredReviews(t, ctx, client, repo, true), 1)+ next := gittest.WriteCommit(t, cfg, repoPath, gittest.WithMessage("next"), gittest.WithParents(head))++ for _, moveTo := range []git.ObjectID{head, next} {+ _, err := client.UpdateReview(ctx, &gitalypb.UpdateReviewRequest{Repository: repo, Id: "7", HeadCommitId: moveTo.String()})+ require.Equal(t, codes.FailedPrecondition, status.Code(err), "%v", err)+ require.ErrorContains(t, err, lost)+ require.ErrorContains(t, err, `review "7"`)+ }++ got, err := client.GetReview(ctx, &gitalypb.GetReviewRequest{Repository: repo, Id: "7"})+ require.NoError(t, err)+ require.Equal(t, head.String(), got.GetReview().GetHeadCommitId())+ _, err = client.CloseReview(ctx, &gitalypb.CloseReviewRequest{Repository: repo, Id: "7"})+ require.NoError(t, err)+}+// TestReviewRootsOutsideAuthoritativeMode: with the storage core off or in// shadow mode the reviews are files, and the service still writes the// keep-around references: housekeeping keeps an open review's commit after itsdiff --git a/internal/graft/corestore/reviewroots/reviewroots.go b/internal/graft/corestore/reviewroots/reviewroots.goindex d0519a1..4bb17eb 100644--- a/internal/graft/corestore/reviewroots/reviewroots.go+++ b/internal/graft/corestore/reviewroots/reviewroots.go@@ -29,10 +29,23 @@ import ("github.com/feder-cr/git-ai/internal/graft/review/store")+// MissingCommitsError is the error of Protect for commits that are not in the+// repository: a commit can only be kept while it exists. It happens for reviews+// written before the REVIEW ROOTS RULE whose commit was lost since; the storage+// core cannot bring it back either (nothing kept it live).+type MissingCommitsError struct {+ Commits []git.ObjectID+}++func (e *MissingCommitsError) Error() string {+ return fmt.Sprintf("commits %v are not in the repository", e.Commits)+}+// Protect writes the keep-around reference of every commit that does not have-// it, in one reference transaction. The commits must be in the repository. A-// commit whose reference exists with its value is left alone, so Protect of-// protected commits writes nothing.+// it, in one reference transaction. A commit that is not in the repository is a+// *MissingCommitsError and nothing is written. A commit whose reference exists+// with its value is left alone, so Protect of protected commits writes+// nothing.func Protect(ctx context.Context, repo *localrepo.Repo, commits []git.ObjectID) error {names := make([]string, 0, len(commits))byName := make(map[string]git.ObjectID, len(commits))@@ -63,6 +76,18 @@ func Protect(ctx context.Context, repo *localrepo.Repo, commits []git.ObjectID)return nil}+ unprotected := make([]git.ObjectID, 0, len(missing))+ for _, name := range missing {+ unprotected = append(unprotected, byName[name])+ }+ absent, err := gitops.MissingObjects(ctx, repo, unprotected)+ if err != nil {+ return fmt.Errorf("look up the commits: %w", err)+ }+ if len(absent) > 0 {+ return &MissingCommitsError{Commits: absent}+ }+updater, err := updateref.New(ctx, repo)if err != nil {return fmt.Errorf("start the keep-around update: %w", err)@@ -212,9 +237,17 @@ type BackfillReport struct {// Protected is the number of commits whose keep-around reference was// written.Protected int- // Missing are the IDs of the open reviews with a commit that is not in the- // repository (or not an object ID): it cannot be protected any more.- Missing []string+ // Missing are the commits of open reviews that are not in the repository+ // (or not object IDs): they cannot be protected any more. Such a review+ // can still be read and closed; UpdateReview refuses it with+ // FailedPrecondition naming the same review and commit.+ Missing []MissingRoot+}++// MissingRoot is a commit of an open review that the repository does not have.+type MissingRoot struct {+ ReviewID string+ Commit string}// Backfill writes the keep-around references of the open reviews that do not@@ -240,7 +273,7 @@ func Backfill(ctx context.Context, repo *localrepo.Repo, reviews store.ReviewStocontinue}if hash.ValidateHex(commit) != nil {- report.Missing = appendOnce(report.Missing, r.GetId())+ report.Missing = append(report.Missing, MissingRoot{ReviewID: r.GetId(), Commit: commit})continue}oid := git.ObjectID(commit)@@ -271,7 +304,7 @@ func Backfill(ctx context.Context, repo *localrepo.Repo, reviews store.ReviewStofor _, oid := range missing {absent[oid] = struct{}{}for _, id := range byCommit[oid] {- report.Missing = appendOnce(report.Missing, id)+ report.Missing = append(report.Missing, MissingRoot{ReviewID: id, Commit: oid.String()})}}@@ -285,7 +318,10 @@ func Backfill(ctx context.Context, repo *localrepo.Repo, reviews store.ReviewStoreturn report, err}report.Protected = len(present)- sort.Strings(report.Missing)+ sort.Slice(report.Missing, func(i, j int) bool {+ a, b := report.Missing[i], report.Missing[j]+ return a.ReviewID < b.ReviewID || (a.ReviewID == b.ReviewID && a.Commit < b.Commit)+ })return report, nil}@@ -305,12 +341,3 @@ func openRoots(reviews store.ReviewStore) (map[string]struct{}, error) {}return used, nil}--func appendOnce(ids []string, id string) []string {- for _, existing := range ids {- if existing == id {- return ids- }- }- return append(ids, id)-}diff --git a/internal/graft/corestore/reviewroots/reviewroots_test.go b/internal/graft/corestore/reviewroots/reviewroots_test.goindex 4dd6a97..6126d64 100644--- a/internal/graft/corestore/reviewroots/reviewroots_test.go+++ b/internal/graft/corestore/reviewroots/reviewroots_test.go@@ -109,9 +109,14 @@ func TestProtect(t *testing.T) {require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c3}))require.Equal(t, keepAroundOf(c1, c2, c3), f.keepArounds())- // A commit that is not in the repository cannot be protected.+ // A commit that is not in the repository cannot be protected; nothing is+ // written, also for the commits that are present.absent := git.ObjectID(strings.Repeat("a", gittest.DefaultObjectHash.EncodedLen()))- require.Error(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{absent}))+ c4 := f.write("c4", c3)+ err := reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c4, absent})+ var missing *reviewroots.MissingCommitsError+ require.ErrorAs(t, err, &missing)+ require.Equal(t, []git.ObjectID{absent}, missing.Commits)require.Equal(t, keepAroundOf(c1, c2, c3), f.keepArounds())}@@ -127,6 +132,16 @@ func TestSweep(t *testing.T) {f.review(c2.String(), c0.String(), gitalypb.Review_STATE_CLOSED)require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c0, c1, c2, c3}))+ // GitLab's own keep-around references are not review roots: no sweep+ // touches them.+ gitlab := git.ReferenceName("refs/keep-around/" + c2.String())+ gittest.WriteRef(t, testcfg.Build(t), f.path, gitlab, c2)+ defer func() {+ refs, err := gitops.ReadRefs(f.ctx, f.repo)+ require.NoError(t, err)+ require.Equal(t, corestore.Target(c2), refs[gitlab.String()], "GitLab's keep-around reference is kept")+ }()+// Candidates only: the open review's head is kept, the closed review's goes.report, err := reviewroots.Sweep(f.ctx, f.repo, f.reviews, []git.ObjectID{c1, c2})require.NoError(t, err)@@ -163,7 +178,10 @@ func TestBackfill(t *testing.T) {report, err := reviewroots.Backfill(f.ctx, f.repo, f.reviews)require.NoError(t, err)require.Equal(t, 1, report.Protected, "the open review's head; its base had its reference")- require.Equal(t, []string{lost.GetId(), garbled.GetId()}, report.Missing)+ require.Equal(t, []reviewroots.MissingRoot{+ {ReviewID: lost.GetId(), Commit: absent},+ {ReviewID: garbled.GetId(), Commit: "not an object ID"},+ }, report.Missing)require.Equal(t, keepAroundOf(c0, c1), f.keepArounds(), "closed reviews are not backfilled")// A second backfill has nothing left to protect.diff --git a/internal/graft/corestore/reviewroots/sweeper.go b/internal/graft/corestore/reviewroots/sweeper.gonew file mode 100644index 0000000..dd974b1--- /dev/null+++ b/internal/graft/corestore/reviewroots/sweeper.go@@ -0,0 +1,155 @@+package reviewroots++import (+ "context"+ "errors"+ "fmt"++ "github.com/prometheus/client_golang/prometheus"++ "github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/localrepo"+ "github.com/feder-cr/git-ai/internal/gitaly/storage"+ "github.com/feder-cr/git-ai/internal/graft/corestore"+ "github.com/feder-cr/git-ai/internal/graft/corestore/dbreview"+ "github.com/feder-cr/git-ai/internal/graft/review/store"+)++// Triggers of a sweep, the label of the sweep metrics.+const (+ // TriggerClose is the sweep of the commits of a review that was just closed.+ TriggerClose = "close"+ // TriggerHousekeeping is the backstop that runs with the object+ // housekeeping of a repository (Housekeep).+ TriggerHousekeeping = "housekeeping"+)++// Sweeper runs the sweeps and the backfill of the review roots and counts+// them, so that a failure that its caller does not return (the sweep of a+// close, which is recorded already) is visible, and a persistent one shows as+// a growing counter.+type Sweeper struct {+ deleted prometheus.Counter+ protected prometheus.Counter+ missing prometheus.Counter+ failures *prometheus.CounterVec+}++var _ prometheus.Collector = (*Sweeper)(nil)++// DefaultSweeper is the sweeper of the process; gitaly serve registers it.+var DefaultSweeper = NewSweeper()++// NewSweeper returns a sweeper with its own counters.+func NewSweeper() *Sweeper {+ s := &Sweeper{+ deleted: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "graft_review_roots_deleted_total",+ Help: "Number of keep-around references of review commits deleted because no open review uses them.",+ }),+ protected: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "graft_review_roots_backfilled_total",+ Help: "Number of keep-around references of open-review commits written by the housekeeping backfill.",+ }),+ missing: prometheus.NewCounter(prometheus.CounterOpts{+ Name: "graft_review_roots_missing_total",+ Help: "Number of open-review commits found missing from a repository by the housekeeping backfill.",+ }),+ failures: prometheus.NewCounterVec(prometheus.CounterOpts{+ Name: "graft_review_roots_sweep_failures_total",+ Help: "Number of sweeps of review keep-around references that failed, by trigger (close, housekeeping).",+ }, []string{"trigger"}),+ }+ for _, trigger := range []string{TriggerClose, TriggerHousekeeping} {+ s.failures.WithLabelValues(trigger)+ }+ return s+}++// Describe implements prometheus.Collector.+func (s *Sweeper) Describe(descs chan<- *prometheus.Desc) {+ s.deleted.Describe(descs)+ s.protected.Describe(descs)+ s.missing.Describe(descs)+ s.failures.Describe(descs)+}++// Collect implements prometheus.Collector.+func (s *Sweeper) Collect(metrics chan<- prometheus.Metric) {+ s.deleted.Collect(metrics)+ s.protected.Collect(metrics)+ s.missing.Collect(metrics)+ s.failures.Collect(metrics)+}++// Sweep runs Sweep and counts it under trigger.+func (s *Sweeper) Sweep(ctx context.Context, repo *localrepo.Repo, reviews store.ReviewStore, candidates []git.ObjectID, trigger string) (SweepReport, error) {+ report, err := Sweep(ctx, repo, reviews, candidates)+ s.deleted.Add(float64(report.Deleted))+ if err != nil {+ s.failures.WithLabelValues(trigger).Inc()+ }+ return report, err+}++// Housekeep is the backstop of the review roots, run with the object+// housekeeping of the repository (gitaly's daily maintenance) in every mode:+// it writes the keep-around references that open reviews lack (Backfill) and+// deletes those that no open review uses (Sweep of every keep-around+// reference), so that a reference a close could not delete, or one written+// before a crash, does not keep its objects forever. In authoritative mode the+// reviews are the database's and the local repository is caught up first (fresh+// is the node's Freshener); an untracked repository, or one whose import is+// not complete, has no reviews in the database and is skipped. In off and+// shadow mode (core nil or not authoritative) the reviews are the files of the+// repository.+func (s *Sweeper) Housekeep(ctx context.Context, core *corestore.Core, fresh corestore.Freshener, repo *localrepo.Repo, repoProto storage.Repository) (BackfillReport, SweepReport, error) {+ reviews, ok, err := housekeepingStore(ctx, core, fresh, repo, repoProto)+ if err != nil {+ s.failures.WithLabelValues(TriggerHousekeeping).Inc()+ return BackfillReport{}, SweepReport{}, err+ }+ if !ok {+ return BackfillReport{}, SweepReport{}, nil+ }++ backfill, err := Backfill(ctx, repo, reviews)+ s.protected.Add(float64(backfill.Protected))+ s.missing.Add(float64(len(backfill.Missing)))+ if err != nil {+ s.failures.WithLabelValues(TriggerHousekeeping).Inc()+ return backfill, SweepReport{}, fmt.Errorf("backfill: %w", err)+ }++ sweep, err := s.Sweep(ctx, repo, reviews, nil, TriggerHousekeeping)+ if err != nil {+ return backfill, sweep, fmt.Errorf("sweep: %w", err)+ }+ return backfill, sweep, nil+}++// housekeepingStore returns the review store of the repository for Housekeep;+// ok is false when the repository has none to keep.+func housekeepingStore(ctx context.Context, core *corestore.Core, fresh corestore.Freshener, repo *localrepo.Repo, repoProto storage.Repository) (store.ReviewStore, bool, error) {+ if !core.Authoritative() {+ path, err := repo.Path(ctx)+ if err != nil {+ return nil, false, fmt.Errorf("repository path: %w", err)+ }+ return store.Open(path), true, nil+ }++ record, err := core.Meta.GetRepository(ctx, corestore.KeyOf(repoProto))+ switch {+ case errors.Is(err, corestore.ErrRepositoryNotFound):+ return nil, false, nil+ case err != nil:+ return nil, false, fmt.Errorf("storage core: %w", err)+ case record.HeadSeq == 0:+ return nil, false, nil+ }+ if err := fresh.EnsureFresh(ctx, repoProto); err != nil {+ return nil, false, fmt.Errorf("catch up: %w", err)+ }+ return dbreview.New(ctx, core.Meta, record.ID), true, nil+}diff --git a/internal/graft/corestore/reviewroots/sweeper_test.go b/internal/graft/corestore/reviewroots/sweeper_test.gonew file mode 100644index 0000000..93f0975--- /dev/null+++ b/internal/graft/corestore/reviewroots/sweeper_test.go@@ -0,0 +1,126 @@+package reviewroots_test++import (+ "errors"+ "testing"++ "github.com/prometheus/client_golang/prometheus"+ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/graft/corestore"+ "github.com/feder-cr/git-ai/internal/graft/corestore/coretest"+ "github.com/feder-cr/git-ai/internal/graft/corestore/reviewroots"+ "github.com/feder-cr/git-ai/internal/graft/review/store"+ "github.com/feder-cr/git-ai/proto/go/gitalypb"+)++// counter returns the value of the counter family name of the sweeper (with+// the trigger label when trigger is not empty).+func counter(t *testing.T, s *reviewroots.Sweeper, name, trigger string) float64 {+ t.Helper()++ reg := prometheus.NewPedanticRegistry()+ require.NoError(t, reg.Register(s))+ families, err := reg.Gather()+ require.NoError(t, err)+ for _, family := range families {+ if family.GetName() != name {+ continue+ }+ for _, metric := range family.GetMetric() {+ if trigger == "" {+ return metric.GetCounter().GetValue()+ }+ for _, label := range metric.GetLabel() {+ if label.GetName() == "trigger" && label.GetValue() == trigger {+ return metric.GetCounter().GetValue()+ }+ }+ }+ }+ return 0+}++// brokenStore is a review store whose listing fails.+type brokenStore struct{ store.ReviewStore }++func (brokenStore) List(string, bool, int) ([]*gitalypb.Review, error) {+ return nil, errors.New("the review files cannot be read")+}++func TestSweeperCounts(t *testing.T) {+ t.Parallel()++ f := newFixture(t)+ c0 := f.write("c0")+ require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c0}))++ s := reviewroots.NewSweeper()+ _, err := s.Sweep(f.ctx, f.repo, brokenStore{}, nil, reviewroots.TriggerClose)+ require.Error(t, err)+ require.Equal(t, float64(1), counter(t, s, "graft_review_roots_sweep_failures_total", reviewroots.TriggerClose))+ require.Zero(t, counter(t, s, "graft_review_roots_sweep_failures_total", reviewroots.TriggerHousekeeping))++ report, err := s.Sweep(f.ctx, f.repo, f.reviews, nil, reviewroots.TriggerClose)+ require.NoError(t, err)+ require.Equal(t, 1, report.Deleted)+ require.Equal(t, float64(1), counter(t, s, "graft_review_roots_deleted_total", ""))+ require.Equal(t, float64(1), counter(t, s, "graft_review_roots_sweep_failures_total", reviewroots.TriggerClose))+}++// TestHousekeepWithoutTheStorageCore is the backstop in off and shadow mode:+// the reviews are the files of the repository; the references open reviews+// lack are written, and a leaked one (a close whose sweep failed) is deleted.+func TestHousekeepWithoutTheStorageCore(t *testing.T) {+ t.Parallel()++ for _, mode := range []corestore.Mode{corestore.ModeOff, corestore.ModeShadow} {+ t.Run(string(mode), func(t *testing.T) {+ t.Parallel()++ f := newFixture(t)+ c0 := f.write("c0")+ c1 := f.write("c1", c0)+ leaked := f.write("leaked", c0)+ f.review(c1.String(), c0.String(), gitalypb.Review_STATE_OPEN_UNSPECIFIED)+ require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{leaked}))++ var core *corestore.Core+ if mode == corestore.ModeShadow {+ core = coretest.Open(t, coretest.Options{Mode: corestore.ModeShadow})+ }++ s := reviewroots.NewSweeper()+ backfill, sweep, err := s.Housekeep(f.ctx, core, corestore.NopFreshener{}, f.repo, f.repo)+ require.NoError(t, err)+ require.Equal(t, 2, backfill.Protected)+ require.Empty(t, backfill.Missing)+ require.Equal(t, reviewroots.SweepReport{Deleted: 1, Kept: 2}, sweep)+ require.Equal(t, keepAroundOf(c0, c1), f.keepArounds())++ require.Equal(t, float64(2), counter(t, s, "graft_review_roots_backfilled_total", ""))+ require.Equal(t, float64(1), counter(t, s, "graft_review_roots_deleted_total", ""))+ require.Zero(t, counter(t, s, "graft_review_roots_sweep_failures_total", reviewroots.TriggerHousekeeping))+ })+ }+}++// TestHousekeepSkipsUntrackedRepositories: in authoritative mode the reviews+// are the database's; a repository the database does not track has none, and+// its local references are left alone.+func TestHousekeepSkipsUntrackedRepositories(t *testing.T) {+ t.Parallel()++ f := newFixture(t)+ c0 := f.write("c0")+ require.NoError(t, reviewroots.Protect(f.ctx, f.repo, []git.ObjectID{c0}))++ core := coretest.Open(t, coretest.Options{Mode: corestore.ModeAuthoritative})+ s := reviewroots.NewSweeper()+ backfill, sweep, err := s.Housekeep(f.ctx, core, corestore.NopFreshener{}, f.repo, f.repo)+ require.NoError(t, err)+ require.Equal(t, reviewroots.BackfillReport{}, backfill)+ require.Equal(t, reviewroots.SweepReport{}, sweep)+ require.Equal(t, keepAroundOf(c0), f.keepArounds())+}diff --git a/internal/graft/corestore/sqlstore/reviewroots_test.go b/internal/graft/corestore/sqlstore/reviewroots_test.goindex fcc590e..9d44bc3 100644--- a/internal/graft/corestore/sqlstore/reviewroots_test.go+++ b/internal/graft/corestore/sqlstore/reviewroots_test.go@@ -129,6 +129,14 @@ func TestOpenReviewRootsCannotBeReleased(t *testing.T) {_, err = commit(ctx, s, repo.ID, update(corestore.KeepAroundRef(string(closed)), closed, ""))require.NoError(t, err)+ // GitLab's own keep-around reference of the same commit is not a review+ // root: it is not guarded.+ gitlab := "refs/keep-around/" + string(head)+ _, err = commit(ctx, s, repo.ID, create(gitlab, head))+ require.NoError(t, err)+ _, err = commit(ctx, s, repo.ID, update(gitlab, head, ""))+ require.NoError(t, err)+// Shadow mode records what git did: forced changesets are not refused._, err = s.CommitChangeset(ctx, corestore.ChangesetInput{RepoID: repo.ID,diff --git a/internal/graft/corestore/types.go b/internal/graft/corestore/types.goindex 8951076..5723064 100644--- a/internal/graft/corestore/types.go+++ b/internal/graft/corestore/types.go@@ -328,10 +328,13 @@ func parsePackKey(key string) (repoID int64, sum [32]byte, owned, ok bool) {}// KeepAroundPrefix is the prefix of the references that keep the commits of-// open reviews alive: Gitaly hides it from clients (it is neither advertised-// nor writable over the git protocols), housekeeping and garbage collection-// respect it like every reference (doc/graft/PLAN7.md, REVIEW ROOTS RULE).-const KeepAroundPrefix = "refs/keep-around/"+// open reviews alive (doc/graft/PLAN7.md, REVIEW ROOTS RULE). It is under+// refs/keep-around/, which Gitaly hides from clients (neither advertised nor+// writable over the git protocols) and which housekeeping and garbage+// collection respect like every reference. It is a namespace of its own:+// GitLab writes refs/keep-around/<oid> for its own purposes, and the sweeps of+// the review roots must never delete those.+const KeepAroundPrefix = "refs/keep-around/review/"// KeepAroundRef returns the keep-around reference of a commit; its target is// the commit itself.