Git designed for AI agents.
docs(storage): amend PLAN7 after the PM review (pack ownership, lagging GC boundary)
Federico Eliabe50089d313f
diff --git a/doc/graft/PLAN7.md b/doc/graft/PLAN7.mdindex ef00f78..353ec23 100644--- a/doc/graft/PLAN7.md+++ b/doc/graft/PLAN7.md@@ -1,12 +1,15 @@# Graft phase 7: pack garbage collection and checkpoint compaction (binding plan)-Status: binding contract for WP71–WP79 (storage core milestone 3). It extends-the phase-6 foundation (`doc/graft/PLAN6.md`, which stays binding: the CAS-semantics, the write path, the read path and catch-up, the REFTABLE, IMPORT,-STALE CACHE and LOCK AND HISTORY rules, and RULES 1–15 all still apply). This-plan adds tables, interface methods and packages; it never edits an applied-migration and never changes a phase-6 exported signature except where a rule-below says so.+Status: binding contract for WP71–WP79 (storage core milestone 3). First+version 85189d4; **amended 2026-10-18 after the PM review** (changes A–H, the+answers to the open questions, and the "No patches: refactor the cause" section+of PM.md). It extends the phase-6 foundation (`doc/graft/PLAN6.md`, which stays+binding: the CAS semantics, the write path, the read path and catch-up, the+REFTABLE, IMPORT, STALE CACHE and LOCK AND HISTORY rules, and RULES 1–15). This+plan adds tables, interface methods and packages, and **refactors two phase-6+contracts** (the pack key format and `BlobStore.Put`, see the PACK OWNERSHIP+RULE), authorized by the PM provided every phase-6 storage test is green+afterwards. It never edits a released migration.Owner workstream: **m3** (`internal/graft/corestore/**`,`internal/gitaly/service/storagecore/**`, the storage sub-commands of@@ -15,543 +18,694 @@ Owner workstream: **m3** (`internal/graft/corestore/**`,## Problem-`doc/graft/STORAGE.md`, "Limits", states two gaps left after the first-release:--1. **Packs are never deleted.** Every reference transaction uploads a- content-addressed pack before it commits (`doc/graft/PLAN6.md`, write path).- Nothing ever removes one. Three kinds accumulate forever in- `repos/<repo_id>/packs/`:- * **orphans** of writes that uploaded a pack and then did not commit the- changeset (a transaction aborted after the Preparing phase, a node that- died before the database commit — `graft_storage_local_commit_failures_total`- and every aborted Preparing pack);+`doc/graft/STORAGE.md`, "Limits", states two gaps left after the first release:++1. **Packs are never deleted.** Every reference transaction uploads a pack+ before it commits (PLAN6, write path), and nothing ever removes one:+ * **orphans** of writes that uploaded a pack and then did not commit (an+ aborted transaction after the Preparing phase, a node that died before the+ database commit);* **superseded** changeset packs, whose objects a later consolidated statealready contains;- * the packs of **removed repositories** (`RemoveRepository` deletes the- database rows through `ON DELETE CASCADE`, but the bucket keeps every pack- under the removed repository's prefix).+ * the packs of **removed repositories** (`RemoveRepository` cascades the+ database rows, the bucket keeps `repos/<repo_id>/`).2. **No checkpoint compaction.** Hydration (`syncer.hydrate`) and the dry- hydration of `graft storage verify --connectivity` replay **every** changeset- from sequence 1: they download and index every pack the repository ever had.- Catch-up of a node that fell behind does the same from its watermark. The- cost of a fresh node and of a connectivity check grows without bound with the- history of the repository.--The database is authoritative and `repo_id` is never reused, so both can be-solved without ever risking the correctness of a read or a write: a consolidated-**checkpoint** gives hydration a bounded starting point, and a crash-safe,-idempotent **mark and sweep** reclaims the packs that no reader needs, with the-database as the single source of truth over what is live.+ hydration of `graft storage verify --connectivity` replay every changeset+ from sequence 1. Their cost grows without bound with the history.++A third, latent problem was found in review and is the subject of the PACK+OWNERSHIP RULE: **bucket objects are silently shared between writes.** Pack keys+are content-addressed and `BlobStore.Put` returns success without writing when+the key exists, so a write can commit a changeset whose pack is an object it did+not write and whose liveness is decided by other rows. Any collector, however+careful, can then delete an object that a write in flight relies on. Phase 6 never+deletes, so the defect was harmless; phase 7 must remove its cause before it+deletes anything.## Goal and non-goals-Deliver, for the storage core in authoritative mode (and, for the parts that-make sense without a source of truth, in shadow mode — see the SHADOW SCOPE-rule):--* **Checkpoints.** Per repository, a checkpoint at sequence `N` = one- consolidated, content-addressed pack holding every object reachable from the- references as of `N`, plus the reference snapshot at `N`, recorded in the- database **atomically after the pack is uploaded**. Hydration and catch-up- start from the newest checkpoint `<= head` and replay only changesets `> N`.- Created by a background task on configurable thresholds and on demand- (`graft storage checkpoint` + RPC). Writes are never blocked while a- checkpoint is built.-* **Pack GC.** Mark and sweep, idempotent, crash-safe after every step, the- database authoritative. Reclaims (a) orphan packs, (b) changeset packs covered- by a kept checkpoint, (c) packs of removed repositories, and the superseded- checkpoint packs themselves. `graft storage gc` (+ `--dry-run`) and the- background task.+* **Pack ownership.** Every pack object in the bucket is written by, and belongs+ to, exactly one database row (a changeset or a checkpoint). No write relies on+ an object it did not upload.+* **Checkpoints.** Per repository, a checkpoint at sequence `N` = one consolidated+ pack holding every object reachable from the references at `N` and from the+ head and base commits of the reviews open at the time of the build, plus the+ reference snapshot at `N`, recorded atomically after the pack is uploaded.+ Hydration starts from the newest checkpoint and replays only changesets `> N`.+ Created on demand (`graft storage checkpoint` + RPC) and by the background task+ when it is enabled. Writes are never blocked while a checkpoint is built.+* **Pack GC.** Mark and sweep, database authoritative, idempotent, crash-safe+ after every step. Reclaims orphans, changeset packs covered by a checkpoint old+ enough that no reader can still need them, superseded checkpoint packs, and the+ packs of removed repositories. `graft storage gc` (+ `--dry-run`), `graft+ storage gc --removed`, and the background task.Out of scope (stated as limits or future work in STORAGE.md): per-directory-permissions; S3/GCS/Azure testing (no server here — file/mem buckets only);-renaming anything; **local-repository housekeeping** (no `git gc`/repack of a-node's cache — the cache is rebuildable from a checkpoint, so an operator-reclaims local disk by removing and re-hydrating a cache; we do not add local-maintenance here); changelog-row / changeset-metadata compaction (we delete-packs, never changeset or changelog rows, so `graft log` and the changelog-cursors are untouched).+permissions; S3/GCS/Azure testing (no server here, file and mem buckets only);+renaming anything; local-repository housekeeping (no `git gc` of a node's cache;+the cache can be rebuilt from a checkpoint); compaction of changeset, changelog or+review rows (only packs are deleted).## Design### Overview-The expensive, immutable state is the **packs** in the bucket. The cheap state-is the **changeset metadata** (`graft_changesets`, `graft_changeset_refs`) and-the **changelog** (`graft_changelog`). Phase 7 reclaims packs and never touches-the metadata or the changelog: every changeset row stays, so the WAL, the-changelog and their consumers' cursors are unaffected. A checkpoint supersedes-the *packs* of the changesets up to its sequence, not their rows.+The expensive, immutable state is the packs in the bucket. The cheap state is the+changeset metadata (`graft_changesets`, `graft_changeset_refs`) and the changelog+(`graft_changelog`). Phase 7 deletes packs and never touches those rows, so the+WAL, the changelog and their consumers' cursors are unaffected. A checkpoint+supersedes the *packs* of the changesets up to its sequence, not their rows.```-seq: 1 2 3 ... N (checkpoint) N+1 ... head-packs: p1 p2 p3 ... [reclaimable once checkpoint N exists]- C_N (consolidated) p_{N+1} ... p_head-rows: all changeset rows kept (WAL + changelog), forever+seq: 1 2 3 ... N (checkpoint) N+1 ... head+packs: p1 p2 p3 ... pN p_{N+1} ... p_head+ C_N (consolidated pack of the checkpoint)+rows: every changeset row is kept (WAL + changelog)```-A **checkpoint at N** is: a consolidated pack `C_N` (objects reachable from the-references at N) uploaded to the bucket, plus the reference snapshot at N, plus-a row in `graft_checkpoints`, committed in one transaction after `C_N` is-uploaded. Because `C_N` holds the full reachable closure at N, the individual-packs `p1..pN` are then redundant for reconstructing any state `>= N`.--* **Hydration** uses the newest checkpoint `<= head`: it indexes `C_N`, sets the- references to the snapshot at N, then replays changesets `(N, head]` exactly- as the phase-6 hydration replays `(0, head]`.-* **Catch-up** of a node whose watermark is `< N` cannot download the reclaimed- `p` packs, so it **re-hydrates** from the checkpoint. A node at `>= N` catches- up from its watermark as before (those packs are never reclaimed).-* **GC** keeps every pack that a hydration or catch-up still needs: the kept- checkpoint packs and every changeset pack with sequence `> C` (the newest kept- checkpoint's sequence). It deletes the rest, orphans only after a grace- period.+`p1..pN` are reclaimed only once the checkpoint `N` is old enough that no+hydration, catch-up or verification that could still read them is running (the+GC BOUNDARY rule). Every pack has exactly one owner row (the PACK OWNERSHIP+RULE), so "is this pack still needed" is a question about its owner row only.++### PACK OWNERSHIP RULE (change A, binding)++**Root cause.** The dedup race reported in review (a write's `Put` finds the key+and skips; GC deletes the key; the write commits a changeset referencing a missing+pack) is not a timing window to be narrowed. Its cause is structural: pack keys+are content-addressed and shared, `Put`-exists makes a write depend on an object+it did not write, and that object's liveness belongs to other rows (an older+covered changeset, or a failed write's orphan). The phase-6 orphan grace was a+time assumption, not an enforced invariant. Both are removed at their source.++**Decision: per-row ownership** (direction 1 of the PM review). Rejected:+database fencing of deletions with a doomed set (direction 2), see "Rejected+alternatives".++1. **Owned pack keys.** A pack key is+ `repos/<repo_id>/packs/<sha256>-<nonce>.pack`, where `<sha256>` is the SHA-256+ of the bytes (64 lowercase hex, still verified on every `Put` and `Get`, RULE+ 22) and `<nonce>` is 16 random bytes (`crypto/rand`, 32 lowercase hex) drawn+ for that one upload. Every writer (the voter, `ImportRepository`, the+ checkpoint builder) uploads under a key it just generated+ (`corestore.NewPackKey(repoID, sum)`). There is no cross-row dedup: two writes+ with byte-identical packs store two objects. Legacy keys+ `repos/<repo_id>/packs/<sha256>.pack`, written by v0.1.0 and still referenced+ by its rows, stay readable (`ParsePackKey` accepts both forms) and are never+ written again.+2. **`BlobStore.Put` is strictly create-only.** `Put` writes with+ `blob.WriterOptions{IfNotExist: true}`; an existing key is the new error+ `corestore.ErrBlobExists`, never success. The `Exists` pre-check is removed (one+ bucket round trip less per write). A fresh nonce never collides in practice; if+ it does, the writer fails rather than share an object. This is a phase-6+ contract change: the blobstore tests "Put idempotent" and "concurrent Puts of+ the same key" assert the old requirement and are rewritten to the new one+ (exactly one concurrent `Put` of a key succeeds, the others return+ `ErrBlobExists`, and the stored content is that of the winner).+3. **Database fence for writers that are not upgraded.** A v0.1.0 binary cannot+ start against a migrated database (sql-migrate v1.8.1 refuses an unknown+ applied migration, and the store does not set `IgnoreUnknown`), but a node+ already running v0.1.0 keeps writing with shared keys. Migration 2 therefore+ installs a trigger on `graft_changesets` that rejects an insert whose+ `pack_key` is not NULL and not an owned key+ (`pack_key NOT LIKE 'repos/%/packs/%-%.pack'`). A write from an old binary that+ carries a pack fails at its database commit (authoritative: git aborts it;+ shadow: counted as a shadow error). Writes without a pack are unaffected.+ Operational consequence, documented: upgrade every node together; until a node+ runs the new binary, its pack-carrying writes fail.+4. **In-flight writes have an enforced deadline.** Without sharing, the only+ object a write relies on is its own upload, which is an unreferenced object+ until the write commits. The writer and the collector are bound by one rule:+ * **Writer.** Let `W = gc_grace − gc_margin`. The writer takes the time `t_r`+ (monotonic clock) of the repository-record read (`GetRepository`) that+ precedes its upload. It uploads the pack and commits the row that references+ it under a context whose deadline is `t_r + W`. When the deadline passes, the+ `Put` is cancelled (gocloud discards a cancelled writer, nothing is stored) or+ the commit transaction is not started, and the write fails like any storage+ failure (authoritative: `Unavailable`, git aborts; shadow: counted, the git+ write proceeds). The voter's Prepared phase uses the pack of the Preparing+ phase only when that pack's deadline has not passed; otherwise it builds and+ uploads a new owned pack itself (the existing "Preparing did not prepare"+ path), with a new `t_r` from its own record read. `ImportRepository` and the+ checkpoint builder re-read the record immediately before their upload, so a+ long pack build does not consume the deadline.+ * **Collector.** An unreferenced pack is deleted only when the collector itself+ **observed** it unreferenced at least `gc_grace` earlier, measured on the+ database clock (`graft_gc_orphans.first_seen`, see the CLOCK RULE), and it is+ still unreferenced in the same database snapshot whose time decides the age.+ The bucket's modification time is never used.+5. **Proof (no live pack is deleted).** Let `t_s` be the database time of the+ snapshot in which the collector finds key `K` unreferenced with+ `first_seen(K) <= t_s − gc_grace`. `first_seen(K)` was recorded after a listing+ that returned `K`, so the upload of `K` completed before `first_seen(K)`, and+ the writer's record read `t_r` was earlier still. The writer can commit only+ before `t_r + W < first_seen(K) + gc_grace − gc_margin <= t_s − gc_margin`. A+ commit before `t_s` is visible in the snapshot, so a committed row would+ reference `K`. Since none does, no writer will ever commit `K`, and deleting it+ is safe. The only clock assumption is that the drift of the clock **rates**+ (the writer's monotonic clock against the database clock) over `gc_grace` is+ smaller than `gc_margin`; absolute skew between machines does not enter.+ `gc_margin` also absorbs the latency of a commit that is in flight when the+ writer's deadline passes (a commit message sent just before the deadline may+ still complete after it).+6. **Covered and superseded packs need no time.** A covered changeset pack (its+ owner row has `seq <= C_gc`) or the pack of a dropped checkpoint is owned by a+ committed row, so no writer relies on it (new writers never reuse a key, rule+ 1), and its readers are excluded by the GC BOUNDARY rule. It is deleted in the+ run that finds it.+7. **Cost on the write path.** No database access is added. The `Put` loses its+ `Exists` round trip; the nonce and the deadline are negligible; the trigger is+ one `LIKE` per inserted changeset; byte-identical packs are stored twice (rare:+ retried identical pushes). The preparing and prepared histograms and the bucket+ bytes are measured before and after in the WP79 benchmark window.++### CLOCK RULE (binding)++Every time that decides a GC action is a **database time**: `graft_checkpoints.created_at`,+`graft_gc_orphans.first_seen`, `graft_gc_removed.removed_at` and the lease times+are written with the database clock (Postgres `now()` in the statement; SQLite+`time.Now()` of the host, which every node shares because SQLite is single-host),+and every "older than" comparison is made against the database time read in the+same transaction (`now()` on Postgres; on SQLite `time.Now()` taken before the+read transaction begins, so that every commit before it is visible). The writer's+deadline is a duration on its own monotonic clock. The only assumption is the+bounded rate drift of RULE 5 above, covered by `gc_margin`.++### GC BOUNDARY rule (changes B and C, binding)++Two boundaries are computed from the checkpoints, on the database clock:++* **`C_gc`** = the sequence of the newest checkpoint created at least `gc_grace`+ ago (0 when there is none). The collector reclaims changeset packs with+ `seq <= C_gc` and the packs of checkpoints with `seq < C_gc`. It keeps every+ changeset pack with `seq > C_gc` and every checkpoint with `seq >= C_gc`+ (`checkpoint_retain` of the first version is removed: the lag makes it+ unnecessary).+* **`C_sync`** = the sequence of the newest checkpoint created at least+ `gc_grace − sync_timeout − gc_margin` ago (0 when there is none). A sync round+ (catch-up or hydration, bounded by `sync_timeout` through its context) decides+ with `C_sync`: **re-hydrate when the watermark is below `C_sync`; otherwise+ catch up** (change C: a node that is a few changesets behind when a checkpoint+ is created catches up normally; the checkpoint does not count for it until it+ is about `gc_grace − sync_timeout` old).++Why no reader ever misses a pack. A checkpoint becomes `C_gc` only `gc_grace`+after its creation. For a collector run during a sync round that starts at `t0`+and ends before `t0 + sync_timeout` to reclaim a pack the round reads, the+checkpoint that defines its `C_gc` must have been created before `t0 + sync_timeout+− gc_grace`, so it was already `gc_grace − sync_timeout` old at `t0` (up to the+rate drift covered by `gc_margin`), so its sequence is `<= C_sync`. A catch-up+with watermark `>= C_sync` reads only packs `> C_sync`, a hydration reads the+newest checkpoint (whose sequence is `>= C_sync >= C_gc`) and the packs after it.+Configuration validation requires `gc_grace > sync_timeout + gc_margin`.++Consequence: under the invariant a sync never meets a reclaimed pack, so a+missing pack is corruption and stays an Internal error naming the pack (as in+phase 6). The restart on a missing covered pack of the first review is not added:+with these boundaries it would be unreachable code. (This refines B and C of the+PM review; confirmed with the PM before WP76.)++### Liveness is per key (change G)++Liveness is decided per key from the rows that reference it: a key is live when+**any** changeset with `seq > C_gc` or any checkpoint with `seq >= C_gc`+references it. The snapshot therefore keeps, per key, the **maximum** referencing+sequence. With owned keys every key has one row, but legacy v0.1.0 keys can be+shared by several changesets, and the maximum makes them correct.### What happens to unreachable objects (binding decision)-The checkpoint pack is built with `git pack-objects` over the object-ID targets-of the references at N, non-thin: it holds exactly the objects **reachable from-the references at N**. Objects that were reachable at some earlier sequence but-not at N — commits orphaned by a force-update, a deleted branch or tag, a-rebased-away lineage — are **not** carried into the checkpoint pack, and once the-changeset packs that introduced them are reclaimed they are gone from object-storage.--Justification. Hydration and catch-up only ever reconstruct the state at or-after a checkpoint; no node ever needs a pre-checkpoint, now-unreachable object-to reach `head`. `graft storage verify --connectivity` checks reachability from-the **current** references, which the checkpoint pack plus the packs after it-satisfy. The changeset log keeps its metadata (operation, actor, old/new object-IDs), so the history questions the log answers keep working; only the git-*objects* of superseded history below a checkpoint are pruned — exactly as-`git gc` prunes unreachable objects in an ordinary repository. Stated as a limit-in STORAGE.md. (Open question for the PM: whether the checkpoint pack should also-retain objects reachable from the tips of **open reviews**; see "Open questions".)+The checkpoint pack holds exactly the objects reachable from:+* the object-ID targets of the references at `N`;+* the `head_commit_id` and `base_commit_id` of every review that is **open** when+ the checkpoint is built (answer 2 of the review).++Objects reachable only from earlier states (commits orphaned by a force-update,+deleted branches or tags, rebased-away lineages) and not from an open review are+not carried into the checkpoint pack; once the changeset packs that introduced+them are reclaimed, they are gone from object storage, as `git gc` prunes+unreachable objects. Hydration and catch-up only reconstruct states at or after a+checkpoint, verification checks reachability from the current references, and+the changeset log keeps its metadata. **Documented limit:** the diff of a closed or+merged review whose commits are reachable from no reference at a checkpoint may+no longer be available.++An open review whose head or base commit is not in the building node's local+repository (it can only happen when the commit was pruned by an earlier+checkpoint and the review was opened later on a node whose old cache still had+it) is a review whose commits the storage core no longer holds. The build fails+with `FailedPrecondition` naming the review IDs (metadata `review_id`), counts+`graft_storage_checkpoint_review_objects_missing_total`, and records nothing; GC+of the existing checkpoints is unaffected. The fix at the source belongs in the+review service (refuse, in authoritative mode, a review whose commits are not in+the storage core), which is not m3's file: raised with the PM.### Data model (new migration, both dialects)-A new migration `20261018000000_graft_storage_gc`, appended to-`sqliteMigrations` and `postgresMigrations` in-`internal/graft/corestore/sqlstore/migrations.go` (the phase-6 migration is-never edited). A v0.1.0 database (migration 1 only) gets these tables created at-`Open`; with zero checkpoints and zero tombstones every phase-7 path degrades to-the phase-6 behaviour (hydration replays from 1; GC still reclaims orphans).+Migration `20261018000000_graft_storage_gc`, appended to `sqliteMigrations` and+`postgresMigrations` in `internal/graft/corestore/sqlstore/migrations.go`. A+v0.1.0 database gets it at `Open`; with no checkpoints, observations or tombstones+every phase-7 path degrades to phase-6 behaviour.```sql--- A checkpoint: a consolidated pack holding every object reachable from the--- references at `seq`, plus (in graft_checkpoint_refs) the reference snapshot.+-- A checkpoint: the consolidated pack and (graft_checkpoint_refs) the reference+-- snapshot at seq.CREATE TABLE graft_checkpoints (- repo_id <int> NOT NULL REFERENCES graft_repositories (repo_id) ON DELETE CASCADE,- seq <int> NOT NULL, -- the changeset sequence the checkpoint is at (1..head_seq)- pack_key TEXT NOT NULL, -- repos/<repo_id>/packs/<sha256>.pack of the consolidated pack- pack_bytes <int> NOT NULL DEFAULT 0,- created_at <ts> NOT NULL,+ repo_id <int> NOT NULL REFERENCES graft_repositories (repo_id) ON DELETE CASCADE,+ seq <int> NOT NULL, -- 1..head_seq+ pack_key TEXT NOT NULL, -- an owned key+ pack_bytes <int> NOT NULL DEFAULT 0,+ created_at <ts> NOT NULL, -- database clockPRIMARY KEY (repo_id, seq));--- The reference snapshot at a checkpoint: name -> target. Never holds an absent--- value (only the references that exist at `seq`). `target` is a hex object ID--- or "ref:<name>" for a symbolic reference, exactly like graft_refs.+-- The reference snapshot at a checkpoint (no absent values). target is a hex+-- object ID or "ref:<name>", as in graft_refs.CREATE TABLE graft_checkpoint_refs (- repo_id <int> NOT NULL,- seq <int> NOT NULL,- ordinal <int> NOT NULL,- name BLOB NOT NULL, -- BYTEA on Postgres; may not be valid UTF-8- target TEXT NOT NULL,+ repo_id <int> NOT NULL,+ seq <int> NOT NULL,+ ordinal <int> NOT NULL,+ name BLOB NOT NULL, -- BYTEA on Postgres+ target TEXT NOT NULL,PRIMARY KEY (repo_id, seq, ordinal),FOREIGN KEY (repo_id, seq) REFERENCES graft_checkpoints (repo_id, seq) ON DELETE CASCADE);--- One maintenance lease per repository: the node building a checkpoint or--- running GC for it. Acquired by compare-and-swap (expires_at in the past is--- free). Observability + exclusivity only; never a correctness gate for reads--- or writes.+-- The collector's observations of unreferenced packs (PACK OWNERSHIP RULE 4).+-- Written only by the collector, never on the write path.+CREATE TABLE graft_gc_orphans (+ repo_id <int> NOT NULL REFERENCES graft_repositories (repo_id) ON DELETE CASCADE,+ pack_key TEXT NOT NULL,+ first_seen <ts> NOT NULL, -- database clock+ PRIMARY KEY (repo_id, pack_key)+);++-- One maintenance lease per repository (exclusivity and efficiency; GC and+-- checkpoint correctness do not depend on it, see change E).CREATE TABLE graft_gc_leases (- repo_id <int> NOT NULL PRIMARY KEY REFERENCES graft_repositories (repo_id) ON DELETE CASCADE,- holder TEXT NOT NULL, -- node_id- acquired_at <ts> NOT NULL,- expires_at <ts> NOT NULL+ repo_id <int> NOT NULL PRIMARY KEY REFERENCES graft_repositories (repo_id) ON DELETE CASCADE,+ holder TEXT NOT NULL, -- node_id+ acquired_at <ts> NOT NULL, -- database clock+ expires_at <ts> NOT NULL -- database clock);--- A tombstone for a removed repository, written in the same transaction as the--- repository row's deletion. No foreign key: the repository row is gone. GC--- deletes the bucket prefix repos/<repo_id>/ and then the tombstone.+-- A tombstone of a removed repository, written in the transaction that deletes+-- the repository row. No foreign key: the row is gone.CREATE TABLE graft_gc_removed (repo_id <int> NOT NULL PRIMARY KEY,- storage TEXT NOT NULL, -- for logs/observability only- relative_path TEXT NOT NULL, -- "- removed_at <ts> NOT NULL+ storage TEXT NOT NULL, -- for logs+ relative_path TEXT NOT NULL, -- for logs+ removed_at <ts> NOT NULL -- database clock);```-`<int>` = `INTEGER` on SQLite / `BIGINT` on Postgres; `<ts>` = RFC 3339 text in-UTC on SQLite / `TIMESTAMPTZ` on Postgres; names are `BLOB`/`BYTEA`, as the-phase-6 tables. `repo_id` of a checkpoint and of a tombstone is never reused, so-a removed-and-recreated repository never shares a checkpoint or a tombstone with-its predecessor.+Plus the owned-key trigger (PACK OWNERSHIP RULE 3), both dialects:++```sql+-- SQLite+CREATE TRIGGER graft_changesets_owned_pack BEFORE INSERT ON graft_changesets+WHEN NEW.pack_key IS NOT NULL AND NEW.pack_key NOT LIKE 'repos/%/packs/%-%.pack'+BEGIN+ SELECT RAISE(ABORT, 'storage core: the pack key is not an owned key; every node must run the same version');+END;++-- Postgres+CREATE FUNCTION graft_changesets_owned_pack() RETURNS trigger LANGUAGE plpgsql AS $$+BEGIN+ IF NEW.pack_key IS NOT NULL AND NEW.pack_key NOT LIKE 'repos/%/packs/%-%.pack' THEN+ RAISE EXCEPTION 'storage core: the pack key is not an owned key; every node must run the same version'+ USING ERRCODE = 'check_violation';+ END IF;+ RETURN NEW;+END $$;+CREATE TRIGGER graft_changesets_owned_pack BEFORE INSERT ON graft_changesets+ FOR EACH ROW EXECUTE FUNCTION graft_changesets_owned_pack();+```++`<int>` = `INTEGER` on SQLite / `BIGINT` on Postgres; `<ts>` = RFC 3339 text in UTC+on SQLite / `TIMESTAMPTZ` on Postgres. The `Down` steps drop the trigger (and the+Postgres function) and the five tables. WP71 checks that sql-migrate executes the+trigger bodies (they contain `;` inside `BEGIN ... END` and `$$ ... $$`) as single+statements on both drivers, with a test.-Go types (new, in `internal/graft/corestore/types.go`):+### Types and pack keys (`internal/graft/corestore/types.go`)```go-// Checkpoint is a consolidated state of a repository at a changeset sequence:-// one pack with every object reachable from the references at Seq.+// Checkpoint is a consolidated state of a repository at a changeset sequence.type Checkpoint struct {RepoID int64Seq uint64PackKey stringPackBytes int64- CreatedAt time.Time+ CreatedAt time.Time // database clock}-// RemovedRepository is a tombstone of a removed repository whose bucket packs-// are still to be reclaimed.+// RemovedRepository is a tombstone whose bucket prefix is still to be reclaimed.type RemovedRepository struct {- RepoID int64- Key RepoKey- RemovedAt time.Time+ RepoID int64+ Key RepoKey+ RemovedAt time.Time // database clock}++// GCState is what the collector decides on, read in one database snapshot.+type GCState struct {+ Now time.Time // database time of the snapshot+ Packs map[string]uint64 // changeset pack key -> greatest referencing seq+ Checkpoints []Checkpoint // ascending by Seq+ Orphans map[string]time.Time // observed unreferenced key -> first_seen+}++var (+ ErrBlobExists = errors.New("storage core: blob already exists")+ ErrCheckpointNotFound = errors.New("storage core: checkpoint not found")+)++// NewPackKey returns a new owned pack key for a pack of the repository with the+// given SHA-256: "repos/<repo_id>/packs/<hex sha256>-<hex nonce>.pack", with a+// random 16-byte nonce.+func NewPackKey(repoID int64, sum [32]byte) string++// ParsePackKey returns the repository ID and the SHA-256 of an owned key or of a+// legacy key "repos/<repo_id>/packs/<hex sha256>.pack", and false for anything+// else. Only canonical spellings are accepted (lowercase hex, no leading zeros).+func ParsePackKey(key string) (repoID int64, sum [32]byte, ok bool)++// IsOwnedPackKey reports whether key is an owned key (not a legacy one).+func IsOwnedPackKey(key string) bool```-### BlobStore additions (RULE: GC needs listing and deletion)+`PackKey(repoID, sum)` (the legacy constructor) is removed: no caller may produce a+legacy key any more. Every call site (voter, `shadowpack.go`, import, tests) moves+to `NewPackKey`.-`corestore.BlobStore` (in `core.go`) gains two methods; `blobstore` implements-them on the gocloud bucket, `coretest`/test doubles follow:+### BlobStore changes (`core.go`, `blobstore`)```go-// BlobInfo describes a stored object for garbage collection.+// BlobInfo describes a stored object.type BlobInfo struct {- Key string- Size int64- ModTime time.Time+ Key string+ Size int64}type BlobStore interface {- // ... existing Put, Get, Exists, Close ...-- // List calls fn for every object whose key starts with prefix, in no- // defined order. It lists objects only, never directory markers. fn may- // return an error to stop the iteration (returned as-is).+ // Put stores the content under key, which must not exist: an existing key is+ // ErrBlobExists and nothing is written. A pack key's content must hash to its+ // SHA-256 (ErrBlobCorrupt otherwise, nothing stored). A cancelled context+ // stores nothing.+ Put(ctx context.Context, key string, r io.Reader) error+ // Get, Exists, Close: unchanged.++ // List calls fn for every object whose key starts with prefix, in no defined+ // order, objects only (no directory entries). An error from fn stops the+ // iteration and is returned.List(ctx context.Context, prefix string, fn func(BlobInfo) error) error- // Delete removes the object. A key that does not exist is success- // (idempotent), so a re-run of GC never fails on an already-deleted pack.+ // Delete removes the object; a missing key is success.Delete(ctx context.Context, key string) error}```-Implementation (`blobstore`): `List` uses `bucket.List(&blob.ListOptions{Prefix:-prefix})` and the iterator's `*blob.ListObject` (`Key`, `Size`, `ModTime`,-`IsDir`), skipping `IsDir` entries (we list without a delimiter, so there are-none, but we guard). `Delete` uses `bucket.Delete`, mapping `gcerrors.NotFound`-to success. `ModTime` is the object's last-modified time; for the fileblob-driver it is the file mtime, for mem it is the write time (both used in tests);-for S3/GCS/Azure it is the provider's object mtime (untested here, see limits).-File buckets may leave empty directories behind after the last pack under them-is deleted; this is harmless and documented (the keys, which are what matters,-are gone).--### MetaStore additions--`corestore.MetaStore` (in `metastore.go`) gains the methods below; `sqlstore`-implements them on both dialects, with the phase-6 patterns (`bind` for-placeholders, `withTx`/`withReadTx` with the retry on busy/serialization,-`timeArg`/`scanTime`, `bytesArg`/`targetArg`, name chunks of-`maxNamesPerQuery`). `DeleteRepository` changes behaviour (writes a tombstone).+`List` uses `bucket.List(&blob.ListOptions{Prefix: prefix})` and skips `IsDir`+entries; `Delete` maps `gcerrors.NotFound` to success. `BlobInfo` carries no+modification time on purpose (the CLOCK RULE: GC never reads bucket times). A+file bucket may keep empty directories after the last key under them is deleted;+the keys, which are what matters, are gone (documented).++### MetaStore additions (`metastore.go`, `sqlstore`)++Implemented on both dialects with the phase-6 patterns (`bind`, `withTx` /+`withReadTx` with the busy/serialization retry, `timeArg`/`scanTime`,+`bytesArg`/`targetArg`, name chunks of `maxNamesPerQuery`).```go-// CheckpointInput is the state recorded by CreateCheckpoint.+// CheckpointInput is the state CreateCheckpoint records.type CheckpointInput struct {- Seq uint64 // 1..head_seq, strictly greater than the newest checkpoint's Seq- Refs map[string]Target // the reference snapshot at Seq; no absent values; no empty map- PackKey string // the already-uploaded consolidated pack (never empty)+ Seq uint64 // 1..head_seq, greater than the newest checkpoint's Seq+ Refs map[string]Target // the reference snapshot at Seq: no absent value, not empty+ PackKey string // an owned key, already uploadedPackBytes int64}type MetaStore interface {- // ... existing methods ...-- // CreateCheckpoint records a checkpoint and its reference snapshot in one- // transaction. It locks the repository row, verifies Seq <= head_seq and- // Seq > the newest existing checkpoint's Seq (else an error, nothing- // written), inserts graft_checkpoints and graft_checkpoint_refs, and- // returns the stored Checkpoint. A checkpoint at Seq that already exists is- // returned unchanged (idempotent; the lease makes a concurrent build at the- // same Seq a non-case, but a resumed build after a crash is safe).- // ErrRepositoryNotFound for an unknown repository.- CreateCheckpoint(ctx context.Context, repoID int64, in CheckpointInput) (Checkpoint, error)- // NewestCheckpoint returns the checkpoint of the repository with the- // greatest Seq; ok is false when there is none.+ // ... phase-6 methods ...++ // CreateCheckpoint records a checkpoint and its snapshot in one transaction+ // with created_at = the database time. It locks the repository row and+ // requires Seq <= head_seq, Seq > the newest checkpoint's Seq and an owned+ // PackKey of the repository (else an error, nothing written). A checkpoint at+ // Seq that exists already is returned unchanged with created=false (its pack+ // is kept; the caller's pack becomes an orphan). ErrRepositoryNotFound for an+ // unknown repository.+ CreateCheckpoint(ctx context.Context, repoID int64, in CheckpointInput) (cp Checkpoint, created bool, err error)+ // NewestCheckpoint returns the checkpoint with the greatest Seq; ok is false+ // when there is none.NewestCheckpoint(ctx context.Context, repoID int64) (cp Checkpoint, ok bool, err error)- // ListCheckpoints returns every checkpoint of the repository, ascending by- // Seq.- ListCheckpoints(ctx context.Context, repoID int64) ([]Checkpoint, error)- // CheckpointRefs returns the reference snapshot at the checkpoint (name ->- // target). ErrCheckpointNotFound when there is no checkpoint at seq.+ // SyncBoundary returns the Seq of the newest checkpoint created at least age+ // ago on the database clock, 0 when there is none (the C_sync of the GC+ // BOUNDARY rule).+ SyncBoundary(ctx context.Context, repoID int64, age time.Duration) (uint64, error)+ // CheckpointRefs returns the snapshot of the checkpoint at seq;+ // ErrCheckpointNotFound when there is none.CheckpointRefs(ctx context.Context, repoID int64, seq uint64) (map[string]Target, error)- // DeleteCheckpoint removes the checkpoint row and its snapshot (cascade). It- // returns false when there was none. The caller deletes the pack afterwards- // (a crash in between leaves an orphan pack, reclaimed by the orphan sweep).+ // DeleteCheckpoint removes the checkpoint row and its snapshot; false when+ // there was none.DeleteCheckpoint(ctx context.Context, repoID int64, seq uint64) (bool, error)- // ChangesetPacks calls fn for every changeset of the repository with a- // non-empty pack key, ascending by sequence, reading only (seq, pack_key).- // It is the lean scan GC marks live packs with (ListChangesets would carry- // every reference change).- ChangesetPacks(ctx context.Context, repoID int64, fn func(seq uint64, packKey string) error) error-- // AcquireMaintenanceLease takes the repository's maintenance lease for- // holder until now+ttl, by compare-and-swap: it succeeds when there is no- // lease or the current one has expired. RenewMaintenanceLease extends it if- // holder still holds it; ReleaseMaintenanceLease frees it if holder holds- // it. All three are no bar to reads or writes; they only serialize GC and- // checkpoint builds across nodes.+ // GCSnapshot reads, in one read transaction, the database time, the changeset+ // pack keys with their greatest referencing sequence, the checkpoints and the+ // orphan observations of the repository. ErrRepositoryNotFound if unknown.+ GCSnapshot(ctx context.Context, repoID int64) (GCState, error)+ // RecordOrphans stores first_seen = the database time for every key that has+ // no observation yet (existing observations are kept).+ RecordOrphans(ctx context.Context, repoID int64, keys []string) error+ // ForgetOrphans removes the observations of the keys (deleted, or referenced+ // since they were observed).+ ForgetOrphans(ctx context.Context, repoID int64, keys []string) error++ // AcquireMaintenanceLease takes the lease for holder until the database time+ // plus ttl, by compare-and-swap: it succeeds when there is no lease, the lease+ // has expired, or holder holds it already. RenewMaintenanceLease extends it+ // when holder still holds it (held=false otherwise). ReleaseMaintenanceLease+ // frees it when holder holds it. They never bar a read or a write.AcquireMaintenanceLease(ctx context.Context, repoID int64, holder string, ttl time.Duration) (acquired bool, err error)RenewMaintenanceLease(ctx context.Context, repoID int64, holder string, ttl time.Duration) (held bool, err error)ReleaseMaintenanceLease(ctx context.Context, repoID int64, holder string) error- // ListRepositories returns every repository registered in the storage- // (its head sequence > 0), for the background maintenance task. keyPrefix- // empty lists all storages' repositories; otherwise only the given storage.+ // ListRepositories returns every repository of the given storage whose import+ // is complete (head_seq > 0), ascending by ID. An empty storage lists the+ // repositories of every storage.ListRepositories(ctx context.Context, storage string) ([]Repository, error)- // ListRemovedRepositories returns the tombstones of removed repositories- // whose bucket packs are still to be reclaimed.- ListRemovedRepositories(ctx context.Context) ([]RemovedRepository, error)- // DeleteRemovedRepository removes a tombstone after its bucket prefix was- // cleared. Idempotent.+ // ListRemovedRepositories returns the tombstones of the given storage (every+ // storage when empty) whose removed_at is at least minAge ago on the database+ // clock (change F).+ ListRemovedRepositories(ctx context.Context, storage string, minAge time.Duration) ([]RemovedRepository, error)+ // DeleteRemovedRepository removes a tombstone; idempotent.DeleteRemovedRepository(ctx context.Context, repoID int64) error}```-`ErrCheckpointNotFound` is a new sentinel in `types.go`.+**DeleteRepository (changed).** In the transaction that deletes the repository row+it reads the row's `repo_id` and inserts the `graft_gc_removed` tombstone with+`removed_at` = the database time. The cascade removes changesets, references,+changelog, reviews, checkpoints, observations and the lease; the tombstone stays.+An unknown key writes nothing and returns false, as before. Signature unchanged.-**DeleteRepository (changed, binding).** It now writes a `graft_gc_removed`-tombstone `(repo_id, storage, relative_path, removed_at)` in the **same-transaction** that deletes the repository row, reading the `repo_id` before the-delete. The cascade removes the repository's changesets, references, changelog,-reviews, checkpoints and lease; the tombstone survives (no FK) so that GC can-reclaim the bucket prefix afterwards. A key the database does not know writes no-tombstone and returns false, as before. The method keeps its signature-`(bool, error)`; the gate's `removeRepository` is unchanged.+### Core additions (`core.go`, `open/open.go`)-### gitops addition+`corestore.Core` gains `GCGrace` and `GCMargin time.Duration` (from the config,+also in shadow mode: GC runs there too, so the writers' deadline applies) and the+method `func (c *Core) WriteDeadline() time.Duration` = `GCGrace − GCMargin` (the+`W` of the PACK OWNERSHIP RULE).-One new function in `internal/graft/corestore/gitops/gitops.go`, with a test;-nothing else in gitops changes:+### gitops addition```go// BuildPackFrom writes a self-contained (non-thin) pack of every object-// reachable from tips (the consolidated pack of a checkpoint). The tips are the-// object-ID targets of the references at the checkpoint sequence; symbolic and-// zero targets are ignored. It is BuildFullPack restricted to the given tips:-// `git rev-list --objects --stdin <tips>` (no --not) piped into-// `git pack-objects --stdout`, hashed into a file in tmpDir. An empty tip set-// returns an empty Pack.+// reachable from tips: `git rev-list --objects --stdin` (the tips on stdin, no+// --not) piped into `git pack-objects --stdout`, hashed into a file in tmpDir.+// Zero tips are ignored; no tips is an empty Pack. A tip that is not in the+// repository is an error that IsMissingObject recognizes.func BuildPackFrom(ctx context.Context, repo *localrepo.Repo, tips []git.ObjectID, tmpDir string) (Pack, error)++// MissingObjects returns the tips that are not in the repository (one+// `cat-file --batch-check` run).+func MissingObjects(ctx context.Context, repo *localrepo.Repo, tips []git.ObjectID) ([]git.ObjectID, error)```-### Checkpoint creation (new package `internal/graft/corestore/checkpoint`)+### Writers (WP73, change A on the write path)-`checkpoint.Checkpointer` builds and records a checkpoint for one repository. It-is used by the background maintenance task and by the `CreateCheckpoint` RPC.+* **Voter.** `preparePack` records `t_r` after its `GetRepository` and uploads with+ `NewPackKey` under a context with deadline `t_r + W`; `preparedPack` keeps the+ deadline. `prepare` (Prepared) uses the prepared pack only when the repository ID+ matches **and** its deadline has not passed; otherwise it calls `buildAndUpload`+ with a new `t_r` from the record it has just read. The `CommitChangeset` call runs+ under the pack's deadline (the earlier of the request context and `t_r + W`). A+ transaction without a pack has no deadline. Shadow mode: the same, inside the+ shadow timeout (which is far shorter than `W`).+* **ImportRepository.** After building the full pack it re-reads the record+ (`GetRepository`, which also confirms the registration still exists), uploads+ with `NewPackKey` and commits changeset 1 under `t_r + W`. A resumed import uploads+ a new owned pack; the pack of the interrupted attempt is an orphan.+* **Checkpoint builder.** See below; same rule.++### Checkpoint creation (`internal/graft/corestore/checkpoint`)```gofunc New(core *corestore.Core, logger log.Logger, locator storage.Locator,repoFactory localrepo.Factory, fresh corestore.Freshener) *Checkpointer// Create builds a checkpoint of repo at the current database head and records-// it. It requires authoritative mode and a local copy that EnsureFresh can-// bring to the head (the objects of the checkpoint come from the local cache).-// Returns the created Checkpoint and whether one was created (false = a-// checkpoint already exists at the head, nothing to do).-func (c *Checkpointer) Create(ctx context.Context, repo storage.Repository) (corestore.Checkpoint, bool, error)+// it. created is false when a checkpoint at the head exists already.+func (c *Checkpointer) Create(ctx context.Context, repo storage.Repository) (cp corestore.Checkpoint, created bool, err error)++// FaultHook is called at "after_upload" and "after_commit" (tests only).+var FaultHook func(point string) error```-Binding order (crash-safe; `FaultHook`-style test points at each `//!` below):--1. Require `core.Authoritative()` (else `FailedPrecondition`). `EnsureFresh` the- local copy. Read `(refs, N) = core.Meta.Refs(repoID)` — the reference- snapshot and the head sequence in **one consistent database snapshot**. If a- checkpoint at `N` already exists (`NewestCheckpoint`), return it, `created =- false`.-2. The tips are the object-ID targets of `refs` (skip symbolic and absent).- `pack = gitops.BuildPackFrom(ctx, local, tips, tmpDir)` from the **local**- repository (fresh to `>= N`, so it holds every object reachable from `refs`;- nothing prunes the local cache). An empty repository (no object tips) records- a checkpoint with no pack is **not** allowed — a checkpoint must have a pack;- a repository whose references are only symbolic/empty is a degenerate case- handled by recording the snapshot with an empty pack key is disallowed, so we- skip creating a checkpoint (return `created = false`) when there are no object- tips. (Hydration of such a repository from 1 is already trivial.)-3. `//! before_upload` — upload `pack` under `corestore.PackKey(repoID,- pack.SHA256)` (`BlobStore.Put`, create-only).-4. `//! after_upload` — `core.Meta.CreateCheckpoint(repoID, CheckpointInput{Seq:- N, Refs: refs, PackKey, PackBytes})`. The snapshot `refs` is exactly the- references read in step 1, so it is consistent with `N`.-5. `//! after_commit` — return the checkpoint, `created = true`.--A crash after step 3 and before step 4 leaves the consolidated pack as an orphan-(no checkpoint row references it), reclaimed by the orphan sweep. A re-run-rebuilds from the head at that time (possibly a larger `N`) and commits. The-build never holds a reference lock and never blocks a write: it reads a database-snapshot and the local objects, and records the checkpoint in its own-transaction, while writes keep appending changesets `> N` (CKPT-2).--### Pack GC (new package `internal/graft/corestore/gc`)--`gc.Collector` runs the mark and sweep for one repository and the removed--repository cleanup. Used by the background task and by `CollectGarbage`.+Binding order:++1. Require authoritative mode (else `FailedPrecondition`, SHADOW SCOPE) and a+ reftable local repository. Read `(refs, N) = Meta.Refs(repoID)` (one snapshot).+ If `NewestCheckpoint` is at `N`: return it, `created = false`.+2. **Change D:** `EnsureFresh(repo)`. EnsureFresh catches up to a head read after+ the call arrived, which is `>= N`; then read the local watermark and require+ `wm.RepoID == repoID && wm.Seq >= N` (else `Internal`: the Freshener contract is+ broken, never retried around). The local repository now holds every object of+ the changesets `<= N`, hence every object reachable from `refs`.+3. Tips = the object-ID targets of `refs` (symbolic and absent skipped) plus the+ `head_commit_id` and `base_commit_id` of every open review+ (`ListReviews(repoID, "", false, 0)`, each value validated as an object ID of+ the repository's hash; empty values skipped). `MissingObjects` on the review+ tips: any missing → `FailedPrecondition` naming the reviews (see "unreachable+ objects"), metric, nothing uploaded. A missing reference tip is impossible after+ step 2 and is `Internal`.+4. `pack = BuildPackFrom(local, tips, tmpDir)`. No object tips → return+ `created = false` (nothing to consolidate; hydration of such a repository is+ already trivial).+5. Re-read the record (`t_r`), upload under `NewPackKey(repoID, pack.SHA256)` with+ deadline `t_r + W`. `//! after_upload`.+6. `CreateCheckpoint(repoID, {Seq: N, Refs: refs, PackKey, PackBytes})` under the+ same deadline. `//! after_commit`.++A crash after 5 leaves the pack as an orphan (reclaimed by the orphan protocol); a+re-run starts over from the head at that time. The build holds no reference lock+and never blocks or fails a write (CKPT-2).++**Change E (fencing).** `CreateCheckpoint` does **not** check the maintenance lease+in its transaction, and checkpoint correctness does not rely on the orphan grace+being a guess either: it relies on (a) per-sequence idempotency (a second build+at the same `N` returns the existing checkpoint, its own owned pack becomes an+orphan), (b) monotonic sequences (a checkpoint at a smaller `N` than the newest+is refused), and (c) the enforced orphan protocol for the loser's pack. Two+checkpoints at different sequences built by two nodes whose leases overlapped are+both valid states. The lease only prevents wasted builds.++### Pack GC (`internal/graft/corestore/gc`)```gofunc New(core *corestore.Core, logger log.Logger) *Collector-// Report is what a collection did (or would do, for a dry run).type Report struct {- CheckpointSeq uint64 // the newest kept checkpoint, 0 when none- PacksDeleted int // covered + orphan + dropped-checkpoint packs- BytesReclaimed int64+ Boundary uint64 // C_gcCoveredDeleted int- OrphansDeleted int- OrphansKeptInGrace intCheckpointsDropped int- RemovedRepos int // removed-repository prefixes reclaimed+ OrphansDeleted int+ OrphansObserved int // newly observed, kept for now+ OrphansInGrace int // observed earlier, not old enough yet+ PacksDeleted int+ BytesReclaimed int64 // sizes from List+ RemovedRepos intDryRun bool}-// CollectRepository reclaims the dead packs of one repository: changeset packs-// covered by the newest kept checkpoint, dropped-checkpoint packs, and orphans-// older than the grace period. The caller holds the repository's maintenance-// lease. grace is config gc_grace; retain is config checkpoint_retain.-func (c *Collector) CollectRepository(ctx context.Context, repoID int64, grace time.Duration, retain int, dryRun bool) (Report, error)+// CollectRepository reclaims the dead packs of one repository. The caller holds+// the repository's maintenance lease (efficiency only).+func (c *Collector) CollectRepository(ctx context.Context, repoID int64, dryRun bool) (Report, error)-// CollectRemoved reclaims the bucket prefix of every removed-repository-// tombstone and deletes the tombstone. No lease (the repository is gone,-// repo_id never reused, no writer); idempotent.-func (c *Collector) CollectRemoved(ctx context.Context, dryRun bool) (Report, error)+// CollectRemoved reclaims the bucket prefixes of the tombstones of the storage+// (every storage when empty) that are older than gc_grace, then deletes the+// tombstones.+func (c *Collector) CollectRemoved(ctx context.Context, storage string, dryRun bool) (Report, error)++// FaultHook is called at "after_drop_row", "after_delete", "mid_sweep" and+// "after_prefix" (tests only).+var FaultHook func(point string) error```-`CollectRepository` (binding; the database is authoritative at every step, and-no live pack is ever deleted):--1. `checkpoints = ListCheckpoints(repoID)` ascending. Choose the **kept** set:- always the newest; also the most recent `retain` checkpoints; also any- checkpoint younger than `grace` (so an in-flight hydration started before a- newer checkpoint appeared is safe). `C` = the newest checkpoint's sequence (0- when there is none). The checkpoints **to drop** are the rest (older than the- newest, beyond `retain`, and older than `grace`).-2. `packKeys = {}`; `ChangesetPacks(repoID, func(seq, key){ packKeys[key] = seq })`.- Build the live set:- * every kept checkpoint's `PackKey`;- * every `key` in `packKeys` with `seq > C`.- The covered set = every `key` in `packKeys` with `seq <= C`.-3. Drop the superseded checkpoints: for each, `DeleteCheckpoint` (row) first,- then its pack becomes an orphan for step 4 (we do **not** delete the pack- before the row: a crash in between would leave a row pointing at a missing- pack, which a hydration might pick — the row goes first, the pack is an orphan- afterwards). In a dry run, nothing is deleted; the pack is counted as a- dropped-checkpoint pack.-4. Sweep: `Blobs.List("repos/<repoID>/packs/", func(obj){ ... })`. For each- `obj.Key`:- * in the live set → keep;- * in the covered set (`packKeys[key] <= C`) → delete (covered by a kept- checkpoint);- * otherwise it is an **orphan** (no changeset row and no kept checkpoint- references it — a dead write's pack, or a just-dropped checkpoint's pack):- delete only when `now - obj.ModTime >= grace`, else count it in- `OrphansKeptInGrace` and keep it.- `dryRun` deletes nothing and only counts.-5. Return the `Report`. Idempotent: the live/covered sets are recomputed from- the database each run, and `Blobs.Delete` of an already-deleted key is a- no-op, so a crashed sweep converges on the next run and a re-run is cheap.--`CollectRemoved`: `ListRemovedRepositories`; for each tombstone,-`Blobs.List("repos/<repo_id>/", delete each key)`, then-`DeleteRemovedRepository(repo_id)`. A crash after a partial prefix delete leaves-the tombstone; the next run finishes (idempotent). No lease.--### Hydration from a checkpoint (syncer, binding change)--`internal/graft/corestore/syncer/syncer.go`, `hydrate`, now seeds from the-newest usable checkpoint instead of always from sequence 1. The phase-6 WP55-RULE (catch-up never moves a reference backwards; the watermark only claims-sequences whose effects are local) and the install/stale-cache handling are-unchanged.--`hydrate(ctx, repo, r, hash, path)`:--1. Choose the seed: iterate `ListCheckpoints(r.ID)` from the greatest `Seq`- downward and pick the first whose `Seq <= r.HeadSeq` and whose pack- `Blobs.Exists`/can be fetched. Call it `cp` (seed sequence `S = cp.Seq`,- snapshot `snap = CheckpointRefs(r.ID, S)`). If there is none, `S = 0`, no- seed pack, no snapshot — the phase-6 replay from 1.-2. `InitBare` the build repository. When seeded, index `cp.PackKey` into it- (downloaded and SHA-256-verified exactly like a changeset pack; the pack key- is validated to belong to `r.ID`), and validate every name/target of `snap`.-3. `f = newFetched(S)`; `fetch(ctx, buildRepo, tmpDir, r.ID, hash, f,- r.HeadSeq)` — downloads and indexes the changeset packs `(S, head]` and folds- their reference changes, as today.-4. Compute the final reference set as of head: start from `snap` (the references- at `S`), then apply `f`'s changes — a changed reference's final value- overwrites, an absent final value deletes it. `applyBuild(combined)`.-5. `WriteWatermark(buildPath, {r.ID, f.head})`, then `install` as today.--The number of changesets replayed is `head - S`, bounded by the checkpoint-cadence, instead of `head`. If no checkpoint's pack can be fetched **and** the-replay from 1 needs a pack that GC reclaimed, hydration fails with an Internal-error naming the missing pack (a genuine inconsistency); the newest checkpoint's-pack is always present by GC-3, so this is the corruption case only.--### Catch-up below a checkpoint (syncer, binding change)--A node whose watermark `wm.Seq < C` (the newest checkpoint) cannot download the-reclaimed packs `(wm.Seq, C]`, so it must re-hydrate from the checkpoint rather-than catch up. In `sync()`, the condition that chooses hydration gains this-case:--* read `cp, hasCp = NewestCheckpoint(r.ID)` once per round (cheap);-* hydrate when `!ok || wm.RepoID != r.ID || !localCurrent || (hasCp && wm.Seq <- cp.Seq)`; otherwise catch up (then `wm.Seq >= cp.Seq`, and every pack `(wm.Seq,- head]` is live, so it exists).--Re-hydration rebuilds the local copy from the checkpoint and replaces it through-the existing `install` path (which moves the stale copy aside under the-repository lock). This is correct because the commit point is the database: every-committed write is in the database at a sequence `<= head`, so it is either in-the checkpoint (`<= S`) or replayed from `(S, head]`. It discards the local-objects of a far-behind cache (wasteful but rare — an actively-served node stays-near the head and never falls below a checkpoint). `checkReplaceable` (the-shadow-history guard, IMPORT RULE) still runs first and is unaffected-(authoritative mode writes no file changelog or file reviews).--Belt-and-suspenders in `fetch`: if a changeset pack with `seq <= C` is-`ErrBlobNotFound` during a catch-up (a lagging round races a concurrent GC that-reclaimed a covered pack), return `errRestart` so `sync` re-evaluates and-hydrates from the checkpoint. A missing pack with `seq > C` stays a hard-Internal error (real corruption). `fetch` is given the newest checkpoint-sequence for this test.--### Verify after compaction (service/storagecore/verify_objects.go, binding change)--`checkStoredObjects` (the dry hydration of `graft storage verify---connectivity`) replays from the newest checkpoint instead of from 1, so it-keeps passing after GC reclaimed the covered packs:--* `cp, ok = NewestCheckpoint(record.ID)`. When `ok`, index `cp.PackKey` into the- empty verification repository, then index the changeset packs for `seq in (C,- head]`. When not, index all changeset packs `1..head` (as today).-* Then `git rev-list --objects --quiet` from the object-ID targets of the- database references, as today.--A checkpoint pack that is missing or corrupt, or a missing changeset pack `>-C`, is reported as a connectivity failure beginning `storage core objects:`-exactly as a missing changeset pack is today.--### Background maintenance task (new, `internal/graft/corestore/maintenance`)--`maintenance.Runner` runs the checkpoints and GC on a timer. Wired in-`internal/cli/gitaly/serve.go` (my section only), started with the server and-stopped on shutdown (as `bundleGenerationStrategy.Start(ctx)` / `stop()`), only-when the storage core is authoritative and `maintenance = "auto"`.+`CollectRepository` (binding):++1. `st = GCSnapshot(repoID)`. `C_gc` = the greatest `Seq` among `st.Checkpoints`+ with `CreatedAt <= st.Now − gc_grace` (0 when none).+2. **Drop superseded checkpoints:** for each checkpoint with `Seq < C_gc`:+ `DeleteCheckpoint` (row first), `//! after_drop_row`, then `Blobs.Delete` of its+ pack (owned by that row only, and no reader uses a checkpoint below `C_gc`).+ A crash in between leaves an unreferenced pack, reclaimed by the orphan protocol.+3. **Live set** (per key, change G): kept checkpoint packs (`Seq >= C_gc`) and+ every key of `st.Packs` whose greatest sequence is `> C_gc`. **Covered set:**+ keys of `st.Packs` whose greatest sequence is `<= C_gc`, minus the live set.+4. **Sweep** `Blobs.List("repos/<repoID>/packs/")`, for each key:+ * live → keep;+ * covered → `Blobs.Delete`;+ * referenced by no row: if `st.Orphans[key] <= st.Now − gc_grace` →+ `Blobs.Delete` and then forget the observation; if it has a younger+ observation → keep (in grace); if it has none → collect it for+ `RecordOrphans`.+ `//! mid_sweep` after the first deletion. Finally `RecordOrphans(new)` and+ `ForgetOrphans` for the observations whose key is now referenced or was not+ listed.+5. Dry run: steps 1 and 3, and the sweep only counts (no delete, no observation+ written).++Every decision comes from one snapshot and the PACK OWNERSHIP proof; `Delete` of a+missing key is success; a crashed run converges on the next one. Two collectors+racing on one repository (lease expired) both make only safe decisions: deletes+are idempotent, observations are inserted with "do nothing on conflict".++### Removed repositories (change F)++`CollectRemoved`: `ListRemovedRepositories(storage, gc_grace)`; for each tombstone,+`Blobs.List("repos/<repo_id>/")` and delete every key, `//! after_prefix`, then+`DeleteRemovedRepository`. A tombstone is swept only once `removed_at` is at least+`gc_grace` old: a writer whose record read found the repository did so before+`removed_at`, and its upload is bounded by `t_r + W`, so no upload can arrive in the+prefix after the sweep (the PACK OWNERSHIP RULE makes this an enforced bound, not+an assumption). A crash after a partial prefix deletion leaves the tombstone; the+next run finishes it. No lease (the repository is gone and its ID is never reused).++### Hydration from a checkpoint (syncer, WP76)++`hydrate` seeds from `NewestCheckpoint`:++1. Seed `S` = the newest checkpoint's `Seq` and its snapshot (`CheckpointRefs`),+ or `S = 0` without one.+2. `InitBare`; when seeded, download, verify (SHA-256 against the key, `repo_id`+ of the key) and index the checkpoint pack, and validate every name and target of+ the snapshot (RULE 22).+3. `f = newFetched(S)`; `fetch(..., f, r.HeadSeq)` downloads and indexes the packs+ `(S, head]` and folds their reference changes.+4. Final references = the snapshot, overwritten by `f`'s final values (an absent+ final value deletes); `applyBuild` once; watermark `{r.ID, f.head}`; `install`.++A missing checkpoint pack or a missing pack `> S` is an Internal error naming the+key (corruption, GC BOUNDARY rule).++### Catch-up and the sync boundary (syncer, WP76, change C)++In `sync()`, each round reads `C_sync = SyncBoundary(r.ID, gc_grace − sync_timeout −+gc_margin)` and hydrates when `!ok || wm.RepoID != r.ID || !localCurrent ||+wm.Seq < C_sync`; otherwise it catches up as in phase 6 (the WP55 RULE unchanged).+`checkReplaceable` still runs before any hydration.++### Verify after compaction (`verify_objects.go`, WP76)++`checkStoredObjects` indexes the newest checkpoint's pack and the changeset packs+`(S, head]` (all packs `1..head` without a checkpoint), then runs the reachability+check as today, and also checks the open reviews' head and base commits. It runs+under a deadline of `W` from its start (a dry hydration that read checkpoint `S`+must end before a checkpoint newer than `S` can become `C_gc`); a missing pack is+reported as a connectivity failure starting with `storage core objects:`.++### Background maintenance (`internal/graft/corestore/maintenance`, WP77)```gofunc New(core *corestore.Core, cp *checkpoint.Checkpointer, collector *gc.Collector,@@ -559,80 +713,66 @@ func New(core *corestore.Core, cp *checkpoint.Checkpointer, collector *gc.Collecfunc (r *Runner) Start(ctx context.Context) (stop func())```-Each tick (every `maintenance_interval`):--1. `CollectRemoved` (tombstones; any node, no lease).-2. For each repository `ListRepositories` reports in this node's storages that- this node has a **local copy with a current watermark** (so it can build a- checkpoint from local objects):- * `AcquireMaintenanceLease(repoID, node_id, lease_ttl)`; skip the repository- when another node holds it.- * `EnsureFresh`.- * If `head_seq - newestCheckpointSeq >= checkpoint_changesets` **or** the pack- bytes of the changesets since the newest checkpoint `>= checkpoint_pack_bytes`- (summed from `ChangesetPacks` / `ListChangesets` pack bytes), `Checkpointer.Create`.- * `Collector.CollectRepository(repoID, gc_grace, checkpoint_retain, false)`.- * `RenewMaintenanceLease` around a long build; `ReleaseMaintenanceLease` at- the end (a crash lets the lease expire after `lease_ttl`).--A node without a local copy of a repository never builds its checkpoint (it has-no objects), but any node may run GC of it (GC needs only the database and the-bucket). In practice the node that serves a repository maintains it.--### Configuration (`internal/gitaly/config/config_graft_storage.go`, my section)+**Off by default** (answer 3): it runs only with `maintenance = "auto"` and in+authoritative mode; the CLI is always available. Each tick (`maintenance_interval`):+`CollectRemoved` for this node's storages; then, for each repository of+`ListRepositories` that this node has a current local copy of:+`AcquireMaintenanceLease` (skip when held elsewhere), checkpoint when+`head_seq − newest checkpoint seq >= checkpoint_changesets` or the pack bytes of+the changesets after the newest checkpoint `>= checkpoint_pack_bytes`, then+`CollectRepository`, renewing during a long build and releasing at the end. Wired in+`serve.go` (the storage core block only), started with the server and stopped on+shutdown.-New keys on `[graft.storage]` (`GraftStorage` struct), all optional:+### Configuration (`config_graft_storage.go`, my section)```toml[graft.storage]-# ... existing mode, database, bucket, node_id, sync_timeout, shadow_timeout ...-maintenance = "auto" # off | auto — background checkpoints + GC (authoritative only). Default auto.-maintenance_interval = "1h" # how often the background task scans repositories (Go duration)-checkpoint_changesets = 1000 # create a checkpoint after this many changesets since the last one (0 = disabled)-checkpoint_pack_bytes = 268435456 # ...or after this many bytes of changeset packs since the last one (0 = disabled)-checkpoint_retain = 2 # number of most-recent checkpoints to keep (>= 1)-gc_grace = "1h" # minimum age before an orphan or a dropped-checkpoint pack is deleted (Go duration)-lease_ttl = "10m" # maintenance lease duration, renewed during a build+# ... mode, database, bucket, node_id, sync_timeout, shadow_timeout ...+maintenance = "off" # off | auto: background checkpoints and GC (authoritative only). Default off.+maintenance_interval = "1h" # how often the background task runs+checkpoint_changesets = 1000 # checkpoint after this many changesets since the newest one (0 = never)+checkpoint_pack_bytes = 268435456 # ...or after this many pack bytes since the newest one (0 = never)+gc_grace = "1h" # age before an unreferenced pack is deleted and a checkpoint becomes C_gc+gc_margin = "5m" # safety margin of the writers' deadline (gc_grace - gc_margin) and the boundaries+lease_ttl = "10m" # maintenance lease duration, renewed during a build```-Defaults (new `Default*` constants): `maintenance` = `"auto"`,-`maintenance_interval` = 1h, `checkpoint_changesets` = 1000,-`checkpoint_pack_bytes` = 256 MiB, `checkpoint_retain` = 2, `gc_grace` = 1h,-`lease_ttl` = 10m. Validation (extend `GraftStorage.validate`): `maintenance in-{"", "off", "auto"}`; durations positive; `checkpoint_retain >= 1`;-thresholds `>= 0`; it is not an error to set them in shadow/off mode (they are-ignored, documented). `gc_grace` must comfortably exceed the Preparing→commit-window (bounded by `shadow_timeout` / the 10 s reftable lock wait +-`sync_timeout`); the default 1h does. Accessor helpers mirror-`SyncTimeoutOrDefault`.+Defaults are `Default*` constants with the values above. Validation (extends+`GraftStorage.validate`): `maintenance in {"", "off", "auto"}`; durations positive;+thresholds `>= 0`; `gc_margin < gc_grace`; **`gc_grace > sync_timeout + gc_margin`**+(GC BOUNDARY rule). `checkpoint_retain` of the first version does not exist.### RPCs (`proto/storage_core.proto`)-`StorageCoreService` is in the gate's `Passthrough`, so these RPCs are **not**-intercepted (no `AuthoritativeMutators` entry needed); they handle freshness-themselves.+`StorageCoreService` is in the gate's `Passthrough`, so these RPCs are not+intercepted and need no `AuthoritativeMutators` entry.```proto// CreateCheckpoint builds a consolidated checkpoint of the repository at the-// current head and records it, so that hydration and catch-up start from it.-// Requires authoritative mode.+// current head and records it. Authoritative mode only.rpc CreateCheckpoint(CreateCheckpointRequest) returns (CreateCheckpointResponse) {option (op_type) = { op: MUTATOR };}-// CollectGarbage reclaims the storage core packs the repository no longer needs-// (orphans past the grace period, and packs covered by a kept checkpoint) and,-// as a cheap global step, the packs of removed repositories. dry_run reports-// what it would delete without deleting anything.+// CollectGarbage reclaims the packs this repository no longer needs: orphans+// observed at least gc_grace ago, packs covered by a checkpoint at least gc_grace+// old, and superseded checkpoints. It does not touch other repositories.rpc CollectGarbage(CollectGarbageRequest) returns (CollectGarbageResponse) {option (op_type) = { op: MUTATOR };}+// CollectRemovedRepositories reclaims the packs of the repositories of a storage+// that were removed at least gc_grace ago.+rpc CollectRemovedRepositories(CollectRemovedRepositoriesRequest) returns (CollectRemovedRepositoriesResponse) {+ option (op_type) = { op: MUTATOR scope_level: STORAGE };+}message CreateCheckpointRequest { Repository repository = 1 [(target_repository)=true]; }message CreateCheckpointResponse {- uint64 checkpoint_sequence = 1; // the sequence the checkpoint is at- bool created = 2; // false when a checkpoint already existed at the head+ uint64 checkpoint_sequence = 1;+ bool created = 2; // false: a checkpoint at the head existed alreadyint64 pack_bytes = 3;- uint32 references = 4; // references in the snapshot+ uint32 references = 4;+ uint32 review_commits = 5; // open-review head/base commits included}message CollectGarbageRequest {@@ -640,342 +780,310 @@ message CollectGarbageRequest {bool dry_run = 2;}message CollectGarbageResponse {- uint64 checkpoint_sequence = 1; // newest kept checkpoint, 0 when none- uint32 packs_deleted = 2;- int64 bytes_reclaimed = 3;- uint32 covered_deleted = 4;- uint32 orphans_deleted = 5;- uint32 orphans_kept_in_grace = 6;- uint32 checkpoints_dropped = 7;- uint32 removed_repositories = 8;- bool dry_run = 9;+ uint64 boundary = 1; // C_gc+ uint32 covered_deleted = 2;+ uint32 checkpoints_dropped = 3;+ uint32 orphans_deleted = 4;+ uint32 orphans_observed = 5;+ uint32 orphans_in_grace = 6;+ uint32 packs_deleted = 7;+ int64 bytes_reclaimed = 8;+ bool dry_run = 9;+}++message CollectRemovedRepositoriesRequest {+ string storage_name = 1 [(storage)=true];+ bool dry_run = 2;+}+message CollectRemovedRepositoriesResponse {+ uint32 repositories = 1;+ uint32 packs_deleted = 2;+ int64 bytes_reclaimed = 3;+ bool dry_run = 4;}```-Handlers (`internal/gitaly/service/storagecore/checkpoint.go`, `gc.go`):-`CreateCheckpoint` → off/shadow → `FailedPrecondition`; MVCC/non-reftable →-`FailedPrecondition`; otherwise `Checkpointer.Create`. `CollectGarbage` → off →-`FailedPrecondition`; resolves the repository, `AcquireMaintenanceLease`-(contending → `Aborted`/`FailedPrecondition` "another maintenance operation is-in progress"), `CollectRepository` + `CollectRemoved`, release the lease. In-shadow mode `CollectGarbage` runs the orphan + removed-repo sweeps only (no-checkpoints exist, so there is no covered set) — see SHADOW SCOPE.+Handlers (`service/storagecore/{checkpoint,gc}.go`): off → `FailedPrecondition`;+`CreateCheckpoint` in shadow → `FailedPrecondition`; `CollectGarbage` acquires the+lease (held elsewhere → `FailedPrecondition` "another maintenance operation of+this repository is in progress"), collects, releases. WP78 checks the exact+`op_type` options the repository's proto lint accepts for a storage-scoped mutator+and follows the existing storage-scoped RPCs.### SDK and CLI-SDK (`sdk/storage_gc.go`, mirroring `sdk/storage.go`): `func (r *Repo)-StorageCheckpoint(ctx) (*StorageCheckpoint, error)` and `func (r *Repo)-StorageGC(ctx, dryRun bool) (*StorageGC, error)` with plain result structs.+SDK (`sdk/storage_gc.go`): `func (r *Repo) StorageCheckpoint(ctx)+(*StorageCheckpoint, error)`, `func (r *Repo) StorageGC(ctx, dryRun bool)+(*StorageGC, error)`, and `func (c *Client) StorageGCRemoved(ctx, storage string,+dryRun bool) (*StorageGCRemoved, error)`.-CLI (`cmd/graft/storage.go`, extending the dispatcher and the usage strings):+CLI (`cmd/graft/storage.go`):```graft storage checkpoint [-repo R] [--json]-graft storage gc [-repo R] [--dry-run] [--json]+graft storage gc [-repo R] [--dry-run] [--json]+graft storage gc --removed [-storage S] [--dry-run] [--json]```-Human output: `checkpoint` → `checkpoint at changeset N: K references, pack B-bytes` (or `already checkpointed at changeset N` when `created=false`); `gc` →-one line per category and a total, with `(dry run)` when `--dry-run`.+`gc -repo R` touches only that repository (answer 1); `--removed` is the storage-wide+sweep of removed repositories and takes no `-repo`.## Rejected alternatives-* **Store the reference snapshot by folding changeset metadata at hydration- time** instead of a `graft_checkpoint_refs` table. Rejected: folding is- `O(N)` metadata rows per hydration, and a self-contained snapshot makes the- checkpoint directly verifiable. The snapshot is `O(refs)` rows written once.-* **Delete changeset and changelog rows below a checkpoint** (full WAL- compaction). Rejected: the changelog is the ordered log clients follow and its- cursors must stay valid (`ListChangesets`/`WatchChangesets`); the rows are- cheap; and keeping them lets catch-up fold reference history above a checkpoint- without a special case. We compact packs, not rows.-* **A pending-record table for in-flight uploads** instead of a grace period on- orphans. Rejected for v1: it adds a database write to every write path (RULE- 4: shadow must not slow a write), while the Preparing→commit window is tightly- bounded and a generous `gc_grace` covers it at zero write-path cost. Kept as a- fallback if a bucket's mtime proves unreliable (noted as a risk).-* **Additive catch-up from a checkpoint** (index the checkpoint pack into the- existing cache of a node below the checkpoint, then replay) instead of a full- re-hydration. Rejected for v1: it entangles the checkpoint seed with the WP55- "never revert a local write" compare-and-swap logic; re-hydration is provably- correct and the case is rare. Noted as a possible later optimization.-* **Carry unreachable objects into the checkpoint pack** (pack `--all` of the- building node's local cache, not just the refs-at-N closure). Rejected: it- would defeat compaction (the cache accumulates every object ever seen) and no- reader needs those objects to reach head. See "unreachable objects" above.-* **No lease; rely on GC idempotency for concurrency.** Rejected: idempotent GC- is safe to race, but a GC on one node must not run while a checkpoint build on- another uploads a not-yet-committed checkpoint pack (it would look like an- orphan). One maintenance lease per repository makes checkpoint build and GC- mutually exclusive and is the invariant the assignment requires. (Orphan grace- still protects the checkpoint pack as defence in depth.)-* **Full-bucket scan to find removed repositories** (list `repos/<id>/` prefixes- and match against `graft_repositories`). Rejected: `O(all objects)` and racy- against concurrent creation. A tombstone written in the delete transaction is- crash-safe and bounded.+* **Database fencing of deletions** (a `graft_gc_doomed` set serialized with+ `CommitChangeset`, direction 2 of the PM review). It keeps objects shared between+ rows and adds a lookup to every commit under the reftable lock. It is not complete+ by itself: after the collector deletes the blob and removes the intent, a writer+ whose `Put` skipped the key before the deletion can commit with neither the intent+ nor the blob present, so the writer needs an additional protocol (re-upload after+ the intent clears, and keep re-checking) whose correctness depends on the order of+ bucket operations outside the transaction. The in-flight orphans of each write still+ need a time rule on top. Ownership removes the sharing instead, needs no write-path+ database access, and makes the in-flight bound an enforced deadline.+* **Bucket modification times for orphans** (the first version). An object's mtime+ is set by the bucket provider's clock, with no bound on skew against the+ collector, untested for S3/GCS/Azure here, and under the old `Put` an existing+ object's mtime is not refreshed. Replaced by the collector's own observations on+ the database clock.+* **A pending record per in-flight upload.** Rigorous, but one more database write+ per transaction on the write path (RULE 4), and it still needs an expiry for+ records of dead writers, i.e. the same deadline. The deadline alone suffices.+* **Restart a sync when a covered pack is missing** (B of the first review).+ Under the GC BOUNDARY rule a sync never meets a reclaimed pack, so this path would+ be unreachable; a missing pack is corruption and is reported as such.+* **`checkpoint_retain`.** The lagging boundary already keeps every checkpoint a+ reader can use; a count adds nothing.+* **Fold the reference snapshot from changeset metadata at hydration time**: `O(N)`+ rows per hydration; the snapshot table is `O(refs)` rows written once.+* **Delete changeset and changelog rows below a checkpoint**: the changelog cursors+ must stay valid and the rows are cheap.+* **Additive catch-up from a checkpoint** (index the checkpoint pack into a+ far-behind cache): entangles the seed with the WP55 compare-and-swap logic;+ re-hydration is correct and the case is rare (the node lagged by about+ `gc_grace − sync_timeout`).+* **Carry unreachable objects into the checkpoint pack**: defeats compaction.+* **Full-bucket scan to find removed repositories**: `O(objects)` and racy; the+ tombstone is crash-safe and bounded.## Invariants (binding; each has a test)-* **I1 (preserved).** Every object reachable from the current references of a- repository is in some pack the storage core keeps: the newest checkpoint's- pack holds the closure at its sequence, and every changeset pack after it is- kept, so the closure at head is covered.-* **GC-1.** GC never deletes a pack referenced by a changeset with sequence `>`- the newest kept checkpoint (they are in the live set).-* **GC-2.** GC never deletes a pack a write in progress may still commit: an- orphan is deleted only when older than `gc_grace`, which exceeds the- Preparing→commit window.-* **GC-3.** GC never deletes a pack a node hydrating from a kept checkpoint- reads: the newest checkpoint is always kept, and an older one is dropped only- when a newer exists and it is older than `gc_grace` (> `sync_timeout`, the- hydration budget).-* **GC-4.** At most one checkpoint build or GC per repository across nodes at a- time (the `graft_gc_leases` compare-and-swap).-* **GC-5.** A node whose watermark is below the newest checkpoint still reaches- head: it re-hydrates from the checkpoint; a covered pack missing mid-catch-up- restarts the sync into hydration.-* **CKPT-1.** A checkpoint at `N` holds every object reachable from the- references at `N`; hydrating from it and replaying `(N, head]` yields exactly- the state at head (references and connectivity).-* **CKPT-2.** Building a checkpoint never blocks a write and never fails one: it- reads a database snapshot and local objects outside any reference lock and- records the checkpoint in its own transaction.-* **COMPAT-1.** A v0.1.0 database (migration 1 only) works unchanged after- migration 2: with no checkpoints hydration replays from 1, and GC still- reclaims orphans and removed-repository packs.+* **OWN-1.** Every pack object is written by exactly one writer and referenced by at+ most one row (legacy keys excepted); `Put` never succeeds on an existing key.+* **OWN-2.** A writer commits a row referencing its pack only before `t_r + W`; no+ writer of a not-upgraded binary can commit a pack-carrying changeset (trigger).+* **I1 (preserved).** Every object reachable from the current references, and the+ head and base commits of the open reviews at the last checkpoint, is in a kept+ pack.+* **GC-1.** The collector never deletes a pack referenced by a changeset with+ `seq > C_gc` or a checkpoint with `seq >= C_gc`.+* **GC-2.** The collector never deletes a pack a write in progress may still commit+ (PACK OWNERSHIP proof).+* **GC-3.** No sync round and no verification ever needs a pack the collector+ deletes during it (GC BOUNDARY rule).+* **GC-4.** At most one checkpoint build or collection per repository at a time+ under normal operation (the lease); correctness does not depend on it.+* **GC-5.** A node whose watermark is below `C_sync` re-hydrates from the newest+ checkpoint; a node at or above it catches up without re-hydrating.+* **CKPT-1.** Hydrating from a checkpoint at `N` and replaying `(N, head]` yields+ exactly the state at head.+* **CKPT-2.** Building a checkpoint never blocks or fails a write.+* **REM-1.** A removed repository's prefix is swept only when its tombstone is at+ least `gc_grace` old, and no upload can arrive in it afterwards.+* **COMPAT-1.** A v0.1.0 database works after migration 2: no checkpoints means+ replay from 1; legacy keys stay readable and collectable.* **CRASH/IDEMPOTENT.** Every GC and checkpoint step is crash-safe and a re-run- converges; the database is authoritative over what is live; a `Delete` of a- missing key is a no-op.-* **HISTORY (preserved).** No changeset, changelog or review row is deleted by- compaction; `graft log`, `WatchChangesets`, reviews and the connectivity- check keep working after GC and compaction.--## Crash and race cases (each tested with an injected fault)--1. Checkpoint crash after the pack upload, before the row commit → orphan- checkpoint pack; reclaimed by the orphan sweep; a re-run rebuilds and commits.-2. Checkpoint crash after the row commit → the checkpoint exists; nothing to do.-3. GC crash mid-sweep → some dead packs remain; the next run reclaims them; no- live pack was deleted.-4. GC crash after `DeleteCheckpoint` (row), before the pack delete → the pack is- an orphan; reclaimed by the orphan sweep.-5. Removed-repository crash after a partial prefix delete, before the tombstone- delete → the tombstone remains; the next run finishes (idempotent).-6. A write uploads its pack (Preparing) while GC sweeps → the pack is younger- than `gc_grace`, so it is kept; after the commit it is a live changeset pack.-7. A checkpoint build on node A while GC runs on node B → prevented by the single- maintenance lease; the orphan grace also protects the in-flight checkpoint- pack.-8. A node catching up from `wm < C` while GC reclaims `(wm, C]` → the sync detects- `wm < C` and hydrates from the checkpoint; a covered pack missing mid-fetch- restarts the sync.-9. A node hydrating from the newest checkpoint while GC runs → GC keeps the- newest checkpoint pack and every pack `> C`, so the hydration is unaffected.-10. `RemoveRepository` with a concurrent hydration leaving a stale local copy →- the STALE CACHE RULE moves the local copy aside; the tombstone drives the- bucket cleanup; `repo_id` is never reused, so a recreated repository is- independent.-11. Two nodes race `AcquireMaintenanceLease` → exactly one wins; the other skips.-12. A checkpoint build node crashes holding the lease → the lease expires after- `lease_ttl`; GC and future builds resume.--## Test plan (multilevel; SQLite and Postgres everywhere a store is involved)--* **sqlstore unit** (`sqlstore_test`, both dialects via `coretest.Dialects()`):- `CreateCheckpoint` (monotonic sequence, `Seq <= head_seq`, idempotent at the- same sequence, snapshot round-trip including non-UTF-8 names and symbolic- targets); `NewestCheckpoint`/`ListCheckpoints`/`CheckpointRefs`/`DeleteCheckpoint`- (cascade of the snapshot); `ChangesetPacks` (only non-empty keys, ascending);- lease acquire/renew/release with a CAS race between two `Store`s and expiry;- `DeleteRepository` writes a tombstone and cascades, re-registration gets a new- `repo_id` and a fresh tombstone space; `ListRemovedRepositories`/- `DeleteRemovedRepository`; `ListRepositories`.-* **blobstore unit** (`blobstore_test`, file + mem): `List` under a prefix with- correct `ModTime`/`Size`, directory markers skipped; `Delete` idempotent on a- missing key; `List`/`Delete` leave other keys untouched; S3 variant behind- `GRAFT_TEST_S3_URL` only.-* **gitops unit** (`gitops_test`, SHA-1 and SHA-256): `BuildPackFrom` yields a- self-contained pack whose index into an empty repository makes the tips'- history connectivity-complete (`git fsck`); an empty tip set is an empty pack.-* **checkpoint unit** (`checkpoint_test`, `coretest` + `gittest`): build a- checkpoint from a source repository, hydrate a fresh repository from the- checkpoint and replay, assert references and `git fsck` connectivity equal the- source at head; an object unreachable before the checkpoint is absent after- compaction (the documented behaviour); `created=false` when a checkpoint- already exists at head.-* **gc unit** (`gc_test`, `coretest` + `gittest`, both dialects): an orphan is- kept within grace and reclaimed after it; covered packs (`<= C`) are deleted,- packs `> C` and the kept checkpoint pack survive; a dropped old checkpoint's- pack is reclaimed; a removed-repository prefix is cleared and its tombstone- removed; `--dry-run` deletes nothing and reports the same counts; a re-run is- a no-op.-* **-race** (`syncer_test`/`gc_test`/e2e): GC concurrent with continuous lands- (every land keeps committing, no live pack is deleted, no CAS or connectivity- break); GC concurrent with a hydration (the hydration completes); a checkpoint- build concurrent with lands (writes are not blocked and the checkpoint is- correct); re-hydration of a node forced below the checkpoint.-* **crash injection** (`checkpoint.FaultHook`, `gc.FaultHook`, and the real-binary- `CrashEnv` for e2e): each fault point of §"Crash and race cases" 1–5 and 12,- each followed by a re-run that converges and a `VerifyRepository- --connectivity` that passes.+ converges; the database decides liveness.+* **HISTORY.** No changeset, changelog or review row is deleted by compaction.++## Crash and race cases (each tested)++1. Two writes with byte-identical packs (the dedup race of the review): each uploads+ its own key; the collector deletes the first's covered key; the second write's+ changeset and pack are intact (test with a fault hook between the second write's+ upload and its commit, the collector deleting in between, SQLite and Postgres).+2. A write whose upload is older than its deadline at Prepared time: it rebuilds and+ re-uploads a new owned pack; one whose commit cannot start before the deadline+ fails like a storage failure (test with a shortened `gc_grace`).+3. An unreferenced pack observed, then committed by its writer before its deadline:+ the next run forgets the observation and keeps the pack.+4. A not-upgraded writer inserts a legacy key: the trigger rejects it (test inserts a+ changeset with a legacy key through `CommitChangeset` bypassing `NewPackKey`).+5. Checkpoint crash after the upload, before the row: the pack is an orphan; a re-run+ commits a new checkpoint.+6. Collector crash after `DeleteCheckpoint`, before the pack deletion; mid-sweep;+ after a partial prefix deletion of a removed repository: the next run converges.+7. A node one changeset behind when a checkpoint is created catches up and does not+ re-hydrate (change C).+8. A node whose watermark is below `C_sync` re-hydrates and reaches head.+9. Hydration while the collector runs; GC concurrent with continuous lands; a+ checkpoint build concurrent with lands (all `-race`).+10. A checkpoint build on a copy that lags behind the head (change D): EnsureFresh+ brings it to `>= N` and the checkpoint is complete.+11. A write in flight on a repository being removed: its pack lands before the sweep+ (bound `t_r + W`), the sweep at `removed_at + gc_grace` deletes it.+12. Two nodes race for the lease: one wins; a holder that crashes lets the lease+ expire after `lease_ttl`.+13. An open review whose commits are missing locally: the checkpoint fails with+ `FailedPrecondition` naming the review; GC still runs.++## Test plan (multilevel; SQLite and Postgres wherever a store is involved)++* **sqlstore unit** (both dialects): every new method; `CreateCheckpoint`+ monotonicity, idempotency (`created=false`), owned-key and `seq <= head`+ checks, snapshot round trip with non-UTF-8 names and symbolic targets;+ `GCSnapshot` maxima per key with legacy shared keys; `RecordOrphans` keeps an+ existing `first_seen`; lease CAS between two `Store`s and expiry on the database+ clock; `DeleteRepository` tombstone and cascade; `ListRemovedRepositories` age+ filter; `SyncBoundary`; the trigger rejects a legacy key and accepts an owned one+ and NULL; migration 2 on a migration-1 database (fixture) and the trigger bodies+ executed as single statements.+* **blobstore unit** (file and mem): `Put` on an existing key is `ErrBlobExists` and+ leaves the content unchanged; concurrent `Put`s of one key: exactly one wins;+ `List` by prefix, sizes, no directory entries; `Delete` idempotent.+* **types unit**: `NewPackKey`/`ParsePackKey`/`IsOwnedPackKey` round trips, legacy+ parsing, non-canonical spellings refused, nonce uniqueness.+* **gitops unit** (SHA-1 and SHA-256): `BuildPackFrom` is self-contained (`fsck` after+ indexing into an empty repository); `MissingObjects`.+* **voter/import unit**: owned keys on every upload; the deadline (prepared pack+ expired → rebuilt; commit past the deadline → refused); the phase-6 voter, import,+ syncer, gate and storagecore suites stay green.+* **checkpoint unit**: hydration from a checkpoint equals the source at head+ (references, `fsck`); open-review commits survive the deletion of their branch;+ closed-review commits do not (documented limit); lagging copy (change D); missing+ review objects → `FailedPrecondition`; `created=false`.+* **gc unit**: every branch of the sweep; covered vs live per key; dropped+ checkpoints; observations (new, in grace, expired, referenced since); removed+ repositories younger and older than `gc_grace`; dry run deletes and writes+ nothing; re-run is a no-op.+* **-race**: crash/race cases 1, 9 and 12.+* **crash injection**: every `FaultHook` point of the checkpoint and gc packages,+ and the voter's `after_upload`, each followed by a re-run that converges and a+ `VerifyRepository --connectivity` that passes.* **e2e** (`internal/graft/e2e/storage_gc_test.go`, two and three real gitaly- processes sharing one database and one bucket, SQLite and Postgres): node A- writes many changesets; `graft storage checkpoint`; `graft storage gc`; a fresh- node B hydrates from the checkpoint (asserted: it downloads the checkpoint pack- and only the changeset packs after it, via `graft_storage_sync_*` and the- bucket), serves, and `VerifyRepository --connectivity` is consistent; a node- that fell below the checkpoint re-hydrates; a repository removed on one node- has its packs reclaimed; the changelog and reviews are intact after GC.-* **migration**: open a database created with migration 1 only (fixture), assert- migration 2 applies, and that hydration, catch-up, checkpoint, GC and verify- all work with zero checkpoints and zero tombstones.-* **measurement** (ask the PM for a benchmark window — the machine is shared):- bucket size and fresh-node hydration time before and after a checkpoint + GC,- on a repository with many changesets. Numbers go into STORAGE.md only from a- real run.+ processes, one database and one bucket, SQLite and Postgres): writes on A,+ `graft storage checkpoint`, `graft storage gc` with a shortened `gc_grace`, a fresh+ node B hydrates from the checkpoint (asserted from the downloaded keys and the+ `graft_storage_sync_*` metrics), serves, verify is consistent; a node below+ `C_sync` re-hydrates; `graft storage gc --removed` reclaims a removed repository;+ changelog and reviews intact.+* **measurement** (WP79, in the benchmark window the PM schedules): bucket size and+ fresh-node hydration time before and after a checkpoint and GC on a repository+ with many changesets; the preparing/prepared histograms before and after WP73.## Acceptance criteria-* After `checkpoint_changesets` writes, a checkpoint plus GC leaves the bucket- holding the newest checkpoint pack and the packs after it, not the whole- history (measured bucket size drops; orphans and removed-repository packs are- gone).-* A fresh node's hydration time is a function of `head - newestCheckpointSeq`,- not of `head` (measured; constant as history grows past a checkpoint).-* `graft storage verify --connectivity` reports `consistent` after GC and- compaction; `graft log`, `WatchChangesets` and reviews are unchanged.+* After a checkpoint and GC the bucket holds the kept checkpoint packs and the packs+ after `C_gc`, not the whole history; orphans and removed-repository packs are gone+ (measured).+* Fresh-node hydration time depends on `head − newest checkpoint`, not on `head`+ (measured).+* `graft storage verify --connectivity` is consistent after GC and compaction;+ `graft log`, `WatchChangesets` and reviews are unchanged.* A v0.1.0 database migrates and every storage path works.-* All phase-6 storage tests stay green; the new unit, `-race`, crash and e2e- tests pass on SQLite and Postgres.-* No live pack is ever deleted (GC-1..GC-5), proven by the `-race` and crash- tests.+* Every phase-6 storage test is green (the blobstore `Put` tests rewritten to the+ new contract, stated in the report); the new tests pass on SQLite and Postgres.+* No live pack is ever deleted (OWN-1, OWN-2, GC-1..GC-3), proven by the race and+ crash tests.## SHADOW SCOPE rule (binding)-Checkpoints and background maintenance are **authoritative-mode only**: a-checkpoint's purpose is to seed hydration, which does not happen in shadow or-off mode, and its reference snapshot is the database's view, which is not-authoritative in shadow mode. `CreateCheckpoint` and the background `Runner`-therefore refuse / do nothing outside authoritative mode. **GC** of orphans and-of removed-repository packs is useful in shadow mode too (shadow uploads a pack-per recorded write, and a dead shadow write leaves an orphan), so `CollectGarbage`-and `CollectRemoved` run in shadow mode as well, doing the orphan and-removed-repository sweeps only (there are no checkpoints, hence no covered set).-Off mode opens no bucket: every entry point returns `FailedPrecondition`.--## Work split: what Opus writes, what Haiku drafts--Per the engineering charter, Opus (the main chat) writes everything that deletes-data, is concurrent, is a compare-and-swap, or is crash recovery; Haiku drafts-scoped support from an exact spec, actively monitored, output checked by Opus.--**Opus writes:**--* the migration (`sqlstore/migrations.go`) and all new `sqlstore` methods- (checkpoint transaction, lease CAS, tombstone, the changed `DeleteRepository`,- `ChangesetPacks`, `ListRepositories`, `ListRemovedRepositories`);-* the `BlobStore` `List`/`Delete` interface and the `blobstore` implementation;-* `checkpoint.Checkpointer` (upload-then-commit, crash points);-* `gc.Collector` (the mark and sweep that deletes packs, and removed-repository- cleanup);-* the syncer changes (hydration from a checkpoint, the below-checkpoint- re-hydration decision, the covered-pack-missing restart);-* the `maintenance.Runner` (the lease-guarded concurrent background task);-* the `verify_objects.go` change;-* `gitops.BuildPackFrom`;-* the config validation, the proto, the service handlers, and the `serve.go`- wiring;-* the integration, `-race` and crash tests, and the e2e test.--**Haiku drafts (exact spec, Opus verifies):**--* the spec tests `zz_*_spec_test.go` for every WP (read only this file and the- foundation signatures), under `<workspace>/spec-tests7/<WP>/...`;-* the CLI subcommand plumbing in `cmd/graft/storage.go` (flag binding, dispatch,- human/JSON output) mirroring the existing `import|state|verify`;-* the SDK result structs and methods mirroring `sdk/storage.go`;-* first drafts of the STORAGE.md and CONFIGURATION.md prose and tables (Opus- edits for correctness and fills the measured numbers);-* table-driven unit-test scaffolding for the pure `sqlstore` methods from the- exact specs above (Opus owns the concurrency and deletion tests).+Checkpoints and the background task are authoritative-mode only: a checkpoint seeds+hydration, which does not happen in shadow mode, and its snapshot is the database's+view, which is not authoritative there. `CollectGarbage` and+`CollectRemovedRepositories` also run in shadow mode (orphans and removed+repositories only; no checkpoints means no covered set). The pack ownership rule,+the trigger and the writers' deadline apply in both modes. Off mode opens no bucket:+every entry point returns `FailedPrecondition`.++## Work packages and split++| WP | content | depends on |+|---|---|---|+| WP71 | migration 2 (tables, trigger), every new `sqlstore` method, `DeleteRepository` tombstone | — |+| WP72 | `BlobStore` `List`/`Delete`, strict `Put`, owned pack keys in `types.go` | — |+| WP73 | writers: owned keys and the deadline in the voter, `shadowpack.go`, import; `Core` fields | WP71, WP72 |+| WP74 | `gitops.BuildPackFrom`/`MissingObjects`, `checkpoint` package | WP71–WP73 |+| WP75 | `gc` package | WP71–WP73 |+| WP76 | syncer (seed, `C_sync`), `verify_objects.go` | WP71, WP74 |+| WP77 | config, `maintenance` package, `serve.go` wiring | WP74–WP76 |+| WP78 | proto, handlers, SDK, CLI | WP74, WP75 |+| WP79 | e2e, STORAGE.md and CONFIGURATION.md, benchmark | all |++**Opus writes** everything that deletes data, is concurrent, is a compare-and-swap+or is crash recovery: WP71–WP77 entirely, the handlers of WP78, and the race, crash+and e2e tests. **Haiku drafts** from an exact spec, monitored and checked by Opus:+the `zz_*_spec_test.go` spec tests of each WP (from this file and the signatures+only, in `/home/feder/m3-work/spec-tests7/<WP>/...`), the CLI and SDK plumbing of+WP78, table-driven scaffolding for the pure `sqlstore` methods, and first drafts of+the STORAGE.md and CONFIGURATION.md prose (Opus fixes and fills the measured+numbers). Work files live in `/home/feder/m3-work/`, never in `/tmp` (PM.md).## Risks-* **Bucket mtime reliability.** The orphan grace uses `BlobInfo.ModTime`. File- and mem buckets are exact; S3/GCS/Azure are untested here. A provider with a- coarse or wrong mtime could, in principle, reclaim a fresh orphan. Mitigation:- a generous default grace (1h ≫ the write window); the pending-record- alternative is kept in reserve. Documented as an S3-untested limit.-* **`pack-objects` non-determinism.** Two nodes could build byte-different- checkpoint packs for the same `N`; the maintenance lease makes concurrent- builds a non-case, and a resumed build after a crash simply uploads a fresh- content-addressed pack.-* **Re-hydration discards a far-behind cache's local objects** (correct but- wasteful); rare, because an actively-served node stays near head.-* **Unreachable-object pruning** can make a *closed/abandoned* review's diff- unavailable if its commits are reachable from no reference at the checkpoint;- usually reviews target live branches. Open question below.-* **Unbounded local-cache growth** (no local `git gc`) is out of scope; an- operator reclaims local disk by removing and re-hydrating a cache. Noted in- STORAGE.md.-* **Lease held by a crashed node** blocks maintenance of that repository until- `lease_ttl` expires (bounded; default 10m).-* **Large-bucket listing** for removed-repository cleanup is `O(objects under- the prefix)`; per-repository GC is bounded by one repository's prefix.+* **Mixed-version clusters.** Old running nodes fail their pack-carrying writes as+ soon as migration 2 is applied (intended, the trigger); operators must upgrade every+ node together. Documented in STORAGE.md.+* **Lost dedup.** Byte-identical packs are stored once per write; measured.+* **Writer deadline.** A write slower than `gc_grace − gc_margin` (55 min by default)+ fails; an import of a very large repository must fit in it after its pack build+ (the deadline starts after the build).+* **Unreachable-object pruning** loses the diff of closed reviews whose commits are+ unreachable at a checkpoint (documented limit).+* **Missing open-review objects** block checkpoints of the repository until the+ review is closed; the root fix is in the review service (open question).+* **Re-hydration** discards a far-behind cache's objects (rare).+* **Lease held by a crashed node** delays maintenance by at most `lease_ttl`.+* **S3/GCS/Azure untested**; the design reads no bucket times, so the clock risk of+ the first version is gone, but `List` and conditional `Put` are exercised on file+ and mem buckets only.++## Resolved questions (PM review of 85189d4)++1. `graft storage gc -repo R` touches only that repository; removed repositories are+ swept by `graft storage gc --removed` and by the background task.+2. The checkpoint closure includes the head and base commits of open reviews.+3. `maintenance` defaults to `off` in this release.+4. The benchmark window is requested at WP79.## Open questions for the PM-1. **`graft storage gc` scope.** This plan makes `CollectGarbage` repository-- scoped (matching `import`/`verify`'s `-repo`) and folds the removed-repository- tombstone sweep into every call as a cheap global step. Would you prefer a- separate storage-scoped `graft storage gc --all` that iterates every- repository in a storage plus the tombstones? (The background task already- covers everything; this is only about the operator CLI shape.)-2. **Open-review objects.** Should the checkpoint pack also retain objects- reachable from the tips of **open** reviews, so a review's diff survives- compaction even when its branch was deleted? It adds a `graft_reviews` read- and extra tips to the pack build. Default in this plan: no (documented limit).-3. **Default `maintenance = "auto"`.** The background task runs by default in- authoritative mode. Do you want it `off` by default for the first milestone-3- release (operators opt in), with only the CLI available?-4. **Benchmark window.** I need a quiet window on the shared machine to measure- bucket size and hydration time before/after for the STORAGE.md numbers (the- other workstream's tests skew it, as the LOCK AND HISTORY benchmark noted).+1. **Refinement of B and C** (GC BOUNDARY rule): decide catch-up versus re-hydration+ with `C_sync` and keep a missing pack an Internal error, instead of adding a+ restart on a missing covered pack, which this boundary makes unreachable. Confirm+ before WP76.+2. **Open reviews whose commits the storage core no longer holds**: the plan fails+ the checkpoint and names them. The source of the condition is the review service+ accepting such a review in authoritative mode; should p4b/the PM own a change+ there, or may m3 add that check?## Foundation touchpoints and ownership-New files (mine): `internal/graft/corestore/checkpoint/**`,-`internal/graft/corestore/gc/**`, `internal/graft/corestore/maintenance/**`,-`internal/gitaly/service/storagecore/{checkpoint,gc}.go`,-`sdk/storage_gc.go`, `internal/graft/e2e/storage_gc_test.go`.-Changed (mine): `sqlstore/**`, `corestore/{core,types,metastore}.go`,-`blobstore/**`, `gitops/gitops.go`, `syncer/syncer.go`,-`service/storagecore/verify_objects.go`, `cmd/graft/storage.go`,-`proto/storage_core.proto` (+ generated code), `config_graft_storage.go`,-`doc/graft/STORAGE.md`.-Shared (my section only, listed in the report): `internal/cli/gitaly/serve.go`-(the storage core block — wire the maintenance runner),-`doc/graft/CONFIGURATION.md` (the `[graft.storage]` section).-I do not touch the land path (`internal/graft/landlane/**`, `land*.go`): the-checkpoint reads a database snapshot and the local objects, and GC touches only-the bucket and the database, so phase 7 needs no land-path change. If-implementation proves otherwise I stop and tell the PM.--## RULES (binding for every WP, in addition to PLAN6 RULES 1–15)--16. **Packs only.** Compaction deletes packs, never changeset, changelog or- review rows. The changelog and its cursors are untouched.-17. **Database authoritative over liveness.** GC computes the live set from the- database every run and never deletes a pack a committed changeset `> C` or a- kept checkpoint references.-18. **Grace before orphan deletion.** An unreferenced pack is deleted only when- older than `gc_grace`.-19. **One maintenance operation per repository across nodes** (the lease).-20. **Checkpoints are authoritative-mode only**; GC of orphans and removed- repositories also runs in shadow mode (SHADOW SCOPE).-21. **Crash-safe and idempotent**: upload before record; row before pack delete;- tombstone before prefix delete; a `Delete` of a missing key is success.-22. **Untrusted data** (RULE 10 extended): a checkpoint pack and its snapshot- read from the database or the bucket are validated (pack SHA-256 against its- key and `repo_id`; names and targets with git's rules) before anything- touches a local repository.+New (m3): `corestore/checkpoint/**`, `corestore/gc/**`, `corestore/maintenance/**`,+`service/storagecore/{checkpoint,gc}.go`, `sdk/storage_gc.go`,+`internal/graft/e2e/storage_gc_test.go`. Changed (m3): `sqlstore/**`,+`corestore/{core,types,metastore}.go`, `blobstore/**`, `gitops/gitops.go`,+`voter/**`, `syncer/syncer.go`, `open/open.go`, `service/storagecore/{import,verify_objects}.go`,+`cmd/graft/storage.go`, `proto/storage_core.proto` (+ generated code),+`config_graft_storage.go`, `doc/graft/STORAGE.md`. Shared, my section only:+`internal/cli/gitaly/serve.go`, `doc/graft/CONFIGURATION.md`. No change to the land+path: the voter is m3's; the land paths call it unchanged.++## RULES (binding, in addition to PLAN6 RULES 1–15)++16. **Packs only.** Compaction deletes packs, never changeset, changelog or review rows.+17. **Ownership.** Every upload uses a fresh owned key; `Put` never succeeds on an+ existing key; no code produces a legacy key.+18. **Enforced deadline.** A writer uploads and commits only before `t_r + W`.+19. **Observed age on the database clock.** An unreferenced pack is deleted only after+ the collector observed it unreferenced at least `gc_grace` earlier, decided in one+ snapshot; bucket times are never read.+20. **Lagging boundaries.** The collector uses `C_gc`; a sync decides with `C_sync`.+21. **Checkpoints are authoritative-mode only**; GC of orphans and removed repositories+ also runs in shadow mode.+22. **Untrusted data** (RULE 10 extended): checkpoint packs, snapshots and review+ commit IDs read from the database or the bucket are validated before they touch a+ local repository.+23. **Crash-safe order**: upload before record; row before pack deletion; prefix before+ tombstone deletion; `Delete` of a missing key is success.+24. **No fallback paths**: a pack the invariants say exists and does not is reported as+ corruption, never routed around.