Git designed for AI agents.
Graft phase 6: the storage core (binding plan)
Status: binding contract for WP51–WP61. The Opus foundation (this file, the
internal/graft/corestore types and interfaces, the config, the proto and the
wiring stubs) is fixed. A work package changes a foundation file only when a
rule below says so. Anything that seems to need a change elsewhere goes into
needs_outside_change in the WP report; it is not done silently.
Goal
Make the repository state of Graft live in a strongly consistent database plus object storage, the way Scion does, so that Gitaly servers become re-creatable caches that scale horizontally:
- Every reference change is a changeset appended to a per-repository
write-ahead log in the database (
graft_changesets), with a per-reference compare-and-swap againstgraft_refs, in one database transaction. - The objects of a changeset are uploaded before the changeset is
committed, as one content-addressed, immutable pack in object storage
(
repos/<repo_id>/packs/<sha256>.pack). - The Graft changelog (ChangesetService) and reviews are written in the same database, atomically with the reference change for the changelog.
- Any node can serve any repository: before serving, it catches up its local git repository (the cache) from the database and the object storage, or builds it from nothing (hydrate).
Honest limit, to be stated in the docs: git still runs on a local repository. "Stateless" means that the local repository is a cache that any node can rebuild from the database and the object storage, not that nodes keep no local state. There is no virtual filesystem in this phase.
This phase delivers milestone 1 (shadow mode, import, verify) and milestone 2 (authoritative mode, any node serves and hydrates). Garbage collection of packs, checkpoint compaction and per-directory permissions are milestone 3, after the first release.
Modes ([graft.storage] mode)
off(default, also the empty string): nothing changes. No database or bucket is opened.shadow: every reference transaction is also committed to the database and its pack uploaded, but the database never rejects a write: on a CAS mismatch the changeset is recorded withforced = trueand thegraft_storage_shadow_divergences_totalcounter is incremented. Any shadow error (database down, upload failure, timeout, panic) is logged and counted (graft_storage_shadow_errors_total) and the git write proceeds. The file changelog and the file review store stay the source of truth. Reads are not affected. Shadow mode works with or without Praefect.authoritative: the database is the source of truth. A reference transaction commits only if its database CAS succeeds (commit point = the Prepared phase of the reference-transaction hook). Reads and writes first catch up the local repository (EnsureFresh). The changelog and reviews are served from the database. Requiresgraft.standalone = true; it is rejected by config validation otherwise (Praefect is not supported in authoritative mode, see RULE 13).
Configuration (foundation, internal/gitaly/config/config_graft.go)
[graft.storage]
mode = "authoritative" # off | shadow | authoritative
database = "sqlite:///var/opt/graft/meta.db" # or postgres://user@host/db?sslmode=disable
bucket = "file:///var/opt/graft/objects" # gocloud URL: file://, mem://, s3://, gs://, azblob://
node_id = "node-a" # default: os.Hostname()
sync_timeout = "10s" # EnsureFresh budget per RPC
shadow_timeout = "5s" # shadow work budget per transaction
Credentials are never in the TOML: Postgres uses the standard PG*
environment variables and .pgpass (pgx honours them), buckets use the
standard cloud SDK environment (AWS_*, GOOGLE_APPLICATION_CREDENTIALS,
AZURE_*), as internal/backup/sink.go does.
Data model (foundation, internal/graft/corestore/sqlstore/migrations.go)
The schema is fixed by the foundation; WP51 implements it. Both dialects
(SQLite and Postgres) have the same tables and columns. Names in Go are in
internal/graft/corestore/types.go.
graft_repositories(repo_id PK, storage, relative_path, object_format, head_seq, log_seq, next_review_id, created_at, UNIQUE(storage, relative_path)).repo_idis never reused: a repository removed and created again gets a new one, which invalidates the local caches of the old one.graft_refs(repo_id, name BLOB, target TEXT, seq, PK(repo_id, name)): the current references.targetis a hex object ID orref:<name>for a symbolic reference.HEADis a row like any other.graft_changesets(repo_id, seq, operation, actor, node_id, pack_key NULL, pack_bytes, forced, created_at, PK(repo_id, seq)): the WAL.seqis dense per repository and starts at 1.graft_changeset_refs(repo_id, seq, ordinal, name BLOB, old_target NULL, new_target NULL, PK(repo_id, seq, ordinal)): the effective reference changes of a changeset.old_targetis the actual previous database value (NULL = absent),new_targetNULL = deleted.graft_changelog(repo_id, log_seq, entry TEXT, PK(repo_id, log_seq)): the Graft changelog (heads and tags only),entry=changelog.EntryJSON. Written in the same transaction as the changeset.graft_nodes(repo_id, node_id, applied_seq, updated_at, PK(repo_id, node_id)): how far each node's cache is (observability only, never used for decisions).graft_reviews(repo_id, review_id, version, target_branch BLOB, closed, body TEXT, updated_at, PK(repo_id, review_id)): reviews in authoritative mode (body= protojson ofgitalypb.Review).
All child tables reference graft_repositories with ON DELETE CASCADE.
Reference CAS semantics (RULE 5, binding)
The reference-transaction hook receives <old> <new> <ref> lines. git
reports the all-zero old value both for "must not exist" and for "forced".
The voter tells them apart in the Prepared phase, where git holds the locks
and the local repository still shows the old values:
| hook old value | local ref now | RefUpdate.Expect |
database check |
|---|---|---|---|
non-zero X (or ref:T) |
any | ExpectValue, Old = X |
current must equal X |
| zero | absent | ExpectValue, Old = "" |
ref must be absent |
| zero | exists | ExpectAny |
none (forced) |
A changeset whose updates all leave the database unchanged (for example a
forced deletion of an absent reference) writes nothing and returns
CommitResult{NoOp: true}. On mismatch the store returns *ErrRefConflict
listing every mismatching reference (expected and actual) and writes nothing
— unless ChangesetInput.Force is set (shadow mode), in which case it writes
the changeset with forced = true and returns the conflicts in
CommitResult.Conflicts.
The voter converts *ErrRefConflict into a
git/updateref.MismatchingStateError (first conflict; ActualObjectID = the
database value, zero OID if absent) wrapped in the hook error, so that
UserLand's lost-race retry (land.go, updateLandedReference) classifies
it as a conflict exactly like a git lock-time mismatch.
Files backend (superseded rule, see "REFTABLE RULE"): the earlier text said to skip force-deletions-only transactions on the files backend. That was wrong: for a deletion without an expected old value the loose half is force-only too, so both halves were skipped and the database kept a reference git had deleted. The binding rule is now the REFTABLE RULE below.
Write path (WP54, binding order)
In GitLabHookManager.ReferenceTransactionHook, Prepared phase, after
enforceGraftPolicy and before the Praefect vote (foundation wires the call):
- Parse all lines (all references, not only heads and tags; symbolic values
ref:<target>are accepted). Skip per the files-backend rule. - Resolve the repository in the database by
(storage, relative_path). Shadow: not tracked → skip silently (countergraft_storage_untracked_writes_total). Authoritative: not tracked →FailedPrecondition(the gate normally stops this earlier). - Compute the expectations (table above) with one
for-each-refrun. - Build the pack: objects reachable from the new non-zero object targets and
not reachable from any current local reference
(
git rev-list --objects --stdinwith--not --all, piped intogit pack-objects --stdout), written to a temporary file while hashing (SHA-256). Empty → no pack. - Upload the pack (
BlobStore.Put, create-only; an existing key is success). MetaStore.CommitChangesetwith the updates, operation/actor/members meta (from the same helper the file changelog uses), node ID, pack key and size, andChangelogmeta in authoritative mode.- Authoritative: on
*ErrRefConflictreturn the mismatching-state error → git aborts the transaction. On any other error returnUnavailable(structerr.NewUnavailable) → git aborts. On success remember(repo_id, seq)for the Committed/Aborted phases (process-local map keyed by SHA-256 of relative path + hook input, TTL 15 minutes, at most 4096 entries). - Committed phase: call
LocalApplier.NoteCommitted(repoPath, repoID, seq)so the node's watermark can advance without downloading its own pack. Never blocks (TryLock semantics, see syncer). - Aborted phase after a successful database commit (git failed to commit
locally): log at error level and count
graft_storage_local_commit_failures_total; the database is the truth and the next EnsureFresh repairs the local cache.
In authoritative mode the file changelog is not written (the database changelog replaces it); in shadow mode it is written as today.
Repositories that use the MVCC reference backend are skipped in shadow mode
(counter graft_storage_mvcc_skipped_total) and rejected with
FailedPrecondition in authoritative mode.
Read path and catch-up (WP55, binding)
Syncer.EnsureFresh(ctx, repo) (authoritative only; no-op otherwise):
- Read
Repositoryby key. Not found → if a local repository exists it is untracked →FailedPrecondition("repository is not imported into the storage core"); otherwiseNotFound. - Read the local watermark
<gitdir>/graft/storage/applied(JSON{"repo_id":N,"seq":M}). Missing local repository, or watermark with a differentrepo_id→ hydrate (below).seq == head_seq→ return. - Catch up: list changesets
(seq, head_seq], download and verify their packs,git index-pack --stdineach into the repository, then compute the final value of every changed reference as of head_seq and apply them in oneupdate-ref --stdintransaction without hooks (updateref.WithDisabledTransactions, never the UpdaterWithHooks), then write the watermark (temp file + fsync + rename) and report it (ReportApplied, best effort, at most once per second per repository). - Hydrate: create a bare repository with the recorded object format and the
reftable backend in the storage's temporary directory, apply all
changesets from 1, set
HEAD, write the watermark, then rename it into place. A local repository with a differentrepo_idis moved away to the storage's temporary directory first (never deleted in place). - Single flight per repository per node; a per-repository mutex protects the
apply + watermark section. Lock contention from git (a concurrent local
write holds the reference lock) is retried with backoff until
sync_timeout, thenUnavailable. - The watermark only advances contiguously:
NoteCommitted(seq)withseq == applied+1advances it (and any already-noted higher contiguous seqs), otherwise the seq is remembered in memory and the next catch-up covers the gap.
Untrusted input (RULE 10): reference names read from the database are checked
with git's reference-name rules (git.ValidateReference or equivalent) and
targets with the repository's object hash before anything touches the local
repository; a pack's SHA-256 must match its key.
Gate: which RPCs run in authoritative mode (WP56, binding)
A unary and a stream interceptor (internal/graft/corestore/gate), installed
on every listener through server.WithUnaryInterceptor /
WithStreamInterceptor (foundation wiring), apply only in authoritative
mode:
/gitaly.HookService/*and/gitaly.StorageCoreService/*are passed through untouched (no EnsureFresh: hooks run while git holds locks).- Repository-scoped RPCs (from
protoregistry.GitalyProtoPreregistered):- MUTATOR not in the allowlist →
FailedPrecondition("<method> is not supported in authoritative storage mode"). - Everything else →
EnsureFreshfirst, then the handler.
- MUTATOR not in the allowlist →
RepositoryService/CreateRepository: after the handler succeeded, register the repository with an initial changeset containing itsHEAD(operationcreate_repository). If the registration says the repository exists, the local repository is moved to the temporary directory andAlreadyExistsis returned. CreateRepository does not call EnsureFresh first.RepositoryService/RemoveRepository: delete the database row first, then run the handler; a not-tracked repository is just removed locally.- Storage- and server-scoped RPCs pass through.
The allowlist lives in internal/graft/corestore/gate/policy.go (foundation
seeds it). WP56 proves every entry with an authoritative-mode test (the RPC's
reference changes reach the database: VerifyRepository reports no
divergence afterwards) and removes an entry it cannot prove, reporting it.
Adding an entry needs that proof.
Land retries: the foundation adds a call to deps.GraftStorageSyncer
(EnsureFresh, no-op when nil) before every retry attempt of the single land
path and the batch executor, so a retry after a database conflict sees the
new tip.
Import, state and verify (WP57)
StorageCoreService (proto/storage_core.proto, foundation):
ImportRepository(mutator): registers a local repository that the database does not know, in one transaction: changeset 1 with operationimport, all references (includingHEAD), one pack with every object reachable from them, the file changelog entries with their sequences preserved (so consumers' cursors stay valid) and the file reviews. Fails withAlreadyExistswhen tracked. Works in shadow and authoritative mode.GetStorageState(accessor): mode, tracked, repository ID, head sequence, this node's applied sequence, node ID.VerifyRepository(accessor): authoritative → EnsureFresh first; then compares every local reference with the database (divergences listed) and, when asked, runsgit fsck --connectivity-only.graft storage import|state|verifysubcommands of thegraftCLI, through the SDK.
Changelog and reviews in the database (WP58, WP59)
- WP58:
changelog.Sourceinterface (Scan, Subscribe) with the file implementation unchanged and a database implementation (Scan fromgraft_changelog; Subscribe = in-process notification from the voter plus polling, every 250 ms while a watcher is active). ChangesetService uses the database source in authoritative mode. - WP59:
review/store.ReviewStoreinterface extracted from*store.Store(same method set) and a database implementation (next_review_idallocation andversionoptimistic concurrency with retry inUpdate). The review server uses it in authoritative mode.
Prerequisites (WP60)
internal/git/mvcc/manifest.go:ParseManifestmust not panic on crafted TOC offsets (uint64 >= 2^63), plus fuzz corpus entries.- Verify that a fetched manifest hashes to its requested name.
updateref.parseStderr: classify the MVCC backend's lock errors like reftable's.update_with_hooks.go: sendReferenceTransactionAbortedwhenPrepareor the Prepared hook fails (not only whenCommitfails).- Replace the stale
doc/mvcc_rpc_flow.mdwith a short note pointing to the storage core docs.
Proof and docs (WP61)
- Multi-node end-to-end tests: two (and three) real gitaly processes with
separate storage directories, one database and one bucket:
- Write on A, read on B.
- Concurrent lands on the same branch from both nodes: no lost update, every landed request in the database changelog exactly once.
- A fresh node with an empty disk hydrates and serves.
- A repository created on A and removed on B.
- Crash injection (test hooks in the voter): after the upload and before the database commit (orphan pack only, no changeset); after the database commit and before the git commit (node behind, repaired by the next EnsureFresh).
- Shadow mode over the existing benchmark: zero divergences.
- Bench: voter overhead and lands per minute with SQLite and Postgres, 1, 10 and 50 writers.
doc/graft/STORAGE.md: architecture, modes, configuration, operations (import, verify, rollback), limits, and the clean-room note.
Foundation map (fixed; read before coding)
| What | Where |
|---|---|
Shared types, errors, PackKey, IsChangelogRef, Watermark |
internal/graft/corestore/types.go, core.go |
MetaStore (+ ChangesetInput, CommitResult, InitialState, review errors) |
internal/graft/corestore/metastore.go |
BlobStore, Freshener, LocalApplier, Core, NopFreshener, KeyOf |
internal/graft/corestore/core.go |
In-process changelog notifications (SubscribeChangelog, NotifyChangelog) |
internal/graft/corestore/notify.go |
| Schema (both dialects) | internal/graft/corestore/sqlstore/migrations.go |
| Opening the core from config | internal/graft/corestore/open/open.go |
Test helpers (SQLite / per-test Postgres DB, file bucket, Open) |
internal/graft/corestore/coretest/coretest.go |
Config [graft.storage] + validation |
internal/gitaly/config/config_graft_storage.go |
| Proto | proto/storage_core.proto (generated) |
Hook wiring (StorageVoter, WithStorageVoter, changelogMetaOf) |
internal/gitaly/hook/storage_voter.go, referencetransaction.go |
Dependencies (GraftStorage, GraftStorageSyncer, getters) |
internal/gitaly/service/dependencies.go |
Land retries call ensureStorageFresh |
internal/gitaly/service/operations/{server,land,land_batch_server}.go |
| Production wiring | internal/cli/gitaly/serve.go |
Test-server wiring (testserver.WithGraftStorage(core)) |
internal/testhelper/testserver/gitaly.go |
changelog.Source + FileSource |
internal/graft/changelog/source.go |
store.ReviewStore |
internal/graft/review/store/iface.go |
Praefect registration of ImportRepository |
internal/praefect/coordinator.go |
The stubs of the implementation packages (sqlstore, blobstore, gitops,
voter, syncer, gate, service/storagecore) have the binding exported
signatures and doc comments. A WP replaces the stub bodies of its package; it
does not change an exported signature (add unexported helpers freely, and new
exported identifiers only when this plan names them).
Test environment
- Postgres for tests runs in Docker (
graft-pg,127.0.0.1:54329, trust auth).GRAFT_TEST_POSTGRES_DSN=postgres://[email protected]:54329/postgres?sslmode=disableis exported by~/scion/env.sh, sogtest.shruns the Postgres variants.coretest.DatabaseURL(t, sqlstore.Postgres)creates a fresh database per test. Never use thepostgresdatabase itself for data. - There is no S3 server in this environment: bucket tests use
file://andmem://; S3 tests run only whenGRAFT_TEST_S3_URLis set. - Import cycles:
testserverimportsvoter,syncerandgate, andcoretestimportssqlstoreandblobstore. Tests ofsqlstore,blobstore,gitops,voter,syncerandgatethat usecoretestortestserverare in the external test package (package voter_test, ...). - Test servers with the storage core:
testserver.RunGitalyServer(t, cfg, setup.RegisterAll, testserver.WithGraftStorage(coretest.Open(t, coretest.Options{...}))). Authoritative mode needscfg.Graft.Standalone = trueand a standalone GitLab client (seeinternal/graft/e2e/e2e_test.go,startGitaly). Two "nodes" are two servers built from twotestcfg.Build(t)configurations (separate storage directories, same storage namedefault) whose cores shareDatabaseURLandBucketURL.
Work packages
Waves: WP51, WP52, WP53 and WP60 start at once; WP54, WP55, WP58 and WP59 draft against the signatures while wave 1 builds and complete once it is merged; WP56 and WP57 after WP54 and WP55; WP61 last. Spec-test writers for every WP start at once.
WP51 — MetaStore on SQLite and Postgres
Owns: internal/graft/corestore/sqlstore/** (the table definitions in
migrations.go are fixed).
Open:ParseDSN; SQLite with the documented DSN parameters (sql.Open("sqlite3", path+"?_journal_mode=WAL&_synchronous=FULL&_busy_timeout=10000&_foreign_keys=on&_txlock=immediate")); Postgres withsql.Open("pgx", dsn). Apply the migrations withmigrate.MigrationSet{TableName: migrationTable}and the dialect names"sqlite3"/"postgres"(seeinternal/gitaly/storage/relational/sqlite.gofor the pattern).Openpings the database.- Write queries once with
?placeholders and rebind them to$1..$nfor Postgres in one helper. Times: SQLite stores RFC 3339 text with nanoseconds in UTC; PostgresTIMESTAMPTZ. Booleans: 0/1 on SQLite. - Transactions that fail with SQLite busy/locked errors, Postgres serialization failures (40001) or deadlocks (40P01) are retried up to 5 times with a short jittered backoff; every other error is returned.
CommitChangeset(one transaction):- Lock the repository row (Postgres
SELECT ... FOR UPDATE; SQLite's immediate transaction already holds the write lock).ErrRepositoryNotFoundif absent. - Validate the input: at least one update, non-empty and unique names,
Expectknown,ExpectAnyupdates ignoreOld. Invalid input is a plain error (no write). - Read the current targets of the named references (chunks of at most 500 names) and evaluate every expectation (CAS table). Conflicts are listed in update order.
- Conflicts without
Force→ roll back, return&ErrRefConflict{...}. - Effective updates = updates whose current value differs from
New. None → roll back, returnCommitResult{NoOp: true, Conflicts: ...}(no sequence is used). seq = head_seq + 1; insert the changeset (forced= Force and at least one conflict), itsgraft_changeset_refsrows (ordinal from 0, actual old value, NULL for absent), upsert or delete thegraft_refsrows (seq= the new seq).- If
Changelogis set: one entry per effective update withIsChangelogRef(name)and neither value symbolic, sequenceslog_seq+1..., OldOID/NewOID = actual old / new value or the zero OID of the repository's object format (40 or 64 zeros), Operation, Actor (strings.ToValidUTF8(actor, "�")), Rewritten, Members, MembersTruncated from the meta, RecordedAt from the meta or now (UTC). Storejson.Marshal(entry)(the Entry's own MarshalJSON keeps non-UTF-8 names). - Update
head_seqandlog_seq; commit; return the result.
- Lock the repository row (Postgres
RegisterRepository: one transaction: insert the repository (unique violation →ErrRepositoryExists, also when found by a SELECT first), then the imported changelog entries (sequences must be > 0 and strictly increasing, else error;log_seq= last), the imported reviews (IDs must be decimal without leading zeros; version 1;closed= state is notReview_STATE_OPEN_UNSPECIFIED;body= protojson;next_review_id= max + 1), then the initial changeset through the same code asCommitChangeset(every update must beExpectValuewith an emptyOld, else error). A new repository always gets a newrepo_id(never reused, also afterDeleteRepository).Refsreadshead_seqand all references in one read transaction (PostgresREPEATABLE READ, read only).ListChangesets,ScanChangelog,ReportApplied(upsert), reviews as documented on the interface.GetReview/UpdateReviewreturnErrReviewNotFoundfor a non-numeric ID.UpdateReviewwith a stale version returnsErrReviewVersionConflict; with an unknown IDErrReviewNotFound.- Tests (external package
sqlstore_test, every test for eachcoretest.Dialects()): every row of the CAS table; forced commits; NoOp; ref names with non-UTF-8 bytes round trip; changelog entry contents (zero OIDs, symbolic skipped, only heads/tags);RegisterRepositorywith history;DeleteRepositorycascades and re-registration gets a new ID;ListChangesetsranges; reviews CRUD, ordering and version conflicts; concurrency: 32 goroutines × 20 commits each withExpectValueon the same reference (each reads the current value first and retries on conflict) end with dense sequences and exactly 640 changesets; 32 goroutines on 32 different references never conflict; twoStores on the same database (two "nodes") behave like one.
WP52 — BlobStore on gocloud
Owns: internal/graft/corestore/blobstore/**.
Open: register the drivers asinternal/backup/sink.godoes (fileblob, memblob, s3blob, gcsblob, azureblob); forfile://create the directory (os.MkdirAll, mode 0o700) before opening.Put: if the key exists, return nil without readingr. Otherwise write withblob.WriterOptions{IfNotExist: true}and treatgcerrors.FailedPreconditionas success. If the driver reportsgcerrors.Unimplementedfor IfNotExist, fall back to a plain write (keys are content-addressed, so the content is the same) and document it. For a pack key, hash the content while writing; on mismatch cancel the write (cancel the writer's context beforeClose, so nothing is committed) and returnErrBlobCorrupt.Get:gcerrors.NotFound→ErrBlobNotFound; for a pack key return a reader that hashes the content and returnsErrBlobCorruptinstead ofio.EOFon mismatch (and on a short read).- Tests (external package): file and mem buckets; Put idempotent; concurrent
Puts of the same key; corrupt content rejected on Put and on Get (write a
wrong file directly into the file bucket); missing key; keys with nested
directories; S3 variant only with
GRAFT_TEST_S3_URL.
WP53 — git operations
Owns: internal/graft/corestore/gitops/**.
Implement every function as documented in gitops.go. Binding details:
- Run git only through
repo.Exec/repo.ExecAndWaitwithgitcmd.Command(never a shell). Revisions are passed as arguments only after validating them as object IDs of the repository's hash. BuildPack:git rev-list --objects --not --all --not <tips...>(the two--notflags make--allexcluded and the tips included), its output piped intogit pack-objects --stdout(no--thin), written toos.CreateTemp(tmpDir, "graft-pack-*")through a SHA-256 hasher. No output lines → no file, empty Pack.BuildFullPack:git rev-list --objects --allpiped the same way.IndexPack:git index-pack --stdin(no--strict), stdin = r.ReadRefs:git for-each-ref --format=%(refname)%00%(objectname)%00%(symref)plusHEAD(git symbolic-ref --quiet HEAD, or its object ID when detached).ReadRefValues: the same, filtered to the exact names.ApplyRefs: oneupdateref.New(ctx, repo, updateref.WithDisabledTransactions())transaction (Update(name, oid, ""),Delete,UpdateSymbolicReferencefor symbolic targets including HEAD). If git reports a file/directory conflict for a transaction that deletes and creates, retry as two transactions (deletions first).IsLockErrorrecognizesupdateref.AlreadyLockedError,updateref.ErrPackedRefsLockedand reftable lock failures.InitBare:git init --bare --object-format=<f> --ref-format=reftable <path>throughgitCmdFactory.NewWithoutRepo.ValidateName:HEAD, orgit.ValidateReference.ValidateTarget:hash.ValidateHexfor object IDs;ref:+ValidateNamefor symbolic.- Watermark file JSON
{"repo_id":N,"seq":M};ReadWatermarkof a missing file isok=false, err=nil; of a corrupt file an error. - Tests (external package,
gittestrepositories, SHA-1 and SHA-256 viagittest.ObjectHashIsSHA256-aware helpers): pack of new objects only, pack indexes into an empty repository and makes the tips' history complete with the earlier packs (git fsck --connectivity-only); empty pack; full pack; ReadRefs with symbolic refs and detached HEAD; ApplyRefs sets, deletes, symbolic HEAD, rejectsrefs/../x,HEAD:x, names with spaces and invalid OIDs without touching the repository; lock contention detected; watermark round trip and atomicity.
WP54 — voter (write path)
Owns: internal/graft/corestore/voter/**. Depends on WP51–WP53 (draft
against the signatures first).
Implement "Write path" exactly. Binding details:
- Parse hook input lines
<old> SP <new> SP <ref>; values are hex object IDs of the payload's object format orref:<target>. Malformed input: authoritative → error (abort); shadow → counted and skipped. - The repository is
repoFactory.Build(payload.Repo)(it carries the quarantine when there is one); the database key iscorestore.KeyOf(payload.Repo); the pack's temporary directory islocator.TempDir(payload.Repo.GetStorageName()). - Reference backend (
repo.ReferenceBackend, compare by Name): see the REFTABLE RULE (authoritative: only reftable; shadow: files processed, MVCC skipped). There is no force-deletions-only skip any more. - Every reference name of the hook input is checked with
gitops.ValidateNamein the parser; an invalid name is malformed input (authoritative: abort; shadow: counted and skipped). - The pending map (relative path + SHA-256 of the input → repo ID, seq) has a
TTL of 15 minutes and at most 4096 entries (oldest evicted).
Committedpops it and callsapplier.NoteCommittedwith the repository path fromlocator.GetRepoPath(ctx, payload.Repo, storage.WithRepositoryVerificationSkipped())(strip the quarantine: build the path from storage and relative path only).Abortedpops it and, if it existed, logs at error level and incrementsgraft_storage_local_commit_failures_total. - After a commit that appended changelog entries, call
corestore.NotifyChangelog(repoID). - Conflicts: return
fmt.Errorf("storage core: %w", updateref.MismatchingStateError{...})for the first conflict (absent values as the zero OID). Other storage errors:structerr.NewUnavailable("storage core: %w", err). - Shadow: the whole Prepared work runs under
context.WithTimeout(ctx, core.ShadowTimeout)and arecover; it never returns an error. - Faults for tests: an exported package variable
var FaultHook func(point string) error(nil in production) called with"after_upload"and"after_commit"; a non-nil error is returned from Prepared as if the process had died at that point (authoritative: abort). TheCrashEnvvariable (read once inNew) makes the process exit with code 75 at the same points, for real-binary tests. - Metrics (one
prometheus.Collector):graft_storage_changesets_total{mode,result}with resultscommitted,noop,conflict,error,skipped;graft_storage_shadow_divergences_total,graft_storage_shadow_errors_total,graft_storage_untracked_writes_total,graft_storage_mvcc_skipped_total,graft_storage_local_commit_failures_total,graft_storage_pack_bytes_total, histogramgraft_storage_prepared_seconds. - Tests (external package
voter_test,coretest.Open,gittest): callPrepared/Committed/Aborteddirectly with agitcmd.HooksPayloadbuilt for the repository; every CAS-table case; conflict error matcheserrors.As(err, &updateref.MismatchingStateError{}); shadow with a closed database returns nil and counts an error; shadow divergence is recorded as forced; untracked repository; pack content (index it into an empty repository); NoOp; files-backend force-deletion skip; authoritative changelog entries with meta;NoteCommittedcalled with the right seq;FaultHookat both points. Plus one end-to-end test throughtestserver.WithGraftStoragein authoritative mode:UserCreateBranchthen the database has the branch; a CAS conflict injected by committing directly to the database makesUserLandretry and succeed.
WP55 — syncer (catch-up and hydration)
Owns: internal/graft/corestore/syncer/**. Depends on WP51–WP53.
Implement "Read path and catch-up" exactly. Binding details:
- Repository path:
locator.GetRepoPath(ctx, repo, storage.WithRepositoryVerificationSkipped()); the local repository exists when that directory exists andlocator.ValidateRepositoryaccepts it. - Single flight with
golang.org/x/sync/singleflightkeyed by storage + relative path; the per-repositorysync.Mutex(in a map, entries removed when idle) covers apply + watermark.EnsureFreshruns undercontext.WithTimeout(ctx, core.SyncTimeout). - Catch-up applies packs in sequence order (
Blobs.Get→gitops.IndexPack), then onegitops.ApplyRefswith the final values as of head, thengitops.WriteWatermark.gitops.IsLockError→ retry with backoff (10 ms doubling to 500 ms) until the timeout, thenUnavailable. - Hydration builds in
locator.TempDir(storage)/<random>/repo.git, thenos.MkdirAllof the target's parent andos.Rename. A stale local repository (otherrepo_id) is renamed into the temporary directory first and removed in the background. NoteCommittedrecords the seq in an in-memory set and, if the per-repository mutex is free (TryLock), advances the watermark through contiguous seqs;EnsureFreshalso advances through them before downloading anything.- Errors: database unavailable →
Unavailable; untracked with a local repository →FailedPrecondition("repository is not imported into the storage core"); untracked without →NotFound("repository not found"). Appliedreads the watermark.ReportAppliedat most once per second per repository, errors only logged.- Metrics:
graft_storage_sync_total{result}(fresh,caught_up,hydrated,error),graft_storage_sync_seconds,graft_storage_sync_changesets_total,graft_storage_sync_bytes_total. - Tests (external package): changesets produced with
gitops.BuildPackandMetaStore.CommitChangesetfrom a source repository; EnsureFresh on an empty storage hydrates an identical repository (refs, HEAD, fsck); catch-up after more changesets; repo_id mismatch re-hydrates; untracked/not-found errors; concurrent EnsureFresh calls do one sync; NoteCommitted contiguity (out of order, with gaps); lock contention (hold a reference lock with a startedupdate-reftransaction) retries and then succeeds; corrupt pack in the bucket fails without touching references; an invalid reference name in the database is rejected.
WP56 — gate (authoritative-mode interceptors)
Owns: internal/graft/corestore/gate/** (policy.go may only lose entries,
or gain a special case described below). Depends on WP54, WP55.
Implement "Gate" exactly. Binding details:
- Method lookup:
registry.LookupMethod(info.FullMethod); unknown methods and non-repository scopes pass through. Target repository:methodInfo.TargetRepo(req). Streams: wrap thegrpc.ServerStreamso that the firstRecvMsgreads the request, runs the checks and EnsureFresh, and then returns it to the handler. CreateRepository: after the handler succeeded, read the object format (repo.ObjectHash(ctx)→Format),HEAD(gitops.ReadRefs), register withInitialState{Changeset: &{Updates: [HEAD], Operation: "create_repository", Actor: identity.Actor(ctx, "unknown"), NodeID}}and write the watermark{repoID, 1}.ErrRepositoryExists→ move the new local repository into the temporary directory, returnAlreadyExists.RemoveRepository:DeleteRepositoryfirst; if it was tracked and the local handler returns NotFound, return success.- Proof tests (external package
gate_test,testserver.WithGraftStorage, authoritative): one table-driven test with a representative call for every allowlist entry, each followed by "local references == database references" (gitops.ReadRefsvsMetaStore.Refs). An entry that cannot be proven is removed frompolicy.goand reported. IfRepositoryService/WriteRefofHEADis not reported by the hook (noref:line reaches the voter), keep the entry and add a post-handler step in the gate that commits the newHEADtarget as a changeset (operationwrite_ref). Plus: a blocked mutator returnsFailedPrecondition; an accessor on a second node sees the first node's write; HookService calls are not intercepted; an untracked repository returnsFailedPrecondition.
WP57 — StorageCoreService, SDK and CLI
Owns: internal/gitaly/service/storagecore/**, sdk/storage*.go,
cmd/graft/storage*.go plus one registration line in cmd/graft.
Depends on WP51–WP55.
ImportRepository: storage core off →FailedPrecondition; MVCC backend →FailedPrecondition; tracked →AlreadyExists. Buildgitops.BuildFullPackinlocator.TempDir, upload, read refs (gitops.ReadRefs), the file changelog (changelog.Scan(repoPath, 0, ...)) and the file reviews (store.Open(repoPath).List("", true, 0)), and callRegisterRepositorywith changeset 1 (operationimport, actoridentity.Actor(ctx, "import"), every referenceExpectValuewith emptyOld, pack key and size, no changelog meta), the changelog history and the reviews. Write the local watermark{repoID, 1}(in both modes).GetStorageState: as in the proto;applied_sequencefromdeps.GetGraftStorageSyncer().Applied.VerifyRepository: authoritative →EnsureFreshfirst; comparegitops.ReadRefswithMetaStore.Refs, divergences sorted by name;check_connectivity→git fsck --connectivity-only --no-dangling.- SDK:
func (r *Repo) StorageImport(ctx) (*StorageImport, error),StorageState(ctx) (*StorageState, error),StorageVerify(ctx, connectivity bool) (*StorageVerify, error)with plain Go result structs mirroring the responses (followsdk/review.go). - CLI:
graft storage import|state|verify [--connectivity] [--json](followcmd/graft/review.go). - Tests: service tests in shadow and authoritative mode (import with changelog and reviews, state before/after, verify clean and with an injected divergence, connectivity), SDK tests with the existing fakes pattern, CLI tests.
WP58 — changelog from the database
Owns: internal/graft/corestore/dblog/**, internal/gitaly/service/changeset/**.
Depends on WP51.
dblog.New(meta corestore.MetaStore, repoID int64) changelog.Source: Scan pages throughScanChangelog(after, 500); Subscribe forwardscorestore.SubscribeChangelog(repoID)and a 250 ms ticker into one coalesced channel, stopped by the returned function.- ChangesetService:
ListChangesetsandWatchChangesetsread achangelog.Source: authoritative →dblog.Newfor the repository's ID (GetRepository(corestore.KeyOf(repo)); untracked →FailedPrecondition); otherwisechangelog.FileSource(repoPath). Behaviour otherwise unchanged (existing tests stay green). - Tests: dblog unit tests on SQLite and Postgres; service tests in authoritative mode: lands produce dense entries with members; watch gets a live entry within 1 s; an entry committed by a second core (another node) reaches the watcher through polling.
WP59 — reviews in the database
Owns: internal/graft/corestore/dbreview/**,
internal/gitaly/service/review/server.go (store selection) and the minimal
type changes in internal/gitaly/service/review/*.go that it needs.
Depends on WP51.
dbreview.New(ctx, meta, repoID) store.ReviewStorewith exactly the semantics of*store.Store(readinternal/graft/review/store/store.go): Create sets ID, CreatedAt, UpdatedAt and clears Requirements/Approved;ErrReviewNotFound→store.ErrNotFound; List order and filters; Update re-reads and re-runs fn up to 10 times onErrReviewVersionConflict, then returns an error.- The review server's
openReporeturns astore.ReviewStore: authoritative →dbreview.New(untracked →FailedPrecondition), otherwisestore.Open(path). - Tests: one conformance suite run against both the file store and the database store (SQLite and Postgres); concurrent Updates from two stores on the same database lose no update; service tests in authoritative mode.
WP60 — prerequisites
Owns: internal/git/mvcc/manifest.go, internal/git/mvcc/cache.go (manifest
hash check only), internal/git/updateref/updateref.go (parseStderr only),
internal/gitaly/hook/updateref/update_with_hooks.go (Aborted calls only),
their tests and fuzz corpora, doc/mvcc_rpc_flow.md.
Implement "Prerequisites" exactly, each with a regression test (fuzz corpus
entries under testdata/fuzz/<FuzzName>/ in the go test fuzz v1 format
for the manifest panic). update_with_hooks.go: when updater.Prepare or
the Prepared hook fails, call the Aborted hook (errors only logged) before
returning, as the Commit-failure path already does.
WP61 — proof, benchmark and docs
Owns: internal/graft/e2e/storage_*_test.go, examples/graft/demo/bench-storage.sh,
doc/graft/STORAGE.md, the [graft.storage] section of
doc/graft/CONFIGURATION.md, and a phase-6 block in the development smoke script.
Runs after the integration of WP51–WP60.
Implement "Proof and docs" exactly. Multi-node tests are in-process (two or
three test servers sharing one database and bucket, see "Test
environment"), on SQLite and Postgres. Crash points use voter.FaultHook.
The benchmark script follows examples/graft/demo/bench-batch.sh (real
binaries, SQLite and Postgres, 1/10/50 writers, results table). STORAGE.md
covers architecture, modes, configuration, operations (import, verify,
switching shadow → authoritative and back), the honest limits (git still
needs a local repository; no GC of packs yet; Praefect unsupported in
authoritative mode; S3 untested here) and the clean-room note.
Spec-test writers
Each WP gets an independent spec-test writer that reads ONLY this file and
the foundation files named in "Foundation map" (signatures and doc comments),
never the implementation. Tests go to
<workspace>/spec-tests6/<WP>/<repo-relative package dir>/zz_<name>_spec_test.go,
function names start with TestSpec, 4–10 focused tests on the edge cases
this plan states, in the package named in the WP's test bullet (external
_test packages where "Test environment" requires it). They cannot compile
the code (it is written in parallel): copy signatures exactly.
Verified evidence (Opus, binding for WP54, WP56 and the review)
Facts checked by the orchestrator with the bundled git (_build/bin/gitaly-git-master,
2.56.0-rc1) and by reading the code. Use them; do not re-derive them differently.
Hook input for symbolic references
git symbolic-ref HEAD refs/heads/x, andsymref-update HEAD refs/heads/xrun with--no-deref(whatlocalrepo.SetDefaultBranchandgitops.ApplyRefsdo), report one line<zero> ref:refs/heads/x HEADin preparing, prepared and committed. The old value of a symbolic reference is always the zero OID, so the voter treats it asExpectAnywhen HEAD exists (CAS table).- Without
--no-deref,symref-update HEADfollows HEAD and turns the branch HEAD points to into the symbolic reference; the hook then reports that branch's name (<zero> ref:refs/heads/x refs/heads/main). This is what git really changed, so the voter records the line literally (no special case). No caller in the tree does this: everyUpdateSymbolicReferencecaller usesupdateref.WithNoDeref(). - When a branch moves, the hook reports only the branch line, never HEAD (HEAD is a symbolic reference in the database and follows by itself).
- git refuses one transaction that updates HEAD and the branch HEAD currently points to,
even with
--no-deref("multiple updates for 'HEAD' (including one via its referent)"). Sogitops.ApplyRefsapplies the other references first and HEAD alone afterwards; this split is required by git, not a workaround. - With the files backend a deletion also produces an
abortedstate for the packed-refs transaction before the loose one; the voter's Aborted without a pending entry is a no-op.
Which RPCs change references without the reference-transaction hook
updateref.New installs the hook by default (internal/git/updateref/updateref.go
WithRefTxHook, unless WithDisabledTransactions); UpdaterWithHooks drives it
manually; receive-pack runs it. Every OperationService writer, ConflictsService/
ResolveConflicts, RefService/UpdateReferences and DeleteRefs, RepositoryService/WriteRef
(including HEAD via SetDefaultBranch), FetchRemote, FetchBundle, FetchSourceBranch,
ReplicateRepository, CleanupService/ApplyBfgObjectMapStream, and SSH/HTTP receive-pack go
through the hook. The repository-scoped RPCs that write references WITHOUT the hook are:
RepositoryService/CreateRepositoryFromURL, CreateFork, CreateRepositoryFromBundle
(clones with hooks disabled), CleanupService/RewriteHistory (its staging repository; the
live repository is then updated through a hooked fetch) and ObjectPoolService/ FetchIntoObjectPool (refs/dangling/* in the pool). None of them is in
gate.AuthoritativeMutators; they stay blocked in authoritative mode. Adding a fetch-type
RPC to the allowlist later still needs the WP56 proof test.
WP55 RULE: catch-up never moves a reference backwards (binding, added by Opus after the WP55 gate)
Defect found by the gate: catch-up read the local references, then applied the final values "as of head" unconditionally. A local writer that committed a newer changeset N > head in between was overwritten (the branch went backwards), and the watermark then advanced through N. A compare-and-swap on the values read just before applying is not enough either: if the local read happens after N committed locally but head was read before N, the CAS passes and the branch still regresses.
Binding protocol for every catch-up step (EnsureFresh and hydration's final apply on an existing repository):
- Snapshot first: read the current local values of every reference the step
may change (
gitops.ReadRefValues), BEFORE reading the database head used as the target of this step. - Then read the head
Hand the changesets(watermark, H]; compute the final value of each changed reference as ofH. - Apply with expectations: apply only those references, each with its snapshot value as the expected old value, in one transaction (superseded: see the amendment and rulings below). Absent in the snapshot = must not exist.
- If the transaction fails because a reference no longer has its expected value
(
gitops.IsRefChanged(err)), a local write committed in between: start the step again from 1 (bounded bysync_timeout; this is the correct design, a concurrent writer always makes progress). Lock contention (IsLockError) retries the same way. - Watermark: after a successful apply set it to
H, drop every noted seq<= H, then advance through contiguous noted seqs> H(they are local commits that happened after the database commit, so the local repository contains them).
Why this is correct: the database commit of a local write happens before its git
commit. A write whose git commit is before the snapshot (1) committed in the
database before H was read (2), so N <= H and the final values include it. A
write whose git commit is after the snapshot changes a reference away from its
snapshot value, so the CAS (3) fails, or the write's own git CAS runs on top of the
H state; either way nothing goes backwards.
[SUPERSEDED by the WP55 RULE amendment and the ORCHESTRATOR RULINGS below: do not add it] gitops addition (WP55 may add it; new file internal/graft/corestore/gitops/apply_if.go
and its tests; nothing else in gitops changes):
// ApplyRefsIf is ApplyRefs with an expected current value for every reference:
// expected[name] is the value the reference must have (empty Target = must not
// exist). It validates names and targets like ApplyRefs, applies the references
// other than HEAD in one --no-deref update-ref transaction with old values, and
// HEAD afterwards in its own transaction (git refuses HEAD together with its
// referent). A reference whose current value differs from its expectation fails
// the transaction with an error that IsRefChanged recognizes, and nothing of
// that transaction is applied. Every name in refs must have an entry in expected.
func ApplyRefsIf(ctx context.Context, repo *localrepo.Repo, refs, expected map[string]corestore.Target) error
// IsRefChanged reports whether err from ApplyRefsIf means that a reference no
// longer had its expected value (updateref.MismatchingStateError,
// updateref.ReferenceAlreadyExistsError, or git's "is at X but expected Y").
func IsRefChanged(err error) bool
For a symbolic expected value of HEAD use symref-update with an old target
(ref form) only if the bundled git accepts it with --no-deref (test it; the
orchestrator observed the ref old-target form being rejected without
--no-deref). If it does not work, apply HEAD with an expectation checked by a
read immediately before, and say so in the doc comment (HEAD changes are rare and
are not written by the land paths).
WP55 RULE amendment (Opus review of the fix-round design, binding): what is binding is the
INVARIANT, not the exact steps above: a catch-up never writes a value over a local reference whose
current value is the effect of a local commit with a sequence greater than the target of the
apply, and the watermark only claims sequences whose effects are in the local repository. The
fix-round design in syncer.go (fetch the range; under the lock read the local values; a value the
range does not explain makes it read the head again and extend the range, or repair when the head
did not move; before applying, extend the range through newer changesets of THIS node that touch a
moved reference; apply with the read values as expected old values; a changed reference restarts the
check) satisfies the invariant, because the database commit of a local write precedes its git commit
and the writer holds the reference lock from prepared to commit. It is accepted in place of the
snapshot-first steps. Binding consequences:
- ONE compare-and-swap implementation (see the ORCHESTRATOR RULINGS below: it is the syncer's own transaction code; gitops gets no ApplyRefsIf). No dead code: nothing unused may remain (e.g. an unused tickets field, doc paragraphs that describe a different algorithm than the code).
- The package documentation describes the algorithm the code implements.
- Symbolic references: see the next item (immediate targets); a two-level symref chain must not make every catch-up "repair" the reference.
ORCHESTRATOR RULINGS for WP55 (override any gate finding that says otherwise; read this main-tree copy of PLAN6.md, not the copy inside your worktree, which is older):
- (Revised after fix round 2 removed
gitops/apply_if.go.) Do NOT recreateapply_if.go. The single expected-old-value implementation is the syncer's own transaction code (checkedTransaction/applyChecked); there must be no second one. - That code must check a symbolic old value UNDER THE LOCK too: a move from a symbolic value
(in practice HEAD) is written as
symref-update <ref> NUL <new> NUL ref NUL <old-target>(or the oid form for an object-ID old value) inside the--no-dereftransaction, so a concurrent SetDefaultBranch/WriteRef of HEAD makes git fail the transaction (IsRefChanged-equivalent classification: re-check). Remove the read-then-ApplyRefsfallback for symbolic moves. Verify with the bundled git (_build/bin/gitaly-git-master) that this form works with--no-derefand add a regression test where HEAD changes between the check and the apply. - REFTABLE RULE for the syncer is still missing: a local repository that is not reftable is a stale
cache (like a repo_id mismatch) in both
sync()andinstall()'s restart condition (no loop), with the test described in the REFTABLE RULE. - Allowed gitops changes in this round (and nothing else in gitops): the immediate-symref-target
change in
ReadRefs/ReadRefValues(next item), reporting dangling symbolic references inReadRefValues/ReadRefs(their value isref:<target>even when the target does not exist), andValidateTargetrejecting a symbolic targetref:HEAD. Each with a test. - Call
forgetPendingfor the repository BEFORE the hydrated repository is renamed into place. - No dead code, package doc describes the code, the gate's concurrent regression test passes on SQLite and Postgres, and the whole syncer and gitops packages are green.
Symbolic references in the snapshot (binding): the snapshot and the expectations compare the IMMEDIATE value of a reference. and currently resolve chains of symbolic references (HEAD -> refs/heads/a -> refs/heads/b reports the final target), so a CAS on such a reference can never match. WP55 may change those two functions in so that a symbolic reference reports its immediate target (; verify empirically what for-each-ref %(symref) reports and use a form that gives the immediate target), with a test of a two-level chain. This is the only other gitops change allowed.
Process note: only ONE agent edits the WP55 worktree at a time. An agent that is told to stop editing leaves the files as they are and ends with a report.
Also binding for WP55 (from the same gate):
NoteCommittedis a no-op unless the core is authoritative (shadow mode keeps no pending set and writes no watermark).- Single flight: a caller must not be satisfied by a flight whose head read happened before the caller arrived (read-your-writes across nodes). Record the time each flight read its head; a caller that arrived later starts or joins a new flight after the current one ends.
- Regression test (required): the gate's repro, in package
syncer_test, on SQLite and Postgres: a goroutine imitating the hook onrefs/heads/main(Start/Update/Prepare, CommitChangeset with ExpectValue, Commit, NoteCommitted), a goroutine committing ExpectAny changesets on another reference (another node) every few ms, and two goroutines looping on EnsureFresh for a few seconds: zero CAS conflicts onmain, and after a final EnsureFresh the local references equalMetaStore.Refsand the watermark equals the head.
REFTABLE RULE: authoritative mode uses reftable repositories only (binding, added by Opus after the WP54 gate)
Root cause of the WP54 gate finding: the files backend deletes a reference with
two reference transactions (packed-refs half, then loose half) whose hook input
can be identical (<zero> <zero> ref for a forced deletion), so the hook cannot
tell a lone forced deletion from the packed-refs half of a checked one. reftable
uses one transaction. In authoritative mode the local repositories are caches the
storage core controls (hydration already creates reftable repositories), so the
format is fixed instead of guessing transaction pairs.
- Voter (WP54), authoritative: a repository whose backend is not reftable
(files or MVCC) →
FailedPrecondition("authoritative storage mode requires a reftable repository"); nothing is committed. - Voter (WP54), shadow: files-backend repositories are processed like reftable
ones (every transaction, including force-only ones; a forced deletion is
ExpectAnyand idempotent). A conflict whose actual value is absent and whose new value is absent (the reference was already deleted, typically by the packed-refs half) is not a divergence: do not incrementgraft_storage_shadow_divergences_totalfor it. MVCC is still skipped. - Gate (WP56), CreateRepository in authoritative mode: if the request's
reference_backendis unspecified, run the handler withgitcmd.ContextWithReferenceBackend(ctx, git.ReferenceBackendReftables)so the repository is created with--ref-format=reftable; if it asks for any other backend →FailedPrecondition. Test: the created repository is reftable. - Syncer (WP55): in authoritative mode a local repository whose backend is not
reftable is a stale cache, handled exactly like a
repo_idmismatch (moved away into the temporary directory, then hydrated as reftable). - Import (WP57): importing a files-backend repository is allowed (the pack and
the references do not depend on the format). In authoritative mode the next
EnsureFresh replaces the local copy with a reftable one (syncer rule above);
document this in STORAGE.md together with
git refs migrate --ref-format=reftableas the in-place alternative. - Tests: WP54's sentinel test that asserted the old files-backend behaviour is
replaced by: authoritative files-backend write → FailedPrecondition; shadow
files-backend
DeleteRefswithout old values → the database no longer has the reference and no divergence is counted; reftableDeleteRefs(authoritative) → the database no longer has the reference.
IMPORT RULE: a resumable import; head_seq 0 means "import not complete" (binding, added by Opus after the WP57 gate)
Root cause of the WP57 gate findings: the plan asked for an import in ONE transaction, but the pack key contains the repository ID, which exists only after the registration. The import therefore has to register first, upload, then commit; a crash in between left a registration that could neither be retried (AlreadyExists) nor served, and the syncer could move the only local copy away. Holding the database transaction during the upload is rejected (it would block every SQLite writer for the whole upload); a staging pack key is rejected (the syncer rightly refuses a pack key of another repository). The state is made explicit instead:
- State: a tracked repository with
HeadSeq == 0is an import that is not complete. No other path creates such a row: the gate's CreateRepository always registers with changeset 1. - ImportRepository (WP57):
- Untracked →
RegisterRepository(key, format, InitialState{Changelog: history, Reviews: reviews})with NO changeset (history and reviews are recorded atomically with the row). Tracked withHeadSeq == 0→ resume with its ID (history and reviews are already there; do not register again). Tracked withHeadSeq > 0→AlreadyExists. - Build the full pack, upload it under
PackKey(id, sha). CommitChangesetchangeset 1: every referenceExpectValuewith emptyOld, operationimport, the pack key and size. A conflict here (a concurrent import resumed first) is returned asAlreadyExists.- Write the local watermark
{id, 1}. On any failure, nothing is rolled back and nothing is deleted: the error tells the operator to run ImportRepository again, which resumes. RemoveabandonImport(and any delete-by-key).
- Untracked →
- Syncer (WP55 code, changed in the WP57 fix round): a tracked repository with
HeadSeq == 0→FailedPrecondition("repository import is not complete; run ImportRepository again"). It never hydrates, catches up, moves away or deletes a local repository in that state. - Voter (WP54 code, changed in the WP57 fix round): a repository with
HeadSeq == 0is treated like an untracked one (authoritative:FailedPrecondition; shadow: skipped and counted as untracked). - Allowed files for this rule in the WP57 fix round:
internal/gitaly/service/storagecore/**,internal/graft/corestore/syncer/syncer.go(+ a test file),internal/graft/corestore/voter/voter.go(+ a test file). Nothing else. - Tests: interrupted import (registration done, no changeset) is resumed by a second ImportRepository on SQLite and Postgres and ends with VerifyRepository clean; EnsureFresh on a head-0 repository returns FailedPrecondition and leaves the local repository untouched (same path, same references); an authoritative write to a head-0 repository is rejected.
STALE CACHE RULE: caches of removed repositories (binding, added by Opus after the WP56 gate)
Root cause of the WP56 gate finding: after RemoveRepository on one node, another node keeps its cached copy; that copy was then treated as an untracked repository (FailedPrecondition "not imported") and blocked CreateRepository (AlreadyExists). The watermark file is the proof that a local repository is a cache managed by the storage core:
- A local repository WITH a watermark whose key is not in the database (or whose watermark names a repo_id the database row does not have, which is already the repo_id-mismatch rule) is a stale cache. A local repository WITHOUT a watermark whose key is not in the database is untracked (unchanged: FailedPrecondition "not imported").
- Syncer: EnsureFresh on a stale cache of a removed repository moves it into the storage's
temporary directory (removed in the background, never deleted in place) and returns
NotFound("repository not found"). - Gate, CreateRepository: before running the handler, a stale cache at the target path (watermark present, key not in the database) is moved away the same way, so the creation proceeds.
- Allowed files in the WP56 fix round:
internal/graft/corestore/gate/**andinternal/graft/corestore/syncer/syncer.go(+ a test file). The WP57 fix round may change other parts ofsyncer.gofor the IMPORT RULE at the same time: keep your change local to the untracked/stale branch of the code, so the merge of the two is mechanical. - Tests: two nodes sharing a database and bucket: create on A, read on B (hydrates), remove on A; then on B RepositoryExists reports false (NotFound internally), CreateRepository succeeds and registers a new repo_id, and a write on B is visible on A after A's catch-up.
The repository key is normalized by corestore.KeyOf (cleaned relative path), so two spellings
of one path are one key (fixed in the foundation by the orchestrator).
LOCK AND HISTORY RULE (binding, added by Opus after the WP61 benchmark)
A. No slow work under the reference lock; reftable waits like packed-refs
Finding: in authoritative mode 50-84% of concurrent writes to one repository failed with "reference is already locked". Two root causes:
- Configuration gap inherited from Gitaly:
internal/git/gitcmd/command_factory.gogives repository-wide reference locks a long wait (core.packedRefsTimeout=10000, with the comment that context cancellation still ends the wait early) and loose references 1 s, but never setsreftable.lockTimeout, so reftable's single, repository-wide lock waits only git's default 100 ms. Fix: addConfigPair{Key: "reftable.lockTimeout", Value: "10000"}next to the packed-refs one, with a comment that reftable's lock is repository-wide like packed-refs. This is the policy Gitaly already applies to repository-wide locks, not a new workaround. - Design flaw in the write path: the voter built the pack, uploaded it and committed to the
database while git held the repository-wide reftable lock (Prepared phase), 40-150 ms per
write. Fix: build and upload the pack in the Preparing phase, before git takes the locks
(the hook manager calls Preparing before
updater.Prepare()on the manual path, and the bundled git emitspreparingfor its own transactions):hook.StorageVotergainsPreparing(ctx, payload, changes)(no error that aborts: a failure is remembered and Prepared then does the work itself, as before), and the hook manager calls it forReferenceTransactionPreparing.- The voter keeps the prepared pack in its pending structure keyed like the Prepared/Committed entries (relative path + SHA-256 of the input), with the same TTL and size bound; an entry that is never consumed (transaction aborted before Prepared) is dropped, its uploaded pack is an orphan like any other (GC is milestone 3).
- Prepared uses the pack prepared for the same input when there is one; only the expectations
read (
for-each-ref, cheap) and the database commit run under the lock. Correctness: the pack holds the objects reachable from the new tips and not from the references at Preparing time; every object reachable from those references is in an earlier pack (invariant I1), so excluding fewer references than at Prepared time only adds objects, never loses one. - Shadow mode follows the same split (the shadow timeout covers both phases).
- Tests: an authoritative-mode test with 10 concurrent
UserCommitFilesonto 10 different branches of one repository: all succeed, and the database has all 10 branches (SQLite and Postgres). A voter unit test that Preparing+Prepared uploads exactly one pack and that Prepared without a preceding Preparing still works. A test that the Prepared phase does no upload when Preparing prepared the pack (countBlobs.Putcalls with a wrapping BlobStore). - Benchmark: rerun
examples/graft/demo/bench-storage.shand replace the table and the "Concurrent writers fail" paragraph and limit indoc/graft/STORAGE.mdwith the new measured numbers (numbers from the real run only).
B. Shadow history reaches the database before authoritative mode serves it
Finding: shadow mode does not append the database changelog and reviews live in files, so after switching to authoritative the database changelog reuses the sequences of file-log entries written in the shadow period (consumer cursors break) and shadow-period reviews are invisible.
- ImportRepository on a tracked repository (
HeadSeq > 0) no longer returns AlreadyExists: it brings the history written since the import into the database: every file-changelog entry with a sequence greater than the database'slog_seq(sequences preserved, appended in order) and every file review whose ID is not in the database (next_review_id= max + 1). References are not touched. It is idempotent and returns the counts in the existing response fields (sequence= current head). Operators run it once while switching (writers stopped), whichgraft storage importdoes. - New MetaStore method (foundation interface + sqlstore implementation, both dialects):
ImportHistory(ctx, repoID int64, changelog []changelog.Entry, reviews []*gitalypb.Review) (entries, reviews int, err error)in one transaction: entries must have sequences strictly increasing and greater than the currentlog_seq(else an error, nothing written); reviews whose ID already exists are skipped (superseded by the REVIEW AMENDMENT below). - Guard in authoritative mode: the gate refuses to serve a tracked repository whose local file
changelog has entries beyond the database's
log_seq(FailedPrecondition("the shadow-mode history of this repository is not in the storage core yet: run graft storage import")). The check reads the last sequence of the file log (cheap: the changelog package's offset index) and is cached per repository per process once it passed. - REVIEW AMENDMENT (binding, added by Opus after the phase-6 review): skipping every existing
ID lost the shadow-mode changes of a review that the first import had copied (closed, landed,
approved or edited in shadow mode, where the database is never written), and the guard above did
not see review-only shadow history (the review service never writes the changelog), so a switch
without the import let
InsertReviewhand out the IDs of shadow-mode reviews again. One rule,corestore.ReviewNeedsImport(file, db *gitalypb.Review) bool(new exported function ininternal/graft/corestore/reviewimport.go), decides both: a file review needs importing when the database has no review with its ID, or when the database review is the same review (equalcreated_at: an earlier import copied it) and the file'supdated_atis later. A database review changed later (authoritative mode), an unchanged one and a different review that only shares the ID are kept.ImportHistoryinserts the new ones and replaces the changed ones withversion + 1(UPDATE ... WHERE version = <read version>), counting both. The guard also lists the file reviews and the database reviews and refuses the repository (same message, metadatareview_id) while any file review needs importing; the result is cached like the changelog check (authoritative mode never writes the file review store). - STORAGE.md: replace the documented workaround with this procedure.
Ownership for this fix round (one Opus agent, worktree IT)
internal/git/gitcmd/command_factory.go (one config pair), internal/gitaly/hook/storage_voter.go
and referencetransaction.go (Preparing call), internal/graft/corestore/voter/**,
internal/graft/corestore/metastore.go + sqlstore/** (ImportHistory), internal/gitaly/service/storagecore/**,
internal/graft/corestore/gate/** (the guard), internal/graft/changelog/** (only a helper that
returns the last sequence of the file log, if one is missing), doc/graft/STORAGE.md,
doc/graft/CONFIGURATION.md, tests of these packages. Nothing else.
RULES (binding for every WP)
- Clean room. Never read anything under
ee/, and never rungit show,git log -p,git difforgit checkoutonee/paths or on commits that touched them. Do not copy code from_build/deps/git-mvcc(GPL); reading its documentation is fine. - Vendor neutral. Only gocloud URLs for buckets and
database/sqlwith the existingmattn/go-sqlite3andjackc/pgx/v5/stdlibdrivers. No new go.mod dependencies. - Commit point. In authoritative mode the database commit in the Prepared phase is the only commit point. A changeset row never references a pack that is not completely uploaded.
- Shadow never hurts. In shadow mode no storage-core error, timeout or
panic may fail or noticeably delay a git write: everything runs under
shadow_timeoutandrecover, and is logged and counted. - CAS semantics exactly as in the table above.
- No sync inside hooks. EnsureFresh never runs for HookService or inside a reference transaction. Hook phases never block on the syncer's per-repository mutex.
- Watermark =
(repo_id, seq), advanced only contiguously, written atomically after the references are applied. - Typed errors as listed:
*ErrRefConflict→ mismatching-state error; database unavailable →Unavailable; untracked →FailedPrecondition. - Both dialects. SQLite and Postgres pass the same conformance suite.
Postgres tests use
GRAFT_TEST_POSTGRES_DSNand skip only when it is unset; the Opus gate runs them against a Docker Postgres. - Untrusted data. Everything read from the database or the bucket is validated before it touches a local repository (see read path).
- Metrics are named
graft_storage_*and registered once. - MVCC repositories are skipped (shadow) or rejected (authoritative).
- Authoritative mode requires standalone mode and is not supported behind Praefect (validated at startup, documented).
- Tests: every WP has unit tests; a separate Haiku writes spec tests
(
zz_*_spec_test.go) from this file only. - Process (as in earlier phases): work only in your worktree, own only
the files listed for your WP, no commits, report
needs_outside_change.