Skip to content

Git designed for AI agents.

doc: add the phase 4b plan (PLAN8) with the profiling evidence

Federico Eliad2115efcc090
diff --git a/doc/graft/PLAN8.md b/doc/graft/PLAN8.md
new file mode 100644
index 0000000..0186c32
--- /dev/null
+++ b/doc/graft/PLAN8.md
@@ -0,0 +1,551 @@
+# Graft phase 4b: land batch latency, root causes first (binding plan)
+
+Status: proposal of the p4b workstream, waiting for the PM's approval. Nothing in
+this plan is implemented. Owner: p4b (Opus). Branch `p4b-pipelined-batches`.
+
+Phase 4a is in [PLAN4.md](PLAN4.md); its acceptance and the measured misses are in
+[BENCH.md](BENCH.md#measured-phase-4a-results) and
+[LIMITATIONS.md](LIMITATIONS.md#phase-4a-limitations). The early completion rules
+are in [LAND-BATCHING.md](LAND-BATCHING.md#early-completion). The storage core
+write path is in [STORAGE.md](STORAGE.md) and [PLAN6.md](PLAN6.md).
+
+## 1. Goal and acceptance
+
+Meet the phase 4a acceptance that the final phase 4a tree misses
+(`examples/graft/demo/acceptance-phase4.sh`): with batching on, `bench-batch.sh` at
+20, 50 and 100 agents of 5 changes, conflict rate 0.2, seed 1:
+
+* A1: attempts/landed = 1.00 at every size;
+* A2: failures = planned conflicts only (`failed == conflicts`);
+* A3: p99(on) <= p99(off) at every size;
+* stretch S: p99(on-100) <= 0.5 x p99(off-100);
+* T: lands per minute with batching on do not drop below the phase 4a run of
+ record at any size (250.9 at 50 agents, 257.6 at 100), because the README claim
+ (2x at 50 agents, 2.9x at 100) rests on them.
+
+The plan started as "pipelined batches". The profile below shows that the latency
+does not come from the lane running batches one after another, but from two
+structural defects inside each batch. This plan fixes those. Pipelining is kept as
+a gated second stage (section 9), decided by the measurement after stage 1.
+
+## 2. Profiling evidence
+
+### 2.1 Method
+
+* Temporary probes (histograms `graft_prof_*`), never committed, around every step
+ of a batch attempt: the lane (collect, gap between batches, execute, deliver),
+ each member (peel, the ancestry checks, the merge-commit check, the merge base,
+ the rebase engine's resolve, merge base, todo list, commit reads, merge-tree,
+ commit write), the publish (pre-receive, quarantine migrate, update hook,
+ update-ref start, prepare, commit, every reference-transaction phase,
+ post-receive, the changeset-log ticket wait and append), and the number of files
+ each migrate moves. Updates of the benchmark's private branches
+ (`UserCommitFiles`) are labelled apart from the batch updates.
+* A driver runs only the batching-on server (config of `bench-batch.sh`, port 9252)
+ with 50 then 100 agents x 5 changes, conflict rate 0.2, seed 1, scrapes
+ `/metrics` before, after and every 5 s, and samples the load average.
+* Machine: Intel Core Ultra 7 255H, 16 logical CPUs, WSL2 kernel 6.6, ext4 on a
+ virtual disk, bundled git 2.56.0-rc1. Shared with the m3 workstream: the load
+ average is given with every run. These are diagnostic runs (one each, batching
+ on only). They do not replace the acceptance run, which needs a benchmark window.
+* Storage core off, as in `bench-batch.sh`.
+
+### 2.2 Runs (batching on, diagnostic)
+
+| run | tree | load (1 min) | 50 agents: p50 / p99 ms, lands/min | 100 agents: p50 / p99 ms, lands/min |
+|---|---|---|---|---|
+| 4a record (BENCH.md) | phase 4a final | 8-38 | 8959 / 12658, 251 | 17292 / 23253, 258 |
+| run1 | 4a + probes | 4.4-6.5 | 11663 / 19786, 178 | 14843 / 21261, 287 |
+| run2 | 4a + finer probes | 2-3 | 5883 / 8768, 378 | 16074 / 26312, 256 |
+| run4 | experiment E (2.6) | 1.8 (50), 8-10 (100) | 2510 / 5118, 716 | 10453 / 19567, 410 |
+| run5 | experiment E (2.6) | 5-7.5 | 2317 / 3100, 901 | 5645 / 14847, 580 |
+
+In every run A1 and A2 held (failures = 50 and 108 planned conflicts), except run3,
+the first version of experiment E (see 2.6).
+
+### 2.3 Where a batch spends its time (run2, 100 agents)
+
+* The lane is busy all the time: the batches executed for 91.4 s of a 91.9 s run.
+ The gap between two batches summed to 16 ms, the collect wait to 13 ms. So the
+ latency is queueing behind earlier batches, as phase 4a found.
+* A batch already publishes with **one** reference transaction: 179 updates for
+ 197 batch executions (the others changed nothing). Members do not run their own
+ reference transaction or hooks. The "about 100 ms per member" of phase 4a is
+ the work in the quarantine before the publish, not a reference update.
+* Mean batch: 464 ms = members 285 ms (2.5 members of 111 ms) + publish 195 ms
+ (+ tip 3.7 ms, quarantine 0.6 ms, cleanup 1.4 ms).
+
+One member, 111 ms:
+
+| step | ms | what runs |
+|---|---:|---|
+| peel the commit ID | 3.9 | `git rev-parse --verify <id>^{commit}` |
+| already landed? | 8.0 | `git merge-base --is-ancestor commit tip` |
+| fast-forward? | 5.8 | `git merge-base --is-ancestor tip commit` |
+| merges in range? | 6.9 | `git rev-list --merges tip..commit` |
+| related? | 6.6 | `git merge-base --all tip commit` |
+| engine: resolve both again | 5.2 | 2 x `git rev-parse` |
+| engine: merge base again | 6.2 | `git merge-base tip commit` |
+| engine: todo list | 6.3 | `git rev-list --right-only ... tip...commit` |
+| engine: read commits | 1.4 | long-lived `cat-file` |
+| engine: merge | 46.2 | long-lived `merge-tree --stdin` |
+| engine: write the commit | 18.2 (x 0.78) | long-lived `hash-object --stdin-paths` |
+
+Nine short git processes (49 ms) answer questions that one range query answers,
+and four of them repeat questions the stepper has already answered (the engine
+re-resolves, re-computes the merge base and re-lists the range because its
+`Rebase` keeps the full semantics of `localrepo.Rebase`). The long-lived helpers,
+which phase 4a introduced to avoid process starts, still cost 60 ms per member.
+
+Publish of a batch, 195 ms (lane updates only):
+
+| step | ms |
+|---|---:|
+| quarantine migrate | 170.2 (9.2 files per migrate) |
+| reference-transaction committed (changeset-log append 12 ms, ticket wait) | 10.0 |
+| update-ref prepare (lock and fsync) | 8.2 |
+| update-ref start | 3.5 |
+| pre-receive, update, prepared, commit, post-receive | 1.6 |
+
+### 2.4 Micro-benchmarks (copy of the run2 repository, 393 commits on the branch, load 2.6)
+
+| measurement | result |
+|---|---|
+| write + fsync of a 300-byte file | 6.4 ms mean (p90 9.5) |
+| `git rev-parse --verify <id>^{commit}` (process start) | 4 ms |
+| `git merge-base --is-ancestor agent tip` | 7 ms |
+| `git rev-list --boundary --parents tip..agent` | 5 ms |
+| `merge-tree --stdin`, 100 merges, `core.fsync=objects` | 10.4 ms per merge |
+| same with `core.fsync=none` | 0.5 ms per merge |
+| `hash-object -w --stdin-paths`, 100 commits, `core.fsync=objects` | 9.2 ms per commit |
+| same with `core.fsync=none` | 0.2 ms per commit |
+
+Gitaly runs every git command with `core.fsync=objects,derived-metadata,reference`
+and `core.fsyncMethod=fsync`. A batch writes every new object (2-3 trees from
+merge-tree and 1 commit per member) as a loose object, fsynced when written; then
+`quarantine.Migrate` hard-links each one into the repository and fsyncs the file
+again and its fan-out directory (`internal/git/quarantine/quarantine.go`,
+`migrate`). About three fsyncs per object, about 12 per member.
+
+### 2.5 Other observations
+
+* `UserCommitFiles` (the agents' commits, outside the lane) takes 0.3-0.86 s per
+ request. Its time is the same migrate and the changeset-log ticket wait (68-106
+ ms): every reference update of the repository, the benchmark's private branches
+ included, appends to one changeset log with an fsync, in order. That is in the
+ agent latency of both columns (on and off) and is not lane work.
+* Adaptive sizing keeps batches small (2.5 members per attempt at 100 agents)
+ because every batch takes longer than `target_latency` (500 ms). That is a
+ consequence of the slow members, not a cause. This plan does not tune it.
+* run3 (first version of experiment E) failed 16 lands with "migrating quarantined
+ objects: sync directory: ... objects/0e: no such file or directory": it left the
+ emptied fan-out directories in the quarantine, `Migrate` re-created them in the
+ repository and fsynced them, and the housekeeping of the same repository removed
+ the empty directories in between. The design below never leaves an empty fan-out
+ directory in a quarantine, so a batch migrate creates no fan-out directory at all
+ (it moves only `pack/` files). The race window in `Migrate` itself is upstream
+ code outside p4b; it is reported to the PM (section 14).
+
+### 2.6 Experiment E (temporary, not committed)
+
+Two diagnostic changes, switched by environment variables of the server: the
+merge-tree and hash-object helpers of a batch ran with `core.fsync=none`, and
+before the publish the stepper packed every loose object of the quarantine into
+one pack (`git pack-objects`) and removed the loose files and their emptied
+fan-out directories. The nine short processes per member were not changed.
+
+| | run2 (4a) | run5 (E), load 5-7.5 |
+|---|---:|---:|
+| member, ms | 111 | 42 (merge 2.1, write 2.8; the nine processes ~37) |
+| publish, ms | 195 | 202 at 100 agents (pack 84, migrate 67, committed 26, prepare 17), 85 at 50 |
+| migrate files per batch | 9.2 | 3 (pack, idx, rev) |
+| p99 50 / 100 agents, ms | 8768 / 26312 | 3100 / 14847 |
+| lands/min 50 / 100 | 378 / 256 | 901 / 580 |
+
+The member cost fell by the fsync share exactly as the micro-benchmarks predict, and
+the migrate went from scaling with the members to three files. What remains per
+member is the nine process starts, and per batch the publish (pack, migrate,
+update-ref, changeset log), whose cost is fsync-bound and rises with the load of the
+machine.
+
+### 2.7 Root causes
+
+* **R1, fsync amplification of the batch's objects.** Objects that live in a
+ quarantine until the publish are written loose and fsynced one by one, then
+ fsynced again one by one (plus a directory each) by the migrate. About 12 fsyncs
+ per member; with 6.4 ms per fsync that is 60 ms of member time and 68 ms of
+ publish time per member. The durability that matters (objects on disk before the
+ reference points to them) needs one durable write of the batch's objects, not
+ three per object.
+* **R2, the land decision is computed piecewise and twice.** The stepper decides
+ "already landed", "fast-forward", "merges in range" and "related" with four
+ separate history walks, then the engine walks again to get the merge base and the
+ todo list. One `rev-list --parents tip..commit` answers all of them (proof in
+ 3.1). The duplication exists because the engine's only entry point re-derives what
+ its caller knows.
+* Not a cause: the lane's serial order. Member i is replayed onto the tip that holds
+ members 1..i-1, so the member work of one batch has a real dependency chain. The
+ idle time of the lane is ~0. The one thing that could overlap without breaking
+ that dependency is the publish of batch N with the member work of batch N+1
+ (section 9).
+
+## 3. Design (stage 1)
+
+### 3.1 One range query per member (fixes R2)
+
+For a request on an existing branch, the stepper:
+
+1. Peels the commit ID through the object info reader of the attempt
+ (`catfile.ObjectInfoReader.Info(ctx, "<id>^{commit}")`), instead of
+ `rev-parse --verify`. The ID is full hex (it was parsed with `FromHex`), so
+ there is no abbreviation to resolve. Missing or not peelable to a commit gives
+ the same `InvalidArgument` ("cannot resolve commit ID", `tipIndependent`).
+2. Lists the range once: `Range(ctx, tip, commit)` = the commits reachable from
+ `commit` and not from `tip`, each with all of its parents (`git rev-list
+ --parents tip..commit`, one process).
+3. Decides with a pure function `rebase.PlanLanding(tip, commit, rng)`:
+ * range empty <=> `commit` is reachable from `tip` => **already landed**;
+ * `tip` is a parent of a commit of the range <=> `tip` is reachable from
+ `commit` => **fast-forward**. (If `tip` is reachable from `commit`, the commit
+ just above `tip` on a path from `commit` is in the range and has `tip` as a
+ parent; the converse is immediate.) Merge commits in the range are allowed
+ here, as today;
+ * otherwise **diverged**. For the rebase strategy, `plan.Replay()` returns:
+ * `ErrMergesInRange` when a commit of the range has two or more parents
+ (= `rev-list --merges tip..commit` non-empty), checked first, as today;
+ * else the range is a chain (every commit has at most one parent). Its bottom
+ commit either has a parent, which is outside the range and so reachable
+ from `tip` (a common ancestor exists), or is a root, in which case no
+ ancestor of `commit` is reachable from `tip`: `ErrUnrelated` (= no merge
+ base);
+ * else the todo list: the chain from its bottom to `commit` (= `rev-list
+ --right-only --no-merges --topo-order --reverse tip...commit` without
+ `--cherry-pick`, which `RebaseWithReappliedCherryPicks` already drops).
+ * The other strategies use only already landed / fast-forward / diverged, then
+ `computeLandedTip` as today (merge, `ff_only`).
+4. Replays the todo list with `Engine.Replay` (3.3). The engine no longer resolves,
+ recomputes the merge base or lists the range.
+
+The early completion rules, the empty-commit rule, the policy admission, conflict
+resolution and every error message and code stay as they are; only how the facts
+are obtained changes. Grafts and replace references are honoured by `rev-list`
+exactly as by `merge-base` and the old `rev-list` calls.
+
+`[graft.land] graph_ancestry`: the checker of the attempt gains `Range`. The git
+checker runs the `rev-list`; the graph checker walks the stored graph index in
+process (from `commit`, down the parents, stopping at commits reachable from `tip`
+by the index's `IsAncestor`), and keeps its rule of refusing shallow, grafted or
+replaced repositories (fallback to git, logged, as today). Both must return the same
+set (equivalence test). The single-land path keeps `IsAncestor` and `MergeBase`
+(see question Q2).
+
+### 3.2 Durable once: the batch publishes its objects as one pack (fixes R1)
+
+* The merge-tree stream and the commit writer of a batch attempt write loose
+ objects **without fsync** (`core.fsync=none` on these two helper processes only,
+ section 6). They write nothing but loose objects into the attempt's quarantine.
+ `core.fsync` cannot be narrowed relative to Gitaly's value: verified with the
+ bundled git, a later `-c core.fsync=-loose-object` replaces the whole value
+ instead of subtracting from it, and `objects,...,-loose-object` still fsyncs
+ loose objects. So the value for these processes is stated, with the reason
+ above, not derived.
+* Before the compare-and-swap of a batch (`Update` and `UpdateSolo` call one shared
+ `publish` helper), the stepper packs **every loose object found in the
+ quarantine's object directory** into one pack inside the quarantine
+ (`git pack-objects <quarantine>/pack/pack` with the object IDs on stdin, run
+ with Gitaly's normal fsync configuration, so the pack and its index are
+ fsynced), then removes those loose files and the emptied fan-out directories.
+ A batch whose quarantine has no loose object (fast-forwards only) packs nothing.
+* `quarantine.Migrate` (unchanged) then moves only the `pack/` files into the
+ repository, fsyncing them and the directories, before `update-ref` runs. The
+ reference transaction, its hooks, the voter and the database CAS see exactly the
+ same state as today.
+
+Durability invariant D (section 4): every object reachable from the new tip and
+not from the old one is in the repository's object store, durable, before the
+reference transaction prepares. Proof: git writes every new object of the attempt
+into the quarantine's object directory (its primary object directory), including
+the objects of the resolver; the pack holds every loose object of that directory;
+`pack-objects` exits 0 only after writing and fsyncing pack and index; the loose
+files are removed only after that; anything in the quarantine that is not a loose
+object is migrated by `Migrate` with its own fsync, as today.
+
+A failure to pack fails the publish of that attempt with `Internal` (the
+`landUpdateFailed` outcome: members fail with the error, the requests after the
+first member that are not members are landed again on the real tip, as today). The
+loose objects stay untouched in that case and the quarantine is discarded with the
+attempt. There is no fallback to migrating loose objects: a pack-objects failure
+is an I/O or resource fault that the loose path would not fix.
+
+### 3.3 The engine: one replay loop, no second derivation
+
+* `Engine.Replay(ctx, upstream, todo, opts...)` is the replay loop that
+ `Engine.rebaseCommits` is today (empty-commit rules, committer option, full
+ commit messages, `*localrepo.RebaseConflictError`), taking the todo list from the
+ caller.
+* `Engine.Rebase` is deleted: after this plan nothing in production calls it. The
+ equivalence tests compare `Range` + `PlanLanding` + `Replay` with
+ `localrepo.Rebase` instead.
+* `rebase.New(repo, rebase.WithDeferredDurability())` starts the two helpers with
+ `localrepo.WithoutObjectFsync()`. The option says what it means: the caller makes
+ the objects durable before a reference points to them. The batch stepper is the
+ only caller that passes it.
+
+### 3.4 What does not change
+
+The lane (`internal/graft/landlane`), batch formation, adaptive sizing, the
+`max_rebuilds` loop, the fallback after a hook rejection, early completion,
+a batch of one as `land`, changelog operations, actors and members, the policy
+(admission in process and the hooks), the synthetic `graft-batch` identity, the
+reference update path (`UpdaterWithHooks`, untouched), the storage core (voter
+Preparing/Prepared, database CAS, `ensureStorageFresh` before rebuilds), the
+single-land path (unless Q2 says otherwise) and every response field.
+
+## 4. Invariants (each has a test, section 10)
+
+* I1 Per-reference CAS and order: a batch publishes with one CAS from the tip it
+ read; requests land in FIFO order; a lost CAS rebuilds the batch.
+* I2 Early completion rules of LAND-BATCHING.md, unchanged: no request is reported
+ landed unless its commit is on the branch.
+* I3 Same decisions: for any (tip, commit) the plan gives the same outcome as the
+ phase 4a sequence (already landed, fast-forward, merges error, unrelated error,
+ todo list, conflict) and the same landed commit IDs (byte-identical commits).
+* I4 Changeset log: `land_batch` with `graft-batch` and members, a batch of one as
+ `land` with its actor and single member; no change in meaning.
+* I5 Policy and hooks: same admission, same hook runs, same skip rule.
+* I6 Storage core: every batch update goes through the voter and the database CAS;
+ `TestStorage*` pass in shadow and authoritative mode.
+* D Durability: section 3.2. Plus: after every publish the quarantine holds no
+ loose object and no empty fan-out directory; `git fsck --connectivity-only` of
+ the repository is clean after any run.
+
+## 5. Failure and race cases
+
+| case | behaviour |
+|---|---|
+| pack-objects fails (disk full, killed, ctx done) | publish fails, `landUpdateFailed` rules; loose objects untouched; quarantine discarded; ref unchanged |
+| crash after the pack is written, before Migrate | quarantine is temporary and removed; nothing reachable; same as today |
+| crash after Migrate, before update-ref | unreferenced pack in the repository; housekeeping removes it like any unreferenced pack (as after a failed push) |
+| crash after the database commit (authoritative), before git commits | unchanged: the next EnsureFresh repairs (PLAN6); the pack is in the repository |
+| housekeeping runs during Migrate | Migrate touches only `objects/pack` (always present): no fan-out directory is created, so the run3 race cannot happen for batches |
+| lost CAS / rebuild | new attempt, new quarantine, new pack; the old quarantine (with its pack) is discarded |
+| resolver writes objects | they are loose objects of the quarantine, packed with the rest |
+| request cancelled mid-batch | unchanged: still lands, caller gets Canceled |
+| range of a commit with a very long history | `rev-list` lists it once (today it is walked five times) |
+| graph index unusable (shallow, grafts, replace refs) | graph checker falls back to git, logged, as today |
+
+## 6. Config and API changes (exact)
+
+No new configuration keys, no change of defaults. `graph_ancestry` also covers the
+range query of a batch member (doc change in CONFIGURATION.md and LAND-BATCHING.md).
+
+```go
+// package ancestry (internal/graft/ancestry)
+// RangeCommit is a commit of a range with all of its parents, as git rev-list --parents prints it.
+type RangeCommit struct {
+ ID git.ObjectID
+ Parents []git.ObjectID
+}
+// Added to Checker, implemented by the git and the graph checker:
+// Range returns the commits reachable from head and not from base (git rev-list base..head), each
+// with all of its parents, in no particular order. It is empty when head is reachable from base.
+Range(ctx context.Context, base, head git.ObjectID) ([]RangeCommit, error)
+
+// package rebase (internal/graft/rebase)
+type LandKind int
+const (
+ LandAlreadyLanded LandKind = iota + 1 // commit is reachable from tip
+ LandFastForward // tip is reachable from commit
+ LandDiverged // neither: the change must be rebased or merged
+)
+type LandPlan struct{ /* unexported: kind, tip, commit, range */ }
+var (
+ ErrMergesInRange = errors.New("commit range contains merge commits")
+ ErrUnrelated = errors.New("commit shares no history with the tip")
+)
+// PlanLanding decides how commit lands onto tip from the range tip..commit (ancestry.Checker.Range).
+func PlanLanding(tip, commit git.ObjectID, rng []ancestry.RangeCommit) (LandPlan, error)
+func (p LandPlan) Kind() LandKind
+// Replay returns the commits to replay onto the tip for the rebase strategy, oldest first, or
+// ErrMergesInRange / ErrUnrelated. Only for LandDiverged.
+func (p LandPlan) Replay() ([]git.ObjectID, error)
+
+type Option func(*engineConfig)
+// WithDeferredDurability starts the helpers without fsync of the loose objects they write. The
+// caller makes those objects durable before any reference points to them.
+func WithDeferredDurability() Option
+func New(repo *localrepo.Repo, opts ...Option) *Engine
+// Replay replays todo (oldest first) onto upstream with the empty-commit rules, the committer
+// option and the conflict errors of localrepo.Rebase.
+func (e *Engine) Replay(ctx context.Context, upstream git.ObjectID, todo []git.ObjectID, opts ...localrepo.RebaseOption) (git.ObjectID, error)
+// Engine.Rebase is removed.
+
+// package localrepo (merge_tree_stream.go, commit_writer.go)
+type HelperOption func(*helperConfig)
+// WithoutObjectFsync runs the helper with core.fsync=none: it writes only loose objects, and the
+// caller makes them durable.
+func WithoutObjectFsync() HelperOption
+func (repo *Repo) NewMergeTreeStream(ctx context.Context, opts ...HelperOption) (*MergeTreeStream, error)
+func (repo *Repo) NewCommitWriter(ctx context.Context, opts ...HelperOption) (*CommitWriter, error)
+
+// package operations (new file land_batch_pack.go)
+// packAttemptObjects packs every loose object of the attempt's quarantine object directory into one
+// pack in that directory and removes the loose files and the emptied fan-out directories. It
+// returns the number of packed objects (0: nothing to pack).
+func packAttemptObjects(ctx context.Context, repo *localrepo.Repo, objectDir string) (int, error)
+```
+
+Removed as dead code: `serverLandBatchStepper.computeTip`'s mirror of the rebase
+branch (replaced by the plan), `Engine.Rebase`, the per-member `IsAncestor` calls of
+the stepper. `rejectMergeCommitsInRange` and `hasMergeBase` stay for the single-land
+path (unless Q2).
+
+## 7. Metrics
+
+`graft_land_batch_phase_seconds{phase}` gains `plan` (peel + range + decision, per
+member), `replay` (per member) and `pack` (per publish). `land_member` keeps its
+meaning (the whole member). New counter `graft_land_batch_packed_objects_total`.
+`bench-batch.sh` adds the new phases to its "time per phase" table.
+
+## 8. Work packages
+
+* WP81 (Opus): `ancestry.Range` (git and graph), `rebase.PlanLanding`, equivalence
+ tests against the phase 4a sequence.
+* WP82 (Opus): `localrepo` helper option, `rebase.Engine` options, `Replay`,
+ removal of `Rebase`, equivalence with `localrepo.Rebase`.
+* WP83 (Opus): stepper integration (peel, plan, replay), `packAttemptObjects`, the
+ shared `publish`, metrics, dead-code removal.
+* WP84 (Haiku, from an exact spec written by Opus after WP81-83 compile): spec
+ tests `TestSpec*` for `PlanLanding` cases and `packAttemptObjects`; then the
+ `bench-batch.sh` phase rows. Checked by Opus.
+* WP85 (Opus + Haiku drafts): the measurement in the PM's window and the docs
+ (LAND-BATCHING, BENCH, LIMITATIONS, CONFIGURATION section of `graph_ancestry`),
+ numbers only from the real run.
+
+## 9. Stage 2: pipelined publish (gated, not part of the approval of stage 1)
+
+After stage 1, a batch is about 8-10 ms per member (the range query plus the
+replay without fsync) and a fixed publish of 85-200 ms (pack, migrate, update-ref,
+changeset log; fsync-bound, rising with the machine's load). That publish is
+inherent per-batch work. It can overlap with the member work of the next batch:
+batch N+1 lands its members onto the unpublished tip of N, in its own quarantine
+whose alternates are N's quarantine and then the repository (so an object moved by
+N's migrate is found in one of them at every instant: the link precedes the
+removal), and publishes only after N's CAS succeeded; if N fails, N+1 is rebuilt
+on the real tip. Tip-dependent early answers of N+1 wait for N's publish.
+
+Gate G1: after the stage 1 acceptance run, stage 2 is proposed (as a binding
+amendment of this plan, with its own invariants and tests) only if A3 or S fails,
+or if they pass with less than 20% margin in either of the two runs, and the
+phase table shows the publish as the largest share of the lane time. Otherwise it
+is not built: it would add two batches in flight per lane for a gain the
+acceptance does not need.
+
+## 10. Test plan
+
+Unit and package tests:
+
+* `ancestry`: `Range` git vs graph on the seeded random DAGs of `TestGraphMatchesGitOnRandomDAG`
+ (>= 200 commits, merges, criss-cross, several roots), every sampled pair; empty
+ range; replace refs make the graph checker fall back.
+* `rebase`: `PlanLanding` table (already landed, equal, fast-forward with merges in
+ the range, diverged chain, merges in range, unrelated chain, root in range);
+ oracle test: for seeded random DAGs, `Range`+`PlanLanding` equals the phase 4a
+ sequence (`IsAncestor` x2, `rev-list --merges`, `merge-base --all`, the engine's
+ todo `rev-list`) for every pair; `Replay` equals `localrepo.Rebase` (OIDs,
+ conflict errors and contents) on the existing equivalence corpus; the helper
+ option: the helpers' command carries `core.fsync=none` (test hook on the command
+ options), output identical with and without it.
+* `operations`: every existing `land_batch*` and `land*` test passes unchanged;
+ new: peel through the info reader (tag peeled, blob and tree rejected with the
+ same error, missing object), one range query per member (count `rev-list` and
+ `merge-base` processes through the existing rebaser/test hooks: 1 and 0),
+ `packAttemptObjects` (all loose objects in one pack, none left loose, no empty
+ fan-out directory, objects readable, nothing to pack for fast-forwards, failure
+ injected through the command hook -> `landUpdateFailed`, ref unchanged, loose
+ objects untouched), invariant D after a batch (`fsck --connectivity-only`, the new
+ pack is in `objects/pack`), batch of one also packs.
+* `-race` on `internal/graft/landlane/...`, `internal/graft/rebase/...`,
+ `internal/graft/ancestry/...` and the `TestLandBatch*`/`TestUserLand*` tests of
+ `internal/gitaly/service/operations`.
+
+Integration (`internal/graft/e2e`): `TestPhase4*`, `TestChaosLanding` (not
+`-short`), `TestStorage*` (shadow and authoritative, SQLite and Postgres; S3 skips,
+no server here), and a new e2e test: batched lands with the storage core in
+authoritative mode, `VerifyRepository --connectivity` clean afterwards.
+
+Benchmark (needs a PM window): `acceptance-phase4.sh` twice on the stage 1 tree;
+plus one informational `bench-batch.sh` run with `[graft.storage] mode = "shadow"`
+(SQLite) to report the voter's share, not part of the acceptance.
+
+## 11. Acceptance of this plan
+
+* Both acceptance runs: A1, A2, A3 PASS at 20, 50 and 100 agents; S reported (target
+ PASS); T: lands per minute at or above the 4a record at every size.
+* Per member (phase table): `plan` <= 10 ms and `replay` <= 5 ms mean at 50 and
+ 100 agents; `pack` + migrate scale with batches, not with members.
+* All tests of section 10 green, `go vet`, `gofmt -l` clean, no probe left.
+* Docs updated with the measured numbers and the load of each run.
+
+Predicted (from 2.3-2.6, not a promise): member ~10 ms, publish 85-200 ms, lane
+capacity of tens of lands per second, p99 at 100 agents in the low seconds; the
+next bottleneck becomes `UserCommitFiles` and the per-repository changeset log
+(2.5), which are outside this workstream.
+
+## 12. Rejected alternatives
+
+* Pipelining first (the original plan): it hides at most the publish share (40% of
+ a batch in run2) and leaves R1 and R2. After stage 1 it is gated (section 9).
+* One reference transaction for all members: already the case (2.3).
+* Rebasing disjoint members in parallel: the members of a batch depend on each
+ other's tips; parallel tree merges would need a merge implementation outside git
+ with byte-identical results. Not needed after stage 1.
+* Tuning `window`, `max_size` or `target_latency`: adaptive sizing reacts to slow
+ members (2.5); tuning would hide R1 and R2 (forbidden by the charter).
+* Turning off object fsync globally or in `quarantine.Migrate`: durability of every
+ other write path depends on it, and it is not p4b's code.
+* Keeping loose objects without fsync and letting Migrate fsync them: removes the
+ write-time fsync but keeps two fsyncs per object in the publish (run2 migrate: 170
+ ms for 9 files).
+* An in-process range walk without the graph index: a reimplementation of git's
+ revision walk; `rev-list` costs 5 ms and the graph index already gives an
+ in-process path when enabled.
+
+## 13. Split of the work
+
+Opus writes everything on the land decision, the engine, the stepper, the
+publish (durability) and every concurrency-related test, and reviews all Haiku
+output. Haiku (one at a time, never two in this worktree) drafts the `TestSpec*`
+files from an exact spec, the `bench-batch.sh` table rows and the first drafts of
+the doc sections, all checked by Opus.
+
+## 14. Risks and open questions
+
+Risks:
+
+* One small pack per batch in the repository, like one pack per push: housekeeping
+ repacks sooner (the runs show `repack` at 0.4-1.8 s per run under `UserLand`). To
+ watch in the acceptance runs; not a correctness issue.
+* The numbers are from a shared WSL2 machine with fsync at 6 ms; the gain on other
+ disks is smaller or larger, the structure is the same.
+* The equivalence of `Range`+`PlanLanding` rests on the proof in 3.1; the oracle
+ test on random DAGs is the guard.
+
+Questions for the PM:
+
+* Q1: approve stage 1 as the p4b deliverable, with stage 2 behind gate G1?
+* Q2: the single-land path (`land.go`) has the same duplicated decision (its
+ `landAttempt` walks history four times and rebases with `localrepo.Rebase`,
+ LIMITATIONS 4a item 2). Moving it onto `Range`/`PlanLanding`/`Replay` removes the
+ duplication (and `rejectMergeCommitsInRange`, `hasMergeBase`). It also makes the
+ "off" column faster, so it changes the baseline of A3. Proposal: do it as WP86
+ after the stage 1 acceptance runs, and report the acceptance before and after.
+ In p4b or later?
+* Q3: a benchmark window for two acceptance runs (about 12 minutes each) once WP81-85
+ are green.
+* Q4 (not p4b's code, for information): `quarantine.Migrate` creates a target fan-out
+ directory and fsyncs it after migrating into it; housekeeping can remove an empty
+ directory in between (seen in run3 with empty source directories). Batches no
+ longer hit it after stage 1; other writers might, rarely. And the per-repository
+ changeset-log order (ticket wait 68-106 ms per `UserCommitFiles`) is now the
+ largest cost of an agent's commit.