Git designed for AI agents.
changelog, hook: follow-ups of the re-review; BENCH.md current results first
Federico Elia2f0d1c32e53f
diff --git a/doc/graft/BENCH.md b/doc/graft/BENCH.mdindex 2cab0c3..162da75 100644--- a/doc/graft/BENCH.md+++ b/doc/graft/BENCH.md@@ -4,6 +4,206 @@changes on the same branches. It uses the public SDK, so it exercises the sameRPCs as an agent.+## Current results++These are the results of the current code, newest first. Older results, which the+changes since replaced, are under [History](#history-superseded) at the end.++The headline of the [README](../../README.md#benchmark), 50 agents landing 5+changes each on one branch with 20% planned conflicts, measured with+`acceptance-phase4.sh` (two runs):++| 50 agents, 250 changes | classic (batching off) | gitai (batching on) |+|---|---|---|+| changes landed | 100-101 | 200 |+| attempts per landed change | 27 | 1.0 |+| lands per minute | 240-272 | 1,862-2,544 |+| p99 latency | 6.3-7.0 s | 1.5-1.8 s |++### `UserCommitFiles` in fewer git processes (WP96)++WP96 ([PLAN9 section 15](PLAN9.md#15-wp96-the-git-processes-of-usercommitfiles-design-for-approval))+cut the git processes of a `UserCommitFiles` request with one file from 10 to 15 to+6. Window of record: before `d584aac`, after `9db0ac7`, 50 agents of 20 changes each+(1000 calls per run), runs ordered before, after, after, before, twice, each started+once the machine was idle (load below 2). Medians of the four runs on each side:++| | lands/min | p50 (ms) | p95 (ms) | p99 (ms) | server mean (ms) |+|---|---:|---:|---:|---:|---:|+| before | 2134 | 217 | 486 | 600 | 237 |+| after | 2185 | 177 | 358 | 444 | 186 |++Every before/after pair is lower after in p50, p95 and p99. The per-run table and+where the time went are in PLAN9 section 15.4.++### The write path (PLAN9)++PLAN9 ([PLAN9.md](PLAN9.md)) changed the write path that every reference update+shares: the changeset log records the transactions that commit together with one+fsync (group commit, in ticket order), and the quarantine migration fsyncs each+directory once instead of every file. The window of 2026-10-19 measured+`UserCommitFiles`, the RPC that pays for both, with the per-RPC latency of+`graft-bench` (`rpcs`), against a baseline built from the same history without+PLAN9 (`b6058e5` + `6aecff1`, the tree of v0.2.0's land code), runs interleaved+base, p5w, base, p5w, each one server with batching on, `--mode land`, 50 then 100+agents of 5 changes, conflict rate 0.2, seed 1, on the same machine (load 0.4 at the+start, 12.7 at the end). The p5w tree is `64b29bb`.++Server-side mean of `UserCommitFiles` (`grpc_server_handling_seconds`), and the+client-side percentiles:++| run | agents | mean (ms) | p50 (ms) | p95 (ms) | p99 (ms) | lands/min |+|---|---:|---:|---:|---:|---:|---:|+| base 1 | 50 | 594 | 466 | 1152 | 1259 | 1427 |+| p5w 1 | 50 | 433 | 379 | 1440 | 1517 | 1342 |+| base 2 | 50 | 775 | 589 | 1420 | 4268 | 833 |+| p5w 2 | 50 | 430 | 349 | 882 | 927 | 1877 |+| base 1 | 100 | 455 | 339 | 1273 | 1458 | 1499 |+| p5w 1 | 100 | 367 | 295 | 1061 | 1089 | 1876 |+| base 2 | 100 | 642 | 404 | 2116 | 2717 | 816 |+| p5w 2 | 100 | 426 | 322 | 1098 | 1146 | 1481 |++The committed phase of a transaction (`graft_changeset_ticket_wait_seconds`, from the+committed hook to its entries being durable) averaged 22 to 27 ms, with 2.5 to 3.1+transactions per group write and 13 to 18 ms per write; before PLAN9 the profile+measured a ticket wait of 92 to 99 ms plus a 7 ms append.++Against the acceptance of PLAN9 (section 11):++| criterion | result |+|---|---|+| mean at least 150 ms lower at 50 agents | met: 684 -> 432 ms on average (-252); per pair -161 and -344 |+| mean at least 100 ms lower at 100 agents | met on average: 548 -> 396 ms (-152); per pair -88 and -216 |+| p99 not higher than the baseline | not met in every pair: p5w 1 at 50 agents 1517 ms against base 1 1259 ms; lower in the other three pairs |+| committed phase at most 20 ms mean | not met: 22 to 27 ms |+| `acceptance-phase4.sh` twice on the p5w tree | A1, A2, A3 and the stretch goal pass in both runs (p99 on-100 3397 and 2338 ms against 0.5 x off of 4527 and 3817 ms) |+| lands/min not below phase 4b | 1377 / 1184 at 20 agents, 2628 / 2856 at 50, 2066 / 2785 at 100; above the phase 4b window 3 runs at 50 and 100 agents, below them at 20 agents in run 2 (1184 against 1366 and 1476) |++The runs are short (250 and 500 calls per point, 8 to 30 s each) and noisy: base 2+ran at half the throughput of base 1 (833 against 1427 lands/min at 50 agents) as+the load rose, so the per-pair numbers vary more than the effect at 100 agents. The+committed phase misses its target by a few milliseconds: a group write still costs+one fsync of the log (13 to 18 ms under this load) after the transaction's own wait+for the head of the queue. Outputs: `/home/feder/p5w-work/window` (scripts+`window.sh`, `summary.py`, `means.py`).++### Batched landing (phase 4b)++Phase 4b ([PLAN8.md](PLAN8.md)) fixed the two causes of the latency that the+profile of the phase 4a tree found: about 12 fsyncs per batched request (each new+object fsynced when written and again by the quarantine migration) and nine short+git processes per request where one listing of the change's commits answers every+question. It also removed a second git process that the command factory started+for most git commands (`gitcmd` attr.tree detection), which speeds up every+command of the server, so the "off" column below is measured on that corrected+base too.++Runs of record: two runs of `acceptance-phase4.sh` in one benchmark window, on+the tree of commit `1899a0d`, with the defaults (`AGENT_COUNTS="20 50 100"`,+`CHANGES=5`, `RATE=0.2`, `SEED=1`, `window = "10ms"`, `max_size = 50`,+`target_latency = "500ms"`, `graph_ancestry = false`). Same machine as above+(Intel Core Ultra 7 255H, 16 logical CPUs, WSL2, ext4 on a virtual disk, an+fsync of a small file takes about 6 ms), with no other workload: the load average+was 4.2 when the first run started and rose to 38 and 48 during the runs, which is+the benchmark itself (both servers and up to 100 agents).++Both runs pass every criterion, the stretch goal included:++| criterion | 20 agents | 50 agents | 100 agents |+|---|---|---|---|+| attempts/landed (on), run 1 / run 2 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |+| failed = planned conflicts (on) | 22 / 22 | 50 / 50 | 108 / 108 |+| p99 on vs off (ms), run 1 | 1347 vs 7437 | 2525 vs 12030 | 4854 vs 10077 |+| p99 on vs off (ms), run 2 | 1193 vs 6741 | 4839 vs 12746 | 2897 vs 11905 |+| stretch: p99 on-100 <= 0.5 x off-100 | | | 4854 <= 5038 (run 1), 2897 <= 5952 (run 2) |++The stretch goal passed by 3.7% in run 1 and by 51% in run 2.++| metric (batching on), run 1 / run 2 | 20 agents | 50 agents | 100 agents |+|---|---|---|---|+| lands/min | 1020 / 1070 | 1230 / 944 | 1616 / 2325 |+| lands/min, phase 4a run of record | 412 | 251 | 258 |+| p50 landed (ms) | 911 / 739 | 1700 / 1980 | 2653 / 1824 |+| batch size mean | 8.2 / 8.3 | 13.4 / 8.9 | 13.9 / 18.5 |+| duration | 4.6 s / 4.4 s | 9.8 s / 12.7 s | 14.6 s / 10.1 s |++With batching off the runs took 17 to 41 s and landed 61 to 116 changes, as before:+the off column still loses most lands to the 20-attempt limit at 50 and 100 agents.++Time per phase, mean per observation, run 1 / run 2:++| phase | 20 agents | 50 agents | 100 agents |+|---|---|---|---|+| land_member (per request) | 8.5 / 7.6 ms | 7.4 / 8.5 ms | 7.5 / 8.1 ms |+| plan (per request) | 3.3 / 3.1 ms | 3.1 / 3.3 ms | 3.1 / 3.6 ms |+| replay (per rebased request) | 4.8 / 4.2 ms | 3.9 / 4.8 ms | 4.1 / 4.2 ms |+| pack (per batch) | 55 / 47 ms | 75 / 69 ms | 53 / 45 ms |+| update (per batch) | 126 / 136 ms | 212 / 210 ms | 175 / 135 ms |+| total (per batch) | 238 / 233 ms | 351 / 335 ms | 321 / 324 ms |++Reading it:++- A batched request costs about 8 ms instead of 80 to 137 ms in phase 4a: one+ `git rev-list` for the decision and a replay through the long-lived merge-tree+ and hash-object processes, without fsync per object.+- The lane is still busy for most of a run, but each batch now spends most of its+ time publishing: packing the objects of the batch and the update (the+ compare-and-swap, the hooks and the changeset log). At 100 agents the publish+ is about 70% of a batch. The update took 135 to 212 ms here, against 20 to 30 ms+ (without the quarantine migration) in a profile at lower load; why it grew under+ load is being profiled.+- The variance between runs is large: p99 at 50 agents was 2525 ms in run 1 and+ 4839 ms in run 2, p99 at 100 agents 4854 and 2897 ms. The benchmark saturates+ the machine (load average 38 to 48 during the runs), so single runs are not+ precise; compare runs of one window, and rerun before comparing across trees.+- The stretch goal passed with a margin of only 3.7% in run 1.+- The off column moved too. At 100 agents its p99 (10077 and 11905 ms) is lower+ than in the phase 4a run of record (13368 ms); at 50 agents it is higher (12030+ and 12746 ms against 9939 ms). One known change applies to it: every git command+ of the server now starts one process instead of two for most commands. Why the+ off p99 at 50 agents rose is not explained; its p99 covers only the 97 changes+ that landed, out of 250.++#### Final tree (window 3)++After window 1 the unbatched land was moved onto the same landing attempt as a+batch (PLAN8 WP86). A second window (tree `59d8fda`) passed every criterion but+showed a regression of the "off" column at 100 agents (p99 33.9 and 34.8 s, 60+lands/min): every unbatched attempt published its objects as a pack, and under+the contention of 100 agents about 98% of the ~6600 attempts lose the+compare-and-swap and each left a pack in the repository (120 packs at the end of+the run against 4 in window 1, `pack-objects` 1484 s and `update-ref` 840 s of+real time against 1 s and 56 s). Unbatched lands now publish their few objects+loose (PLAN8 section 13a). The runs of record of the final tree, commit+`9bf4742`, are window 3 (same workload and defaults as above; load average 10.7+at the start, 53 and 57 at the end of the runs):++| criterion | 20 agents | 50 agents | 100 agents |+|---|---|---|---|+| attempts/landed (on), run 1 / run 2 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |+| failed = planned conflicts (on) | 22 / 22 | 50 / 50 | 108 / 108 |+| p99 on vs off (ms), run 1 | 819 vs 4707 | 1880 vs 8267 | 4640 vs 14641 |+| p99 on vs off (ms), run 2 | 748 vs 4813 | 9171 vs 17868 | 5390 vs 26692 |+| stretch: p99 on-100 <= 0.5 x off-100 | | | 4640 <= 7321 (run 1), 5390 <= 13346 (run 2) |++| metric, run 1 / run 2 | 20 agents | 50 agents | 100 agents |+|---|---|---|---|+| lands/min, batching on | 1366 / 1476 | 1690 / 620 | 1459 / 1678 |+| lands/min, batching off | 332 / 336 | 230 / 151 | 158 / 114 |+| p50 landed on (ms) | 599 / 639 | 1326 / 1794 | 2927 / 2537 |+| member (land_member, mean) | 10.8 / 11.1 ms | 14.9 / 21.1 ms | 14.2 / 13.1 ms |+| pack per batch (mean) | 39 / 34 ms | 51 / 210 ms | 62 / 50 ms |+| update per batch (mean) | 65 / 52 ms | 89 / 155 ms | 114 / 107 ms |++All criteria pass in both runs, the stretch goal included (by 37% and 60%). The 50+agents of run 2 are an outlier on both sides (pack 210 ms and update 155 ms per+batch, p99 on 9171 ms, lands/min 620): the load average was above 40 during that+part of the run. The off column at 100 agents recovered from window 2 (p99 14641 and+26692 ms, 158 and 114 lands/min; window 1: 10077 and 11905 ms, about 170 lands/min)+and its repository ended with 3 packs and 232 loose objects, but it still varies a+lot between runs and was slower than in window 1 in run 2, for a reason not+established.+## What it does- Starts `--agents` concurrent agents (goroutines). Each agent commits@@ -212,7 +412,7 @@ Each server exposes its metrics on its own `prometheus_listen_addr` (set bythe mean `land_member` time per batch. `plan` and `replay` are parts of`land_member`; `pack` precedes `update`. The metric names are in [LAND-BATCHING](LAND-BATCHING.md#metrics).-### Phase 4a acceptance run+### Acceptance run (`acceptance-phase4.sh`)`examples/graft/demo/acceptance-phase4.sh [WORKDIR]` runs `bench-batch.sh` withthe same workload (20, 50 and 100 agents, 5 changes each, conflict rate 0.2,@@ -238,7 +438,48 @@ Markdown, the Prometheus scrapes and `combined.md`). On exit, also when a run isinterrupted, it stops any Gitaly server of `WORKDIR` that is still running anddeletes the agent tokens in `WORKDIR/tokens`.-Phase 3 baseline (the table from PLAN4, measured with batching on and off):+## Reading the numbers++- Compare `land` and `land-resolve` at the same `--conflict-rate`. The first+ shows the cost of conflicts as failures, the second their cost as latency.+- `attempts` close to `landed` means little contention on the branch tip.+ High values mean many retries, so increase `--max-attempts`, reduce the+ number of agents landing on one branch (`--branches` shows the effect), or+ enable batching.+- A `failed_attempts_histogram` entry at the `max_attempts` of the server+ means lands gave up because the branch kept moving.+- `batch_size_mean` above 1 means concurrent lands shared one reference+ update. With batching disabled every land reports 1.+- In `queue` mode, `p95` and `p99` grow with the number of changes ahead in+ the queue, times `--test-duration`. That is the cost of serialization, which+ the rebase-based land avoids for changes that do not conflict.+- Results depend on the machine, the storage and whether hooks run. Report+ the full JSON output with any number you quote.++## Limits of the benchmark++- Each target branch is shared by the agents assigned to it. Changes on+ different branches never conflict with each other, so `--branches` measures+ independent contention, not cross-branch interaction.+- The simulated test in `queue` mode does not run the tests. Use+ `graft-verify` for real post-merge checks.+- The shared file conflicts by design. Real repositories conflict less often,+ so the conflict rate is a stress knob, not a forecast.+- The attempts of a failed land are known only when it gave up because the+ branch kept moving. Other failures (merge conflicts, hook rejections) report+ no attempts.+++## History (superseded)++These results describe code that has since changed. Phase 4b (PLAN8) superseded+the phase 4a results: it removed the per-object fsyncs and the extra git processes+of a batched land, and its runs pass every acceptance criterion (see+[Batched landing (phase 4b)](#batched-landing-phase-4b)).++### Phase 3 baseline++The table from PLAN4, measured with batching on and off:| metric | off-20 | on-20 | off-50 | on-50 | off-100 | on-100 ||---|---|---|---|---|---|---|@@ -246,7 +487,7 @@ Phase 3 baseline (the table from PLAN4, measured with batching on and off):| p50 landed (ms) | 954 | 3320 | 5484 | 8152 | 5703 | 15950 || p99 landed (ms) | 5049 | 9868 | 17669 | 15317 | 16775 | 21808 |-#### Measured phase 4a results+### Measured phase 4a resultsRun of record: `acceptance-phase4.sh /tmp/graft-acc4-final3` on the final phase4a tree (all work packages and fixes merged), with the defaults@@ -377,202 +618,3 @@ Reading it:- Compared with phase 3 (table above), p99 with batching on dropped at 20 agents(9868 to 2789 ms) and 50 agents (15317 to 12658 ms), and rose slightly at 100agents (21808 to 23253 ms), on a noisy machine.--#### Measured phase 4b results--Phase 4b ([PLAN8.md](PLAN8.md)) fixed the two causes of the latency that the-profile of the phase 4a tree found: about 12 fsyncs per batched request (each new-object fsynced when written and again by the quarantine migration) and nine short-git processes per request where one listing of the change's commits answers every-question. It also removed a second git process that the command factory started-for most git commands (`gitcmd` attr.tree detection), which speeds up every-command of the server, so the "off" column below is measured on that corrected-base too.--Runs of record: two runs of `acceptance-phase4.sh` in one benchmark window, on-the tree of commit `1899a0d`, with the defaults (`AGENT_COUNTS="20 50 100"`,-`CHANGES=5`, `RATE=0.2`, `SEED=1`, `window = "10ms"`, `max_size = 50`,-`target_latency = "500ms"`, `graph_ancestry = false`). Same machine as above-(Intel Core Ultra 7 255H, 16 logical CPUs, WSL2, ext4 on a virtual disk, an-fsync of a small file takes about 6 ms), with no other workload: the load average-was 4.2 when the first run started and rose to 38 and 48 during the runs, which is-the benchmark itself (both servers and up to 100 agents).--Both runs pass every criterion, the stretch goal included:--| criterion | 20 agents | 50 agents | 100 agents |-|---|---|---|---|-| attempts/landed (on), run 1 / run 2 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |-| failed = planned conflicts (on) | 22 / 22 | 50 / 50 | 108 / 108 |-| p99 on vs off (ms), run 1 | 1347 vs 7437 | 2525 vs 12030 | 4854 vs 10077 |-| p99 on vs off (ms), run 2 | 1193 vs 6741 | 4839 vs 12746 | 2897 vs 11905 |-| stretch: p99 on-100 <= 0.5 x off-100 | | | 4854 <= 5038 (run 1), 2897 <= 5952 (run 2) |--The stretch goal passed by 3.7% in run 1 and by 51% in run 2.--| metric (batching on), run 1 / run 2 | 20 agents | 50 agents | 100 agents |-|---|---|---|---|-| lands/min | 1020 / 1070 | 1230 / 944 | 1616 / 2325 |-| lands/min, phase 4a run of record | 412 | 251 | 258 |-| p50 landed (ms) | 911 / 739 | 1700 / 1980 | 2653 / 1824 |-| batch size mean | 8.2 / 8.3 | 13.4 / 8.9 | 13.9 / 18.5 |-| duration | 4.6 s / 4.4 s | 9.8 s / 12.7 s | 14.6 s / 10.1 s |--With batching off the runs took 17 to 41 s and landed 61 to 116 changes, as before:-the off column still loses most lands to the 20-attempt limit at 50 and 100 agents.--Time per phase, mean per observation, run 1 / run 2:--| phase | 20 agents | 50 agents | 100 agents |-|---|---|---|---|-| land_member (per request) | 8.5 / 7.6 ms | 7.4 / 8.5 ms | 7.5 / 8.1 ms |-| plan (per request) | 3.3 / 3.1 ms | 3.1 / 3.3 ms | 3.1 / 3.6 ms |-| replay (per rebased request) | 4.8 / 4.2 ms | 3.9 / 4.8 ms | 4.1 / 4.2 ms |-| pack (per batch) | 55 / 47 ms | 75 / 69 ms | 53 / 45 ms |-| update (per batch) | 126 / 136 ms | 212 / 210 ms | 175 / 135 ms |-| total (per batch) | 238 / 233 ms | 351 / 335 ms | 321 / 324 ms |--Reading it:--- A batched request costs about 8 ms instead of 80 to 137 ms in phase 4a: one- `git rev-list` for the decision and a replay through the long-lived merge-tree- and hash-object processes, without fsync per object.-- The lane is still busy for most of a run, but each batch now spends most of its- time publishing: packing the objects of the batch and the update (the- compare-and-swap, the hooks and the changeset log). At 100 agents the publish- is about 70% of a batch. The update took 135 to 212 ms here, against 20 to 30 ms- (without the quarantine migration) in a profile at lower load; why it grew under- load is being profiled.-- The variance between runs is large: p99 at 50 agents was 2525 ms in run 1 and- 4839 ms in run 2, p99 at 100 agents 4854 and 2897 ms. The benchmark saturates- the machine (load average 38 to 48 during the runs), so single runs are not- precise; compare runs of one window, and rerun before comparing across trees.-- The stretch goal passed with a margin of only 3.7% in run 1.-- The off column moved too. At 100 agents its p99 (10077 and 11905 ms) is lower- than in the phase 4a run of record (13368 ms); at 50 agents it is higher (12030- and 12746 ms against 9939 ms). One known change applies to it: every git command- of the server now starts one process instead of two for most commands. Why the- off p99 at 50 agents rose is not explained; its p99 covers only the 97 changes- that landed, out of 250.--##### Final tree (window 3)--After window 1 the unbatched land was moved onto the same landing attempt as a-batch (PLAN8 WP86). A second window (tree `59d8fda`) passed every criterion but-showed a regression of the "off" column at 100 agents (p99 33.9 and 34.8 s, 60-lands/min): every unbatched attempt published its objects as a pack, and under-the contention of 100 agents about 98% of the ~6600 attempts lose the-compare-and-swap and each left a pack in the repository (120 packs at the end of-the run against 4 in window 1, `pack-objects` 1484 s and `update-ref` 840 s of-real time against 1 s and 56 s). Unbatched lands now publish their few objects-loose (PLAN8 section 13a). The runs of record of the final tree, commit-`9bf4742`, are window 3 (same workload and defaults as above; load average 10.7-at the start, 53 and 57 at the end of the runs):--| criterion | 20 agents | 50 agents | 100 agents |-|---|---|---|---|-| attempts/landed (on), run 1 / run 2 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |-| failed = planned conflicts (on) | 22 / 22 | 50 / 50 | 108 / 108 |-| p99 on vs off (ms), run 1 | 819 vs 4707 | 1880 vs 8267 | 4640 vs 14641 |-| p99 on vs off (ms), run 2 | 748 vs 4813 | 9171 vs 17868 | 5390 vs 26692 |-| stretch: p99 on-100 <= 0.5 x off-100 | | | 4640 <= 7321 (run 1), 5390 <= 13346 (run 2) |--| metric, run 1 / run 2 | 20 agents | 50 agents | 100 agents |-|---|---|---|---|-| lands/min, batching on | 1366 / 1476 | 1690 / 620 | 1459 / 1678 |-| lands/min, batching off | 332 / 336 | 230 / 151 | 158 / 114 |-| p50 landed on (ms) | 599 / 639 | 1326 / 1794 | 2927 / 2537 |-| member (land_member, mean) | 10.8 / 11.1 ms | 14.9 / 21.1 ms | 14.2 / 13.1 ms |-| pack per batch (mean) | 39 / 34 ms | 51 / 210 ms | 62 / 50 ms |-| update per batch (mean) | 65 / 52 ms | 89 / 155 ms | 114 / 107 ms |--All criteria pass in both runs, the stretch goal included (by 37% and 60%). The 50-agents of run 2 are an outlier on both sides (pack 210 ms and update 155 ms per-batch, p99 on 9171 ms, lands/min 620): the load average was above 40 during that-part of the run. The off column at 100 agents recovered from window 2 (p99 14641 and-26692 ms, 158 and 114 lands/min; window 1: 10077 and 11905 ms, about 170 lands/min)-and its repository ended with 3 packs and 232 loose objects, but it still varies a-lot between runs and was slower than in window 1 in run 2, for a reason not-established.--#### Measured PLAN9 results--PLAN9 ([PLAN9.md](PLAN9.md)) changed the write path that every reference update-shares: the changeset log records the transactions that commit together with one-fsync (group commit, in ticket order), and the quarantine migration fsyncs each-directory once instead of every file. The window of 2026-10-19 measured-`UserCommitFiles`, the RPC that pays for both, with the per-RPC latency of-`graft-bench` (`rpcs`), against a baseline built from the same history without-PLAN9 (`b6058e5` + `6aecff1`, the tree of v0.2.0's land code), runs interleaved-base, p5w, base, p5w, each one server with batching on, `--mode land`, 50 then 100-agents of 5 changes, conflict rate 0.2, seed 1, on the same machine (load 0.4 at the-start, 12.7 at the end). The p5w tree is `64b29bb`.--Server-side mean of `UserCommitFiles` (`grpc_server_handling_seconds`), and the-client-side percentiles:--| run | agents | mean (ms) | p50 (ms) | p95 (ms) | p99 (ms) | lands/min |-|---|---:|---:|---:|---:|---:|---:|-| base 1 | 50 | 594 | 466 | 1152 | 1259 | 1427 |-| p5w 1 | 50 | 433 | 379 | 1440 | 1517 | 1342 |-| base 2 | 50 | 775 | 589 | 1420 | 4268 | 833 |-| p5w 2 | 50 | 430 | 349 | 882 | 927 | 1877 |-| base 1 | 100 | 455 | 339 | 1273 | 1458 | 1499 |-| p5w 1 | 100 | 367 | 295 | 1061 | 1089 | 1876 |-| base 2 | 100 | 642 | 404 | 2116 | 2717 | 816 |-| p5w 2 | 100 | 426 | 322 | 1098 | 1146 | 1481 |--The committed phase of a transaction (`graft_changeset_ticket_wait_seconds`, from the-committed hook to its entries being durable) averaged 22 to 27 ms, with 2.5 to 3.1-transactions per group write and 13 to 18 ms per write; before PLAN9 the profile-measured a ticket wait of 92 to 99 ms plus a 7 ms append.--Against the acceptance of PLAN9 (section 11):--| criterion | result |-|---|---|-| mean at least 150 ms lower at 50 agents | met: 684 -> 432 ms on average (-252); per pair -161 and -344 |-| mean at least 100 ms lower at 100 agents | met on average: 548 -> 396 ms (-152); per pair -88 and -216 |-| p99 not higher than the baseline | not met in every pair: p5w 1 at 50 agents 1517 ms against base 1 1259 ms; lower in the other three pairs |-| committed phase at most 20 ms mean | not met: 22 to 27 ms |-| `acceptance-phase4.sh` twice on the p5w tree | A1, A2, A3 and the stretch goal pass in both runs (p99 on-100 3397 and 2338 ms against 0.5 x off of 4527 and 3817 ms) |-| lands/min not below phase 4b | 1377 / 1184 at 20 agents, 2628 / 2856 at 50, 2066 / 2785 at 100; above the phase 4b window 3 runs at 50 and 100 agents, below them at 20 agents in run 2 (1184 against 1366 and 1476) |--The runs are short (250 and 500 calls per point, 8 to 30 s each) and noisy: base 2-ran at half the throughput of base 1 (833 against 1427 lands/min at 50 agents) as-the load rose, so the per-pair numbers vary more than the effect at 100 agents. The-committed phase misses its target by a few milliseconds: a group write still costs-one fsync of the log (13 to 18 ms under this load) after the transaction's own wait-for the head of the queue. Outputs: `/home/feder/p5w-work/window` (scripts-`window.sh`, `summary.py`, `means.py`).--## Reading the numbers--- Compare `land` and `land-resolve` at the same `--conflict-rate`. The first- shows the cost of conflicts as failures, the second their cost as latency.-- `attempts` close to `landed` means little contention on the branch tip.- High values mean many retries, so increase `--max-attempts`, reduce the- number of agents landing on one branch (`--branches` shows the effect), or- enable batching.-- A `failed_attempts_histogram` entry at the `max_attempts` of the server- means lands gave up because the branch kept moving.-- `batch_size_mean` above 1 means concurrent lands shared one reference- update. With batching disabled every land reports 1.-- In `queue` mode, `p95` and `p99` grow with the number of changes ahead in- the queue, times `--test-duration`. That is the cost of serialization, which- the rebase-based land avoids for changes that do not conflict.-- Results depend on the machine, the storage and whether hooks run. Report- the full JSON output with any number you quote.--## Limits of the benchmark--- Each target branch is shared by the agents assigned to it. Changes on- different branches never conflict with each other, so `--branches` measures- independent contention, not cross-branch interaction.-- The simulated test in `queue` mode does not run the tests. Use- `graft-verify` for real post-merge checks.-- The shared file conflicts by design. Real repositories conflict less often,- so the conflict rate is a stress knob, not a forecast.-- The attempts of a failed land are known only when it gave up because the- branch kept moving. Other failures (merge conflicts, hook rejections) report- no attempts.diff --git a/doc/graft/LAND-BATCHING.md b/doc/graft/LAND-BATCHING.mdindex bbce348..f9d99b8 100644--- a/doc/graft/LAND-BATCHING.md+++ b/doc/graft/LAND-BATCHING.md@@ -423,7 +423,7 @@ the running tip, so it cannot delay or block the other requests of its batch.The runs of `examples/graft/demo/acceptance-phase4.sh` on the phase 4b trees(three benchmark windows; full tables, phases and the machine in-[BENCH.md](BENCH.md#measured-phase-4b-results)) **meet the phase 4a acceptance**,+[BENCH.md](BENCH.md#batched-landing-phase-4b)) **meet the phase 4a acceptance**,the stretch goal included: one attempt per landed change, only planned conflictsfail, and p99 with batching on is below off at every size. The final tree(`9bf4742`, window 3):diff --git a/doc/graft/LIMITATIONS.md b/doc/graft/LIMITATIONS.mdindex a1e026d..b0f155b 100644--- a/doc/graft/LIMITATIONS.md+++ b/doc/graft/LIMITATIONS.md@@ -238,6 +238,11 @@ Git hooks use private.are, so a reader never sees an entry that a crash could take back. The entriesof the transactions that commit together are written with one fsync, in theorder in which the transactions locked their references.+- When an append fails and cutting off the bytes it wrote fails too (two I/O+ errors in a row), only the process that wrote them remembers that they are not+ entries, and it forgets this when it drops its cache of the log. Another Gitaly+ process, during a graceful restart, can then read those bytes as entries or cut+ entries after them. A later version persists that marker next to the log.- The log is stored per Gitaly node and is not replicated. Behind Praefect,every replica records the updates it applies itself, so sequence numbers andmetadata differ between replicas. Praefect therefore always routes@@ -298,7 +303,7 @@ commitment.## Phase 4b limitations1. The phase 4a acceptance is met by two runs on one machine and one workload- (seed 1, conflict rate 0.2; see [BENCH](BENCH.md#measured-phase-4b-results)).+ (seed 1, conflict rate 0.2; see [BENCH](BENCH.md#batched-landing-phase-4b)).The runs vary a lot, the benchmark saturates the machine, and the stretch goalpassed by only 3.7% in one of the two runs.2. Batches still run one at a time per lane, and each batch spends most of its timediff --git a/doc/graft/PLAN9.md b/doc/graft/PLAN9.mdindex 52da6e6..f9084b8 100644--- a/doc/graft/PLAN9.md+++ b/doc/graft/PLAN9.md@@ -433,8 +433,9 @@ update hook and update-ref); batch publishing (pack mode) and unbatched publishi| a committed phase runs after its ticket expired | (amended after review) the ticket is found and made ready in one step under the queue's lock; an expired or released ticket is not found, so the transaction is appended at once, out of order, and counted; never taken for written (the first version could, in the window between finding the ticket and making it ready, and lost the entries silently) || a transaction with the same hook input in another process | the ticket key holds the invocation of the git process, or of the `UserCommitFiles`/land call that drives the hook in process, so an abort or a commit never takes another transaction's ticket || the group write panics | the panic is the group's error: every member reports its updates lost, the queue stops writing and goes on |-| the log's lock file is not granted (a holder that never releases it) | the append or the read fails after `lockWaitTimeout` (30 s); the append's updates are reported lost |-| an fsync never returns (hung I/O) | not bounded: its group's members wait with it, and later appends fail at `lockWaitTimeout` |+| the log's lock file is not granted (a holder that never releases it) | each wait for the lock file fails after `lockWaitTimeout` (30 s), and that append's updates are reported lost. The bound is per wait, not per call: the appends and readers of the same process queue on the state's mutex behind it, unbounded, and then each waits for the lock file in turn. A failed wait keeps its goroutine blocked in flock(2), on an OS thread, until the holder releases the lock |+| an fsync never returns (hung I/O) | not bounded: its group's members wait with it, the appends of the same process queue behind it, and the appends of other processes fail at `lockWaitTimeout` |+| known, not fixed: a double I/O failure followed by another process's append | when an append fails and cutting its bytes off fails too, the process remembers them as untrusted (`untrustedFrom`). Another process does not know this and can read them as entries, or cut entries after them. The marker is also forgotten when `evictIdleStates` drops the state. A real fix persists the marker next to the log || the committed phase cannot resolve the repository path | the ticket is released (queues are kept by storage and relative path) and the updates are reported lost || another process's cache overlaps (graceful restart) | a cache that knows of bytes after its last entry is never trusted by size: the state is loaded again, so an append never cuts off another process's durable entries (amended after review); a cache without such bytes is also checked against the last entry || the storage mode changes between the prepared and the committed phase | cannot: the mode is the storage core's configuration, fixed for the life of the process |diff --git a/internal/gitaly/hook/changeset_order.go b/internal/gitaly/hook/changeset_order.goindex 5daba95..2499360 100644--- a/internal/gitaly/hook/changeset_order.go+++ b/internal/gitaly/hook/changeset_order.go@@ -3,6 +3,7 @@ package hookimport ("crypto/sha256""fmt"+ "path/filepath""runtime/debug""sync""time"@@ -130,7 +131,8 @@ var changesetTickets = struct {// changesetQueueKey returns the key of the queue of the repository: its storage and relative// path, which every hook payload of the repository carries, whatever its object directories.func changesetQueueKey(repo storage.Repository) string {- return repo.GetStorageName() + "\x00" + repo.GetRelativePath()+ // Cleaned as GetRepoPath joins it, so that a.git, ./a.git and a.git/ share one queue.+ return repo.GetStorageName() + "\x00" + filepath.Clean(repo.GetRelativePath())}// changesetTicketKey returns the key of the transaction that the git process with the given@@ -259,8 +261,7 @@ func recordChangeset(queueKey, repoPath, invocation string, changes []byte, recoif ticket == nil {changesetTickets.Unlock()changelogUnorderedRecordsTotal.Inc()- _, err := appendChangesetGroup(repoPath, []changelog.Record{record})- return err+ return appendChangesetGroupRecovering(repoPath, []changelog.Record{record})}ticket.record = &recordq.signal()diff --git a/internal/gitaly/hook/referencetransaction.go b/internal/gitaly/hook/referencetransaction.goindex 6f91323..c3bc4dc 100644--- a/internal/gitaly/hook/referencetransaction.go+++ b/internal/gitaly/hook/referencetransaction.go@@ -5,6 +5,7 @@ import ("bytes""context""crypto/sha1"+ "errors""fmt""io""strings"@@ -263,7 +264,7 @@ func (m *GitLabHookManager) recordChangesets(ctx context.Context, objectHash gitif metaErr != nil {release()- m.logger.WithError(metaErr).ErrorContext(ctx, "recording changesets: reading changelog metadata")+ m.reportLostChangesets(ctx, updates, fmt.Errorf("reading changelog metadata: %w", metaErr))return}@@ -278,11 +279,13 @@ func (m *GitLabHookManager) recordChangesets(ctx context.Context, objectHash gitrelease()cache := mvcc.CacheByCommitHandle(graftMeta.MVCCCommit)if cache == nil || !cache.AfterCommit(func() {- if _, err := changelog.Append(repoPath, record.Meta, record.Updates); err != nil {+ if err := appendChangesetGroupRecovering(repoPath, []changelog.Record{record}); err != nil {m.reportLostChangesets(ctx, updates, err)}}) {- m.logger.ErrorContext(ctx, "recording changesets: the MVCC cache that commits the updates is gone, not recording them")+ // The cache committed or was cleaned up before the updates could be registered for+ // its commit, so whether they were published cannot be told: they are not recorded.+ m.reportLostChangesets(ctx, updates, errors.New("the MVCC cache that commits the updates is gone"))}return}diff --git a/internal/gitaly/hook/updateref/changeset_order_test.go b/internal/gitaly/hook/updateref/changeset_order_test.goindex 2d21810..efbfc65 100644--- a/internal/gitaly/hook/updateref/changeset_order_test.go+++ b/internal/gitaly/hook/updateref/changeset_order_test.go@@ -5,10 +5,12 @@ import ("context""fmt""io"+ "sync""testing""time""github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/gitcmd""github.com/feder-cr/git-ai/internal/git/gittest""github.com/feder-cr/git-ai/internal/gitaly/config""github.com/feder-cr/git-ai/internal/gitaly/hook"@@ -43,6 +45,67 @@ func (m *pausingHookManager) ReferenceTransactionHook(ctx context.Context, statereturn m.Manager.ReferenceTransactionHook(ctx, state, env, bytes.NewReader(changes))}+// invocationRecordingHookManager records the changelog invocation of every reference-transaction+// hook call, by phase.+type invocationRecordingHookManager struct {+ hook.Manager+ mu sync.Mutex+ invocations map[hook.ReferenceTransactionState][]string+}++func (m *invocationRecordingHookManager) ReferenceTransactionHook(ctx context.Context, state hook.ReferenceTransactionState, env []string, stdin io.Reader) error {+ meta, err := gitcmd.ChangelogMetaFromEnv(env)+ if err != nil {+ return err+ }+ m.mu.Lock()+ m.invocations[state] = append(m.invocations[state], meta.Invocation)+ m.mu.Unlock()++ return m.Manager.ReferenceTransactionHook(ctx, state, env, stdin)+}++// TestUpdaterWithHooks_changesetInvocation checks that a call of UpdateReference drives the+// reference-transaction hook with an invocation of its own, the same in every phase, so that the+// changeset log tells its transaction apart from one with the same updates (hook.changesetTicketKey).+func TestUpdaterWithHooks_changesetInvocation(t *testing.T) {+ t.Parallel()++ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ logger := testhelper.NewLogger(t)+ locator := config.NewLocator(cfg)+ gitCmdFactory := gittest.NewCommandFactory(t, cfg)++ repo, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{+ SkipCreationViaService: true,+ })+ base := gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("main"), gittest.WithMessage("base"))+ a := gittest.WriteCommit(t, cfg, repoPath, gittest.WithParents(base), gittest.WithMessage("a"))++ manager := &invocationRecordingHookManager{+ Manager: hook.NewManager(cfg, locator, logger, gitCmdFactory, nil, nil, gitlab.NewMockClient(+ t, gitlab.MockAllowed, gitlab.MockPreReceive, gitlab.MockPostReceive,+ ), hook.NewPostReceiveRegistry()),+ invocations: map[hook.ReferenceTransactionState][]string{},+ }+ updater := updateref.NewUpdaterWithHooks(cfg, logger, locator, manager, gitCmdFactory, nil)++ reference := git.ReferenceName("refs/heads/main")+ require.NoError(t, updater.UpdateReference(ctx, repo, gittest.TestUser, nil, reference, a, base))+ require.NoError(t, updater.UpdateReference(ctx, repo, gittest.TestUser, nil, reference, base, a))++ phases := []hook.ReferenceTransactionState{hook.ReferenceTransactionPreparing, hook.ReferenceTransactionPrepared, hook.ReferenceTransactionCommitted}+ for call := 0; call < 2; call++ {+ invocation := manager.invocations[hook.ReferenceTransactionPrepared][call]+ require.NotEmpty(t, invocation, "call %d has an invocation", call)+ for _, phase := range phases {+ require.Equal(t, invocation, manager.invocations[phase][call], "call %d, phase %v", call, phase)+ }+ }+ require.NotEqual(t, manager.invocations[hook.ReferenceTransactionPrepared][0], manager.invocations[hook.ReferenceTransactionPrepared][1])+}+func TestUpdaterWithHooks_changesetOrder(t *testing.T) {t.Parallel()diff --git a/internal/graft/changelog/log.go b/internal/graft/changelog/log.goindex ddb950e..cdc8fd2 100644--- a/internal/graft/changelog/log.go+++ b/internal/graft/changelog/log.go@@ -326,9 +326,11 @@ func Append(repoPath string, meta Meta, updates []Update) ([]Entry, error) {// the repository itself, so entries written there would be lost.// - Each Gitaly node keeps its own log and its own sequences. Behind Praefect,// reads may be routed to a replica whose log lags or differs.-// - The wait for the lock is bounded by lockWaitTimeout: an append that does-// not get it fails. An fsync that never returns is not bounded: its append-// holds the lock, and the appends after it fail at lockWaitTimeout.+// - Each wait for the lock file is bounded by lockWaitTimeout: an append that+// does not get it fails. The appends of this process queue on the state's+// mutex before that wait, which is not bounded. An fsync that never returns+// is not bounded either: its append holds the lock, and the appends of other+// processes fail at lockWaitTimeout.func AppendGroup(repoPath string, records []Record) ([][]Entry, error) {entries := make([][]Entry, len(records))@@ -813,11 +815,13 @@ var syncDir = func(dir string) error {return d.Sync()}-// lockWaitTimeout bounds the wait for the log's lock file, which another process holds for one-// append. A holder that never releases it would otherwise block every append and every reader of-// the repository's log: the wait fails instead, and the appends see the error (their updates are-// reported lost). Each wait that failed keeps a goroutine and a descriptor until the holder-// releases the lock. Tests lower it.+// lockWaitTimeout bounds one wait for the log's lock file, which another process holds for one+// append. A holder that never releases it would otherwise block that wait forever: it fails+// instead, and an append sees the error (its updates are reported lost). The bound is per wait+// for the lock file, not per call: the appends and readers of this process that wait for the+// state's mutex (s.mu) behind such a wait are not bounded by it, and each then waits for the+// lock file in turn. A wait that failed keeps its goroutine, blocked in flock(2) on an OS+// thread, and its descriptor until the holder releases the lock. Tests lower it.var lockWaitTimeout = 30 * time.Second// lockFile takes an exclusive flock on the file at path and returns thediff --git a/internal/graft/changelog/log_review_test.go b/internal/graft/changelog/log_review_test.goindex 5562e23..284814a 100644--- a/internal/graft/changelog/log_review_test.go+++ b/internal/graft/changelog/log_review_test.go@@ -1,6 +1,7 @@package changelogimport (+ "bytes""os""path/filepath""sync"@@ -96,6 +97,28 @@ func TestAppend_lockWaitIsBounded(t *testing.T) {require.Equal(t, []string{"a@1", "c@2"}, sequencesAndReferences(t, repoPath))}+// TestAppend_checksTheLastEntryOfACacheOfTheSameSize: a cache without a tail and a file of the same+// size are not enough. The line before the cache's end must still hold the cache's last entry,+// otherwise the state is loaded again.+func TestAppend_checksTheLastEntryOfACacheOfTheSameSize(t *testing.T) {+ t.Parallel()++ repoPath := newRepoPath(t)+ path := Path(repoPath)+ _, err := Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/a", OldOID: zeroOID, NewOID: oidA}})+ require.NoError(t, err)++ // The file now ends with another entry of the same length: sequence 7 instead of 1.+ data, err := os.ReadFile(path)+ require.NoError(t, err)+ require.Equal(t, 1, bytes.Count(data, []byte(`"sequence":1,`)))+ require.NoError(t, os.WriteFile(path, bytes.Replace(data, []byte(`"sequence":1,`), []byte(`"sequence":7,`), 1), 0o644))++ _, err = Append(repoPath, Meta{Operation: "test"}, []Update{{Reference: "refs/heads/b", OldOID: zeroOID, NewOID: oidB}})+ require.NoError(t, err)+ require.Equal(t, []string{"a@7", "b@8"}, sequencesAndReferences(t, repoPath), "the append follows the entry in the file")+}+// TestAppend_doesNotTrustACacheWithATail is the overlap of two processes during a graceful// restart. Process A's cache knows of bytes after its last valid entry, which it would cut off on// its next append. Process B cut them off itself and appended a durable entry, which ends at the