Git designed for AI agents.
Benchmarking agent landings (graft-bench)
graft-bench measures how a Graft server copes with many agents landing
changes on the same branches. It uses the public SDK, so it exercises the same
RPCs as an agent.
Current results
These are the results of the current code, newest first. Older results, which the changes since replaced, are under History at the end.
The headline of the README, 50 agents landing 5
changes each on one branch with 20% planned conflicts, measured with
acceptance-phase4.sh (two runs):
| 50 agents, 250 changes | classic (batching off) | gitai (batching on) |
|---|---|---|
| changes landed | 100-101 | 200 |
| attempts per landed change | 27 | 1.0 |
| lands per minute | 240-272 | 1,862-2,544 |
| p99 latency | 6.3-7.0 s | 1.5-1.8 s |
UserCommitFiles in fewer git processes (WP96)
WP96 (PLAN9 section 15)
cut the git processes of a UserCommitFiles request with one file from 10 to 15 to
6. Window of record: before d584aac, after 9db0ac7, 50 agents of 20 changes each
(1000 calls per run), runs ordered before, after, after, before, twice, each started
once the machine was idle (load below 2). Medians of the four runs on each side:
| lands/min | p50 (ms) | p95 (ms) | p99 (ms) | server mean (ms) | |
|---|---|---|---|---|---|
| before | 2134 | 217 | 486 | 600 | 237 |
| after | 2185 | 177 | 358 | 444 | 186 |
Every before/after pair is lower after in p50, p95 and p99. The per-run table and where the time went are in PLAN9 section 15.4.
The write path (PLAN9)
PLAN9 (PLAN9.md) changed the write path that every reference update
shares: the changeset log records the transactions that commit together with one
fsync (group commit, in ticket order), and the quarantine migration fsyncs each
directory once instead of every file. The window of 2026-10-19 measured
UserCommitFiles, the RPC that pays for both, with the per-RPC latency of
graft-bench (rpcs), against a baseline built from the same history without
PLAN9 (b6058e5 + 6aecff1, the tree of v0.2.0's land code), runs interleaved
base, p5w, base, p5w, each one server with batching on, --mode land, 50 then 100
agents of 5 changes, conflict rate 0.2, seed 1, on the same machine (load 0.4 at the
start, 12.7 at the end). The p5w tree is 64b29bb.
Server-side mean of UserCommitFiles (grpc_server_handling_seconds), and the
client-side percentiles:
| run | agents | mean (ms) | p50 (ms) | p95 (ms) | p99 (ms) | lands/min |
|---|---|---|---|---|---|---|
| base 1 | 50 | 594 | 466 | 1152 | 1259 | 1427 |
| p5w 1 | 50 | 433 | 379 | 1440 | 1517 | 1342 |
| base 2 | 50 | 775 | 589 | 1420 | 4268 | 833 |
| p5w 2 | 50 | 430 | 349 | 882 | 927 | 1877 |
| base 1 | 100 | 455 | 339 | 1273 | 1458 | 1499 |
| p5w 1 | 100 | 367 | 295 | 1061 | 1089 | 1876 |
| base 2 | 100 | 642 | 404 | 2116 | 2717 | 816 |
| p5w 2 | 100 | 426 | 322 | 1098 | 1146 | 1481 |
The committed phase of a transaction (graft_changeset_ticket_wait_seconds, from the
committed hook to its entries being durable) averaged 22 to 27 ms, with 2.5 to 3.1
transactions per group write and 13 to 18 ms per write; before PLAN9 the profile
measured a ticket wait of 92 to 99 ms plus a 7 ms append.
Against the acceptance of PLAN9 (section 11):
| criterion | result |
|---|---|
| mean at least 150 ms lower at 50 agents | met: 684 -> 432 ms on average (-252); per pair -161 and -344 |
| mean at least 100 ms lower at 100 agents | met on average: 548 -> 396 ms (-152); per pair -88 and -216 |
| p99 not higher than the baseline | not met in every pair: p5w 1 at 50 agents 1517 ms against base 1 1259 ms; lower in the other three pairs |
| committed phase at most 20 ms mean | not met: 22 to 27 ms |
acceptance-phase4.sh twice on the p5w tree |
A1, A2, A3 and the stretch goal pass in both runs (p99 on-100 3397 and 2338 ms against 0.5 x off of 4527 and 3817 ms) |
| lands/min not below phase 4b | 1377 / 1184 at 20 agents, 2628 / 2856 at 50, 2066 / 2785 at 100; above the phase 4b window 3 runs at 50 and 100 agents, below them at 20 agents in run 2 (1184 against 1366 and 1476) |
The runs are short (250 and 500 calls per point, 8 to 30 s each) and noisy: base 2
ran at half the throughput of base 1 (833 against 1427 lands/min at 50 agents) as
the load rose, so the per-pair numbers vary more than the effect at 100 agents. The
committed phase misses its target by a few milliseconds: a group write still costs
one fsync of the log (13 to 18 ms under this load) after the transaction's own wait
for the head of the queue. Outputs: /home/feder/p5w-work/window (scripts
window.sh, summary.py, means.py).
Batched landing (phase 4b)
Phase 4b (PLAN8.md) fixed the two causes of the latency that the
profile of the phase 4a tree found: about 12 fsyncs per batched request (each new
object fsynced when written and again by the quarantine migration) and nine short
git processes per request where one listing of the change's commits answers every
question. It also removed a second git process that the command factory started
for most git commands (gitcmd attr.tree detection), which speeds up every
command of the server, so the "off" column below is measured on that corrected
base too.
Runs of record: two runs of acceptance-phase4.sh in one benchmark window, on
the tree of commit 1899a0d, with the defaults (AGENT_COUNTS="20 50 100",
CHANGES=5, RATE=0.2, SEED=1, window = "10ms", max_size = 50,
target_latency = "500ms", graph_ancestry = false). Same machine as above
(Intel Core Ultra 7 255H, 16 logical CPUs, WSL2, ext4 on a virtual disk, an
fsync of a small file takes about 6 ms), with no other workload: the load average
was 4.2 when the first run started and rose to 38 and 48 during the runs, which is
the benchmark itself (both servers and up to 100 agents).
Both runs pass every criterion, the stretch goal included:
| criterion | 20 agents | 50 agents | 100 agents |
|---|---|---|---|
| attempts/landed (on), run 1 / run 2 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |
| failed = planned conflicts (on) | 22 / 22 | 50 / 50 | 108 / 108 |
| p99 on vs off (ms), run 1 | 1347 vs 7437 | 2525 vs 12030 | 4854 vs 10077 |
| p99 on vs off (ms), run 2 | 1193 vs 6741 | 4839 vs 12746 | 2897 vs 11905 |
| stretch: p99 on-100 <= 0.5 x off-100 | 4854 <= 5038 (run 1), 2897 <= 5952 (run 2) |
The stretch goal passed by 3.7% in run 1 and by 51% in run 2.
| metric (batching on), run 1 / run 2 | 20 agents | 50 agents | 100 agents |
|---|---|---|---|
| lands/min | 1020 / 1070 | 1230 / 944 | 1616 / 2325 |
| lands/min, phase 4a run of record | 412 | 251 | 258 |
| p50 landed (ms) | 911 / 739 | 1700 / 1980 | 2653 / 1824 |
| batch size mean | 8.2 / 8.3 | 13.4 / 8.9 | 13.9 / 18.5 |
| duration | 4.6 s / 4.4 s | 9.8 s / 12.7 s | 14.6 s / 10.1 s |
With batching off the runs took 17 to 41 s and landed 61 to 116 changes, as before: the off column still loses most lands to the 20-attempt limit at 50 and 100 agents.
Time per phase, mean per observation, run 1 / run 2:
| phase | 20 agents | 50 agents | 100 agents |
|---|---|---|---|
| land_member (per request) | 8.5 / 7.6 ms | 7.4 / 8.5 ms | 7.5 / 8.1 ms |
| plan (per request) | 3.3 / 3.1 ms | 3.1 / 3.3 ms | 3.1 / 3.6 ms |
| replay (per rebased request) | 4.8 / 4.2 ms | 3.9 / 4.8 ms | 4.1 / 4.2 ms |
| pack (per batch) | 55 / 47 ms | 75 / 69 ms | 53 / 45 ms |
| update (per batch) | 126 / 136 ms | 212 / 210 ms | 175 / 135 ms |
| total (per batch) | 238 / 233 ms | 351 / 335 ms | 321 / 324 ms |
Reading it:
- A batched request costs about 8 ms instead of 80 to 137 ms in phase 4a: one
git rev-listfor the decision and a replay through the long-lived merge-tree and hash-object processes, without fsync per object. - The lane is still busy for most of a run, but each batch now spends most of its time publishing: packing the objects of the batch and the update (the compare-and-swap, the hooks and the changeset log). At 100 agents the publish is about 70% of a batch. The update took 135 to 212 ms here, against 20 to 30 ms (without the quarantine migration) in a profile at lower load; why it grew under load is being profiled.
- The variance between runs is large: p99 at 50 agents was 2525 ms in run 1 and 4839 ms in run 2, p99 at 100 agents 4854 and 2897 ms. The benchmark saturates the machine (load average 38 to 48 during the runs), so single runs are not precise; compare runs of one window, and rerun before comparing across trees.
- The stretch goal passed with a margin of only 3.7% in run 1.
- The off column moved too. At 100 agents its p99 (10077 and 11905 ms) is lower than in the phase 4a run of record (13368 ms); at 50 agents it is higher (12030 and 12746 ms against 9939 ms). One known change applies to it: every git command of the server now starts one process instead of two for most commands. Why the off p99 at 50 agents rose is not explained; its p99 covers only the 97 changes that landed, out of 250.
Final tree (window 3)
After window 1 the unbatched land was moved onto the same landing attempt as a
batch (PLAN8 WP86). A second window (tree 59d8fda) passed every criterion but
showed a regression of the "off" column at 100 agents (p99 33.9 and 34.8 s, 60
lands/min): every unbatched attempt published its objects as a pack, and under
the contention of 100 agents about 98% of the ~6600 attempts lose the
compare-and-swap and each left a pack in the repository (120 packs at the end of
the run against 4 in window 1, pack-objects 1484 s and update-ref 840 s of
real time against 1 s and 56 s). Unbatched lands now publish their few objects
loose (PLAN8 section 13a). The runs of record of the final tree, commit
9bf4742, are window 3 (same workload and defaults as above; load average 10.7
at the start, 53 and 57 at the end of the runs):
| criterion | 20 agents | 50 agents | 100 agents |
|---|---|---|---|
| attempts/landed (on), run 1 / run 2 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |
| failed = planned conflicts (on) | 22 / 22 | 50 / 50 | 108 / 108 |
| p99 on vs off (ms), run 1 | 819 vs 4707 | 1880 vs 8267 | 4640 vs 14641 |
| p99 on vs off (ms), run 2 | 748 vs 4813 | 9171 vs 17868 | 5390 vs 26692 |
| stretch: p99 on-100 <= 0.5 x off-100 | 4640 <= 7321 (run 1), 5390 <= 13346 (run 2) |
| metric, run 1 / run 2 | 20 agents | 50 agents | 100 agents |
|---|---|---|---|
| lands/min, batching on | 1366 / 1476 | 1690 / 620 | 1459 / 1678 |
| lands/min, batching off | 332 / 336 | 230 / 151 | 158 / 114 |
| p50 landed on (ms) | 599 / 639 | 1326 / 1794 | 2927 / 2537 |
| member (land_member, mean) | 10.8 / 11.1 ms | 14.9 / 21.1 ms | 14.2 / 13.1 ms |
| pack per batch (mean) | 39 / 34 ms | 51 / 210 ms | 62 / 50 ms |
| update per batch (mean) | 65 / 52 ms | 89 / 155 ms | 114 / 107 ms |
All criteria pass in both runs, the stretch goal included (by 37% and 60%). The 50 agents of run 2 are an outlier on both sides (pack 210 ms and update 155 ms per batch, p99 on 9171 ms, lands/min 620): the load average was above 40 during that part of the run. The off column at 100 agents recovered from window 2 (p99 14641 and 26692 ms, 158 and 114 lands/min; window 1: 10077 and 11905 ms, about 170 lands/min) and its repository ended with 3 packs and 232 loose objects, but it still varies a lot between runs and was slower than in window 1 in run 2, for a reason not established.
What it does
- Starts
--agentsconcurrent agents (goroutines). Each agent commits--changeschanges on private branches, then lands each one on its target branch. - With
--branches Nthe agents are spread round robin over N target branches (agentilands on branchi mod N), so each branch has its own contention. The default is one branch shared by every agent. - Each change adds its own file, so changes never conflict by path.
- A fraction of the changes (
--conflict-rate) also rewrites a shared file by appending one line at the same place. Two such changes made from the same base conflict when the second one lands on the same branch. - The change plan is deterministic for a given
--seed, so runs are comparable.
Modes
| Mode | What is measured |
|---|---|
land |
UserLand with rebase. A conflicting change fails with merge_conflict. |
land-resolve |
UserLand with resolve_conflicts. Conflicts are resolved by the server's resolver. Requires the resolver configuration of RESOLVE. |
land-batch |
The same requests as land, run against a server that batches concurrent lands of a branch (see LAND-BATCHING). The mode only names the run: whether lands are batched is decided by the server's [graft.land.batch]. The batch of every land is reported in any mode. |
queue |
A simulated serialized merge queue. Changes land one at a time, and each is held for --test-duration (a simulated test) before the next starts. Latency includes waiting in the queue. |
Metrics
--format json and --json-out write these fields. The table output and the
Markdown report show the same values.
| Field | Meaning |
|---|---|
label |
The --label of the run. Omitted when the run has none. |
mode, agents, changes_per_agent, branches, conflict_rate, test_duration_ms |
The configuration of the run. |
changes |
Changes attempted (agents times changes per agent). |
landed |
Changes that reached the branch, including already landed ones. |
already_landed |
Changes that were already reachable from the branch. |
failed, conflicts |
Changes that did not land, and how many of them were merge conflicts. |
observed_conflict_rate |
conflicts divided by changes. |
resolutions |
Conflicting paths the server resolved. |
attempts |
Compare-and-swap attempts the server used, summed over every change, landed or failed. A land that gave up because the branch kept moving counts the attempts it made (the number the server reports in its "giving up after N attempts" error). Other failed lands count none, because the server does not report their attempts. Above landed means the branch moved during lands. |
attempts_per_land |
attempts divided by landed. The attempts wasted on failed changes raise it, so it is the server work per landed change. |
unavailable_retries |
Calls that the server answered with Unavailable and the SDK retried (LandOptions.RetryUnavailable), summed over the landed changes. Each is a whole land request again, the conflict resolver included. A failed land reports none. The benchmark lands with rebase, which the SDK does not retry by default, so it is 0 until rebase lands are retried. |
max_attempts |
The largest number of attempts of any change, landed or failed. |
attempts_histogram |
Landed changes by their number of attempts ({"1": 95, "2": 5}: 95 changes landed at the first attempt). Landed changes that report no attempts are left out. |
failed_attempts_histogram |
Failed changes that report attempts, by their number of attempts. A land that gave up after max_attempts rounds shows up here. |
batch_size_mean, batch_size_max, batch_size_histogram |
The size of the batch each landed change landed in, as the server reports it in batch_size of UserLandResponse. A server without batching reports 1 for every land. Landed changes without a reported batch (an older server) are left out; batch_size_max is then 0. Failed changes are not counted. |
duration_seconds |
Wall-clock duration of the run. |
lands_per_minute |
landed divided by the wall-clock duration of the run. |
p50_ms, p95_ms, p99_ms |
Latency of the LANDED changes, from the start of the commit to the end of the land (nearest rank). Failed changes are not included. |
failed_p95_ms |
p95 latency of the failed changes, measured the same way. 0 when none failed. |
rpcs |
The latency of each RPC as the client measures it around the call: commit_files (the UserCommitFiles of every change) and land (the UserLand of every change that was committed, without the wait of queue mode). Each has calls, failed, and p50_ms, p95_ms, p99_ms of the calls that succeeded (nearest rank). A land that fails on a conflict is a failed call. |
failure_classes |
Failed changes counted by the kind of their error message, with commit IDs removed, so the same conflict on different commits is one kind. |
errors |
Up to five distinct kinds of failure, each as the first full message of its kind. |
Latency includes the commit, so it measures what an agent waits for. It does
not include setup of the branch. The latency percentiles cover landed changes
only: the time a failed change spent before it failed is not the time a landed
change takes. Failed changes are reported separately by failed,
failure_classes and failed_p95_ms.
Usage
Every run needs branches of its own: give each run a fresh --branch.
graft-bench --address unix:/var/lib/graft/run/gitaly.socket \
--repo bench.git --branch bench-land-1 --agents 50 --changes 10 \
--conflict-rate 0.2 --mode land --token "$GRAFT_TOKEN"
graft-bench --address unix:/var/lib/graft/run/gitaly.socket \
--repo bench.git --branch bench-queue-1 --mode queue \
--test-duration 200ms --format json --token "$GRAFT_TOKEN"
graft-bench --address unix:/var/lib/graft/run/gitaly.socket \
--repo bench-new.git --create-repo --branch bench-land-1 --mode land \
--token "$GRAFT_TOKEN"
graft-bench --address unix:/var/lib/graft/run/gitaly.socket \
--repo bench.git --branch bench-batch-1 --branches 4 --agents 100 \
--changes 5 --mode land-batch --label batching-on-100 \
--json-out on-100.json --md-out on-100.md --token "$GRAFT_TOKEN"
Flags:
--address:unix:<path>,tcp://host:portortls://host:port. Required.--repo: relative path of the repository on the storage. Required.--branch(defaultbench): the branch the changes land on. It is notmainby default, because a policy usually protectsmain(the example policy ofexamples/graft/config.tomllets onlyplannerupdate it), and then every land is denied. The actor of the lands is the identity of the token (see CONFIGURATION), so the policy must let it update this branch. With--branchesabove 1 it is the prefix of the branch names, see below.--branches(default 1): the number of target branches. With more than one, the branches are named<branch>-00,<branch>-01, and so on, and the agents are spread over them round robin. It must not exceed--agents.--agents(default 50),--changes(default 10 per agent),--conflict-rate(default 0, from 0 to 1).--mode(defaultland):land,land-resolve,land-batchorqueue.--test-duration: the simulated test time per change,queuemode only.--storage,--seed(default 1),--max-attempts(default 0, the server default).--token(default$GRAFT_TOKEN) or--agent-token(default$GRAFT_AGENT_TOKEN). Use an agent token to measure what an agent sees. A flag overrides the other variable. With no flag,$GRAFT_TOKENis used when both variables are set. Passing both flags is an error.--create-repo: creates the repository first, through the server'sRepositoryService(never by writing into the storage directory). It needs the shared--tokenand fails if the repository exists.--timeout: bounds the whole run (default 10m).--format table(default) orjson: the output on stdout.--label NAME: names the run in reports, such asbatching-on-50. It is thelabelfield, the title of the Markdown report and the column header ofgraft-bench compare. It must not contain|or newlines.--json-out FILE: also writes the result as JSON (the--format jsonobject) to FILE.--md-out FILE: also writes the result as a Markdown report to FILE: a## graft-bench: <label>heading (the mode when there is no label), the metrics as a table, and the failed changes by kind.
Each target branch is created on first use with an initial commit that holds
the shared file. A run adds a file per change under agents/, so a second run
on the same branch would fail every commit: graft-bench refuses a branch
that holds agents/ and exits 1 with "every run needs a fresh --branch". A
repository can be reused with a fresh branch name, which also keeps the
numbers comparable. A branch that exists without agents/ (for example one
created beforehand) is used as it is.
A run cut short by --timeout or an interrupt prints the samples it has,
writes the --json-out and --md-out files, and exits 1 with the reason on
stderr. Its numbers are partial and must not be reported as a complete run.
Comparing runs
graft-bench compare prints saved JSON results (from --json-out or
--format json) side by side, one column per file:
graft-bench compare --format markdown off-50.json on-50.json
--format table(default) prints an aligned text table,--format markdowna Markdown pipe table.- Each column is headed by the
labelof the result, or by the file name when the result has no label. - The rows are the mode, agents, branches, changes, landed, failed, conflict rate, attempts (all changes), attempts per landed change, max attempts (all changes), the attempts histograms of landed and of failed changes, the batch size mean, max and histogram, duration, lands per minute, latency p50, p95 and p99 of the landed changes, and latency p95 of the failed changes.
It exits 2 without a file or with an unknown format, and 1 when a file cannot be read.
Demo
examples/graft/demo/demo.sh runs the land, land-resolve and queue
modes with 50 agents and prints a comparison table. See the header of the
script for the setup it needs.
land-resolve only resolves conflicts when the server has a resolver for the
shared file. The demo ships examples/graft/demo/resolve.toml, which unions
the two sides of shared/log.txt and tries trivial first. The header of
demo.sh gives the [graft.land.resolver] snippet that enables it. Without
that configuration the land-resolve row matches the land row.
Batching benchmark
examples/graft/demo/bench-batch.sh [WORKDIR] measures what batched landing
(see LAND-BATCHING) changes:
- It builds
gitaly(withmake build) andgraft-bench, and writes tokens. - It starts two standalone Gitaly servers from
examples/graft/config.tomlthat differ only in[graft.land.batch] disabled(truefor "off",falsefor "on"), each with its own repositories and unix socket under WORKDIR. - Against each server it runs
graft-bench --mode landwith 20, 50 and 100 agents of 5 changes each, conflict rate 0.2 and seed 1, on a fresh branch per run (the first run creates the repository with--create-repo). Each run is labelledbatching-<off|on>-<agents>and saved with--json-outand--md-out. - It prints the combined
graft-bench compare --format markdowntable of all runs and saves it ascombined.md. - It stops both servers when it exits, also on failure.
WORKDIR defaults to a fresh temporary directory; the results are in
WORKDIR/results. The environment variables AGENT_COUNTS ("20 50 100"),
CHANGES (5), RATE (0.2) and SEED (1) change the workload. The script needs
make, go, openssl and the Gitaly build dependencies (see
examples/graft/quickstart.sh).
Read the combined table column by column: the off and on runs of one agent
count share the workload, so attempts/landed change, the attempts
histograms, max attempts (all changes) and the landed latency show what
batching saves, and the batch size rows show how many lands shared an update.
Because the attempts of lands that gave up count in both columns, the off
column is not flattered by its failures.
Phase breakdown
Each server exposes its metrics on its own prometheus_listen_addr (set by
bench-batch.sh). After each run the script scrapes /metrics and appends a
"time per phase" table to the results. For each phase of a batch attempt
(graft_land_batch_phase_seconds{phase}: quarantine, tip, land_member,
plan, replay, pack, update, total) it shows the sum and the mean, and
the mean land_member time per batch. plan and replay are parts of
land_member; pack precedes update. The metric names are in LAND-BATCHING.
Acceptance run (acceptance-phase4.sh)
examples/graft/demo/acceptance-phase4.sh [WORKDIR] runs bench-batch.sh with
the same workload (20, 50 and 100 agents, 5 changes each, conflict rate 0.2,
seed 1) and checks the acceptance criteria of
PLAN4 against the JSON results:
| Criterion | Check on the batching-on runs (and the off runs for p99) |
|---|---|
| Attempts | attempts_per_land is 1.00 at every agent count |
| Failures | failed equals conflicts: only the planned conflicts fail |
| Latency | p99_ms on is at most p99_ms off, at every agent count |
| Stretch (reported, not required) | p99_ms on at 100 agents is at most half of off |
Each criterion prints PASS or FAIL. The script exits non-zero when one fails,
and it prints the stretch result as STRETCH PASS or STRETCH MISS. The
AGENT_COUNTS, CHANGES, RATE and SEED variables of bench-batch.sh are
passed through.
acceptance-phase4.sh --dry-run WORKDIR runs only the check, on the JSON files
that a previous run left in WORKDIR/results; it builds and starts nothing. Use
it to re-check saved results. The script keeps WORKDIR/results (the JSON, the
Markdown, the Prometheus scrapes and combined.md). On exit, also when a run is
interrupted, it stops any Gitaly server of WORKDIR that is still running and
deletes the agent tokens in WORKDIR/tokens.
Reading the numbers
- Compare
landandland-resolveat the same--conflict-rate. The first shows the cost of conflicts as failures, the second their cost as latency. attemptsclose tolandedmeans little contention on the branch tip. High values mean many retries, so increase--max-attempts, reduce the number of agents landing on one branch (--branchesshows the effect), or enable batching.- A
failed_attempts_histogramentry at themax_attemptsof the server means lands gave up because the branch kept moving. batch_size_meanabove 1 means concurrent lands shared one reference update. With batching disabled every land reports 1.- In
queuemode,p95andp99grow with the number of changes ahead in the queue, times--test-duration. That is the cost of serialization, which the rebase-based land avoids for changes that do not conflict. - Results depend on the machine, the storage and whether hooks run. Report the full JSON output with any number you quote.
Limits of the benchmark
- Each target branch is shared by the agents assigned to it. Changes on
different branches never conflict with each other, so
--branchesmeasures independent contention, not cross-branch interaction. - The simulated test in
queuemode does not run the tests. Usegraft-verifyfor real post-merge checks. - The shared file conflicts by design. Real repositories conflict less often, so the conflict rate is a stress knob, not a forecast.
- The attempts of a failed land are known only when it gave up because the branch kept moving. Other failures (merge conflicts, hook rejections) report no attempts.
History (superseded)
These results describe code that has since changed. Phase 4b (PLAN8) superseded the phase 4a results: it removed the per-object fsyncs and the extra git processes of a batched land, and its runs pass every acceptance criterion (see Batched landing (phase 4b)).
Phase 3 baseline
The table from PLAN4, measured with batching on and off:
| metric | off-20 | on-20 | off-50 | on-50 | off-100 | on-100 |
|---|---|---|---|---|---|---|
| attempts/landed | 10.97 | 1.00 | 29.35 | 1.00 | 56.70 | 1.00 |
| p50 landed (ms) | 954 | 3320 | 5484 | 8152 | 5703 | 15950 |
| p99 landed (ms) | 5049 | 9868 | 17669 | 15317 | 16775 | 21808 |
Measured phase 4a results
Run of record: acceptance-phase4.sh /tmp/graft-acc4-final3 on the final phase
4a tree (all work packages and fixes merged), with the defaults
(AGENT_COUNTS="20 50 100", CHANGES=5, RATE=0.2, SEED=1,
window = "10ms", max_size = 50, target_latency = "500ms",
graph_ancestry = false). Machine: Intel Core Ultra 7 255H, 16 logical CPUs,
WSL2 (kernel 6.6.87.2), ext4 on a virtual disk, git 2.53.0. Other builds and
tests ran on the same machine at the same time (load average 8 to 38 during the
run), so the numbers carry noise. Quote them with the machine, and rerun before
comparing.
Result:
PASS attempts/landed = 1.00 with batching on, 20 agents (landed 78)
PASS failures are planned conflicts only, batching on, 20 agents (failed 22, conflicts 22)
PASS p99 on 2789.0 ms <= off 10022.5 ms, 20 agents
PASS attempts/landed = 1.00 with batching on, 50 agents (landed 200)
PASS failures are planned conflicts only, batching on, 50 agents (failed 50, conflicts 50)
FAIL p99 on 12658.3 ms <= off 9939.0 ms, 50 agents
PASS attempts/landed = 1.00 with batching on, 100 agents (landed 392)
PASS failures are planned conflicts only, batching on, 100 agents (failed 108, conflicts 108)
FAIL p99 on 23252.6 ms <= off 13368.1 ms, 100 agents
STRETCH MISS p99 on-100 23252.6 ms, 0.5 x off-100 6684.1 ms
acceptance-phase4: FAIL
The phase 4a acceptance is not met. Contention stays solved at every size (one attempt per landed change, only the planned conflicts fail), and p99 with batching on is below off at 20 agents, but it is above off at 50 and 100 agents.
The p99 criterion is not stable between runs. All runs below use the same workload:
| run | tree | p99 on/off 20 (ms) | p99 on/off 50 (ms) | p99 on/off 100 (ms) | result |
|---|---|---|---|---|---|
/tmp/graft-acc4 |
phase 4a before the final fixes | 4443 / 5914 | 9607 / 9683 | 17635 / 19381 | PASS (50 agents by 76 ms) |
/tmp/graft-acc4-final2 |
final | 3502 / 6303 | 13235 / 9703 | 24716 / 19848 | FAIL at 50 and 100 |
/tmp/graft-acc4-final3 (of record) |
final | 2789 / 10023 | 12658 / 9939 | 23253 / 13368 | FAIL at 50 and 100 |
A further run on the final tree, made while other test suites loaded the machine heavily (some of its runs took up to 8 times longer than usual), is left out; it failed the p99 criterion at 50 agents. The off column varies most, because its landed latency depends on random retries, and its p99 covers only the changes that landed (115 of 500 at 100 agents in the run of record; 277 gave up after 20 attempts and are not in it).
Combined table of the run of record (results/combined.md):
| metric | batching-off-20 | batching-on-20 | batching-off-50 | batching-on-50 | batching-off-100 | batching-on-100 |
|---|---|---|---|---|---|---|
| mode | land | land | land | land | land | land |
| agents | 20 | 20 | 50 | 50 | 100 | 100 |
| branches | 1 | 1 | 1 | 1 | 1 | 1 |
| changes | 100 | 100 | 250 | 250 | 500 | 500 |
| landed | 62 | 78 | 99 | 200 | 115 | 392 |
| failed | 38 | 22 | 151 | 50 | 385 | 108 |
| conflict rate | 0.220 | 0.220 | 0.200 | 0.200 | 0.216 | 0.216 |
| attempts (all changes) | 706 | 78 | 2766 | 200 | 6548 | 392 |
| attempts/landed change | 11.39 | 1.00 | 27.94 | 1.00 | 56.94 | 1.00 |
| max attempts (all changes) | 20 | 1 | 20 | 1 | 20 | 1 |
| attempts histogram (landed) | 1: 7, 2: 4, 3: 10, 4: 10, 5: 9, 6: 1, 7: 2, 8: 6, 9: 2, 11: 1, 12: 1, 13: 1, 14: 2, 16: 2, 17: 1, 18: 1, 19: 1, 20: 1 | 1: 78 | 1: 8, 2: 15, 3: 9, 4: 9, 5: 6, 6: 5, 7: 9, 8: 3, 9: 2, 10: 3, 11: 1, 12: 6, 13: 6, 14: 2, 15: 3, 16: 3, 17: 4, 20: 5 | 1: 200 | 1: 10, 2: 10, 3: 11, 4: 7, 5: 10, 6: 8, 7: 4, 8: 2, 9: 5, 10: 3, 11: 2, 12: 4, 13: 6, 14: 5, 15: 8, 16: 5, 17: 2, 18: 4, 19: 4, 20: 5 | 1: 392 |
| attempts histogram (failed) | 20: 16 | - | 20: 101 | - | 20: 277 | - |
| batch size mean | 1.00 | 9.86 | 1.00 | 10.95 | 1.00 | 3.80 |
| batch size max | 1 | 18 | 1 | 43 | 1 | 25 |
| batch size histogram | 1: 62 | 2: 12, 3: 11, 4: 12, 6: 2, 12: 12, 17: 14, 18: 15 | 1: 99 | 1: 14, 2: 97, 3: 33, 6: 5, 12: 11, 43: 40 | 1: 115 | 1: 47, 2: 180, 3: 129, 6: 4, 12: 10, 25: 22 |
| duration | 17.51s | 11.36s | 28.02s | 47.82s | 54.49s | 91.30s |
| lands/min | 212.4 | 411.9 | 212.0 | 250.9 | 126.6 | 257.6 |
| latency p50 (landed) | 1518.2ms | 1989.4ms | 2097.0ms | 8958.5ms | 4073.0ms | 17292.1ms |
| latency p95 (landed) | 5786.8ms | 2670.6ms | 7324.4ms | 12475.3ms | 12293.4ms | 22915.5ms |
| latency p99 (landed) | 10022.5ms | 2789.0ms | 9939.0ms | 12658.3ms | 13368.1ms | 23252.6ms |
| latency p95 (failed) | 11338.5ms | 2706.3ms | 9644.9ms | 12466.1ms | 14038.5ms | 22404.3ms |
Time per phase of the run of record (the difference of
graft_land_batch_phase_seconds between the scrapes before and after each
run). The runs with batching off execute no batch, so all their rows are zero
and are left out here:
| run | phase | observations | sum (s) | mean (s) |
|---|---|---|---|---|
| on 20 agents | quarantine | 21 | 0.012507 | 0.000596 |
| on 20 agents | tip | 21 | 0.078462 | 0.003736 |
| on 20 agents | land_member | 100 | 7.968227 | 0.079682 |
| on 20 agents | land_member per batch | 21 | 7.968227 | 0.379439 |
| on 20 agents | update | 19 | 3.002550 | 0.158029 |
| on 20 agents | total | 21 | 11.095477 | 0.528356 |
| on 50 agents | quarantine | 99 | 0.059037 | 0.000596 |
| on 50 agents | tip | 99 | 0.370074 | 0.003738 |
| on 50 agents | land_member | 250 | 28.369638 | 0.113479 |
| on 50 agents | land_member per batch | 99 | 28.369638 | 0.286562 |
| on 50 agents | update | 86 | 18.260168 | 0.212328 |
| on 50 agents | total | 99 | 47.216474 | 0.476934 |
| on 100 agents | quarantine | 228 | 0.140545 | 0.000616 |
| on 100 agents | tip | 228 | 0.877662 | 0.003849 |
| on 100 agents | land_member | 500 | 51.281139 | 0.102562 |
| on 100 agents | land_member per batch | 228 | 51.281139 | 0.224917 |
| on 100 agents | update | 215 | 38.124248 | 0.177322 |
| on 100 agents | total | 228 | 90.788043 | 0.398193 |
Reading it:
- Contention stays solved: every landed change took one attempt, and only the planned conflicts failed. With batching off, 16, 101 and 277 changes gave up after 20 attempts. Batching on lands 1.3 to 3.4 times as many changes, and its throughput (lands/min) is higher at every size.
- The lane is saturated: the
totalsum of the batches (11.1 s, 47.2 s, 90.8 s) is almost the whole run duration (11.4 s, 47.8 s, 91.3 s). Batches run one after another, so a request waits for every batch queued before it, and p50 and p99 with batching on are mostly time in the queue. - Landing the members is the dominating phase:
land_memberis 72%, 60% and 56% of the batch time at 20, 50 and 100 agents, at about 80 to 115 ms per member with the streaming rebase engine, and it is serial. The reference update with its hooks (update, 160 to 210 ms per published batch) is 27%, 39% and 42%. Opening the quarantine and reading the tip are under 1%. - Adaptive sizing is not the cause. A diagnostic run of the batching-on server
alone with
target_latency = "0s"(no adaptation; mean batch sizes 32.7 and 49.6 instead of 10.95 and 3.80) gave p99 12128 ms at 50 agents and 20081 ms at 100 agents, with durations of 52.5 s and 79.5 s: larger batches save update phases, but the serial member work stays. - What phase 4b (pipelined batches) must address: the serial lane. A batch must
be able to land its members onto the unpublished tip of the batch before it
while that batch runs its CAS and hooks, which hides the
updateshare (about 40% at 50 and 100 agents). The per-member cost of about 100 ms is the larger share and the bound that remains after pipelining, so phase 4b must also make it cheaper or parallel (for example, landing members that touch disjoint paths concurrently), and breakland_memberdown in the metrics (ancestry, rebase, commit writes, policy) to find where the 100 ms go. - Compared with phase 3 (table above), p99 with batching on dropped at 20 agents (9868 to 2789 ms) and 50 agents (15317 to 12658 ms), and rose slightly at 100 agents (21808 to 23253 ms), on a noisy machine.