Skip to content

Git designed for AI agents.

docs: progress of the lands on a hot file: the measurements, the rules in the lane, the queue, the limits

Federico Eliadba7b343ebd2
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.md
index 2bfac8c..daed38e 100644
--- a/doc/graft/CONFIGURATION.md
+++ b/doc/graft/CONFIGURATION.md
@@ -201,7 +201,8 @@ cause.
| `verify_per_repository` | int | `1` | Verifications that run at once for one repository (of any resolution: a rule's, an agent's, a model's), inside `verify_concurrency`: every land of a busy branch builds a tree of its own to verify, and one repository must not take the whole pool. A land waits for its repository's turn first, holding nothing of the pool, within the land's deadline. **The cost, stated:** a resolution lands only on the tree that was verified, so a land whose branch moved after its tree was verified is verified again, and on a busy branch the lands take their turns one repository at a time; at the deadline a land ends with the package (a rule's files `NOT_ATTEMPTED` with the reason, a model's `PROPOSED`) and the agent lands again. Lands of rules are batched only when `verify.enabled` is not set. |
| `verify_concurrency` | int | `2` | Verifications that run at once in the Gitaly process (of any resolution: a rule's, an agent's, a model's), all out of the lane, in a pool of their own that exists whenever `verify.enabled` is set, with or without an AI resolver, as `verify.command` takes minutes. A verification waits for a free slot within the land's deadline. The wait is in `graft_resolver_verify_slot_wait_seconds`. |
| `resolve_concurrency` | int | `2` | Resolutions that run at once in the Gitaly process, in a pool apart from the checks', so that long resolutions never delay a check. A resolution waits for a free slot within the land's deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_resolve_slot_wait_seconds`. |
-| `resolve_rounds` | int | `32` | Safety cap on the rounds of a land whose conflicting files keep changing while it is resolved out of the lane. The rounds are bounded by `timeout`; the cap is set well above realistic contention and is no contract. |
+| `resolve_rounds` | int | `32` | Safety cap on the rounds of a land whose conflicting files keep changing while it is resolved out of the lane. The rounds are bounded by `timeout`; a land that loses a round waits for its turn on its files in a queue, so that it does not lose them one after the other (RESOLVE.md, "Out of the lane"), and the cap is not reached in practice. It is no contract. |
+| `inline_timeout` | string (Go duration) | `"1s"` | Bound of the call with which the lane resolves a member's conflicting files itself, as the rules (trivial, union, rerere, a command that is a rule) and never a model: the request carries `rules_only`. A member whose files the rules cannot resolve within it is resolved out of the lane, as any other. `"0s"` turns the inline resolution off. Never used with `verify.enabled`: every tree is then verified out of the lane. |
| `paths_allow` | array of strings | empty (every path) | Globs of paths that may be sent to the resolver, with the syntax of policy refs. |
| `paths_deny` | array of strings | empty | Globs of paths that are never sent. Deny wins over allow. |
| `actors` | array of strings | empty (every caller that may land) | Globs of the identities that may request resolution: agent names, or `shared` for the shared token. Without `[[graft.agent]]` no caller has an identity, so a non-empty list admits nobody (the server warns at startup). |
diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.md
index e614ac8..fde98c9 100644
--- a/doc/graft/PLAN10.md
+++ b/doc/graft/PLAN10.md
@@ -1133,6 +1133,8 @@ whole file (`agree_with`, 15.13): the one filter that chose better proposals tha
| the tree that would land | `landResolvedTree` | operations/land_resolve.go |
| VERIFY, the one box, out of the lane, for every resolution | `landVerifyOutOfLane`, `landVerify`, `landPrecheckVerify` (an agent's), `VerifyPool.Acquire` | operations/land_verify.go, land_reland.go; graft/resolver/check.go |
| verdicts applied in the lane | `landApplyVerdicts`, `landNeedsResolution` | operations/land_resolve.go, land_preresolve.go |
+| the rules resolved in the lane (rules_only, inline_timeout) | `landResolveInline`, `landInlineEnabled`, `Server.landInlineOff`, `Request.RulesOnly`, `Response.RulesOnly`, `unsupportedBy` and the skip of models in `resolveFile` | operations/land_inline.go, land_resolve.go; graft/resolver/protocol.go; tools/.../resolve/resolve.go |
+| the queue of the lands that lose a round | `landTickets`, `landTicketRegistry.acquire/blocked`, `landTakeTicket`, `landTicketsBlock`, the block in `landApplyVerdicts` | operations/land_tickets.go, land_preresolve.go, land_resolve.go |
| unverified in time: proposals | `landAnnotateUnverified`, `landStoppedReason`, `landFinalPass` | operations/land_resolve.go, land_preresolve.go |
| resolver call, bounded slots | `Client.AcquireResolveSlot`, `ResolveSlot.Resolve` | graft/resolver/check.go |
| policy before resolving | `hook.CheckAgentPolicy`, `landPolicyDeniedError` | operations/land_resolve.go, land_proposals.go |
@@ -1277,17 +1279,41 @@ The resolution is bound to the files instead:
changed), the lane returns the attempt with those files, without resolving. UserLand
pre-resolves only them against the new tip, out of the lane, keeps every other verdict, and
enters the lane again. A tip that moves on unrelated files costs no resolver call at all.
-* **Progress, and its limit.** A round is lost only when the lane found a file whose sides
- changed, that is when another land on that file landed. So the lands that contend for a file
- all land, at least one per round, and N contenders need at most N rounds each. The rounds are
- therefore bounded by the resolver's timeout for the whole land only; resolve_rounds (default
- 32) is a safety cap well above realistic contention, documented as a cap and not a contract
- (decided after the 12-concurrent graft-bench test failed with a cap of 3). The limit, stated:
- under continuous new arrivals on the same file, one land can keep losing until its deadline,
- and then ends with the merge conflict and its package (15.9, the final pass): the files that
- changed under it are NOT_ATTEMPTED with the reason "the files changed again before they could be
- resolved" (the old "files kept changing" message is gone, as it carried no package); past the cap
- the same.
+* **Progress, and what it took.** A round is lost when the lane found a file whose sides changed, that is when
+ another land on that file landed. It was written here that the lands that contend for a file all land, one per
+ round, and that was true of the system and false of a land. Measured (graft-bench, 50 agents x 5 changes on one
+ branch, conflict rate 0.2, seed 1, 50 lands on one file, the commit before the fix): every landing invalidates the
+ verdicts of all the others, so after each one they all resolved again: 749 resolver calls and 713 hand-backs from the
+ lane for 50 resolutions. The resolve slots were not the cost (2.3 s of wait in all); the lane was: 957 members in a
+ run in which it was busy for 33 s of 34, 0.41 s of wait for each, and so the clean lands of the branch were slow too.
+ 6 of the 50 lands lost 32 rounds in a row and ended with "the files changed again before they could be resolved"
+ (244 of 250 landed, 364 to 430 lands/min, p99 23 to 27 s, against 250 of 250, about 1,700 to 1,900 and 2 s of
+ v0.2.0, which resolved inside the lane). Three things now give every land its turn:
+ 1. **The rules are resolved in the lane, as in v0.2.0** (inline, `inline_timeout`): a land enters the lane first;
+ what a rule can resolve is resolved there, in its batch, by a request with `rules_only` that graft-resolve answers
+ with its rules and never with a model (the answer echoes `rules_only`; a resolver that does not is an older one, and
+ the lane never asks again for the life of the process). The split follows what the resolution is, not how long it
+ took: a model, a verification, a rule that does not end within `inline_timeout` (default 1 s) or one that cannot
+ resolve a file go out of the lane, to the path below. With the rules in the lane the bench has no hand-back, no
+ ticket and one resolver call per resolution, and its numbers are v0.2.0's.
+ 2. **A queue for the lands that lose a round out of the lane.** A land that lost a round takes a ticket, first in
+ first out, on its files (per file; per branch when `verify.enabled`, since the tree that is verified is the
+ branch's), keeps it until it lands, its deadline ends or it fails, and is the only land of the branch that resolves
+ them again; the others wait without calling the resolver, and the lane hands back any other land with a verdict on a
+ ticketed file, so that the holder's verdict is the one that lands. Measured on the same bench with the rules out of
+ the lane: 92 resolver calls, 8 hand-backs, 250 of 250. A land that waits past its deadline ends with the package
+ ("the time ran out waiting for the other lands on the same files").
+ 3. resolve_rounds (default 32) is only a safety cap: the rounds of a land are bounded by its deadline, and the queue
+ makes it lose at most the rounds that a land which changes its files without a conflict makes it lose.
+ The limits, stated. The queue is in memory and per process, like the lanes: against another node of the same branch
+ (authoritative storage) the lands go on in rounds, without the guarantee. A holder's verdict is invalidated by a land
+ that changes its file cleanly and lands first (each such land lands once, and the holder resolves again, which is
+ cheap for a rule); with a verification every tree is verified out of the lane, and any land that moves the branch
+ while a holder verifies, a clean one included, makes the verified tree stale: on the bench with a verification of 1 s,
+ 50 resolutions cost 130 verifications and 120 to 135 s (113 to 124 lands/min, p99 74 to 94 s), all 250 landed, bounded by
+ the deadline; before the queue the same bench cost 802 verifications for 34 resolutions, 16 lands failed ("could not
+ verify before the branch moved", "the files changed again"), 158 lands/min, p99 53 s. Verifying a batch of resolutions
+ on one combined tree would remove that cost; it is not done.
* **Rules alike.** The operator rules' verdicts (trivial, union, rerere, command) are kept and
reused by the same key, so the policy is the same for every strategy.
* **Slots: two pools.** check_concurrency bounds the checks of agents' resolutions and
diff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.md
index bd488a6..d0a8e72 100644
--- a/doc/graft/RESOLVE.md
+++ b/doc/graft/RESOLVE.md
@@ -250,6 +250,11 @@ Request (server to resolver):
origin of a file, `proposal` a file that is not a rule's is a proposal that lands only on a verified tree, `hold` it
never lands a file with a `hold`, `verify` it verifies the exact tree that lands. A request without it comes from an
older server, and `graft-resolve` gives it no model's resolution (see "Version skew" below).
+- `rules_only` (additive, optional) asks the resolver to run only its rules, the strategies whose answers are a rule's, and
+ never to call a model: the lane of the server sends it when it resolves a member inline (see "Out of the lane"). A
+ resolver of this release that honours it says so with `"rules_only": true` in its response; what the rules do not
+ resolve is left unresolved, as a failure of the chain. A server that gets no echo is talking to an older resolver,
+ whose answer it drops, and it never asks again for the life of the process.
- `ours` is the current branch tip side. `theirs` is the change being landed.
- `tip`, `merge_base` and `messages.tip` are always empty since this version;
they stay in the protocol for compatibility. The server reuses a file's
@@ -538,14 +543,31 @@ branch for the other lands:
the server resolves only that file, out of the lane, and enters the lane
again.
-A round is lost only when another land changed one of the files, so the lands
-that contend for a file all land, one per round. The rounds are bounded by the
-resolver's `timeout` for the whole land, and by `resolve_rounds`, a safety cap
-well above realistic contention. The limit, stated: under continuous new
-arrivals on the same file, one land can keep losing until its deadline; it then
-ends with the merge conflict and its package: the files that changed under it are
-`NOT_ATTEMPTED` with the reason `the files changed again before they could be
-resolved`, the others keep their verdicts (a model's `PROPOSED`).
+A round is lost when another land changed one of the files. The rules do not wait for that:
+
+* **Rules in the lane.** Without `verify.enabled`, a land enters the lane first, and the lane resolves the files that
+ have no verdict itself, as the rules (trivial, union, rerere, a command that is a rule), within `inline_timeout`
+ (default `1s`; `0s` turns it off). It sends the request with `"rules_only": true`, and `graft-resolve` never calls a
+ model for it. What a rule cannot resolve, or does not resolve in time, is handed back and resolved out of the lane as
+ described above. The lands on a hot file ride the same batches as the others, as they did in v0.2.0. A resolver that
+ does not echo `rules_only` is an older one: its answer is dropped and the lane never asks again for the life of the
+ server process.
+* **A queue for the rest.** A land that loses a round out of the lane (a model, or a verification, or a rule that
+ takes more than `inline_timeout`) takes a ticket on its files, first in first out, and keeps it until it ends. It is
+ the only land of the branch that resolves those files again; the others wait without calling the resolver, and the lane
+ hands back any other land with a verdict on them. A land that waits past its deadline ends with the merge conflict and
+ its package, its files `NOT_ATTEMPTED` with the reason `the time ran out waiting for the other lands on the same files`.
+ The tickets are per file, or per branch with `verify.enabled` (the tree that is verified is the branch's). They are in
+ the memory of the server process, like the lanes: with several nodes on one branch (authoritative storage) the lands
+ go on in rounds across nodes, without this order.
+
+The rounds are bounded by the resolver's `timeout` for the whole land, and by `resolve_rounds`, a safety cap that the
+order above does not reach. The limits, stated: a land that holds the ticket of a file loses a round to a land that
+changes the file without a conflict and lands first (once per such land); with a verification every resolution is
+verified out of the lane on its own tree, in turn, and a land that moves the branch meanwhile makes the verified tree
+stale and the verification is repeated (the cost is measured in PLAN10 15.8). In every case the end is bounded by the
+deadline, with the merge conflict and its package: the files that changed under the land are `NOT_ATTEMPTED` with the
+reason `the files changed again before they could be resolved`, the others keep their verdicts (a model's `PROPOSED`).
`verify.command` runs out of the lane, whatever made the resolution (a rule, an agent, a model): on
the exact tree that would land, with a slot of the process's verify pool, before the lane, which only
diff --git a/internal/gitaly/service/operations/land_inline_test.go b/internal/gitaly/service/operations/land_inline_test.go
index 0fa457f..d763eb0 100644
--- a/internal/gitaly/service/operations/land_inline_test.go
+++ b/internal/gitaly/service/operations/land_inline_test.go
@@ -1,6 +1,7 @@
package operations
import (
+ "fmt"
"strings"
"sync"
"testing"
@@ -8,9 +9,14 @@ import (
"github.com/stretchr/testify/require"
+ "github.com/feder-cr/git-ai/internal/featureflag"
"github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
"github.com/feder-cr/git-ai/internal/gitaly/config"
"github.com/feder-cr/git-ai/internal/graft/identity"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
)
// The tests of the inline resolution of the rules in the lane (land_inline.go), with the resolver of
@@ -175,3 +181,63 @@ func TestLandResolveInline_neverForAnAgentThePolicyDenies(t *testing.T) {
require.True(t, s.landResolveInline(ctx, fx.request("Land"), fx.tip, fx.commit, l, pre))
require.Equal(t, 1, unionCalls(t, dir))
}
+
+// TestUserLand_aRuleResolvedInTheLaneRidesTheBatchOfTheCleanLands: through the lanes of a standalone server,
+// a land with a conflict that a rule resolves enters the lane with the clean lands, is resolved there by
+// one request for the rules only, and lands in their batch, not in one of its own.
+func TestUserLand_aRuleResolvedInTheLaneRidesTheBatchOfTheCleanLands(t *testing.T) {
+ t.Parallel()
+
+ dir, argv := unionResolver(t)
+ rcfg := config.GraftResolver{Enabled: true, Command: argv, ResolveRounds: 1}
+ cfg := testcfg.Build(t)
+ cfg.Graft = config.Graft{
+ Standalone: true,
+ Land: config.GraftLand{Batch: config.GraftLandBatch{Window: "100ms"}, Resolver: rcfg},
+ }
+ ctx, cfg, client := setupOperationsServiceWithCfg(t, testhelper.Context(t), cfg)
+ ctx = featureflag.ContextWithFeatureFlag(ctx, featureflag.GPGSigning, true)
+ repo, repoPath := gittest.CreateRepository(t, ctx, cfg)
+ file := func(path, content string) gittest.TreeEntry {
+ return gittest.TreeEntry{Mode: "100644", Path: path, Content: content}
+ }
+ base := gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch(git.DefaultBranch), gittest.WithTreeEntries(file("a", "base\n")))
+ gittest.WriteCommit(t, cfg, repoPath, gittest.WithParents(base), gittest.WithBranch(git.DefaultBranch), gittest.WithTreeEntries(file("a", "ours\n")))
+
+ conflicting := gittest.WriteCommit(t, cfg, repoPath, gittest.WithParents(base), gittest.WithTreeEntries(file("a", "theirs\n")))
+ var clean []git.ObjectID
+ for i := 0; i < 4; i++ {
+ clean = append(clean, gittest.WriteCommit(t, cfg, repoPath, gittest.WithParents(base),
+ gittest.WithTreeEntries(file("a", "base\n"), file(fmt.Sprintf("p%d", i), "p\n"))))
+ }
+
+ var wg sync.WaitGroup
+ sizes := make([]int32, 0, 5)
+ var mu sync.Mutex
+ land := func(commit git.ObjectID, resolve bool) {
+ defer wg.Done()
+ resp, err := client.UserLand(ctx, &gitalypb.UserLandRequest{
+ Repository: repo, User: gittest.TestUser, Branch: []byte(git.DefaultBranch),
+ CommitId: commit.String(), ResolveConflicts: resolve, MergeMessage: []byte("Land " + commit.String()),
+ })
+ require.NoError(t, err)
+ if resolve {
+ mu.Lock()
+ sizes = append(sizes, resp.GetBatchSize())
+ mu.Unlock()
+ }
+ }
+ wg.Add(1)
+ go land(conflicting, true)
+ for _, c := range clean {
+ wg.Add(1)
+ go land(c, true) // they ask for the resolution too: the rest of the clean lands never need it
+ }
+ wg.Wait()
+
+ require.Equal(t, 1, unionCalls(t, dir), "the conflict was resolved by one request")
+ require.Equal(t, 1, inlineCalls(t, dir), "which was the lane's, for the rules only")
+ for _, size := range sizes {
+ require.Greater(t, size, int32(1), "every land rode a batch of several")
+ }
+}