Skip to content

Git designed for AI agents.

graft: land minors of the review (final pass, hand-back, per-repository cap, docs)

Federico Elia7ebbb0c6a894
diff --git a/doc/graft/AGENT-SECURITY.md b/doc/graft/AGENT-SECURITY.md
index 7a14b24..b12cb8f 100644
--- a/doc/graft/AGENT-SECURITY.md
+++ b/doc/graft/AGENT-SECURITY.md
@@ -213,6 +213,21 @@ Two known limits, accepted:
## Resolution and verification are opt-in
+**A model's resolution can land, and the verification is what lets it.** With a
+server AI and a `verify.command`, a resolution that the model made lands if the
+command passes on the merged tree (see [RESOLVE](RESOLVE.md)). That makes the
+command a code execution path: it runs code a model wrote, derived from
+repository content that agents control. A hostile agent can plant text for the
+model to read (a prompt injection), and the model's answer is then executed by
+your tests, as the gitaly user on the server. The tree the command sees is the
+whole tree (no `export-ignore` hides anything from it), so a planted file is
+there for it as well. Run the command inside an isolation boundary, always when
+the AI lands: a container with `--network=none`, the checkout read-only, no
+credentials, resource limits (the example in RESOLVE.md is the default to
+start from), and a separate unprivileged user. The post-merge test command of
+`graft-verify` ([VERIFY](VERIFY.md)) runs the same kind of code on the tip and
+has the same advice. Without a `verify.command` nothing of a model's lands.
+
- Automatic conflict resolution is off by default and can be limited to
specific actors and paths (see [RESOLVE](RESOLVE.md)).
- The verifier writes the green ref and reverts with the operator credential,
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.md
index 2328196..c74ac64 100644
--- a/doc/graft/CONFIGURATION.md
+++ b/doc/graft/CONFIGURATION.md
@@ -193,11 +193,12 @@ cause.
| `command` | array of strings | empty | Program and arguments. It reads one request on stdin and writes the answer on stdout. |
| `url` | string | empty | `http://` or `https://` endpoint that receives the request as a JSON POST. Exactly one of `command` and `url` must be set when `enabled` is true. |
| `token_file` | string | empty | For `url`: file with the bearer token, read at startup. |
-| `timeout` | string | `"120s"` | Bound on one resolver call (Go duration), and on the whole land that the server resolves for `resolve_conflicts`: its rounds (the resolutions, the verifications of a model's resolution, the waits for slots, and the passes through the branch's lane) end at this deadline, which is `timeout` plus `verify.timeout` when `verify.enabled`. A round that the lane hands back after it fails the land with the merge conflict, "files kept changing", naming the files. |
+| `timeout` | string | `"120s"` | Bound on one resolver call (Go duration), and on the whole land that the server resolves for `resolve_conflicts`: its rounds (the resolutions, the verifications of a model's resolution, the waits for slots, and the passes through the branch's lane) end at this deadline, which is `timeout` plus `verify.timeout` when `verify.enabled`. A land that is still unresolved at this deadline does not fail with a deadline: it ends with the merge conflict and its package, the proposals of what is known and the files that changed under it as not attempted, with the reason. |
| `max_files` | int | `20` | Conflicted files per land. A land with more is not sent to the resolver. |
| `max_file_bytes` | int | `1048576` (1 MiB) | Bound on every side of every file sent and on every resolved file accepted. At most 12566528 (about 12 MiB): one file, its three sides and its resolution, must fit one check request. |
| `proposal_ttl` | string | `"168h"` | How long the proposals of a failed land are kept under `refs/graft/proposals/` (Go duration, by the server's clock); at most 20 are kept per branch. Lands with a resolution and the housekeeping pass prune them. |
| `check_concurrency` | int | `2` | Checks of agents' resolutions that run at once in the Gitaly process, across all its listeners (one resolver client is shared by them); each holds a request of up to 64 MiB. A check waits for a free slot within its deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_check_slot_wait_seconds`. |
+| `verify_per_repository` | int | `1` | Verifications of a model's resolution that run at once for one repository, inside `verify_concurrency`: every land of a busy branch builds a tree of its own to verify, and one repository must not take the whole pool. A land waits for its repository's turn first, holding nothing of the pool, within the land's deadline. |
| `verify_concurrency` | int | `2` | Verifications of a model's resolution that run at once in the Gitaly process, out of the lane, in a pool of their own, as `verify.command` takes minutes. A verification waits for a free slot within the land's deadline. The wait is in `graft_resolver_verify_slot_wait_seconds`. |
| `resolve_concurrency` | int | `2` | Resolutions that run at once in the Gitaly process, in a pool apart from the checks', so that long resolutions never delay a check. A resolution waits for a free slot within the land's deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_resolve_slot_wait_seconds`. |
| `resolve_rounds` | int | `32` | Safety cap on the rounds of a land whose conflicting files keep changing while it is resolved out of the lane. The rounds are bounded by `timeout`; the cap is set well above realistic contention and is no contract. |
diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.md
index 04f48f5..dccc8bc 100644
--- a/doc/graft/PLAN10.md
+++ b/doc/graft/PLAN10.md
@@ -1253,15 +1253,32 @@ and the tree it verifies depends on the tip.
* **Tip moved: out again, no resolver call.** When the tree is not a verified one, the lane does not
land. The land goes back out, rebuilds the tree from the same verdicts (the resolver is called only
for files whose sides changed, as in 15.8), verifies the new tree, and enters the lane again. Each
- round costs one verification; the progress argument is the one of 15.8 (a round is lost only when
- another land landed on the branch), and the rounds are bounded by the deadline of the land.
+ round costs one verification.
+* **Liveness, stated plainly: the progress argument of 15.8 does not carry over.** In 15.8 a round is
+ lost only when another land changed the file, so the lands that contend for a file all land. Here a
+ round is lost when ANY land moved the branch, because any land changes the tree that was verified,
+ whatever file it touched. On a busy branch a verified AI land therefore mostly ends as proposals
+ within one verification (the deadline holds a verification's time and little more), and that is the
+ designed, safe outcome: the agent gets the proposal and decides. A verified AI land is opportunistic.
+ Two bounds keep it from costing more than that: a per-repository cap on the verifications at once
+ (verify_per_repository, default 1, inside verify_concurrency), so that one busy repository cannot take
+ the whole pool, and the deadline of the land. A land that waits for its repository's turn holds
+ nothing of the pool, and a wait that outlasts the deadline ends in proposals like the rest.
* **Deadline: a proposal, not a failure.** The deadline of the land is the resolver's timeout plus
verify.timeout when a verify.command is configured. A verified AI land is opportunistic: when the
deadline ends before a tree could be verified and landed, the land does not fail for the agent. The
model's files become PROPOSED with the reason "could not verify before the branch moved again" (or
"before the deadline of <t>" when the verification itself was cut), the proposals are kept (14), and
- the package goes to the agent. This final pass verifies nothing and calls no resolver. A caller that
- gives up is not a deadline: it gets its cancellation.
+ the package goes to the agent. This final pass verifies nothing and calls no resolver. It is the end
+ of every land that had a conflict and could not go on, not only of a model's: the deadline wherever it
+ fell (the listing, the tree, the resolution, the verification), the cap on the rounds, and a land that
+ the lane or its batch kept handing back (a model's resolution whose batch ran out of rebuilds is
+ handed back as a land that needs its new tree verified, not failed with "kept moving"). The files
+ that have a verdict are reported with it (a model's PROPOSED, a rule's RESOLVED, their proposals
+ kept); the files that changed under the land and have none are NOT_ATTEMPTED, with the reason. With
+ nothing known yet it is the merge conflict with its package and no proposals, with the reason. A
+ land that had a conflict never returns a bare deadline to the agent: only a caller that gives up
+ gets its cancellation.
* **A verification that fails** (non-zero exit, or its own timeout) is a verdict, not a deadline: the
files stay PROPOSED and carry "verification failed: exit status N: <the first 4 KiB of the
output>" (the package cuts a reason to 512 bytes). Nothing is verified for a land that cannot land
diff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.md
index 9d54b57..75b5a0f 100644
--- a/doc/graft/RESOLVE.md
+++ b/doc/graft/RESOLVE.md
@@ -47,8 +47,10 @@ actors = ["builder", "planner"]
on the rounds of a land whose files keep changing (see "Out of the lane"
below); the rounds are bounded by `timeout`, and the cap is no contract.
- `verify_concurrency` (default 2): the verifications of a model's resolution
- that run at once in the Gitaly process, in a pool of their own (see "Optional
- verification of a resolution").
+ that run at once in the Gitaly process, in a pool of their own, and
+ `verify_per_repository` (default 1) the ones of one repository, so that a busy
+ repository cannot take the whole pool (see "Optional verification of a
+ resolution").
- A configuration the server could not use is a startup error (see
[CONFIGURATION](CONFIGURATION.md)). A configured resolver is never treated
as no resolver: if it cannot be built later, every land that needs it fails
@@ -455,6 +457,18 @@ exec docker run --rm --network=none --read-only --tmpfs /tmp \
-v "$PWD":/src:ro -w /src golang:1.26 go build ./...
```
+**Since a model's resolution can land through it, the command now runs code
+that a model wrote, from conflicts in repository content that agents control.**
+A hostile agent can put text in a file (a comment, a string, a README) that is
+written for the model to read, a prompt injection, and what the model returns
+is then run by your tests. Treat the command as running untrusted code: for any
+setup where the AI lands (a server AI and a `verify.command`), use the
+container wrapper above as the default, not as an example: `--network=none`,
+the checkout mounted read-only, no credentials in its environment or on its
+file system, resource limits, and a separate unprivileged user. A command that
+needs the network or a secret to pass is a command a model can steer into
+leaking it.
+
It also costs time on every resolution, which is why it is off by default;
the trailers and `graft-verify` remain the safety net.
@@ -1006,7 +1020,7 @@ in blank lines only, and the accepted results that got no syntax verdict.
| check answer | 64 KiB + 8 KiB per file | server |
| checks of agents' resolutions at once | 2 | `check_concurrency` |
| resolutions at once | 2 | `resolve_concurrency` |
-| verifications of a model's resolution at once | 2 | `verify_concurrency` |
+| verifications of a model's resolution at once; of one repository | 2; 1 | `verify_concurrency`; `verify_per_repository` |
| rounds of a land whose files keep changing | 32 (a safety cap; `timeout` bounds the land) | `resolve_rounds` |
| verification run | 5 min, output kept 4 KiB | `verify.timeout` |
| request body read by `graft-resolve` | 128 MiB (`protocol.DefaultMaxRequestBytes`) | `--max-request-bytes` |
diff --git a/doc/graft/VERIFY.md b/doc/graft/VERIFY.md
index 0175359..6fc595f 100644
--- a/doc/graft/VERIFY.md
+++ b/doc/graft/VERIFY.md
@@ -7,6 +7,18 @@ built on the public SDK (`sdk/`). It does not run inside the server and has no
special access to it: it is an ordinary gRPC client that uses the operator's
shared token (see [Permissions](#permissions)).
+## Security note: the test command runs code from the tree
+
+The test command runs code that is in the tip, written by agents, and, where
+the server's AI lands a resolution because the land's own `verify.command`
+passed (see [RESOLVE](RESOLVE.md)), written by a model from repository content
+that agents control. Treat both commands as untrusted code: run them in a
+container with `--network=none`, the work directory read-only where the tests
+allow it, no credentials in the environment or on the file system, and resource
+limits. A command that needs the network or a secret to pass can be steered
+into leaking it (a prompt injection planted in the repository). See
+[AGENT-SECURITY](AGENT-SECURITY.md).
+
## Loop
1. Read the changelog of the branch from the saved cursor with
diff --git a/internal/gitaly/config/config_graft.go b/internal/gitaly/config/config_graft.go
index fce28dc..7922ebe 100644
--- a/internal/gitaly/config/config_graft.go
+++ b/internal/gitaly/config/config_graft.go
@@ -332,6 +332,10 @@ type GraftResolver struct {
// lands only if it passes. It is a pool of its own, as the command takes minutes. Zero means
// DefaultGraftResolverVerifyConcurrency.
VerifyConcurrency int `json:"verify_concurrency" toml:"verify_concurrency,omitempty"`
+ // VerifyPerRepository bounds the verifications that run at once for one repository, inside
+ // VerifyConcurrency, so that one repository cannot take the whole pool: every land of its busy
+ // branches builds a tree of its own to verify. Zero means DefaultGraftResolverVerifyPerRepository.
+ VerifyPerRepository int `json:"verify_per_repository" toml:"verify_per_repository,omitempty"`
// ResolveRounds is a safety cap on the rounds of a land whose conflicting files keep changing
// while it is resolved (PLAN10 15.8). The rounds are bounded by the resolver's timeout; the cap
// is set well above realistic contention and is no contract. Zero means
@@ -523,8 +527,8 @@ func (r GraftResolver) validate() error {
if _, err := r.ProposalTTLOrDefault(); err != nil {
return err
}
- if r.MaxFiles < 0 || r.MaxFileBytes < 0 || r.CheckConcurrency < 0 || r.ResolveConcurrency < 0 || r.VerifyConcurrency < 0 || r.ResolveRounds < 0 {
- return fmt.Errorf("graft land resolver: max_files, max_file_bytes, check_concurrency, resolve_concurrency, verify_concurrency and resolve_rounds must not be negative")
+ if r.MaxFiles < 0 || r.MaxFileBytes < 0 || r.CheckConcurrency < 0 || r.ResolveConcurrency < 0 || r.VerifyConcurrency < 0 || r.VerifyPerRepository < 0 || r.ResolveRounds < 0 {
+ return fmt.Errorf("graft land resolver: max_files, max_file_bytes, check_concurrency, resolve_concurrency, verify_concurrency, verify_per_repository and resolve_rounds must not be negative")
}
// The check of an agent's resolution sends a file's three sides and its resolution in one
// request, which must fit its bound.
@@ -592,6 +596,17 @@ func (r GraftResolver) CheckConcurrencyOrDefault() int {
return DefaultGraftResolverCheckConcurrency
}
+// DefaultGraftResolverVerifyPerRepository is the default of GraftResolver.VerifyPerRepository.
+const DefaultGraftResolverVerifyPerRepository = 1
+
+// VerifyPerRepositoryOrDefault returns VerifyPerRepository or its default.
+func (r GraftResolver) VerifyPerRepositoryOrDefault() int {
+ if r.VerifyPerRepository > 0 {
+ return r.VerifyPerRepository
+ }
+ return DefaultGraftResolverVerifyPerRepository
+}
+
// DefaultGraftResolverVerifyConcurrency is the default of GraftResolver.VerifyConcurrency.
const DefaultGraftResolverVerifyConcurrency = 2
diff --git a/internal/gitaly/config/config_graft_resolver_test.go b/internal/gitaly/config/config_graft_resolver_test.go
index f9fc16a..7d99d87 100644
--- a/internal/gitaly/config/config_graft_resolver_test.go
+++ b/internal/gitaly/config/config_graft_resolver_test.go
@@ -116,6 +116,9 @@ func TestGraftResolverValidateBuildable(t *testing.T) {
"negative verify_concurrency": {
GraftResolver{Command: []string{"graft-resolve"}, VerifyConcurrency: -1}, "must not be negative",
},
+ "negative verify_per_repository": {
+ GraftResolver{Command: []string{"graft-resolve"}, VerifyPerRepository: -1}, "must not be negative",
+ },
"negative resolve_rounds": {
GraftResolver{Command: []string{"graft-resolve"}, ResolveRounds: -1}, "must not be negative",
},
@@ -140,6 +143,8 @@ func TestGraftResolverValidateBuildable(t *testing.T) {
require.Equal(t, 4, GraftResolver{ResolveConcurrency: 4}.ResolveConcurrencyOrDefault())
require.Equal(t, DefaultGraftResolverVerifyConcurrency, GraftResolver{}.VerifyConcurrencyOrDefault())
require.Equal(t, 3, GraftResolver{VerifyConcurrency: 3}.VerifyConcurrencyOrDefault())
+ require.Equal(t, DefaultGraftResolverVerifyPerRepository, GraftResolver{}.VerifyPerRepositoryOrDefault())
+ require.Equal(t, 3, GraftResolver{VerifyPerRepository: 3}.VerifyPerRepositoryOrDefault())
require.Equal(t, DefaultGraftResolverResolveRounds, GraftResolver{}.ResolveRoundsOrDefault())
require.Equal(t, 7, GraftResolver{ResolveRounds: 7}.ResolveRoundsOrDefault())
}
diff --git a/internal/gitaly/service/operations/land.go b/internal/gitaly/service/operations/land.go
index 8a24dcd..eba7917 100644
--- a/internal/gitaly/service/operations/land.go
+++ b/internal/gitaly/service/operations/land.go
@@ -154,7 +154,7 @@ func (s *Server) landWithRetries(ctx context.Context, in *gitalypb.UserLandReque
}
if attempt >= maxAttempts || snapshotPinned {
- return nil, landKeptMovingError(in, conflict, attempt)
+ return nil, landKeptMoving(in, pre, conflict, attempt)
}
if err := waitLandBackoff(ctx, attempt); err != nil {
@@ -402,6 +402,18 @@ func landMergeConflictError(tip, commitID git.ObjectID, conflictingFiles [][]byt
})
}
+// landKeptMoving is the error of a landing that gave up because the branch kept moving. A model's
+// resolution is handed back instead, as a land that needs its new tree verified (it is the same
+// cause: the branch moved on): UserLand goes out again, and at its deadline the agent gets the
+// proposals, not an error about a branch it has nothing to do with. The others get
+// landKeptMovingError.
+func landKeptMoving(in *gitalypb.UserLandRequest, pre *landPreResolution, conflict landConflict, attempts int) error {
+ if pre.hasModel() {
+ return &landNeedsResolution{tip: conflict.newOID, commit: git.ObjectID(in.GetCommitId()), moved: true}
+ }
+ return landKeptMovingError(in, conflict, attempts)
+}
+
// landKeptMovingError returns the FailedPrecondition error for a landing that lost the
// compare-and-swap on every attempt, or that cannot be retried inside the RPC.
func landKeptMovingError(in *gitalypb.UserLandRequest, conflict landConflict, attempts int) error {
diff --git a/internal/gitaly/service/operations/land_batch_server.go b/internal/gitaly/service/operations/land_batch_server.go
index df4bb01..2d9a32b 100644
--- a/internal/gitaly/service/operations/land_batch_server.go
+++ b/internal/gitaly/service/operations/land_batch_server.go
@@ -223,7 +223,7 @@ func landBatchGiveUp(req *landBatchRequest, cause error, attempts int) error {
return structerr.NewInternal("updating ref with hooks: %w", cause)
}
- return landKeptMovingError(req.in, conflict, attempts)
+ return landKeptMoving(req.in, req.preResolution, conflict, attempts)
}
// serverLandBatchStepper implements landBatchStepper on one batch attempt: the landing attempt that
diff --git a/internal/gitaly/service/operations/land_package_test.go b/internal/gitaly/service/operations/land_package_test.go
index 57aa993..d1b3fcf 100644
--- a/internal/gitaly/service/operations/land_package_test.go
+++ b/internal/gitaly/service/operations/land_package_test.go
@@ -413,20 +413,48 @@ func TestUserLand_reland(t *testing.T) {
t.Run("the whole check has one deadline", func(t *testing.T) {
t.Parallel()
- // Each check request takes 0.7 s and the deadline of the whole check is 1 s: the second
- // request runs out of time, and the check fails closed.
+ // No clock decides this test. The check is made of two batches. Every batch must run under the
+ // same deadline, which is set once when the check starts and is the resolver's timeout from
+ // then (so the second request has what the first left, not a timeout of its own), and a check
+ // that ends in its second batch fails closed with the verdicts of the first. The test ends the
+ // check from the hook of the second batch, an event; the deadline itself is read from the
+ // context of each batch.
_, argv := perCallChecker(t, checkAnswer("graft-resolve", "a", true, ""), checkAnswer("graft-resolve", "b", true, ""))
- slow := append([]string{"sh", "-c", `sleep 0.7; exec "$0"`}, argv...)
- fx, s, req := batchedReland(t, slow)
- s.resolveCfg.Timeout = "1s"
+ fx, s, req := batchedReland(t, argv)
+ s.resolveCfg.Timeout = "1h"
client, err := resolver.New(s.resolveCfg)
require.NoError(t, err)
s.conflictResolver = client
- start := time.Now()
- pre, err := s.landPrecheckResolution(fx.ctx, req, fx.repo)
+ callerCtx, cancel := context.WithCancel(fx.ctx)
+ defer cancel()
+ var mu sync.Mutex
+ var deadlines []time.Time
+ var withoutDeadline int
+ // The hook only records: a failed assertion inside it would leave the check stuck with its slot
+ // instead of failing the test.
+ s.landCheckBatchHook = func(ctx context.Context) {
+ deadline, ok := ctx.Deadline()
+ mu.Lock()
+ defer mu.Unlock()
+ if !ok {
+ withoutDeadline++
+ }
+ deadlines = append(deadlines, deadline)
+ if len(deadlines) == 2 {
+ cancel()
+ }
+ }
+
+ before := time.Now()
+ pre, err := s.landPrecheckResolution(callerCtx, req, fx.repo)
+ after := time.Now()
require.NoError(t, err)
- require.Less(t, time.Since(start), 2*time.Second, "the check stops at its deadline")
+ require.Zero(t, withoutDeadline, "a batch runs under the deadline of the check")
+ require.Len(t, deadlines, 2, "the check ran two batches")
+ require.Equal(t, deadlines[0], deadlines[1], "one deadline for the whole check")
+ require.False(t, deadlines[0].Before(before.Add(time.Hour)), "set when the check started, not before")
+ require.False(t, after.Add(time.Hour).Before(deadlines[0]), "set when the check started, not later")
_, err = s.landWithResolution(fx.ctx, req, fx.repo, fx.tip, fx.commit, fx.committer(), pre)
pkg := builtUnresolved(t, fx, err)
require.Contains(t, pkg.GetReason(), "the resolution could not be checked")
diff --git a/internal/gitaly/service/operations/land_preresolve.go b/internal/gitaly/service/operations/land_preresolve.go
index 18085ea..0c93326 100644
--- a/internal/gitaly/service/operations/land_preresolve.go
+++ b/internal/gitaly/service/operations/land_preresolve.go
@@ -5,11 +5,14 @@ import (
"errors"
"fmt"
"strings"
+ "time"
"github.com/feder-cr/git-ai/internal/git"
"github.com/feder-cr/git-ai/internal/graft/resolver"
"github.com/feder-cr/git-ai/internal/structerr"
"github.com/feder-cr/git-ai/proto/go/gitalypb"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
)
// A land with resolve_conflicts is resolved out of the branch's lane (PLAN10 15.8): the resolver
@@ -66,6 +69,20 @@ func newLandPreResolution() *landPreResolution {
return &landPreResolution{verdicts: landVerdicts{}, verified: map[git.ObjectID]bool{}}
}
+// hasModel reports whether a verdict of the land is a model's: the land is a model's resolution
+// that only a verified tree lands.
+func (p *landPreResolution) hasModel() bool {
+ if p == nil {
+ return false
+ }
+ for _, v := range p.verdicts {
+ if v.file.FromModel() {
+ return true
+ }
+ }
+ return false
+}
+
// markVerified records that verify.command passed on tree.
func (p *landPreResolution) markVerified(tree git.ObjectID) {
if p.verified == nil {
@@ -87,9 +104,16 @@ type landNeedsResolution struct {
tip, commit git.ObjectID
paths []string
unverified git.ObjectID
+ // moved says that the branch moved under the batch that was landing a model's resolution until
+ // the batch ran out of attempts: the tree it was about to land is another one, and the land goes
+ // out again as it does for an unverified tree.
+ moved bool
}
func (e *landNeedsResolution) Error() string {
+ if e.moved {
+ return fmt.Sprintf("the branch moved while the resolution of the land at tip %s was being landed", e.tip)
+ }
if e.unverified != "" {
return fmt.Sprintf("the tree %s of the land at tip %s was not verified", e.unverified, e.tip)
}
@@ -182,9 +206,12 @@ func (s *Server) landPreResolve(ctx context.Context, in *gitalypb.UserLandReques
// The rounds are bounded by the deadline of the land, the resolver's timeout plus the verification's
// when one is configured, and by resolve_rounds, a safety cap. A round is lost only when another
// land changed the files or the tip, so the lands that contend for a file all land, one per round.
-// A land that runs out of its deadline fails with the merge conflict naming the files that kept
-// changing; but a model's resolution that could not be verified in time does not fail the agent: its
-// files become proposals, with the reason, in the package (landUnverified).
+//
+// A land that had a conflict never ends on a bare deadline: whatever stopped it, the deadline, the
+// cap on the rounds or a lane that kept handing it back, it ends with the merge conflict and its
+// package (landFinalPass): the proposals for what is known, a model's files PROPOSED with the
+// reason, and NOT_ATTEMPTED with the reason for the files that changed under it. Only a caller that
+// gives up gets its cancellation.
func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequest, committer git.Signature) (*gitalypb.UserLandResponse, error) {
timeout, err := s.resolveCfg.TimeoutOrDefault()
if err != nil {
@@ -207,25 +234,18 @@ func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequest
return ctx.Err() == nil && errors.Is(resolveCtx.Err(), context.DeadlineExceeded)
}
until := fmt.Sprintf("until the deadline of %s", timeout)
- // giveUp ends a land that cannot go on: a model's resolution that was not verified in time goes
- // to the agent as proposals; anything else is the merge conflict of files that kept changing.
- giveUp := func(when, unverifiedReason string) (*gitalypb.UserLandResponse, error) {
- if unverifiedReason != "" {
- return s.landUnverified(ctx, in, committer, pre, unverifiedReason, need, when)
- }
- return nil, need.keptChanging(when)
- }
rounds := s.resolveCfg.ResolveRoundsOrDefault()
for round := 1; round <= rounds; round++ {
if err := s.landPreResolve(resolveCtx, in, committer, pre); err != nil {
- var unfinished *landVerifyUnfinishedError
- switch {
- case errors.As(err, &unfinished) && deadline():
- return s.landUnverified(ctx, in, committer, pre, fmt.Sprintf("could not verify before the deadline of %s", timeout), need, until)
- case need != nil && deadline():
- // A land that lost a round to its files or its tip changing, and then ran out of its
- // deadline while it resolved or verified again, failed because they kept changing.
- return giveUp(until, landNotVerifiedReason(need))
+ // The deadline ended a step of the round: the listing, the tree, the resolution or the
+ // verification. A refusal of the policy is not a deadline.
+ if deadline() && status.Code(err) != codes.PermissionDenied {
+ var unfinished *landVerifyUnfinishedError
+ reason := landStoppedReason(need, timeout)
+ if errors.As(err, &unfinished) {
+ reason = fmt.Sprintf("could not verify before the deadline of %s", timeout)
+ }
+ return s.landFinalPass(ctx, in, committer, pre, reason, need, until)
}
return nil, err
}
@@ -234,29 +254,34 @@ func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequest
return resp, err
}
if deadline() {
- return giveUp(until, landNotVerifiedReason(need))
+ return s.landFinalPass(ctx, in, committer, pre, landStoppedReason(need, timeout), need, until)
}
}
- return giveUp(fmt.Sprintf("after %d rounds", rounds), landNotVerifiedReason(need))
+ return s.landFinalPass(ctx, in, committer, pre, landStoppedReason(need, timeout), need, fmt.Sprintf("after %d rounds", rounds))
}
-// landNotVerifiedReason is why a model's files become proposals when the lane handed the land back
-// for a tree that was not verified: the branch moved on after the verification. It is empty when the
-// lane asked for files to be resolved.
-func landNotVerifiedReason(need *landNeedsResolution) string {
- if need != nil && need.unverified != "" {
+// landStoppedReason is why the land could not go on, as the files it leaves say: the tree it was
+// about to land was not verified before the branch moved again, the files changed again before they
+// could be resolved, or the time ran out before the conflicts were resolved at all.
+func landStoppedReason(need *landNeedsResolution, timeout time.Duration) string {
+ switch {
+ case need != nil && (need.unverified != "" || need.moved):
return "could not verify before the branch moved again"
+ case need != nil:
+ return "the files changed again before they could be resolved"
}
- return ""
+ return fmt.Sprintf("the time ran out before the conflicts were resolved (deadline of %s)", timeout)
}
-// landUnverified ends a land whose model resolution could not be verified in time. A verified AI
-// land is opportunistic, and losing the race is harmless: the agent gets the proposals. It
-// pre-resolves once more, with the request's context, in a new quarantine, verifying nothing and
-// calling no resolver (the verdicts are kept): the files of the model are PROPOSED with the reason,
-// the proposals are kept, and the land fails with the package. need, when set, is the state the
-// lane handed back, for the merge conflict if the land cannot be reported as proposals after all.
-func (s *Server) landUnverified(
+// landFinalPass ends a land that could not go on (landResolving): the agent gets the package, not
+// a deadline. A verified AI land is opportunistic, and losing the race is harmless. It pre-resolves
+// once more with the request's context, in a new quarantine, verifying nothing and calling no
+// resolver: the verdicts that are kept give the proposals (a model's files PROPOSED with the reason,
+// the rules' files RESOLVED), the files without a verdict are NOT_ATTEMPTED with the reason, and the
+// land fails with the package. When there is nothing to report as a conflict after all (the change
+// landed, the merge is clean, or every file has a verdict a rule made), the lane answers; need and
+// when name the files for the merge conflict if even that hands the land back.
+func (s *Server) landFinalPass(
ctx context.Context,
in *gitalypb.UserLandRequest,
committer git.Signature,
@@ -277,8 +302,6 @@ func (s *Server) landUnverified(
case err != nil:
return nil, err
}
- // The land is no longer a conflict to resolve (the change landed, or the merge is clean): the
- // lane answers it.
resp, err := s.landOnce(ctx, in, committer, nil, pre)
if errors.As(err, &needs) {
return nil, needs.keptChanging(when)
diff --git a/internal/gitaly/service/operations/land_preresolve_test.go b/internal/gitaly/service/operations/land_preresolve_test.go
index 545e403..6ce66ed 100644
--- a/internal/gitaly/service/operations/land_preresolve_test.go
+++ b/internal/gitaly/service/operations/land_preresolve_test.go
@@ -381,7 +381,7 @@ func TestUserLand_resolveRounds(t *testing.T) {
require.Equal(t, "merged-b\n", fx.blobAt(t, landed, "b"))
})
- t.Run("files that keep changing fail after resolve_rounds, by name", func(t *testing.T) {
+ t.Run("files that keep changing give the package after resolve_rounds: what is known, and what is not", func(t *testing.T) {
t.Parallel()
dir, argv := resolverScript(t, answerByCall)
@@ -396,22 +396,34 @@ func TestUserLand_resolveRounds(t *testing.T) {
gittest.WithTreeEntries(entries("a", "ours-2\n", "b", "ours\n", "c", "c\n")...))
writeMoves(t, dir, fx, w1, w2)
+ // The hot file a changed under every round; b did not. The agent gets the package, not a bare
+ // "files kept changing": b with the rules' answer (RESOLVED, and its proposal kept), a as not
+ // attempted, with why, and the tip it was last seen at.
_, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
requireMergeConflict(t, err)
- require.ErrorContains(t, err, "files kept changing while the land was resolved, after 2 rounds: a")
conflict := landErrorDetail(t, err).GetMergeConflict()
- require.Equal(t, [][]byte{[]byte("a")}, conflict.GetConflictingFiles())
require.Equal(t, []string{w2.String(), fx.commit.String()}, conflict.GetConflictingCommitIds())
- require.Equal(t, 2, resolverCalls(t, dir))
+ pkg := landConflictDetail(err)
+ require.NotNil(t, pkg, "the package, not a bare conflict")
+ require.Equal(t, w2.String(), pkg.GetTip())
+ hot := packageFile(t, pkg, "a")
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_NOT_ATTEMPTED, hot.GetOutcome())
+ require.Contains(t, hot.GetReason(), "changed again before they could be resolved")
+ require.Nil(t, hot.GetProposal())
+ known := packageFile(t, pkg, "b")
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_RESOLVED, known.GetOutcome())
+ require.Equal(t, "merged-b\n", string(gittest.Exec(t, fx.cfg, "-C", fx.repoPath, "cat-file", "blob", known.GetProposal().GetOid())))
+ require.NotEmpty(t, pkg.GetProposalRef())
+ require.Equal(t, 2, resolverCalls(t, dir), "the final pass calls no resolver")
requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, w2)
})
- t.Run("files that keep changing fail at the deadline, by name", func(t *testing.T) {
+ t.Run("files that keep changing give the package at the deadline", func(t *testing.T) {
t.Parallel()
// A writer changes a and b while every round is resolved, more often than the land can
- // keep up with: the land fails at the resolver's deadline, however many rounds the cap
- // would allow, and says why.
+ // keep up with: the land ends at the resolver's deadline, however many rounds the cap
+ // would allow, with the package that says why: no file has an answer for the tip it ends at.
dir, argv := resolverScript(t, "sleep 0.2\n"+answerByCall)
writeAnswer(t, dir, validAnswer())
rcfg := enabledResolver(argv)
@@ -427,12 +439,18 @@ func TestUserLand_resolveRounds(t *testing.T) {
}
writeMoves(t, dir, fx, writers...)
+ // The bound only says that the land ends: how many rounds fit in 3 s is the speed of the machine.
start := time.Now()
_, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ require.Less(t, time.Since(start), 3*time.Minute)
requireMergeConflict(t, err)
- require.ErrorContains(t, err, "files kept changing while the land was resolved, until the deadline of 3s: a, b")
- require.Less(t, time.Since(start), 30*time.Second)
- require.GreaterOrEqual(t, resolverCalls(t, dir), 2)
+ pkg := landConflictDetail(err)
+ require.NotNil(t, pkg, "the package, not a bare deadline")
+ for _, p := range []string{"a", "b"} {
+ f := packageFile(t, pkg, p)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_NOT_ATTEMPTED, f.GetOutcome(), p)
+ require.NotEmpty(t, f.GetReason(), p)
+ }
require.Less(t, resolverCalls(t, dir), len(writers), "the deadline ended it, not the writers")
})
@@ -472,8 +490,11 @@ func TestUserLand_resolveRounds(t *testing.T) {
continue
}
requireMergeConflict(t, r.err)
- require.True(t, strings.Contains(r.err.Error(), "files kept changing") || strings.Contains(r.err.Error(), "deadline"),
- "land %d fails with its reason: %v", i, r.err)
+ pkg := landConflictDetail(r.err)
+ require.NotNil(t, pkg, "land %d ends with the package: %v", i, r.err)
+ for _, f := range pkg.GetFiles() {
+ require.NotEmpty(t, f.GetReason(), "land %d, %s", i, f.GetPath())
+ }
timedOut++
case <-time.After(60 * time.Second):
require.FailNow(t, "a land hangs", "land %d", i)
diff --git a/internal/gitaly/service/operations/land_proposals.go b/internal/gitaly/service/operations/land_proposals.go
index 0c5fe67..e2ce818 100644
--- a/internal/gitaly/service/operations/land_proposals.go
+++ b/internal/gitaly/service/operations/land_proposals.go
@@ -46,6 +46,10 @@ var landProposalWriteErrors = promauto.NewCounter(prometheus.CounterOpts{
// not follows it (landNoVerifyReason, the output of the verification, or why it could not run).
const landProposedReason = "a model resolved the file: it is proposed (see proposal), and lands by itself only after verify.command passed on the merged tree; land it, edited or not, as your own resolution"
+// landVerifyDisabledReason follows landProposedReason when a verify.command is configured and
+// verify.enabled is false.
+const landVerifyDisabledReason = "verification is disabled (verify.enabled is false), so nothing verifies a model's resolution"
+
// landNoVerifyReason follows landProposedReason when no verify.command is configured.
const landNoVerifyReason = "no verify.command is configured, so nothing verifies a model's resolution"
diff --git a/internal/gitaly/service/operations/land_reland.go b/internal/gitaly/service/operations/land_reland.go
index b739bf7..3d8e40f 100644
--- a/internal/gitaly/service/operations/land_reland.go
+++ b/internal/gitaly/service/operations/land_reland.go
@@ -471,7 +471,7 @@ func (s *Server) landCheckSlotted(
}
}()
if s.landCheckBatchHook != nil {
- s.landCheckBatchHook()
+ s.landCheckBatchHook(ctx)
}
checked, resp, err := s.landCheckBatch(ctx, in, repo, l, actor, resolved, batch, cache, slot, unchecked)
return checked, resp, "checking", err
diff --git a/internal/gitaly/service/operations/land_resolve.go b/internal/gitaly/service/operations/land_resolve.go
index daca754..e805b4a 100644
--- a/internal/gitaly/service/operations/land_resolve.go
+++ b/internal/gitaly/service/operations/land_resolve.go
@@ -148,22 +148,21 @@ func (s *Server) landResolveAt(
if !f.sent() {
continue
}
- mine = append(mine, f)
if v, ok := pre.verdicts[landKeyOf(f.conflict)]; ok {
+ mine = append(mine, f)
reused[f.conflict.path] = v.file
reusedNames = append(reusedNames, v.resolver)
f.outcome = gitalypb.LandConflictFile_OUTCOME_RESOLVED
continue
}
- sent = append(sent, landRequestFile(f))
- }
- // Past its deadline a land only reports what it has: the resolver is not called.
- if pre.unverifiable != "" && len(sent) > 0 {
- paths := make([]string, 0, len(sent))
- for _, f := range sent {
- paths = append(paths, f.Path)
+ if pre.unverifiable != "" {
+ // The land is being ended (landFinalPass): the resolver is not called, and a file with no
+ // verdict at this tip is reported as not attempted, with why.
+ f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_NOT_ATTEMPTED, pre.unverifiable
+ continue
}
- return l, nil, &landNeedsResolution{tip: tip, commit: commitID, paths: paths}
+ mine = append(mine, f)
+ sent = append(sent, landRequestFile(f))
}
// answer is the resolver's accepted answer, the only thing ever written. A refusal is never
@@ -226,6 +225,8 @@ func (s *Server) landResolveAt(
switch {
case pre.unverifiable != "":
landAnnotateProposed(files, pre.unverifiable)
+ case !s.resolveCfg.Verify.Enabled && len(s.resolveCfg.Verify.Command) > 0:
+ landAnnotateProposed(files, landVerifyDisabledReason)
case !s.resolveCfg.Verify.Enabled:
landAnnotateProposed(files, landNoVerifyReason)
case landAllAnswered(files):
@@ -234,7 +235,7 @@ func (s *Server) landResolveAt(
return l, nil, err
}
var verifyErr *landVerifyError
- switch err := s.landVerifyOutOfLane(ctx, repo, tree); {
+ switch err := s.landVerifyOutOfLane(ctx, in.GetRepository(), repo, tree); {
case err == nil:
verified = tree
pre.markVerified(tree)
diff --git a/internal/gitaly/service/operations/land_resolver_shared_test.go b/internal/gitaly/service/operations/land_resolver_shared_test.go
index 03c6a26..ffdbd4b 100755
--- a/internal/gitaly/service/operations/land_resolver_shared_test.go
+++ b/internal/gitaly/service/operations/land_resolver_shared_test.go
@@ -79,7 +79,7 @@ func TestUserLand_relandRobustness(t *testing.T) {
client, err := resolver.New(s.resolveCfg)
require.NoError(t, err)
s.conflictResolver = client
- s.landCheckBatchHook = func() { panic("a batch failed") }
+ s.landCheckBatchHook = func(context.Context) { panic("a batch failed") }
require.PanicsWithValue(t, "a batch failed", func() { _, _ = s.landPrecheckResolution(fx.ctx, req, fx.repo) })
diff --git a/internal/gitaly/service/operations/land_review_test.go b/internal/gitaly/service/operations/land_review_test.go
index 4f9e576..5ba0980 100644
--- a/internal/gitaly/service/operations/land_review_test.go
+++ b/internal/gitaly/service/operations/land_review_test.go
@@ -2,8 +2,10 @@ package operations
import (
"context"
+ "errors"
"os"
"path/filepath"
+ "strings"
"testing"
"time"
@@ -11,7 +13,10 @@ import (
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
+ "github.com/feder-cr/git-ai/internal/git"
"github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/graft/resolver"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
)
// Tests of the review of m6 (9586fd2).
@@ -140,3 +145,45 @@ func TestUserLand_aConflictWithoutAConflictingFileAtTheLanesTip(t *testing.T) {
require.Equal(t, 1, resolverCalls(t, dir), "no round was repeated")
requireBranchAt(t, fx.cfg, fx.repoPath, "main", split)
}
+
+// TestLandKeptMoving: a land that gave up because the branch kept moving is handed back to be verified
+// again when it is a model's resolution (the same cause as a tip that moved), and fails as before when
+// it is not.
+func TestLandKeptMoving(t *testing.T) {
+ t.Parallel()
+
+ in := &gitalypb.UserLandRequest{Branch: []byte("main"), CommitId: strings.Repeat("c", 40)}
+ conflict := landConflict{reference: "refs/heads/main", oldOID: git.ObjectID(strings.Repeat("1", 40)), newOID: git.ObjectID(strings.Repeat("2", 40))}
+ model := &landPreResolution{verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginModel}}}}
+ rules := &landPreResolution{verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginRule}}}}
+ unmarked := &landPreResolution{verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a"}}}}
+
+ for name, pre := range map[string]*landPreResolution{"a model's verdict": model, "a verdict that says nothing": unmarked} {
+ var need *landNeedsResolution
+ require.ErrorAs(t, landKeptMoving(in, pre, conflict, 20), &need, name)
+ require.True(t, need.moved, name)
+ require.Equal(t, strings.Repeat("c", 40), need.commit.String(), name)
+ require.Equal(t, "could not verify before the branch moved again", landStoppedReason(need, time.Minute), name)
+ }
+ for name, pre := range map[string]*landPreResolution{"rules only": rules, "no pre-resolution": nil, "no verdict": {}} {
+ err := landKeptMoving(in, pre, conflict, 20)
+ var need *landNeedsResolution
+ require.False(t, errors.As(err, &need), name)
+ require.Equal(t, codes.FailedPrecondition, status.Code(err), name)
+ require.ErrorContains(t, err, "kept moving", name)
+ }
+
+ // A member of a batch that ran out of rebuilds is the same: the cause is a conflict of the update.
+ req := &landBatchRequest{in: in, preResolution: model}
+ var need *landNeedsResolution
+ require.ErrorAs(t, landBatchGiveUp(req, conflict, 4), &need)
+ require.ErrorContains(t, landBatchGiveUp(&landBatchRequest{in: in, preResolution: rules}, conflict, 4), "kept moving")
+}
+
+func TestLandStoppedReason(t *testing.T) {
+ t.Parallel()
+
+ require.Equal(t, "could not verify before the branch moved again", landStoppedReason(&landNeedsResolution{unverified: "tree"}, time.Minute))
+ require.Equal(t, "the files changed again before they could be resolved", landStoppedReason(&landNeedsResolution{paths: []string{"a"}}, time.Minute))
+ require.Equal(t, "the time ran out before the conflicts were resolved (deadline of 2m0s)", landStoppedReason(nil, 2*time.Minute))
+}
diff --git a/internal/gitaly/service/operations/land_verified_test.go b/internal/gitaly/service/operations/land_verified_test.go
index 8dbf184..0d61a60 100644
--- a/internal/gitaly/service/operations/land_verified_test.go
+++ b/internal/gitaly/service/operations/land_verified_test.go
@@ -7,7 +7,9 @@ import (
"io/fs"
"os"
"path/filepath"
+ "strconv"
"strings"
+ "syscall"
"testing"
"time"
@@ -195,7 +197,7 @@ func TestUserLand_verifiedModelResolution(t *testing.T) {
require.Equal(t, 1, verifyRuns(t, vdir), "once, out of the lane")
})
- t.Run("a verification that times out is a failure with its reason", func(t *testing.T) {
+ t.Run("a verification that times out leaves the files PROPOSED, with its reason", func(t *testing.T) {
t.Parallel()
rdir, rargv := resolverScript(t, "")
@@ -211,6 +213,27 @@ func TestUserLand_verifiedModelResolution(t *testing.T) {
require.Equal(t, 1, verifyRuns(t, vdir))
})
+ t.Run("with a verify.command that is disabled the reason says so", func(t *testing.T) {
+ t.Parallel()
+
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, modelAnswer())
+ vdir, vargv := countingVerify(t, "exit 0")
+ rcfg := withVerify(enabledResolver(rargv), vargv, "")
+ rcfg.Verify.Enabled = false
+ fx := newResolveFixture(t, rcfg)
+
+ _, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ requireMergeConflict(t, err)
+ for _, p := range []string{"a", "b"} {
+ f := packageFile(t, landConflictDetail(err), p)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome(), p)
+ require.Contains(t, f.GetReason(), "verification is disabled (verify.enabled is false)", p)
+ require.NotContains(t, f.GetReason(), "no verify.command is configured", p)
+ }
+ require.Equal(t, 0, verifyRuns(t, vdir))
+ })
+
t.Run("without a verify.command a model's resolution is a proposal", func(t *testing.T) {
t.Parallel()
@@ -431,8 +454,9 @@ fi`)
require.Equal(t, 1, resolverCalls(t, rdir))
}
-// TestUserLand_verifiedModelResolution_poolBoundsVerifications: with verify_concurrency 1, two lands
-// at once never run the operator's command at once.
+// TestUserLand_verifiedModelResolution_poolBoundsVerifications: two lands of one repository at once
+// never run the operator's command at once, though the pool has room for four: one repository takes one
+// slot (verify_per_repository). The bound of the pool itself is tested in the resolver package.
func TestUserLand_verifiedModelResolution_poolBoundsVerifications(t *testing.T) {
t.Parallel()
@@ -446,7 +470,7 @@ fi
sleep 0.3
rmdir "$DIR/lock"`)
rcfg := withVerify(enabledResolver(rargv), vargv, "")
- rcfg.VerifyConcurrency = 1
+ rcfg.VerifyConcurrency = 4
graft := batchingGraft()
graft.Land.Resolver = rcfg
fx := newResolveFixtureIn(t, graft,
@@ -534,12 +558,15 @@ func TestUserLand_verifiedRulesKeepTheirVerificationInTheLane(t *testing.T) {
require.Equal(t, 1, verifyRuns(t, failingDir))
}
-// TestUserLand_verifiedModelResolution_policyAndCancellation: a land whose caller gives up while it
-// verifies fails with the caller's cancellation, and not as proposals.
+// TestUserLand_verifiedModelResolution_callerGivesUp: a land whose caller gives up while it verifies
+// ends with the caller's cancellation, and what the SERVER does with it is as asserted here: the
+// operator's command is killed (it does not run on for a land nobody waits for), nothing is kept as a
+// proposal, and the branch is not touched. The Canceled on the client side proves none of it.
func TestUserLand_verifiedModelResolution_callerGivesUp(t *testing.T) {
t.Parallel()
- fx, _, vdir := verifiedFixture(t, modelAnswer(), "sleep 30", func(r *config.GraftResolver) { r.Verify.Timeout = "1m" })
+ fx, _, vdir := verifiedFixture(t, modelAnswer(), `echo $$ > "$DIR/pid"
+sleep 60`, func(r *config.GraftResolver) { r.Verify.Timeout = "1m" })
ctx, cancel := context.WithCancel(fx.ctx)
req := fx.request("Land")
done := make(chan error, 1)
@@ -547,14 +574,70 @@ func TestUserLand_verifiedModelResolution_callerGivesUp(t *testing.T) {
_, err := fx.client.UserLand(ctx, req)
done <- err
}()
- require.Eventually(t, func() bool { return verifyRuns(t, vdir) == 1 }, 30*time.Second, 10*time.Millisecond)
+ var pid int
+ require.Eventually(t, func() bool {
+ raw, err := os.ReadFile(filepath.Join(vdir, "pid"))
+ if err != nil {
+ return false
+ }
+ pid, err = strconv.Atoi(strings.TrimSpace(string(raw)))
+ return err == nil
+ }, 30*time.Second, 10*time.Millisecond, "the verification never started")
+ require.NoError(t, syscall.Kill(pid, 0), "the verification is running")
+
cancel()
select {
case err := <-done:
require.Error(t, err)
require.Equal(t, codes.Canceled, status.Code(err), "%v", err)
- case <-time.After(30 * time.Second):
+ case <-time.After(60 * time.Second):
require.FailNow(t, "the land did not end when its caller gave up")
}
+ // The server's side: the command is gone, nothing was kept or landed.
+ require.Eventually(t, func() bool { return errors.Is(syscall.Kill(pid, 0), syscall.ESRCH) },
+ 30*time.Second, 20*time.Millisecond, "the verification still runs for a land nobody waits for")
+ require.Empty(t, proposalRefTarget(t, fx), "a land that was cancelled keeps no proposal")
requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)
}
+
+// TestUserLand_verifiedModelResolution_aBusyRepositoryEndsInProposals: at most one verification runs
+// for a repository (verify_per_repository). A land whose turn does not come before its deadline is not
+// an error for the agent: it ends with the package, the model's files PROPOSED, and says why. The
+// operator's command did not run for it.
+func TestUserLand_verifiedModelResolution_aBusyRepositoryEndsInProposals(t *testing.T) {
+ t.Parallel()
+
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, modelAnswer())
+ vdir, vargv := countingVerify(t, "exit 0")
+ rcfg := withVerify(enabledResolver(rargv), vargv, "1s")
+ rcfg.Timeout = "1s"
+ fx := newResolveFixture(t, rcfg)
+ s := fx.server(t, rcfg)
+ s.locator = config.NewLocator(fx.cfg) // the land quarantines the objects it makes
+
+ // Another land of the repository is being verified: its slot is the repository's.
+ busy, err := s.conflictResolver.AcquireVerifySlot(fx.ctx, landVerifyRepoKey(fx.repoProto))
+ require.NoError(t, err)
+
+ _, err = s.landResolving(fx.ctx, fx.request("Land"), fx.committer())
+ requireMergeConflict(t, err)
+ pkg := landConflictDetail(err)
+ require.NotNil(t, pkg, "the package, not a deadline")
+ require.NotEmpty(t, pkg.GetProposalRef())
+ for _, p := range []string{"a", "b"} {
+ f := packageFile(t, pkg, p)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome(), p)
+ require.Contains(t, f.GetReason(), "could not verify before the deadline of 2s", p)
+ require.NotNil(t, f.GetProposal(), p)
+ }
+ require.Equal(t, 0, verifyRuns(t, vdir), "the command did not run for the land that waited")
+ require.Equal(t, 1, resolverCalls(t, rdir))
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)
+ busy.Release()
+
+ // Another repository is not held back by the first one's.
+ other, err := s.conflictResolver.AcquireVerifySlot(fx.ctx, landVerifyRepoKey(&gitalypb.Repository{StorageName: "default", RelativePath: "another.git"}))
+ require.NoError(t, err)
+ other.Release()
+}
diff --git a/internal/gitaly/service/operations/land_verify.go b/internal/gitaly/service/operations/land_verify.go
index 1ea46e7..a239602 100644
--- a/internal/gitaly/service/operations/land_verify.go
+++ b/internal/gitaly/service/operations/land_verify.go
@@ -18,6 +18,7 @@ import (
"github.com/feder-cr/git-ai/internal/git"
"github.com/feder-cr/git-ai/internal/git/gitcmd"
"github.com/feder-cr/git-ai/internal/git/localrepo"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
)
// This file is the optional verification of a resolution ([graft.land.resolver.verify], PLAN10
@@ -109,12 +110,17 @@ func (e *landVerifyUnfinishedError) Error() string {
func (e *landVerifyUnfinishedError) Unwrap() error { return e.err }
+// landVerifyRepoKey is the key of a repository in the verify pool: its storage and relative path.
+func landVerifyRepoKey(repository *gitalypb.Repository) string {
+ return repository.GetStorageName() + "\x00" + repository.GetRelativePath()
+}
+
// landVerifyOutOfLane runs the verification of a model's resolution out of the lane (PLAN10 15.9),
// with a slot of the verify pool. It returns a *landVerifyError when the command fails or times
// out, a *landVerifyUnfinishedError when ctx ended first, and another error when the verification
// could not be run.
-func (s *Server) landVerifyOutOfLane(ctx context.Context, repo *localrepo.Repo, tree git.ObjectID) error {
- slot, err := s.conflictResolver.AcquireVerifySlot(ctx)
+func (s *Server) landVerifyOutOfLane(ctx context.Context, repository *gitalypb.Repository, repo *localrepo.Repo, tree git.ObjectID) error {
+ slot, err := s.conflictResolver.AcquireVerifySlot(ctx, landVerifyRepoKey(repository))
if err != nil {
if ctx.Err() != nil {
return &landVerifyUnfinishedError{err: err}
diff --git a/internal/gitaly/service/operations/server.go b/internal/gitaly/service/operations/server.go
index 9c15cbf..502fbcc 100644
--- a/internal/gitaly/service/operations/server.go
+++ b/internal/gitaly/service/operations/server.go
@@ -38,9 +38,9 @@ type Server struct {
// checkBatchBytes, when set, replaces the bound of the file data of one check request
// (landCheckBatchBytes); only tests set it.
checkBatchBytes int64
- // landCheckBatchHook, when set, runs in every check batch while it holds its slot; only tests
- // set it.
- landCheckBatchHook func()
+ // landCheckBatchHook, when set, runs in every check batch while it holds its slot, with the
+ // context of the batch; only tests set it.
+ landCheckBatchHook func(ctx context.Context)
// landResolveHook, when set, runs in every resolver call while it holds its resolve slot; only
// tests set it.
landResolveHook func()
diff --git a/internal/gitaly/service/repository/create_bundle_from_ref_list_test.go b/internal/gitaly/service/repository/create_bundle_from_ref_list_test.go
index 158ef61..f44571e 100644
--- a/internal/gitaly/service/repository/create_bundle_from_ref_list_test.go
+++ b/internal/gitaly/service/repository/create_bundle_from_ref_list_test.go
@@ -198,10 +198,17 @@ func TestCreateBundleFromRefList_refusesPatternsThatAreMoreThanOneLine(t *testin
require.NoError(t, stream.Send(&gitalypb.CreateBundleFromRefListRequest{Repository: repo, Patterns: [][]byte{[]byte("master")}}))
require.NoError(t, stream.Send(&gitalypb.CreateBundleFromRefListRequest{Patterns: [][]byte{[]byte(pattern)}}))
require.NoError(t, stream.CloseSend())
- for _, err = stream.Recv(); err == nil; _, err = stream.Recv() {
+ // The data of the patterns that were read before the invalid one may have been sent: the
+ // bundle is not finished, and the answer is the error. What must never be in it is a
+ // reference the invalid pattern would have named, which never reached git.
+ var data []byte
+ var resp *gitalypb.CreateBundleFromRefListResponse
+ for resp, err = stream.Recv(); err == nil; resp, err = stream.Recv() {
+ data = append(data, resp.GetData()...)
}
require.Equal(t, codes.InvalidArgument, status.Code(err), "%v", err)
require.ErrorContains(t, err, "a revision can't contain a line break or NUL")
+ require.NotContains(t, string(data), "refs/graft/green/main", "the invalid pattern reached git")
})
}
diff --git a/internal/graft/resolver/check.go b/internal/graft/resolver/check.go
index 3744c44..c5f47b8 100644
--- a/internal/graft/resolver/check.go
+++ b/internal/graft/resolver/check.go
@@ -135,27 +135,65 @@ var verifySlotWait = promauto.NewHistogram(prometheus.HistogramOpts{
Buckets: prometheus.ExponentialBuckets(0.01, 4, 8),
})
+// repoVerifySlots are the verify slots of one repository: its verifications at once, and the
+// number of those holding or waiting for one, which keeps the entry in the map.
+type repoVerifySlots struct {
+ slots chan struct{}
+ users int
+}
+
// VerifySlot is the right to run one verification at a time: the client runs at most
-// verify_concurrency of them at once (AcquireVerifySlot), in a pool apart from the checks' and the
-// resolutions'. Release it when the verification is done.
+// verify_concurrency of them at once, and at most verify_per_repository for one repository
+// (AcquireVerifySlot), in a pool apart from the checks' and the resolutions'. Release it when the
+// verification is done.
type VerifySlot struct {
c *Client
+ repo string
released bool
}
-// AcquireVerifySlot waits for a verify slot. The wait counts against the deadline of ctx: when ctx
-// ends first, it fails, and the land cannot be verified in time.
-func (c *Client) AcquireVerifySlot(ctx context.Context) (*VerifySlot, error) {
+// AcquireVerifySlot waits for a verify slot of repo (a key of the repository: its storage and
+// relative path) and then for one of the pool. The wait counts against the deadline of ctx: when ctx
+// ends first, it fails, and the land cannot be verified in time. The slot of the repository is taken
+// first, so that a land that waits for its repository holds nothing of the pool.
+func (c *Client) AcquireVerifySlot(ctx context.Context, repo string) (*VerifySlot, error) {
if c == nil {
return nil, ErrDisabled
}
start := time.Now()
+ observe := func() { verifySlotWait.Observe(time.Since(start).Seconds()) }
+
+ c.verifyMu.Lock()
+ r, ok := c.verifyRepos[repo]
+ if !ok {
+ r = &repoVerifySlots{slots: make(chan struct{}, c.verifyPerRepo)}
+ c.verifyRepos[repo] = r
+ }
+ r.users++
+ c.verifyMu.Unlock()
+ leave := func() {
+ c.verifyMu.Lock()
+ defer c.verifyMu.Unlock()
+ if r.users--; r.users == 0 {
+ delete(c.verifyRepos, repo)
+ }
+ }
+
+ select {
+ case r.slots <- struct{}{}:
+ case <-ctx.Done():
+ observe()
+ leave()
+ return nil, fmt.Errorf("waiting for a verify slot of the repository: %w", ctx.Err())
+ }
select {
case c.verifySlots <- struct{}{}:
- verifySlotWait.Observe(time.Since(start).Seconds())
- return &VerifySlot{c: c}, nil
+ observe()
+ return &VerifySlot{c: c, repo: repo}, nil
case <-ctx.Done():
- verifySlotWait.Observe(time.Since(start).Seconds())
+ observe()
+ <-r.slots
+ leave()
return nil, fmt.Errorf("waiting for a verify slot of the resolver: %w", ctx.Err())
}
}
@@ -167,6 +205,14 @@ func (s *VerifySlot) Release() {
}
s.released = true
<-s.c.verifySlots
+
+ s.c.verifyMu.Lock()
+ defer s.c.verifyMu.Unlock()
+ r := s.c.verifyRepos[s.repo]
+ <-r.slots
+ if r.users--; r.users == 0 {
+ delete(s.c.verifyRepos, s.repo)
+ }
}
// Check sends one check request with the slot (Client.check).
diff --git a/internal/graft/resolver/client.go b/internal/graft/resolver/client.go
index 59c44fd..f5b54cc 100644
--- a/internal/graft/resolver/client.go
+++ b/internal/graft/resolver/client.go
@@ -10,6 +10,7 @@ import (
"os"
"slices"
"strings"
+ "sync"
"time"
"unicode/utf8"
@@ -105,6 +106,12 @@ type Client struct {
// verifySlots bounds the verifications of a model's resolution that run at once (VerifySlot),
// a third pool, as the operator's command takes minutes (PLAN10 15.9).
verifySlots chan struct{}
+ // verifyRepos are the slots of each repository that has a verification running or waiting, taken
+ // before a verify slot so that one repository cannot hold the whole pool; an entry lives as long
+ // as it has a holder or a waiter. verifyPerRepo is their number.
+ verifyMu sync.Mutex
+ verifyRepos map[string]*repoVerifySlots
+ verifyPerRepo int
// sendRules decide the files that are never sent (SendRules).
sendRules *SendRules
}
@@ -138,8 +145,8 @@ func New(cfg config.GraftResolver) (*Client, error) {
return nil, err
}
- if cfg.MaxFiles < 0 || cfg.MaxFileBytes < 0 || cfg.CheckConcurrency < 0 || cfg.ResolveConcurrency < 0 || cfg.VerifyConcurrency < 0 || cfg.ResolveRounds < 0 {
- return nil, errors.New("graft land resolver: max_files, max_file_bytes, check_concurrency, resolve_concurrency, verify_concurrency and resolve_rounds must not be negative")
+ if cfg.MaxFiles < 0 || cfg.MaxFileBytes < 0 || cfg.CheckConcurrency < 0 || cfg.ResolveConcurrency < 0 || cfg.VerifyConcurrency < 0 || cfg.VerifyPerRepository < 0 || cfg.ResolveRounds < 0 {
+ return nil, errors.New("graft land resolver: max_files, max_file_bytes, check_concurrency, resolve_concurrency, verify_concurrency, verify_per_repository and resolve_rounds must not be negative")
}
// The check of an agent's resolution sends a file's three sides and its resolution in one
// request, which must fit its bound.
@@ -152,17 +159,19 @@ func New(cfg config.GraftResolver) (*Client, error) {
return nil, err
}
c := &Client{
- command: slices.Clone(cfg.Command),
- timeout: timeout,
- maxFiles: cfg.MaxFilesOrDefault(),
- maxFileBytes: cfg.MaxFileBytesOrDefault(),
- pathsAllow: slices.Clone(cfg.PathsAllow),
- pathsDeny: slices.Clone(cfg.PathsDeny),
- actors: slices.Clone(cfg.Actors),
- sendRules: rules,
- checkSlots: make(chan struct{}, cfg.CheckConcurrencyOrDefault()),
- resolveSlots: make(chan struct{}, cfg.ResolveConcurrencyOrDefault()),
- verifySlots: make(chan struct{}, cfg.VerifyConcurrencyOrDefault()),
+ command: slices.Clone(cfg.Command),
+ timeout: timeout,
+ maxFiles: cfg.MaxFilesOrDefault(),
+ maxFileBytes: cfg.MaxFileBytesOrDefault(),
+ pathsAllow: slices.Clone(cfg.PathsAllow),
+ pathsDeny: slices.Clone(cfg.PathsDeny),
+ actors: slices.Clone(cfg.Actors),
+ sendRules: rules,
+ checkSlots: make(chan struct{}, cfg.CheckConcurrencyOrDefault()),
+ resolveSlots: make(chan struct{}, cfg.ResolveConcurrencyOrDefault()),
+ verifySlots: make(chan struct{}, cfg.VerifyConcurrencyOrDefault()),
+ verifyRepos: map[string]*repoVerifySlots{},
+ verifyPerRepo: cfg.VerifyPerRepositoryOrDefault(),
}
if cfg.URL != "" {
diff --git a/internal/graft/resolver/client_test.go b/internal/graft/resolver/client_test.go
index b3570d5..67126c9 100644
--- a/internal/graft/resolver/client_test.go
+++ b/internal/graft/resolver/client_test.go
@@ -768,23 +768,38 @@ func TestAcquireResolveSlot_boundsTheResolutionsAtOnce(t *testing.T) {
require.ErrorIs(t, err, ErrDisabled)
}
-// TestAcquireVerifySlot_boundsTheVerificationsAtOnce: at most verify_concurrency slots are held at
-// once; a wait counts against the caller's deadline; a released slot frees its place once; the pool
-// is apart from the checks' and the resolutions'.
+// timedOut reports whether acquiring a verify slot of repo gives up at once: the wait counts against
+// the context, so a short one ends it.
+func timedOut(t *testing.T, client *Client, repo string) error {
+ t.Helper()
+
+ ctx, cancel := context.WithTimeout(testhelper.Context(t), 50*time.Millisecond)
+ defer cancel()
+ slot, err := client.AcquireVerifySlot(ctx, repo)
+ if err == nil {
+ slot.Release()
+ }
+ return err
+}
+
+// TestAcquireVerifySlot_boundsTheVerificationsAtOnce: at most verify_concurrency slots are held at once
+// in the process, whatever the repositories; a wait counts against the deadline of the caller; a
+// released slot frees its place once; the pool is apart from the checks' and the resolutions'.
func TestAcquireVerifySlot_boundsTheVerificationsAtOnce(t *testing.T) {
cfg := commandConfig([]string{"/bin/true"})
- cfg.VerifyConcurrency = 1
+ cfg.VerifyConcurrency = 2
cfg.CheckConcurrency = 1
cfg.ResolveConcurrency = 1
client := newClient(t, cfg)
ctx := testhelper.Context(t)
- first, err := client.AcquireVerifySlot(ctx)
+ a, err := client.AcquireVerifySlot(ctx, "a")
+ require.NoError(t, err)
+ b, err := client.AcquireVerifySlot(ctx, "b")
require.NoError(t, err)
- waitCtx, cancel := context.WithTimeout(ctx, 50*time.Millisecond)
- defer cancel()
- _, err = client.AcquireVerifySlot(waitCtx)
+ // Two repositories hold the two slots: a third waits until its deadline, then fails.
+ err = timedOut(t, client, "c")
require.ErrorIs(t, err, context.DeadlineExceeded)
require.ErrorContains(t, err, "waiting for a verify slot of the resolver")
@@ -798,25 +813,93 @@ func TestAcquireVerifySlot_boundsTheVerificationsAtOnce(t *testing.T) {
checkSlot.Release()
resolveSlot.Release()
- first.Release()
- first.Release()
- second, err := client.AcquireVerifySlot(ctx)
+ // A slot released twice frees one place only.
+ a.Release()
+ a.Release()
+ require.NoError(t, timedOut(t, client, "c"))
+ c, err := client.AcquireVerifySlot(ctx, "c")
require.NoError(t, err)
- waitCtx2, cancel2 := context.WithTimeout(ctx, 50*time.Millisecond)
- defer cancel2()
- _, err = client.AcquireVerifySlot(waitCtx2)
- require.ErrorIs(t, err, context.DeadlineExceeded, "a second release must not free a second place")
- second.Release()
+ require.ErrorIs(t, timedOut(t, client, "d"), context.DeadlineExceeded, "a second release must not free a second place")
+ b.Release()
+ c.Release()
var none *VerifySlot
none.Release()
var disabled *Client
- _, err = disabled.AcquireVerifySlot(ctx)
+ _, err = disabled.AcquireVerifySlot(ctx, "a")
require.ErrorIs(t, err, ErrDisabled)
require.Equal(t, config.DefaultGraftResolverVerifyConcurrency, cap(newClient(t, commandConfig([]string{"/bin/true"})).verifySlots))
}
+// TestAcquireVerifySlot_oneRepositoryCannotTakeThePool: at most verify_per_repository verifications run
+// at once for one repository (one by default), so that a busy repository, whose every land builds a tree
+// of its own to verify, leaves slots to the others. A land that waits for its repository holds nothing of
+// the pool, and nothing is left of a repository when its last holder and waiter are gone.
+func TestAcquireVerifySlot_oneRepositoryCannotTakeThePool(t *testing.T) {
+ cfg := commandConfig([]string{"/bin/true"})
+ cfg.VerifyConcurrency = 2
+ client := newClient(t, cfg)
+ ctx := testhelper.Context(t)
+ require.Equal(t, config.DefaultGraftResolverVerifyPerRepository, client.verifyPerRepo)
+
+ first, err := client.AcquireVerifySlot(ctx, "busy")
+ require.NoError(t, err)
+
+ // A second verification of the same repository waits, though the pool has a free slot...
+ err = timedOut(t, client, "busy")
+ require.ErrorIs(t, err, context.DeadlineExceeded)
+ require.ErrorContains(t, err, "waiting for a verify slot of the repository")
+
+ // ...and takes nothing of the pool while it waits: another repository gets the free slot, even
+ // while a land waits for the busy one.
+ waiting := make(chan error, 1)
+ go func() {
+ slot, err := client.AcquireVerifySlot(ctx, "busy")
+ if err == nil {
+ slot.Release()
+ }
+ waiting <- err
+ }()
+ require.Eventually(t, func() bool {
+ client.verifyMu.Lock()
+ defer client.verifyMu.Unlock()
+ return client.verifyRepos["busy"] != nil && client.verifyRepos["busy"].users == 2
+ }, 5*time.Second, 5*time.Millisecond, "the second land never waited")
+ other, err := client.AcquireVerifySlot(ctx, "other")
+ require.NoError(t, err, "a land that waits for its repository holds nothing of the pool")
+
+ // The release of the first lets the waiter through.
+ first.Release()
+ require.NoError(t, <-waiting)
+ other.Release()
+
+ client.verifyMu.Lock()
+ require.Empty(t, client.verifyRepos, "nothing is kept of a repository without holder or waiter")
+ client.verifyMu.Unlock()
+
+ // A waiter that gave up leaves no trace either.
+ held, err := client.AcquireVerifySlot(ctx, "gone")
+ require.NoError(t, err)
+ require.Error(t, timedOut(t, client, "gone"))
+ held.Release()
+ client.verifyMu.Lock()
+ require.Empty(t, client.verifyRepos)
+ client.verifyMu.Unlock()
+
+ // Configured, the cap is the number of verifications of one repository at once.
+ cfg.VerifyPerRepository = 2
+ cfg.VerifyConcurrency = 3
+ wide := newClient(t, cfg)
+ one, err := wide.AcquireVerifySlot(ctx, "r")
+ require.NoError(t, err)
+ two, err := wide.AcquireVerifySlot(ctx, "r")
+ require.NoError(t, err)
+ require.ErrorContains(t, timedOut(t, wide, "r"), "of the repository")
+ one.Release()
+ two.Release()
+}
+
// TestResolveSlots_apartFromTheCheckSlots: the resolutions and the checks have pools of their own
// (PLAN10 15.8): every resolve slot held does not delay a check, and every check slot held does not
// delay a resolution.
@@ -854,9 +937,10 @@ func TestAcquireResolveSlot_defaultConcurrency(t *testing.T) {
// negative check_concurrency is.
func TestNew_refusesNegativePools(t *testing.T) {
for name, set := range map[string]func(*config.GraftResolver){
- "resolve_concurrency": func(c *config.GraftResolver) { c.ResolveConcurrency = -1 },
- "verify_concurrency": func(c *config.GraftResolver) { c.VerifyConcurrency = -1 },
- "resolve_rounds": func(c *config.GraftResolver) { c.ResolveRounds = -1 },
+ "resolve_concurrency": func(c *config.GraftResolver) { c.ResolveConcurrency = -1 },
+ "verify_concurrency": func(c *config.GraftResolver) { c.VerifyConcurrency = -1 },
+ "verify_per_repository": func(c *config.GraftResolver) { c.VerifyPerRepository = -1 },
+ "resolve_rounds": func(c *config.GraftResolver) { c.ResolveRounds = -1 },
} {
cfg := commandConfig([]string{"/bin/true"})
set(&cfg)