Skip to content

Git designed for AI agents.

graft: one verification mechanism, out of the lane, for every resolution; the checkout keeps no link target (review of 61dc0e3)

Federico Eliaacb104816142
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.md
index cd89129..9098650 100644
--- a/doc/graft/CONFIGURATION.md
+++ b/doc/graft/CONFIGURATION.md
@@ -198,8 +198,8 @@ cause.
| `max_file_bytes` | int | `1048576` (1 MiB) | Bound on every side of every file sent and on every resolved file accepted. At most 12566528 (about 12 MiB): one file, its three sides and its resolution, must fit one check request. |
| `proposal_ttl` | string | `"168h"` | How long the proposals of a failed land are kept under `refs/graft/proposals/` (Go duration, by the server's clock); at most 20 are kept per branch. Lands with a resolution and the housekeeping pass prune them. |
| `check_concurrency` | int | `2` | Checks of agents' resolutions that run at once in the Gitaly process, across all its listeners (one resolver client is shared by them); each holds a request of up to 64 MiB. A check waits for a free slot within its deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_check_slot_wait_seconds`. |
-| `verify_per_repository` | int | `1` | Verifications of a model's resolution that run at once for one repository, inside `verify_concurrency`: every land of a busy branch builds a tree of its own to verify, and one repository must not take the whole pool. A land waits for its repository's turn first, holding nothing of the pool, within the land's deadline. |
-| `verify_concurrency` | int | `2` | Verifications of a model's resolution that run at once in the Gitaly process, out of the lane, in a pool of their own, as `verify.command` takes minutes. A verification waits for a free slot within the land's deadline. The wait is in `graft_resolver_verify_slot_wait_seconds`. |
+| `verify_per_repository` | int | `1` | Verifications that run at once for one repository (of any resolution: a rule's, an agent's, a model's), inside `verify_concurrency`: every land of a busy branch builds a tree of its own to verify, and one repository must not take the whole pool. A land waits for its repository's turn first, holding nothing of the pool, within the land's deadline. |
+| `verify_concurrency` | int | `2` | Verifications that run at once in the Gitaly process (of any resolution: a rule's, an agent's, a model's), all out of the lane, in a pool of their own that exists whenever `verify.enabled` is set, with or without an AI resolver, as `verify.command` takes minutes. A verification waits for a free slot within the land's deadline. The wait is in `graft_resolver_verify_slot_wait_seconds`. |
| `resolve_concurrency` | int | `2` | Resolutions that run at once in the Gitaly process, in a pool apart from the checks', so that long resolutions never delay a check. A resolution waits for a free slot within the land's deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_resolve_slot_wait_seconds`. |
| `resolve_rounds` | int | `32` | Safety cap on the rounds of a land whose conflicting files keep changing while it is resolved out of the lane. The rounds are bounded by `timeout`; the cap is set well above realistic contention and is no contract. |
| `paths_allow` | array of strings | empty (every path) | Globs of paths that may be sent to the resolver, with the syntax of policy refs. |
diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.md
index 707244f..fc8e19f 100644
--- a/doc/graft/PLAN10.md
+++ b/doc/graft/PLAN10.md
@@ -897,6 +897,8 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
| branch moved / change already landed -> skipped (lane answers)
| review: exact paths, modes, markers, pointer lines, size
| resolver check (syntax + operator checks), bounded slots
+ | VERIFY (below) the exact tree the resolution would land: fails ->
+ | PACKAGE (every file REJECTED, the output); not in time -> PACKAGE
|
|-- resolve_conflicts? --yes--> PRE-RESOLVE, before the lane, in a discarded quarantine (15.8)
| tip T read as the lane reads it; no branch / already landed /
@@ -904,35 +906,38 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
| agent may not update the branch (policy) -> PermissionDenied
| the conflicting files not yet resolved -> one VERDICT per file
| (15.2), each kept by its key (path, base, ours, theirs, modes),
- | resolve slots; a model's answer: verify.command runs on the exact
- | tree that would land, verify slots (15.9); decideLand (15.4):
+ | resolve slots; VERIFY (the one verify box, 15.9): when verify.command
+ | is configured, or the answer holds a model's resolution, it runs on
+ | the exact tree that would land, with a slot of the verify pool
+ | (verify_concurrency, verify_per_repository); decideLand (15.4):
| not landable -> proposals kept (14) and PACKAGE now: the lane
| is not entered; a model's files are PROPOSED,
- | with the verify output when verify failed
+ | a rule's REJECTED, with the verify output when
+ | verify failed; NOT_ATTEMPTED (a rule's) or PROPOSED
+ | (a model's) with the reason when the deadline cut it
| landable ----> the verdicts, and the verified tree, go to the lane
v
LANE (one per branch)
|-- a resolution -----> landWithResolution: tip moved -> refuse, fresh PACKAGE
| applies the precheck's verdicts; any failure -> PACKAGE
- | operator verify.command on the merged tree; fails -> PACKAGE
- | all good -> LAND as agent/<actor> (+ Graft-Checked-By)
+ | the tree it would write is the tree the precheck verified, else PACKAGE
+ | (the lane runs no command) -> LAND as agent/<actor> (+ Graft-Checked-By)
v
3-way merge (rebase or merge) of the tip and the change
|-- clean -------------------------------------------------------------> LAND
|-- conflicts, no resolve_conflicts ------------------------------------> merge_conflict
|-- conflicts, every conflicting file has a verdict with its key, and the lane's
| classification of the file agrees with the verdict's
- | rules only -> operator verify.command (in the lane, for now); passes -> merge commit
- | signed by the rules -> LAND; fails -> PACKAGE (every file REJECTED, reason)
- | a model's file -> decideLand (15.4) on the tree the lane would write: it LANDS only if
- | that tree is the tree verify.command passed on (tree object IDs). The lane
- | runs no verification of a model's resolution. Merge commit signed
- | Graft-Resolved-By: <resolver> + Graft-Checked-By: verify:<command>.
+ | decideLand (15.4) on the tree the lane would write: with a verify.command configured, any
+ | resolution (a rule's, a model's) LANDS only if that tree is the tree
+ | verify.command passed on (tree object IDs), and a model's on no other tree,
+ | configured or not. The lane runs no verification of any resolution. Merge
+ | commit signed Graft-Resolved-By: <resolver> + Graft-Checked-By: verify:<command>.
| Another tree (the tip moved) -> NEEDS VERIFICATION, below
|-- conflicts, a file without such a verdict (new, or its sides changed)
| -> NEEDS RESOLUTION, out of the lane: PRE-RESOLVE only those files against the new tip,
| keeping the other verdicts, and enter the lane again
- |-- a model's answer whose tree is not the verified one (the tip moved)
+ |-- an answer whose tree is not the verified one (the tip moved)
-> NEEDS VERIFICATION, out of the lane: rebuild the tree from the same verdicts (the
resolver is called only for files that changed), VERIFY it, enter the lane again
The rounds are bounded by the deadline of the land (the resolver's timeout, plus verify.timeout when
@@ -1018,7 +1023,8 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
```
decideLand(answer, files, tree, verified): the pre-resolution calls it with the tree it verified,
the lane with the tree it would write
- every file resolved by an operator rule ...... LAND (then verify.command, in the lane)
+ every file resolved by an operator rule ...... LAND, and with a verify.command configured only if it ran
+ and passed on THIS tree (tree object ID equality)
a file a model resolved ...................... LAND only if verify.command ran and passed on THIS
tree: tree object ID equality, not the tip
anything else ................................ PACKAGE (the model's files are proposals)
@@ -1039,7 +1045,7 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
| out of the lane: rounds, deadline | `landResolving`, `landPreResolve` | operations/land_preresolve.go |
| per-file verdicts, decide | `landResolveAt`, `landSetOutcomes`, `decideLand`, `landAnswerProposes`, `landKeyOf` | operations/land_resolve.go, land_proposals.go, land_preresolve.go |
| the tree that would land | `landResolvedTree` | operations/land_resolve.go |
-| verify a model's resolution, out of the lane | `landVerifyOutOfLane`, `landVerify`, `Client.AcquireVerifySlot` | operations/land_verify.go; graft/resolver/check.go |
+| VERIFY, the one box, out of the lane, for every resolution | `landVerifyOutOfLane`, `landVerify`, `landPrecheckVerify` (an agent's), `VerifyPool.Acquire` | operations/land_verify.go, land_reland.go; graft/resolver/check.go |
| verdicts applied in the lane | `landApplyVerdicts`, `landNeedsResolution` | operations/land_resolve.go, land_preresolve.go |
| unverified in time: proposals | `landUnverified`, `landNotVerifiedReason` | operations/land_preresolve.go |
| resolver call, bounded slots | `Client.AcquireResolveSlot`, `ResolveSlot.Resolve` | graft/resolver/check.go |
@@ -1050,7 +1056,7 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
| regions, assemble | `llmRegions`, `llmAssemble` | tools/.../resolve/llm_core.go |
| one call per region, its checks | `llmPatchSystem`, `llmPatchUser`, `llmPatchAnswer`, `llmCheckPatched`, `llmCheckAnswer` | tools/.../resolve/llm_patch.go, llm_core.go |
| proposal mark, no recording | `proposer`, `Resolver.resolveFile` | tools/.../resolve/llm_strategy.go, resolve.go |
-| verify.command | `landVerify`, `verifyRejected` | operations/land_verify.go, land_resolve.go |
+| verify.command's checkout | `landCheckoutTree`, `landMaterializer.add`, `landCheckMaterialized`, `landRemoveCheckout`, `resolver.IsGitDirName` | operations/land_verify.go; graft/resolver/validate.go |
| merge commit, trailers | `landWriteResolution` | operations/land_resolve.go |
| PACKAGE | `landWithPackage`, `landListing.conflictPackage` | operations/land_package.go |
| proposals, hidden reference | `landWriteProposals`, `proposals.Update`, `proposals.Prune`, `landProposalsNotStored` | operations/land_proposals.go; graft/proposals |
@@ -1199,10 +1205,9 @@ The resolution is bound to the files instead (the PM's correction):
long resolutions starve the checks, and fairness inside one pool needs a scheduler for no gain
over two plain semaphores; the memory bound of the process is the sum of the two pools' requests.
Each request waits for its slot within the resolver's timeout and fails closed when it expires.
-* **Still in the lane, stated:** verify.command, the operator's tests on the merged tree, runs in
- the lane for the rules' resolutions and for agents' resolutions alike. For a model's resolution
- it already runs out of the lane (15.9). Moving the other two onto the same mechanism is the next
- step, measured on its own, so that in the end there is one verification mechanism.
+* **Nothing is verified in the lane any more.** This bullet said that verify.command still ran in the
+ lane for the rules' resolutions and for agents'; it does not (15.9): every verification runs out
+ of the lane, on one mechanism.
* **Measured (TestLandResolveBench, GRAFT_LAND_BENCH=1).** Six lands that the server resolves,
all on the same file, and six plain lands that conflict with nothing, at once through the lanes
of a standalone server, with a resolver that takes 500 ms; the same bench on the tree before
@@ -1241,8 +1246,24 @@ The final ruling of 14: an AI resolution lands only through the operator's verif
verification runs the operator's tests, which take minutes, so it cannot run in the lane (15.8),
and the tree it verifies depends on the tip.
+* **One mechanism, for every verification (PM ruling after the review of the checkout).** The
+ verification that was left in the lane for the rules' resolutions and for agents' went out of it
+ too: whatever made the resolution, verify.command runs on the exact tree, out of the lane, with a
+ slot of the process's pool, and the lane compares tree object IDs. Why: in the lane it was bounded
+ by nothing but one per branch, so the number of commands running in the process was unbounded;
+ now every one goes through verify_concurrency and verify_per_repository, and the pool exists
+ whenever verify.enabled is set, with or without an AI resolver. The costs, stated: a rules'
+ land whose tip moved after its tree was verified is verified again (it is a model's land in this
+ respect: liveness is the deadline's, and at the deadline the rules' files are NOT_ATTEMPTED and
+ an agent's resolution gets "could not verify before the deadline of <t>" with its package); an
+ agent's resolution is verified in its precheck, at the only tip the lane accepts it at. The
+ per-process worst case of a verification, so that the operator can size it: inodes and disk at
+ most verify_concurrency x (verify.max_tree_entries inodes, verify.max_tree_bytes of files), heap
+ at most verify_concurrency x (the paths kept, 128 MiB by the charge of the materializer; the
+ entries and the target of no link), and a TMPDIR on tmpfs turns that disk into memory: point it
+ at a disk, or lower the limits.
* **Verification out of the lane, on the exact tree.** For an answer that holds a model's
- resolution, and when nothing else keeps the land from landing (every file resolved or proposed),
+ resolution (or any answer, with a verify.command), and when nothing else keeps the land from landing (every file resolved or proposed),
the pre-resolution builds the tree that would land (`landResolvedTree`: the merged tree of the
tip and the change, the answer's files written in) and runs verify.command on it, in the
quarantine, with a slot of the verify pool (`verify_concurrency`, default 2, a pool of its own:
@@ -1320,9 +1341,13 @@ and the tree it verifies depends on the tip.
a fixed 128 MiB the memory of the paths, each entry and each directory charged twice (the sets that
look for links and for folded names). The limits are checked entry by entry, from the listing
(ls-tree -l), BEFORE that entry's content is read, not from a first pass over the whole listing: the
- listing and the blobs are streamed through two commands, so the memory is that of the paths kept (the
- fixed budget), and an entry that would go over stops the checkout with the earlier ones already
- written. The context is checked at every entry. A name over 255 bytes, a path that with the
+ listing and the blobs are streamed through two commands, and what stays in memory is exactly what
+ the materializer charges against the fixed 128 MiB budget (landMaxPathBytes): each entry its path
+ plus 256 bytes, each distinct prefix twice its length plus 64 (the folded key and the spelling),
+ each directory made its path plus 64, each link its path plus 64, and nothing of the target of a link
+ or the content of a file (the read-back hashes what is on disk). A link target is at most 4095
+ bytes, refused up front. An entry that would go over a limit stops the checkout with the earlier ones
+ already written. The context is checked at every entry. A name over 255 bytes, a path that with the
directory reaches PATH_MAX, a NUL in the target of a link, and a system call that fails (a name
that exists already, no room) are refused with the reason, not errors of the server. The checkout
runs inside verify.timeout, as the command does. A refusal is no error of the server: nothing is
@@ -1336,6 +1361,6 @@ and the tree it verifies depends on the tip.
proposal; a tip that moved gets its new tree verified and lands, with no resolver call; a tip that
moved to the same tree keeps the verification; churn past the deadline gives a proposal, and so does
the cap on the rounds; the lane is free while a verification runs (a plain land passes through it);
- the lane never verifies a model's resolution; the verify pool bounds the verifications at once;
- rules keep their verification in the lane; and the table test of decideLand, whose one rule is
- compared by tree object ID.
+ the lane never verifies; the verify pool bounds the verifications at once, those of rules and of
+ agents' resolutions as well; and the table test of decideLand, whose one rule is compared by tree
+ object ID.
diff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.md
index d7ed9cc..eab20a3 100644
--- a/doc/graft/RESOLVE.md
+++ b/doc/graft/RESOLVE.md
@@ -533,7 +533,14 @@ ends with the merge conflict and its package: the files that changed under it ar
`NOT_ATTEMPTED` with the reason `the files changed again before they could be
resolved`, the others keep their verdicts (a model's `PROPOSED`).
-`verify.command` still runs in the lane, on the merged tree of the lane's tip.
+`verify.command` runs out of the lane, whatever made the resolution (a rule, an agent, a model): on
+the exact tree that would land, with a slot of the process's verify pool, before the lane, which only
+compares the tree it would write with the one that passed. A land whose tip moved after its tree was
+verified is verified again; at the deadline the files of a rules' land are `NOT_ATTEMPTED` with the
+reason, and an agent's resolution gets the package with "could not verify before the deadline". A
+verification costs, per process, at most `verify_concurrency` times the inodes and bytes of
+`verify.max_tree_entries` and `verify.max_tree_bytes` on disk and the paths of the tree in memory;
+with `TMPDIR` on tmpfs that disk is memory.
## The conflict package and the re-land
diff --git a/internal/gitaly/config/config_graft.go b/internal/gitaly/config/config_graft.go
index 046c9ca..10e4857 100644
--- a/internal/gitaly/config/config_graft.go
+++ b/internal/gitaly/config/config_graft.go
@@ -402,6 +402,8 @@ const (
DefaultGraftResolverVerifyMaxTreeBytes int64 = 2 << 30
// DefaultGraftResolverVerifyMaxTreeDepth is the default GraftResolverVerify.MaxTreeDepth.
DefaultGraftResolverVerifyMaxTreeDepth = 64
+ // MaxGraftResolverVerifyMaxTreeDepth is the highest max_tree_depth a configuration may set.
+ MaxGraftResolverVerifyMaxTreeDepth = 256
)
// MaxTreeDepthOrDefault returns MaxTreeDepth or its default.
@@ -445,6 +447,9 @@ func (v GraftResolverVerify) validate() error {
if v.MaxTreeEntries < 0 || v.MaxTreeBytes < 0 || v.MaxTreeDepth < 0 {
return fmt.Errorf("graft land resolver verify: max_tree_entries, max_tree_bytes and max_tree_depth must not be negative (0 means the default)")
}
+ if v.MaxTreeDepth > MaxGraftResolverVerifyMaxTreeDepth {
+ return fmt.Errorf("graft land resolver verify: max_tree_depth %d is over %d: the checks of a path cost more with its depth", v.MaxTreeDepth, MaxGraftResolverVerifyMaxTreeDepth)
+ }
if !v.Enabled {
return nil
}
diff --git a/internal/gitaly/config/config_graft_resolver_test.go b/internal/gitaly/config/config_graft_resolver_test.go
index 69a399f..fcd5577 100644
--- a/internal/gitaly/config/config_graft_resolver_test.go
+++ b/internal/gitaly/config/config_graft_resolver_test.go
@@ -61,13 +61,14 @@ func TestGraftResolverValidateVerify(t *testing.T) {
require.Equal(t, int64(9), GraftResolverVerify{MaxTreeBytes: 9}.MaxTreeBytesOrDefault())
for name, v := range map[string]GraftResolverVerify{
- "no command": {Enabled: true},
- "empty program": {Enabled: true, Command: []string{""}},
- "bad timeout": {Enabled: true, Command: []string{"x"}, Timeout: "soon"},
- "zero timeout": {Enabled: true, Command: []string{"x"}, Timeout: "0s"},
- "negative entries": {MaxTreeEntries: -1},
- "negative bytes": {MaxTreeBytes: -1},
- "negative depth": {MaxTreeDepth: -1},
+ "no command": {Enabled: true},
+ "empty program": {Enabled: true, Command: []string{""}},
+ "bad timeout": {Enabled: true, Command: []string{"x"}, Timeout: "soon"},
+ "zero timeout": {Enabled: true, Command: []string{"x"}, Timeout: "0s"},
+ "negative entries": {MaxTreeEntries: -1},
+ "negative bytes": {MaxTreeBytes: -1},
+ "negative depth": {MaxTreeDepth: -1},
+ "depth over the bound": {MaxTreeDepth: 257},
} {
require.Error(t, GraftResolver{Verify: v}.validate(), name)
}
diff --git a/internal/gitaly/service/operations/land.go b/internal/gitaly/service/operations/land.go
index f795d0b..a1d092a 100644
--- a/internal/gitaly/service/operations/land.go
+++ b/internal/gitaly/service/operations/land.go
@@ -75,7 +75,7 @@ func (s *Server) UserLand(ctx context.Context, in *gitalypb.UserLandRequest) (*g
// The check of an agent's resolution runs an external process: it runs here, before the land
// enters its lane, so that no branch is held while it runs (landPrecheckResolution).
var precheck *landPrecheck
- if in.GetResolution() != nil && s.conflictResolver != nil {
+ if in.GetResolution() != nil && (s.conflictResolver != nil || s.resolveCfg.Verify.Enabled) {
if precheck, err = s.landPrecheckInQuarantine(ctx, in); err != nil {
return nil, err
}
@@ -429,7 +429,7 @@ func landLostRace(in *gitalypb.UserLandRequest, pre *landPreResolution, conflict
// proposals, not an error about a branch it has nothing to do with. The others get
// landKeptMovingError.
func landKeptMoving(in *gitalypb.UserLandRequest, pre *landPreResolution, conflict landConflict, attempts int) error {
- if pre.hasModel() {
+ if pre.mustVerify() {
return &landNeedsResolution{tip: conflict.newOID, commit: git.ObjectID(in.GetCommitId()), moved: true}
}
return landKeptMovingError(in, conflict, attempts)
diff --git a/internal/gitaly/service/operations/land_preresolve.go b/internal/gitaly/service/operations/land_preresolve.go
index f3ce4a7..1e900e7 100644
--- a/internal/gitaly/service/operations/land_preresolve.go
+++ b/internal/gitaly/service/operations/land_preresolve.go
@@ -67,6 +67,9 @@ type landPreResolution struct {
// verdicts kept, only these are the land's now. A verdict of a file that has since changed (its
// key is another) is stale, and decides nothing about the land.
current map[landFileKey]bool
+ // verifyAll says that a verification is configured: every resolution then needs a verified tree
+ // to land, a rule's as well as a model's (15.9).
+ verifyAll bool
}
func newLandPreResolution() *landPreResolution {
@@ -96,6 +99,23 @@ func (p *landPreResolution) hasModel() bool {
return false
}
+// mustVerify reports whether the land has a resolution whose landing needs a verified tree: a
+// model's, or any when a verification is configured. A stale verdict does not count.
+func (p *landPreResolution) mustVerify() bool {
+ if p == nil {
+ return false
+ }
+ if !p.verifyAll {
+ return p.hasModel()
+ }
+ for k := range p.verdicts {
+ if p.current[k] {
+ return true
+ }
+ }
+ return false
+}
+
// markVerified records that verify.command passed on tree.
func (p *landPreResolution) markVerified(tree git.ObjectID) {
if p.verified == nil {
@@ -230,6 +250,7 @@ func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequest
defer cancel()
pre := newLandPreResolution()
+ pre.verifyAll = s.resolveCfg.Verify.Enabled
var need *landNeedsResolution
// deadline reports whether the land ran out of its own timeout, and not its caller.
deadline := func() bool {
diff --git a/internal/gitaly/service/operations/land_preresolve_test.go b/internal/gitaly/service/operations/land_preresolve_test.go
index 6ce66ed..01239cb 100644
--- a/internal/gitaly/service/operations/land_preresolve_test.go
+++ b/internal/gitaly/service/operations/land_preresolve_test.go
@@ -10,6 +10,7 @@ import (
"strconv"
"strings"
"sync"
+ "sync/atomic"
"testing"
"time"
@@ -175,28 +176,45 @@ type heldLane struct {
release func()
}
-func newHeldLane(t *testing.T) heldLane {
+// newHeldLane starts the land of a, which holds the lane. With verify, a verification is configured (it runs
+// out of the lane, 15.9): every land then needs the tree that passed, and lands one by one when the tip moves.
+func newHeldLane(t *testing.T, verify bool) heldLane {
t.Helper()
dir, argv := resolverScript(t, answerByCall)
vdir := t.TempDir()
- // The first verification, the land of a's, waits for the release, at most a minute so that a
- // failed test never hangs; the others pass at once.
- verify := verifyScript(t, fmt.Sprintf(`V='%s'
+ // Every verification runs out of the lane (PLAN10 15.9) and passes at once, except when the test
+ // asks for a failure after the first. The lane is held by the land of a: the first land that
+ // applies its verdicts in it waits for the release, at most a minute so that a failed test never
+ // hangs.
+ verifyArgv := verifyScript(t, fmt.Sprintf(`V='%s'
N=$(cat "$V/calls" 2>/dev/null || echo 0)
N=$((N+1))
echo "$N" > "$V/calls"
-if [ "$N" = 1 ]; then
- i=0
- while [ ! -f "$V/release" ] && [ "$i" -lt 600 ]; do sleep 0.1; i=$((i+1)); done
-fi
if [ "$N" != 1 ] && [ -f "$V/fail" ]; then
echo "tests failed"
exit 3
fi
exit 0`, vdir))
+ var laneCalls atomic.Int32
+ entered, released := make(chan struct{}), make(chan struct{})
+ operationServerHooks.Store(t.Name(), func(s *Server) {
+ s.landLaneHook = func() {
+ if laneCalls.Add(1) != 1 {
+ return
+ }
+ close(entered)
+ select {
+ case <-released:
+ case <-time.After(time.Minute):
+ }
+ }
+ })
graft := batchingGraft()
- graft.Land.Resolver = withVerify(enabledResolver(argv), verify, "")
+ graft.Land.Resolver = enabledResolver(argv)
+ if verify {
+ graft.Land.Resolver = withVerify(graft.Land.Resolver, verifyArgv, "")
+ }
fx := newResolveFixtureIn(t, graft,
entries("a", "base\n", "b", "base\n", "c", "base\n"),
@@ -210,14 +228,15 @@ exit 0`, vdir))
writeAnswerN(t, dir, 1, answerOf("a"))
var once sync.Once
- release := func() {
- once.Do(func() { require.NoError(t, os.WriteFile(filepath.Join(vdir, "release"), nil, 0o600)) })
- }
+ release := func() { once.Do(func() { close(released) }) }
t.Cleanup(release)
first := landAsync(fx, fx.commit, "Land a")
- require.Eventually(t, func() bool { return countIn(t, filepath.Join(vdir, "calls")) == 1 },
- 30*time.Second, 10*time.Millisecond, "the land of a never reached its verification")
+ select {
+ case <-entered:
+ case <-time.After(30 * time.Second):
+ require.FailNow(t, "the land of a never reached the lane")
+ }
return heldLane{fx: fx, dir: dir, vdir: vdir, b: b, c: c, first: first, release: release}
}
@@ -233,14 +252,13 @@ func TestUserLand_resolvesOutOfTheLane(t *testing.T) {
t.Run("the resolver answers while another land holds the lane", func(t *testing.T) {
t.Parallel()
- h := newHeldLane(t)
+ h := newHeldLane(t, false)
writeAnswerN(t, h.dir, 2, answerOf("b"))
second := landAsync(h.fx, h.b, "Land b")
require.Eventually(t, func() bool { return countIn(t, filepath.Join(h.dir, "calls")) == 2 },
30*time.Second, 10*time.Millisecond, "b was not resolved while the lane was held")
requireRunning(t, h.first, "the land of a")
- require.Equal(t, 1, h.verifications(t))
h.release()
first := requireLanded(t, h.first)
@@ -261,7 +279,7 @@ func TestUserLand_resolvesOutOfTheLane(t *testing.T) {
t.Run("an answer that does not land never enters the lane", func(t *testing.T) {
t.Parallel()
- h := newHeldLane(t)
+ h := newHeldLane(t, true)
// A model's answer for b, whose verification fails: b cannot land (15.9). The verification
// runs out of the lane too, while the lane is held.
proposal := answerOf("b")
@@ -277,7 +295,7 @@ func TestUserLand_resolvesOutOfTheLane(t *testing.T) {
requireMergeConflict(t, err)
require.Equal(t, [][]byte{[]byte("b")}, landErrorDetail(t, err).GetMergeConflict().GetConflictingFiles())
requireRunning(t, h.first, "the land of a")
- require.Equal(t, 2, h.verifications(t), "a's, held in the lane, and b's, out of it")
+ require.Equal(t, 2, h.verifications(t), "a's and b's, both out of the lane")
require.Contains(t, packageFile(t, landConflictDetail(err), "b").GetReason(), "verification failed: exit status 3: tests failed")
// The proposal was kept although the lane was busy with a batch: it was written before the
@@ -301,7 +319,7 @@ func TestUserLand_resolvesOutOfTheLane(t *testing.T) {
t.Run("verdicts made before a batch are applied in it, each on the one before", func(t *testing.T) {
t.Parallel()
- h := newHeldLane(t)
+ h := newHeldLane(t, false)
writeAnswerN(t, h.dir, 2, answerOf("b"))
writeAnswerN(t, h.dir, 3, answerOf("c"))
diff --git a/internal/gitaly/service/operations/land_proposals.go b/internal/gitaly/service/operations/land_proposals.go
index e2ce818..a2c1395 100644
--- a/internal/gitaly/service/operations/land_proposals.go
+++ b/internal/gitaly/service/operations/land_proposals.go
@@ -207,17 +207,18 @@ func landTrailerValueOrQuoted(v string) string {
// decideLand is the one decision point of a land that asked the server to resolve its conflicts
// (PLAN10 15.4): it lands only when the resolver answered and every conflicting file is resolved,
-// by a rule (RESOLVED) or by a model (PROPOSED). The one rule about a model: a file that proposes
-// may land only if verify.command ran and passed on the exact tree that lands. tree is the tree
-// the land would write and verified the tree the verification passed on, compared by object ID,
-// not by the tip they were made at; a verified tree is never the empty ID. No other code path
-// lands a model's resolution.
-func decideLand(answer *resolver.Response, files []*landFileState, tree, verified git.ObjectID) bool {
+// by a rule (RESOLVED) or by a model (PROPOSED). The one rule: a resolution lands only on a tree
+// that verify.command ran and passed on, whenever a verification is configured (verifyRequired), and a
+// model's resolution lands on no other, with or without one configured (without, it never lands).
+// tree is the tree the land would write and verified the tree the verification passed on, compared
+// by object ID, not by the tip they were made at; a verified tree is never the empty ID. Every
+// verification, whatever made the resolution, runs out of the lane, so that the lane only compares.
+func decideLand(answer *resolver.Response, files []*landFileState, tree, verified git.ObjectID, verifyRequired bool) bool {
if answer == nil {
return false
}
proposes := landAnswerProposes(answer)
- if proposes && (verified == "" || verified != tree) {
+ if (proposes || verifyRequired) && (verified == "" || verified != tree) {
return false
}
for _, f := range files {
diff --git a/internal/gitaly/service/operations/land_proposals_test.go b/internal/gitaly/service/operations/land_proposals_test.go
index 6bd0754..4dc4aa8 100755
--- a/internal/gitaly/service/operations/land_proposals_test.go
+++ b/internal/gitaly/service/operations/land_proposals_test.go
@@ -352,19 +352,26 @@ func TestDecideLand(t *testing.T) {
rules := &resolver.Response{Resolver: "graft-resolve/trivial,union", Files: []resolver.ResponseFile{
{Path: "a", Origin: resolver.OriginRule}, {Path: "b", Origin: resolver.OriginRule},
}}
- require.True(t, decideLand(rules, resolved(), "", ""), "rules resolved every file")
- require.True(t, decideLand(rules, resolved(), "t1", "t2"), "rules land whatever tree was verified: their verification stays in the lane")
- require.False(t, decideLand(nil, resolved(), "t1", "t1"), "no answer")
+ require.True(t, decideLand(rules, resolved(), "", "", false), "rules resolved every file")
+ require.True(t, decideLand(rules, resolved(), "t1", "t2", false), "rules land whatever tree was verified when no verification is configured")
+ require.False(t, decideLand(nil, resolved(), "t1", "t1", false), "no answer")
left := resolved()
left[1].outcome = gitalypb.LandConflictFile_OUTCOME_NOT_SENT
- require.False(t, decideLand(rules, left, "", ""), "a file left to the agent")
+ require.False(t, decideLand(rules, left, "", "", false), "a file left to the agent")
rejected := resolved()
rejected[0].outcome = gitalypb.LandConflictFile_OUTCOME_REJECTED
- require.False(t, decideLand(rules, rejected, "", ""), "a file the answer is refused for")
+ require.False(t, decideLand(rules, rejected, "", "", false), "a file the answer is refused for")
proposedByRules := resolved()
proposedByRules[0].outcome = gitalypb.LandConflictFile_OUTCOME_PROPOSED
- require.False(t, decideLand(rules, proposedByRules, "", ""), "a PROPOSED file in an answer that does not propose")
+ require.False(t, decideLand(rules, proposedByRules, "", "", false), "a PROPOSED file in an answer that does not propose")
+
+ // With a verification configured the rule holds for rules as well (15.9): their resolution lands
+ // only on the tree that passed, and the lane runs nothing.
+ require.True(t, decideLand(rules, resolved(), "t1", "t1", true), "rules, verified on this very tree")
+ require.False(t, decideLand(rules, resolved(), "t1", "", true), "rules, nothing verified")
+ require.False(t, decideLand(rules, resolved(), "t1", "t2", true), "rules, verified on another tree (the tip moved)")
+ require.False(t, decideLand(rules, resolved(), "", "", true), "rules, no tree at all: empty is not equal")
// THE rule (PLAN10 15.9): a file that proposes may land only if verify.command passed on the exact
// tree that lands, compared by object ID.
@@ -376,18 +383,18 @@ func TestDecideLand(t *testing.T) {
return []*landFileState{{outcome: gitalypb.LandConflictFile_OUTCOME_RESOLVED}, {outcome: gitalypb.LandConflictFile_OUTCOME_PROPOSED}}
}
for desc, answer := range map[string]*resolver.Response{"a file marked as a proposal": flagged, "a file that says nothing": unmarked} {
- require.True(t, decideLand(answer, proposing(), "tree-1", "tree-1"), "%s: verified on this very tree", desc)
- require.False(t, decideLand(answer, proposing(), "tree-1", ""), "%s: nothing verified", desc)
- require.False(t, decideLand(answer, proposing(), "tree-1", "tree-2"), "%s: verified on another tree (the tip moved)", desc)
- require.False(t, decideLand(answer, proposing(), "", ""), "%s: no tree at all: empty is not equal", desc)
- require.False(t, decideLand(answer, resolved(), "tree-1", ""), "%s: even with every outcome RESOLVED", desc)
+ require.True(t, decideLand(answer, proposing(), "tree-1", "tree-1", false), "%s: verified on this very tree", desc)
+ require.False(t, decideLand(answer, proposing(), "tree-1", "", false), "%s: nothing verified", desc)
+ require.False(t, decideLand(answer, proposing(), "tree-1", "tree-2", false), "%s: verified on another tree (the tip moved)", desc)
+ require.False(t, decideLand(answer, proposing(), "", "", false), "%s: no tree at all: empty is not equal", desc)
+ require.False(t, decideLand(answer, resolved(), "tree-1", "", false), "%s: even with every outcome RESOLVED", desc)
leftOver := proposing()
leftOver[0].outcome = gitalypb.LandConflictFile_OUTCOME_FAILED
- require.False(t, decideLand(answer, leftOver, "tree-1", "tree-1"), "%s: verified, but a file is left to the agent", desc)
+ require.False(t, decideLand(answer, leftOver, "tree-1", "tree-1", false), "%s: verified, but a file is left to the agent", desc)
refused := proposing()
refused[1].outcome = gitalypb.LandConflictFile_OUTCOME_REJECTED
- require.False(t, decideLand(answer, refused, "tree-1", "tree-1"), "%s: verified, but a file is refused", desc)
+ require.False(t, decideLand(answer, refused, "tree-1", "tree-1", false), "%s: verified, but a file is refused", desc)
}
}
diff --git a/internal/gitaly/service/operations/land_reland.go b/internal/gitaly/service/operations/land_reland.go
index 3d8e40f..1be7c58 100644
--- a/internal/gitaly/service/operations/land_reland.go
+++ b/internal/gitaly/service/operations/land_reland.go
@@ -149,6 +149,13 @@ type landPrecheck struct {
tip, commit git.ObjectID
result landCheckResult
skipped string
+ // verified is the tree that verify.command passed on, out of the lane (15.9), when a
+ // verification is configured and the resolution passed the checks; verifyErr is why it was not
+ // verified: a *landVerifyError (the command failed, the checkout was refused) or a
+ // plain error (the deadline). The lane only compares the tree it would write with
+ // verified.
+ verified git.ObjectID
+ verifyErr error
}
// landPrecheckResolution runs the resolver's checks of the request's resolution before the land
@@ -164,7 +171,7 @@ type landPrecheck struct {
// is not checked (skipped): the lane refuses or answers it. It returns nil when no resolver is
// configured. A failure to check is the result's failure, with its cause, which the lane reports.
func (s *Server) landPrecheckResolution(ctx context.Context, in *gitalypb.UserLandRequest, repo *localrepo.Repo) (*landPrecheck, error) {
- if s.conflictResolver == nil {
+ if s.conflictResolver == nil && !s.resolveCfg.Verify.Enabled {
return nil, nil
}
objectHash, err := repo.ObjectHash(ctx)
@@ -211,14 +218,85 @@ func (s *Server) landPrecheckResolution(ctx context.Context, in *gitalypb.UserLa
return &landPrecheck{tip: tip, commit: commitID, result: landCheckResult{failure: fmt.Sprintf("listing the conflicts: %v", err), failureErr: err}}, nil
}
+ var result landCheckResult
+ if s.conflictResolver != nil {
+ timeout, err := s.resolveCfg.TimeoutOrDefault()
+ if err != nil {
+ return nil, structerr.NewInternal("resolver timeout: %w", err)
+ }
+ checkCtx, cancel := context.WithTimeout(ctx, timeout)
+ defer cancel()
+ result = s.landCheckResolution(checkCtx, in, repo, review.listing, review.actor, review.resolved)
+ }
+ pc := &landPrecheck{tip: tip, commit: commitID, result: result}
+ if s.resolveCfg.Verify.Enabled && result.failure == "" && !landCheckRejects(result) {
+ if err := s.landPrecheckVerify(ctx, in, repo, review, pc); err != nil {
+ return nil, err
+ }
+ }
+ return pc, nil
+}
+
+// landCheckRejects reports whether the check rejected a file of the resolution: the lane then
+// refuses it, and there is nothing to verify.
+func landCheckRejects(r landCheckResult) bool {
+ for _, v := range r.verdicts {
+ if v.rejected != "" {
+ return true
+ }
+ }
+ return false
+}
+
+// landAgentResponse is the agent's resolution as a resolver answer, the files in the order of the
+// listing: what landWriteResolution writes.
+func landAgentResponse(review landReview) *resolver.Response {
+ resp := &resolver.Response{Version: resolver.ProtocolVersion, Resolver: landAgentResolver(review.actor)}
+ for _, f := range review.listing.files {
+ resp.Files = append(resp.Files, review.resolved[f.conflict.path])
+ }
+ return resp
+}
+
+// landPrecheckVerify verifies the tree of the agent's resolution out of the lane (15.9), with a slot
+// of the verify pool, before the land enters its lane, and records the outcome in pc. The deadline is
+// the resolver's timeout plus the verification's; a verification that the deadline cut is recorded
+// as unfinished, and one that the caller cut is the caller's cancellation. The resolution was made
+// against the tip the branch is at, which the lane insists on, so the tree it would write there is
+// the tree verified here.
+func (s *Server) landPrecheckVerify(ctx context.Context, in *gitalypb.UserLandRequest, repo *localrepo.Repo, review landReview, pc *landPrecheck) error {
timeout, err := s.resolveCfg.TimeoutOrDefault()
if err != nil {
- return nil, structerr.NewInternal("resolver timeout: %w", err)
+ return structerr.NewInternal("resolver timeout: %w", err)
+ }
+ verifyTimeout, err := s.resolveCfg.Verify.TimeoutOrDefault()
+ if err != nil {
+ return structerr.NewInternal("verify timeout: %w", err)
}
- ctx, cancel := context.WithTimeout(ctx, timeout)
+ tree, err := s.landResolvedTree(ctx, repo, review.listing.mergedTree, landAgentResponse(review))
+ if err != nil {
+ return structerr.NewInternal("building the tree to verify: %w", err)
+ }
+ vctx, cancel := context.WithTimeout(ctx, timeout+verifyTimeout)
defer cancel()
- result := s.landCheckResolution(ctx, in, repo, review.listing, review.actor, review.resolved)
- return &landPrecheck{tip: tip, commit: commitID, result: result}, nil
+ var verifyErr *landVerifyError
+ switch err := s.landVerifyOutOfLane(vctx, in.GetRepository(), repo, tree); {
+ case err == nil:
+ pc.verified = tree
+ case errors.As(err, &verifyErr):
+ pc.verifyErr = err
+ default:
+ var unfinished *landVerifyUnfinishedError
+ if errors.As(err, &unfinished) && ctx.Err() == nil {
+ pc.verifyErr = fmt.Errorf("could not verify before the deadline of %s", timeout+verifyTimeout)
+ return nil
+ }
+ if ctx.Err() != nil {
+ return landCallerGaveUp(ctx, err)
+ }
+ return err
+ }
+ return nil
}
// landWithResolution lands commitID onto tip with the caller's resolution of the conflicts
@@ -302,13 +380,48 @@ func (s *Server) landWithResolution(
return landResolution{}, l.unresolved("", "the resolution was rejected: "+landUnresolvedCause(l.files), nil)
}
- resp := &resolver.Response{Version: resolver.ProtocolVersion, Resolver: landAgentResolver(review.actor)}
- for _, f := range l.files {
- resp.Files = append(resp.Files, review.resolved[f.conflict.path])
+ resp := landAgentResponse(review)
+ // The verification ran out of the lane (15.9): the lane compares the tree it would write with
+ // the one that passed, and writes nothing that did not.
+ var built git.ObjectID
+ verified := false
+ if s.resolveCfg.Verify.Enabled {
+ switch reason := landPrecheckUnusable(precheck, tip, commitID); {
+ case reason != "":
+ return landResolution{}, l.notAttempted("the resolution could not be verified: "+reason, nil)
+ case precheck.verifyErr != nil:
+ var verifyErr *landVerifyError
+ if errors.As(precheck.verifyErr, &verifyErr) {
+ return landResolution{}, l.verifyRejected(precheck.verifyErr)
+ }
+ return landResolution{}, l.notAttempted("the resolution could not be verified: "+precheck.verifyErr.Error(), nil)
+ }
+ tree, err := s.landResolvedTree(ctx, repo, l.mergedTree, resp)
+ if err != nil {
+ return landResolution{}, err
+ }
+ if precheck.verified == "" || precheck.verified != tree {
+ return landResolution{}, l.notAttempted("the resolution could not be verified: the tree to land is not the tree that was verified", nil)
+ }
+ built, verified = tree, true
}
- res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, review.actor, resp, checkedBy, "", false)
+ res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, review.actor, resp, checkedBy, built, verified)
res.syntaxUnchecked = syntaxUnchecked
- return res, l.verifyRejected(err)
+ return res, err
+}
+
+// landPrecheckUnusable says why the precheck of a land cannot vouch for the tip and commit the lane
+// has, or "" when it can.
+func landPrecheckUnusable(pc *landPrecheck, tip, commitID git.ObjectID) string {
+ switch {
+ case pc == nil:
+ return "it was not verified before the land"
+ case pc.skipped != "":
+ return fmt.Sprintf("it was not verified before the land, since %s", pc.skipped)
+ case pc.tip != tip || pc.commit != commitID:
+ return fmt.Sprintf("it was verified against tip %s and commit %s, not tip %s and commit %s", pc.tip, pc.commit, tip, commitID)
+ }
+ return ""
}
// landResolutionFiles maps the files of the resolution to the conflicting paths. It returns a
diff --git a/internal/gitaly/service/operations/land_resolve.go b/internal/gitaly/service/operations/land_resolve.go
index 383f445..f797562 100644
--- a/internal/gitaly/service/operations/land_resolve.go
+++ b/internal/gitaly/service/operations/land_resolve.go
@@ -218,17 +218,19 @@ func (s *Server) landResolveAt(
}
}
- // A model's resolution lands only through the operator's verification (15.9): verify.command
- // runs here, out of the lane, on the exact tree that would land, once nothing else stops the
- // land. Whatever keeps a model's file from landing is said in the reason of its file.
+ // A resolution lands only through the operator's verification (15.9): verify.command runs
+ // here, out of the lane, on the exact tree that would land, once nothing else stops the land,
+ // whoever made the resolution: a model's, whose files stay proposals without one, and a rule's,
+ // when a verification is configured. Whatever keeps a file from landing is said in its reason.
+ verifyRequired := s.resolveCfg.Verify.Enabled
var tree, verified git.ObjectID
- if combined != nil && landAnswerProposes(combined) {
+ if combined != nil && (landAnswerProposes(combined) || verifyRequired) {
switch {
case pre.unverifiable != "":
- landAnnotateProposed(files, pre.unverifiable)
- case !s.resolveCfg.Verify.Enabled && len(s.resolveCfg.Verify.Command) > 0:
+ landAnnotateUnverified(files, combined, pre.unverifiable)
+ case !verifyRequired && len(s.resolveCfg.Verify.Command) > 0:
landAnnotateProposed(files, landVerifyDisabledReason)
- case !s.resolveCfg.Verify.Enabled:
+ case !verifyRequired:
landAnnotateProposed(files, landNoVerifyReason)
case landAllAnswered(files):
var err error
@@ -248,15 +250,19 @@ func (s *Server) landResolveAt(
verified = tree
pre.markVerified(tree)
s.logger.WithFields(log.Fields{"branch": string(in.GetBranch()), "tip": tip.String(), "commit": commitID.String(), "tree": tree.String()}).
- InfoContext(ctx, "graft: a model's resolution passed the verification out of the lane")
+ InfoContext(ctx, "graft: a resolution passed the verification out of the lane")
case errors.As(err, &verifyErr):
- landAnnotateProposed(files, verifyErr.Error())
+ if landAnswerProposes(combined) {
+ landAnnotateProposed(files, verifyErr.Error())
+ } else {
+ landRejectResolved(files, verifyErr.Error())
+ }
default:
return l, nil, err
}
}
}
- landed := decideLand(combined, files, tree, verified)
+ landed := decideLand(combined, files, tree, verified, verifyRequired)
// An answer that does not land, because a model made part of it or another file is left, is
// kept for the agent: the files that rules resolved can be landed as they are. The proposals
// are an aid and the package is the contract: when they cannot be kept, the package still goes
@@ -315,6 +321,31 @@ func landAllAnswered(files []*landFileState) bool {
}
// landAnnotateProposed adds why a model's files did not land to the reason of each.
+// landAnnotateUnverified says why a land that needs the verification could not be verified in time:
+// a model's files are PROPOSED with the reason, and the files of an answer made by rules only are
+// NOT_ATTEMPTED with it, since nothing of them lands without the verification.
+func landAnnotateUnverified(files []*landFileState, answer *resolver.Response, why string) {
+ if landAnswerProposes(answer) {
+ landAnnotateProposed(files, why)
+ return
+ }
+ for _, f := range files {
+ if f.outcome == gitalypb.LandConflictFile_OUTCOME_RESOLVED {
+ f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_NOT_ATTEMPTED, why
+ }
+ }
+}
+
+// landRejectResolved marks every resolved file of a rules' answer REJECTED with the reason of the
+// verification that failed: the land does not happen, and the package says why.
+func landRejectResolved(files []*landFileState, why string) {
+ for _, f := range files {
+ if f.outcome == gitalypb.LandConflictFile_OUTCOME_RESOLVED {
+ f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_REJECTED, why
+ }
+ }
+}
+
func landAnnotateProposed(files []*landFileState, why string) {
for _, f := range files {
if f.outcome != gitalypb.LandConflictFile_OUTCOME_PROPOSED {
@@ -410,6 +441,9 @@ func (s *Server) landApplyVerdicts(
committer git.Signature,
pre *landPreResolution,
) (landResolution, error) {
+ if s.landLaneHook != nil {
+ s.landLaneHook()
+ }
l, err := landListConflicts(ctx, repo, tip, commitID, s.resolveCfg.MaxFileBytesOrDefault())
if err != nil {
return landResolution{}, err
@@ -453,12 +487,14 @@ func (s *Server) landApplyVerdicts(
answer := landCombinedAnswer(nil, reused, names, false)
landSetOutcomes(answer, l.files)
- // The one rule about a model's resolution, asked of the tree this tip would land: it lands only
- // if verify.command passed on this very tree (tree OIDs, not tips). When the tip moved, the tree
- // is another one, and the land goes back out of the lane to be verified again. The lane never
- // runs the verification of a model's resolution.
+ // The one rule, asked of the tree this tip would land: it lands only if verify.command passed on
+ // this very tree (tree OIDs, not tips), whoever made the resolution, whenever a verification is
+ // configured, and a model's resolution on no other. When the tip moved, the tree is another one,
+ // and the land goes back out of the lane to be verified again. The lane never runs a
+ // verification: it compares.
+ verifyRequired := s.resolveCfg.Verify.Enabled
var tree, verified git.ObjectID
- if landAnswerProposes(answer) {
+ if landAnswerProposes(answer) || verifyRequired {
if tree, err = s.landResolvedTree(ctx, repo, l.mergedTree, answer); err != nil {
return landResolution{}, err
}
@@ -466,7 +502,7 @@ func (s *Server) landApplyVerdicts(
verified = tree
}
}
- if !decideLand(answer, l.files, tree, verified) {
+ if !decideLand(answer, l.files, tree, verified, verifyRequired) {
need := &landNeedsResolution{tip: tip, commit: commitID, unverified: tree}
if tree == "" {
// Nothing to verify, yet not landable: a file the answer rejects. Resolving again says why.
@@ -481,7 +517,7 @@ func (s *Server) landApplyVerdicts(
checkedBy = s.landVerifyTrailer()
}
res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, checkedBy, tree, verified != "")
- return res, l.verifyRejected(err)
+ return res, err
}
// verifyRejected turns a resolution that the verification rejected into the unresolved land with
@@ -675,9 +711,8 @@ func landUnresolvedCause(files []*landFileState) string {
// tree and commits it with parents [tip, commitID]; checkedBy names what checked it, if anything
// did. The commit message carries the trailers. built is the resolved tree when the caller already
// built it (landResolvedTree). verified says that verify.command passed on that very tree, out of
-// the lane (15.9), so that it is not run again; otherwise the operator's verification runs here,
-// in the lane, on the merged tree, before it lands: that is a resolution made by rules or by an
-// agent, which keeps its verification in the lane for now.
+// the lane (15.9). The lane runs no verification: when one is configured and the tree is not the
+// one it passed on, the resolution is refused here, closed, whatever the caller did.
func (s *Server) landWriteResolution(
ctx context.Context,
in *gitalypb.UserLandRequest,
@@ -707,12 +742,10 @@ func (s *Server) landWriteResolution(
return landResolution{}, err
}
}
- // The optional verification runs on the merged tree, the resolution included, before it lands,
- // unless it already passed on this tree out of the lane.
+ // The verification ran out of the lane, on this very tree, before it got here (15.9): no
+ // resolution lands on a tree it did not pass on, and no code path verifies in the lane.
if s.resolveCfg.Verify.Enabled && !verified {
- if err := s.landVerify(ctx, repo, treeID); err != nil {
- return landResolution{}, err
- }
+ return landResolution{}, errors.New("a resolution reached the lane without the verification of its tree")
}
newTip, err := repo.WriteCommit(ctx, localrepo.WriteCommitConfig{
diff --git a/internal/gitaly/service/operations/land_resolve_helper_test.go b/internal/gitaly/service/operations/land_resolve_helper_test.go
index 5c318c8..5c71f01 100644
--- a/internal/gitaly/service/operations/land_resolve_helper_test.go
+++ b/internal/gitaly/service/operations/land_resolve_helper_test.go
@@ -21,11 +21,10 @@ func (s *Server) resolveLandConflicts(ctx context.Context, in *gitalypb.UserLand
// As the lane does for a model's resolution: the tree is the one verify.command passed on, out
// of the lane, or the answer would not have been landable.
var tree git.ObjectID
- if landAnswerProposes(answer) {
+ if landAnswerProposes(answer) || s.resolveCfg.Verify.Enabled {
if tree, err = s.landResolvedTree(ctx, repo, l.mergedTree, answer); err != nil {
return landResolution{}, err
}
}
- res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, "", tree, pre.isVerified(tree))
- return res, l.verifyRejected(err)
+ return s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, "", tree, pre.isVerified(tree))
}
diff --git a/internal/gitaly/service/operations/land_resolve_test.go b/internal/gitaly/service/operations/land_resolve_test.go
index 94e7914..6d2d007 100644
--- a/internal/gitaly/service/operations/land_resolve_test.go
+++ b/internal/gitaly/service/operations/land_resolve_test.go
@@ -254,8 +254,12 @@ func (fx resolveFixture) server(t *testing.T, rcfg config.GraftResolver) *Server
require.NoError(t, err)
cache := catfile.NewCache(fx.cfg)
t.Cleanup(cache.Stop)
+ var pool *resolver.VerifyPool
+ if rcfg.Verify.Enabled {
+ pool = resolver.NewVerifyPool(rcfg)
+ }
return &Server{
- logger: testhelper.NewLogger(t), gitConfig: fx.cfg.Git, resolveCfg: rcfg, conflictResolver: client,
+ logger: testhelper.NewLogger(t), gitConfig: fx.cfg.Git, resolveCfg: rcfg, conflictResolver: client, verifyPool: pool,
localRepoFactory: localrepo.NewFactory(testhelper.NewLogger(t), config.NewLocator(fx.cfg), gittest.NewCommandFactory(t, fx.cfg), cache),
}
}
diff --git a/internal/gitaly/service/operations/land_verified_test.go b/internal/gitaly/service/operations/land_verified_test.go
index e8a7c7c..15549ca 100644
--- a/internal/gitaly/service/operations/land_verified_test.go
+++ b/internal/gitaly/service/operations/land_verified_test.go
@@ -535,9 +535,9 @@ func TestLandApplyVerdicts_neverVerifiesAModelsResolution(t *testing.T) {
require.Equal(t, 1, verifyRuns(t, vdir))
}
-// TestUserLand_verifiedRulesKeepTheirVerificationInTheLane: a resolution made by rules alone keeps its
-// verify.command in the lane, as before (15.9): the next step moves it.
-func TestUserLand_verifiedRulesKeepTheirVerificationInTheLane(t *testing.T) {
+// TestUserLand_verifiedRulesAreVerifiedOutOfTheLane: a resolution made by rules alone is verified like
+// any other (15.9): out of the lane, once, and the trailer names what ran.
+func TestUserLand_verifiedRulesAreVerifiedOutOfTheLane(t *testing.T) {
t.Parallel()
fx, _, vdir := verifiedFixture(t, validAnswer(), "grep -qx merged-a a", nil)
@@ -547,7 +547,7 @@ func TestUserLand_verifiedRulesKeepTheirVerificationInTheLane(t *testing.T) {
commit, _ := landedCommit(t, fx, resp)
require.Equal(t, 1, verifyRuns(t, vdir))
require.Contains(t, string(commit.GetBody()), "Graft-Resolved-By: graft-resolve/fake\n")
- require.NotContains(t, string(commit.GetBody()), "Graft-Checked-By", "rules are unchanged")
+ require.Contains(t, string(commit.GetBody()), "Graft-Checked-By: verify:verify.sh\n", "the trailer names what ran")
failing, _, failingDir := verifiedFixture(t, validAnswer(), "exit 1", nil)
_, err = failing.client.UserLand(failing.ctx, failing.request("Land"))
@@ -617,7 +617,7 @@ func TestUserLand_verifiedModelResolution_aBusyRepositoryEndsInProposals(t *test
s.locator = config.NewLocator(fx.cfg) // the land quarantines the objects it makes
// Another land of the repository is being verified: its slot is the repository's.
- busy, err := s.conflictResolver.AcquireVerifySlot(fx.ctx, landVerifyRepoKey(fx.repoProto))
+ busy, err := s.verifyPool.Acquire(fx.ctx, landVerifyRepoKey(fx.repoProto))
require.NoError(t, err)
_, err = s.landResolving(fx.ctx, fx.request("Land"), fx.committer())
@@ -637,7 +637,7 @@ func TestUserLand_verifiedModelResolution_aBusyRepositoryEndsInProposals(t *test
busy.Release()
// Another repository is not held back by the first one's.
- other, err := s.conflictResolver.AcquireVerifySlot(fx.ctx, landVerifyRepoKey(&gitalypb.Repository{StorageName: "default", RelativePath: "another.git"}))
+ other, err := s.verifyPool.Acquire(fx.ctx, landVerifyRepoKey(&gitalypb.Repository{StorageName: "default", RelativePath: "another.git"}))
require.NoError(t, err)
other.Release()
}
diff --git a/internal/gitaly/service/operations/land_verify.go b/internal/gitaly/service/operations/land_verify.go
index b4a93b2..98e62b0 100644
--- a/internal/gitaly/service/operations/land_verify.go
+++ b/internal/gitaly/service/operations/land_verify.go
@@ -18,6 +18,9 @@ import (
"strings"
"syscall"
+ "github.com/prometheus/client_golang/prometheus"
+ "github.com/prometheus/client_golang/prometheus/promauto"
+
"github.com/feder-cr/git-ai/internal/command"
"github.com/feder-cr/git-ai/internal/git"
"github.com/feder-cr/git-ai/internal/git/gitcmd"
@@ -39,6 +42,35 @@ import (
// landVerifyOutputBytes bounds the output of the command kept as the reason.
const landVerifyOutputBytes = 4 << 10
+// landVerifyCheckoutFaults counts the checkouts that failed on the server's own file system.
+var landVerifyCheckoutFaults = promauto.NewCounter(prometheus.CounterOpts{
+ Name: "graft_land_verify_checkout_faults_total",
+ Help: "Checkouts of a tree for the verification that failed on the file system of the server (no space, no descriptors, an I/O error, no permission).",
+})
+
+// landVerifyCleanupFailures counts the verification directories that could not be removed.
+var landVerifyCleanupFailures = promauto.NewCounter(prometheus.CounterOpts{
+ Name: "graft_land_verify_cleanup_failures_total",
+ Help: "Verification directories that could not be removed after the verification, even after making the tree writable.",
+})
+
+// landRemoveCheckout removes the directory of a verification. The operator's command runs in it and
+// may leave directories that cannot be removed (read-only, no execute bit): the tree is made writable
+// first, directory by directory before they are read. A directory that still cannot be removed is
+// logged and counted, and leaves a trace in the log: it would otherwise fill the disk unseen.
+func (s *Server) landRemoveCheckout(dir string) {
+ _ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+ if d != nil && d.IsDir() {
+ _ = os.Chmod(p, 0o700)
+ }
+ return nil
+ })
+ if err := os.RemoveAll(dir); err != nil {
+ landVerifyCleanupFailures.Inc()
+ s.logger.WithError(err).WithField("directory", dir).Error("graft: a verification directory could not be removed")
+ }
+}
+
// landVerifyError is a resolution the verification rejected.
type landVerifyError struct {
reason string
@@ -57,11 +89,11 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.
return err
}
- dir, err := os.MkdirTemp("", "graft-verify-")
+ dir, err := os.MkdirTemp(s.verifyTmpDir, "graft-verify-")
if err != nil {
return fmt.Errorf("verification directory: %w", err)
}
- defer func() { _ = os.RemoveAll(dir) }()
+ defer s.landRemoveCheckout(dir)
checkout, home := filepath.Join(dir, "tree"), filepath.Join(dir, "home")
for _, d := range []string{checkout, home} {
if err := os.Mkdir(d, 0o700); err != nil {
@@ -79,8 +111,9 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.
depth: v.MaxTreeDepthOrDefault(),
pathBytes: landMaxPathBytes,
}
- if err := landCheckoutTree(runCtx, repo, tree, checkout, limits); err != nil {
+ if err := landCheckoutTreeThen(runCtx, repo, tree, checkout, limits, s.verifyAfterWriteHook); err != nil {
var refusal *landCheckoutRefusal
+ var fault *landCheckoutFault
switch {
case ctx.Err() != nil:
return ctx.Err()
@@ -88,6 +121,11 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.
return &landVerifyError{reason: fmt.Sprintf("the checkout of the tree and the command did not finish within %s", timeout)}
case errors.As(err, &refusal):
return &landVerifyError{reason: "nothing was verified, the merged tree is not checked out: " + refusal.reason}
+ case errors.As(err, &fault):
+ // The server's own file system failed: its log has the path, the agent gets none.
+ s.logger.WithError(err).Error("graft: the checkout of a tree to verify failed on the file system of the server")
+ landVerifyCheckoutFaults.Inc()
+ return errors.New("checking out the merged tree failed on the file system of the server")
}
return fmt.Errorf("checking out the merged tree: %w", err)
}
@@ -141,7 +179,7 @@ func landVerifyRepoKey(repository *gitalypb.Repository) string {
// out, a *landVerifyUnfinishedError when ctx ended first, and another error when the verification
// could not be run.
func (s *Server) landVerifyOutOfLane(ctx context.Context, repository *gitalypb.Repository, repo *localrepo.Repo, tree git.ObjectID) error {
- slot, err := s.conflictResolver.AcquireVerifySlot(ctx, landVerifyRepoKey(repository))
+ slot, err := s.verifyPool.Acquire(ctx, landVerifyRepoKey(repository))
if err != nil {
if ctx.Err() != nil {
return &landVerifyUnfinishedError{err: err}
@@ -174,15 +212,14 @@ func (s *Server) landVerifyTrailer() string {
}
// landTreeEntry is one entry of a tree as git-ls-tree(1) -r lists it: a file, a symbolic link or a
-// gitlink (a submodule), by path. size and target are filled when it is written: the length of the
-// file, the target of the link. sized and listed are the size that the listing gave (ls-tree -l), when
-// it did.
+// gitlink (a submodule), by path. size is the length of the file once it is written. The target of a
+// link is not kept: the read-back hashes what the link says and compares it with the OID. sized and
+// listed are the size that the listing gave (ls-tree -l), when it did.
type landTreeEntry struct {
mode uint32
oid git.ObjectID
path string
size int64
- target string
sized bool
listed int64
}
@@ -192,8 +229,9 @@ const (
landTreeModeRegular = 0o100000
landTreeModeSymlink = 0o120000
landTreeModeGitlink = 0o160000
- // landMaxLinkTarget bounds the target of a symbolic link, which is read into memory.
- landMaxLinkTarget = 64 << 10
+ // landMaxLinkTarget is the longest target of a symbolic link that Linux takes (PATH_MAX - 1): a
+ // longer one is refused up front, with the reason. Nothing of it is kept after the link is made.
+ landMaxLinkTarget = 4095
// landMaxTreePath bounds a path of the tree, and with it the record that is read of the listing.
landMaxTreePath = 4096
// landNameMax and landPathMax are the limits of a name and of a whole path on the file systems
@@ -206,11 +244,19 @@ func (e landTreeEntry) isGitlink() bool { return e.mode&landTreeModeTypeMask ==
func (e landTreeEntry) isSymlink() bool { return e.mode&landTreeModeTypeMask == landTreeModeSymlink }
func (e landTreeEntry) isRegular() bool { return e.mode&landTreeModeTypeMask == landTreeModeRegular }
-// landMaxPathBytes bounds the bytes of the distinct paths a checkout keeps in memory: of every entry
-// and of every directory made, each charged twice (the sets that look for links and for names that a
-// case-insensitive file system folds together). A few thousand bytes of path times many entries is
-// where a small push could otherwise take the memory of the server.
-const landMaxPathBytes = 128 << 20
+// landMaxPathBytes bounds the memory a checkout keeps for the paths it has seen, counted as the
+// materializer charges it, exactly: an entry costs its path plus landEntryOverhead (the entries kept
+// for the read-back); every distinct prefix of a path in the set of folded names costs twice its
+// length (the folded key and the spelling) plus landKeyOverhead; every directory made costs its path
+// plus landKeyOverhead (the set of directories); every symbolic link costs its path plus
+// landKeyOverhead (the set of links). Nothing else of the tree is kept: not the target of a link, not
+// the content of a file. A few thousand bytes of path times many entries is where a small push could
+// otherwise take the memory of the server.
+const (
+ landMaxPathBytes = 128 << 20
+ landEntryOverhead = 256
+ landKeyOverhead = 64
+)
// landCheckoutLimits bound a checkout; a zero field is no limit (the server always sets them, from
// verify.max_tree_entries, verify.max_tree_bytes, verify.max_tree_depth and landMaxPathBytes).
@@ -329,7 +375,7 @@ func landCheckoutTreeThen(ctx context.Context, repo *localrepo.Repo, tree git.Ob
if afterWrite != nil {
afterWrite()
}
- return landCheckMaterialized(entries, dir)
+ return landCheckMaterialized(ctx, entries, dir)
}
// landFeed reads the entries of the listing (ls-tree -r -z -l), asks cat-file for the blob of each
@@ -469,57 +515,88 @@ type landMaterializer struct {
folded map[string]landFolded // the paths as a case-insensitive file system sees them
count int // entries written, and directories made
bytes int64 // bytes of files and links written
- paths int64 // bytes of the paths kept in memory
+ paths int64 // memory of the paths kept, as landMaxPathBytes counts it
}
func newLandMaterializer(ctx context.Context, dir string, limits landCheckoutLimits) *landMaterializer {
return &landMaterializer{ctx: ctx, dir: dir, limits: limits, links: map[string]bool{}, dirs: map[string]bool{}, folded: map[string]landFolded{}}
}
-// landOSError is the refusal for what the file system would not do: a name too long for it, a name
-// that exists already (two names a case-insensitive file system makes one), no room. The tree is
-// not one this server can write, which is the agent's to know, not an error of the server.
-func landOSError(err error) error {
+// landCheckoutFault is a failure of the server's own file system while it wrote the tree (no space,
+// no descriptors, an I/O error, no permission): not the tree's doing, so no refusal and no proposal,
+// an error of the server with a log and a metric.
+type landCheckoutFault struct {
+ err error
+}
+
+func (e *landCheckoutFault) Error() string {
+ return "checkout failed on the file system: " + e.err.Error()
+}
+func (e *landCheckoutFault) Unwrap() error { return e.err }
+
+// osError sorts what the file system said. What the TREE causes is a refusal with its reason: a name
+// that exists already (two names a case-insensitive file system makes one), a name or a path too
+// long, an illegal byte sequence, a link in the way, a file where a directory is needed. What the
+// SERVER's file system causes is a fault: no space, no descriptors, an I/O error, no permission. The
+// reason names the path relative to the checkout, never the directory of the server.
+func (m *landMaterializer) osError(err error) error {
if err == nil {
return nil
}
- return landRefuse("the tree cannot be written on this file system: %v", err)
+ var errno syscall.Errno
+ if errors.As(err, &errno) {
+ switch errno {
+ case syscall.EEXIST, syscall.ENAMETOOLONG, syscall.EILSEQ, syscall.ELOOP, syscall.ENOTDIR, syscall.EISDIR, syscall.ENOTEMPTY, syscall.EINVAL:
+ return landRefuse("the tree cannot be written on this file system: %s", m.describe(err, errno))
+ }
+ }
+ return &landCheckoutFault{err: err}
}
-// landOSWriter makes the errors of writing a file refusals, and not the errors of reading git.
-type landOSWriter struct{ w io.Writer }
+// describe words a system error with the operation and the path relative to the checkout.
+func (m *landMaterializer) describe(err error, errno syscall.Errno) string {
+ op, p := "", ""
+ var pe *fs.PathError
+ var le *os.LinkError
+ switch {
+ case errors.As(err, &pe):
+ op, p = pe.Op, pe.Path
+ case errors.As(err, &le):
+ op, p = le.Op, le.New
+ }
+ rel, ok := strings.CutPrefix(p, m.dir+string(filepath.Separator))
+ if !ok {
+ return strings.TrimSpace(op + ": " + errno.Error())
+ }
+ return op + " " + rel + ": " + errno.Error()
+}
-func (o landOSWriter) Write(p []byte) (int, error) {
- n, err := o.w.Write(p)
- return n, landOSError(err)
+// landOSWriter sorts the errors of writing a file, and leaves the errors of reading git alone.
+type landOSWriter struct {
+ m *landMaterializer
+ w io.Writer
}
-// landMaterialize writes the entries under dir, reading the contents of the files and links from batch
-// in their order (the answer of git-cat-file(1) --batch to the blobs of the listing, gitlinks left
-// out). See landMaterializer.add for what it refuses. It sets size and target of the entries.
-func landMaterialize(entries []landTreeEntry, batch *bufio.Reader, dir string) error {
- m := newLandMaterializer(context.Background(), dir, landCheckoutLimits{})
- for i := range entries {
- if err := m.add(&entries[i], batch); err != nil {
- return err
- }
- }
- return nil
+func (o landOSWriter) Write(p []byte) (int, error) {
+ n, err := o.w.Write(p)
+ return n, o.m.osError(err)
}
// add writes one entry. It refuses, with a *landCheckoutRefusal:
// - a path that is not clean, or leaves dir;
-// - a path longer than landMaxTreePath, more entries or more bytes than the limits allow;
+// - a path longer than landMaxTreePath, or that with dir reaches PATH_MAX; a name over 255 bytes;
+// - more entries and directories, more bytes or components than the limits allow;
// - a component that names .git on some file system (resolver.IsGitDirName): the operator's
// command may run git in the checkout, and a .git of the tree would be its repository, whose
// configuration and hooks run;
// - an entry under a symbolic link that was written, also when a case-insensitive file system
// would see the same name (a link "A" and an entry "a/x");
-// - a parent that is not a directory.
+// - a parent that is not a directory; a target of a link over 4095 bytes or with a NUL in it.
//
// Each directory is made one component at a time and looked at with Lstat, never with MkdirAll,
// which would follow a link; a file is created with O_EXCL and O_NOFOLLOW; a link is written as a
-// link and never followed.
+// link and never followed. The memory it keeps for the paths is charged before it is taken
+// (landMaxPathBytes).
func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
if err := m.ctx.Err(); err != nil {
return err
@@ -538,18 +615,18 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
if e.sized && !e.isGitlink() && m.limits.bytes > 0 && m.bytes+e.listed > m.limits.bytes {
return landRefuse("the tree has more than %d bytes of files (verify.max_tree_bytes)", m.limits.bytes)
}
-
- parts := strings.Split(e.path, "/")
- if m.limits.depth > 0 && len(parts) > m.limits.depth {
- return landRefuse("the tree has a path of %d components, more than %d (verify.max_tree_depth)", len(parts), m.limits.depth)
+ if e.isSymlink() && e.sized && e.listed > landMaxLinkTarget {
+ return landRefuse("the symbolic link %q has a target of %d bytes, more than the %d that Linux takes", e.path, e.listed, landMaxLinkTarget)
}
- if !e.isGitlink() {
- // An entry is kept in memory by its path, twice (the entries and the folded names).
- if err := m.chargePath(2 * int64(len(e.path))); err != nil {
- return err
+
+ if m.limits.depth > 0 {
+ if depth := strings.Count(e.path, "/") + 1; depth > m.limits.depth {
+ return landRefuse("the tree has a path of %d components, more than %d (verify.max_tree_depth)", depth, m.limits.depth)
}
}
- for _, c := range parts {
+ for rest := e.path; rest != ""; {
+ c, tail, _ := strings.Cut(rest, "/")
+ rest = tail
if len(c) > landNameMax {
return landRefuse("the path %q has a name of more than %d bytes", e.path, landNameMax)
}
@@ -562,21 +639,53 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
return landRefuse("entry %q lies under the symbolic link %q", e.path, p)
}
}
- for i := range parts {
- prefix := strings.Join(parts[:i+1], "/")
- link := i == len(parts)-1 && e.isSymlink()
+ if err := m.chargePath(int64(len(e.path)) + landEntryOverhead); err != nil {
+ return err
+ }
+
+ // The names as a case-insensitive file system sees them: a collision with a link is refused here,
+ // and the new names are charged now but entered only after the directories are made and charged.
+ type foldedName struct {
+ key string
+ landFolded
+ }
+ var fresh []foldedName
+ for end := 0; end < len(e.path); {
+ next := strings.IndexByte(e.path[end:], '/')
+ if next < 0 {
+ end = len(e.path)
+ } else {
+ end += next
+ }
+ prefix := e.path[:end]
+ link := end == len(e.path) && e.isSymlink()
f := resolver.FoldGitName(prefix)
prev, seen := m.folded[f]
switch {
case !seen:
- m.folded[f] = landFolded{spelling: prefix, link: link}
+ if err := m.chargePath(2*int64(len(prefix)) + landKeyOverhead); err != nil {
+ return err
+ }
+ fresh = append(fresh, foldedName{key: f, landFolded: landFolded{spelling: prefix, link: link}})
case prev.spelling != prefix && (prev.link || link):
return landRefuse("%q and %q are one name on a case-insensitive file system, and one is a symbolic link", e.path, prev.spelling)
}
+ end++
+ }
+ enter := func() {
+ for _, n := range fresh {
+ m.folded[n.key] = n.landFolded
+ }
}
+ dir, _ := path.Split(e.path)
+ dir = strings.TrimSuffix(dir, "/")
if e.isGitlink() {
- return m.mkdirAll(parts)
+ if err := m.mkdirAll(e.path); err != nil {
+ return err
+ }
+ enter()
+ return nil
}
if m.dirs[e.path] {
return landRefuse("the entry %q is also a directory of the tree", e.path)
@@ -591,13 +700,14 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
if m.bytes += size; m.limits.bytes > 0 && m.bytes > m.limits.bytes {
return landRefuse("the tree has more than %d bytes of files (verify.max_tree_bytes)", m.limits.bytes)
}
- if err := m.mkdirAll(parts[:len(parts)-1]); err != nil {
+ if err := m.mkdirAll(dir); err != nil {
return err
}
+ enter()
target := filepath.Join(m.dir, filepath.FromSlash(e.path))
if e.isSymlink() {
if size > landMaxLinkTarget {
- return landRefuse("the symbolic link %q has a target of %d bytes", e.path, size)
+ return landRefuse("the symbolic link %q has a target of %d bytes, more than the %d that Linux takes", e.path, size, landMaxLinkTarget)
}
content := make([]byte, size)
if _, err := io.ReadFull(batch, content); err != nil {
@@ -606,10 +716,13 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
if bytes.IndexByte(content, 0) >= 0 {
return landRefuse("the symbolic link %q has a NUL in its target", e.path)
}
+ if err := m.chargePath(int64(len(e.path)) + landKeyOverhead); err != nil {
+ return err
+ }
if err := os.Symlink(string(content), target); err != nil {
- return landOSError(err)
+ return m.osError(err)
}
- e.target, e.size = string(content), size
+ e.size = size
m.links[e.path] = true
} else {
perm := os.FileMode(0o644)
@@ -618,22 +731,23 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
}
f, err := os.OpenFile(target, os.O_WRONLY|os.O_CREATE|os.O_EXCL|syscall.O_NOFOLLOW, perm)
if err != nil {
- return landOSError(err)
+ return m.osError(err)
}
- if _, err := io.CopyN(landOSWriter{f}, batch, size); err != nil {
+ if _, err := io.CopyN(landOSWriter{m: m, w: f}, batch, size); err != nil {
_ = f.Close()
var refusal *landCheckoutRefusal
- if errors.As(err, &refusal) {
+ var fault *landCheckoutFault
+ if errors.As(err, &refusal) || errors.As(err, &fault) {
return err
}
return fmt.Errorf("%q: %w", e.path, err)
}
if err := f.Close(); err != nil {
- return landOSError(err)
+ return m.osError(err)
}
// The mode is the tree's, whatever the umask of the process.
if err := os.Chmod(target, perm); err != nil {
- return landOSError(err)
+ return m.osError(err)
}
e.size = size
}
@@ -651,7 +765,7 @@ func (m *landMaterializer) charge(n int) error {
return nil
}
-// chargePath counts n bytes of paths kept in memory against landMaxPathBytes.
+// chargePath counts n bytes of the memory kept for paths against landMaxPathBytes.
func (m *landMaterializer) chargePath(n int64) error {
if m.paths += n; m.limits.pathBytes > 0 && m.paths > m.limits.pathBytes {
return landRefuse("the paths of the tree take more than %d bytes of memory", m.limits.pathBytes)
@@ -659,32 +773,38 @@ func (m *landMaterializer) chargePath(n int64) error {
return nil
}
-// mkdirAll makes the directory of the given components under dir, one at a time. A component that
-// exists must be a directory by Lstat, which does not follow a link: a link, or a file, in the way
-// is a refusal.
-func (m *landMaterializer) mkdirAll(parts []string) error {
- current := m.dir
- for i, c := range parts {
- current = filepath.Join(current, c)
- key := strings.Join(parts[:i+1], "/")
+// mkdirAll makes the directory rel (slash-separated, relative to dir, empty for dir itself), one
+// component at a time. A component that exists must be a directory by Lstat, which does not follow a
+// link: a link, or a file, in the way is a refusal. Each directory made is charged first.
+func (m *landMaterializer) mkdirAll(rel string) error {
+ for end := 0; end < len(rel); {
+ next := strings.IndexByte(rel[end:], '/')
+ if next < 0 {
+ end = len(rel)
+ } else {
+ end += next
+ }
+ key := rel[:end]
+ end++
if m.dirs[key] {
continue
}
+ current := filepath.Join(m.dir, filepath.FromSlash(key))
info, err := os.Lstat(current)
switch {
case errors.Is(err, fs.ErrNotExist):
- // A directory is an inode and a path kept in memory, charged like the entries.
+ // A directory is an inode and a path kept in memory, charged before it is made.
if err := m.charge(1); err != nil {
return err
}
- if err := m.chargePath(2 * int64(len(key))); err != nil {
+ if err := m.chargePath(int64(len(key)) + landKeyOverhead); err != nil {
return err
}
if err := os.Mkdir(current, 0o755); err != nil {
- return landOSError(err)
+ return m.osError(err)
}
case err != nil:
- return landOSError(err)
+ return m.osError(err)
case !info.IsDir():
return landRefuse("%q is in the way of a directory of the tree, and is not a directory", key)
}
@@ -725,12 +845,27 @@ func landBlobOID(r io.Reader, size int64, like git.ObjectID) (git.ObjectID, erro
return git.ObjectID(hex.EncodeToString(h.Sum(nil))), nil
}
+// landCtxReader stops a read when its context has ended, so that the hashing of a large tree is
+// bounded by the deadline too.
+type landCtxReader struct {
+ ctx context.Context
+ r io.Reader
+}
+
+func (c landCtxReader) Read(p []byte) (int, error) {
+ if err := c.ctx.Err(); err != nil {
+ return 0, err
+ }
+ return c.r.Read(p)
+}
+
// landCheckMaterialized reads dir back and compares it with the entries that were written: every entry
-// is there with its type, its executable bit, the length of its file or the target of its link; the
-// content of every file and every link hashes, as a git blob, to the object ID of the tree; a
-// gitlink is an empty directory; and nothing else is there but the directories that hold the
-// entries. A difference is an error.
-func landCheckMaterialized(entries []landTreeEntry, dir string) error {
+// is there with its type and its executable bit; the content of every file and the target of every
+// link hashes, as a git blob, to the object ID of the tree (the target of a link is not kept: what
+// the link says now is hashed); a gitlink is an empty directory; and nothing else is there but the
+// directories that hold the entries. A difference is an error. The context is looked at at every
+// entry and while a file is hashed: when it ends, the error is the context's, not a difference.
+func landCheckMaterialized(ctx context.Context, entries []landTreeEntry, dir string) error {
want := make(map[string]*landTreeEntry, len(entries))
parents := map[string]bool{}
for i := range entries {
@@ -745,6 +880,9 @@ func landCheckMaterialized(entries []landTreeEntry, dir string) error {
if err != nil {
return err
}
+ if err := ctx.Err(); err != nil {
+ return err
+ }
rel, err := filepath.Rel(dir, full)
if err != nil {
return err
@@ -765,8 +903,8 @@ func landCheckMaterialized(entries []landTreeEntry, dir string) error {
}
case isEntry && e.isSymlink():
got, err := os.Readlink(full)
- if err != nil || info.Mode()&os.ModeSymlink == 0 || got != e.target {
- return fmt.Errorf("the symbolic link %q does not point to %q", name, e.target)
+ if err != nil || info.Mode()&os.ModeSymlink == 0 {
+ return fmt.Errorf("%q is not the symbolic link of the tree", name)
}
if oid, err := landBlobOID(strings.NewReader(got), int64(len(got)), e.oid); err != nil || oid != e.oid {
return fmt.Errorf("the symbolic link %q is not the blob %s of the tree", name, e.oid)
@@ -779,8 +917,11 @@ func landCheckMaterialized(entries []landTreeEntry, dir string) error {
if err != nil {
return err
}
- oid, err := landBlobOID(f, e.size, e.oid)
+ oid, err := landBlobOID(landCtxReader{ctx: ctx, r: f}, e.size, e.oid)
_ = f.Close()
+ if ctx.Err() != nil {
+ return ctx.Err()
+ }
if err != nil || oid != e.oid {
return fmt.Errorf("the content of the file %q is not the blob %s of the tree", name, e.oid)
}
@@ -794,6 +935,9 @@ func landCheckMaterialized(entries []landTreeEntry, dir string) error {
seen++
return nil
})
+ if ctxErr := ctx.Err(); ctxErr != nil {
+ return ctxErr
+ }
if err != nil {
return fmt.Errorf("the checkout differs from the tree: %w", err)
}
diff --git a/internal/gitaly/service/operations/land_verify_pool_test.go b/internal/gitaly/service/operations/land_verify_pool_test.go
new file mode 100644
index 0000000..183a219
--- /dev/null
+++ b/internal/gitaly/service/operations/land_verify_pool_test.go
@@ -0,0 +1,132 @@
+package operations
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
+)
+
+// Tests of PLAN10 15.9 after the ruling that every verification runs out of the lane: whatever made the
+// resolution (a rule, an agent, a model), verify.command runs on the exact tree of the land, with a
+// slot of the process's pool, before the lane; the lane compares the tree it would write with the one that
+// passed, and runs nothing.
+
+// heldPool starts a service whose verifications go through a pool the test can hold: it returns the
+// service's own Server, to take the slot of the fixture's repository with.
+func heldPool(t *testing.T, rcfg config.GraftResolver) (resolveFixture, *Server) {
+ t.Helper()
+
+ var s *Server
+ operationServerHooks.Store(t.Name(), func(served *Server) { s = served })
+ fx := newResolveFixture(t, rcfg)
+ require.NotNil(t, s)
+ require.NotNil(t, s.verifyPool, "a pool whenever verify.enabled is")
+ return fx, s
+}
+
+// TestUserLand_aRulesLandIsVerifiedThroughThePool: the rules' resolution of a land needs the verification
+// too, out of the lane and through the pool of the process: with the repository's slot taken, the
+// verification cannot start before the deadline, the command never runs, nothing lands, and the rules'
+// files are NOT_ATTEMPTED with the reason (the agent gets the package).
+func TestUserLand_aRulesLandIsVerifiedThroughThePool(t *testing.T) {
+ t.Parallel()
+
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, validAnswer())
+ vdir, vargv := countingVerify(t, "exit 0")
+ rcfg := withVerify(enabledResolver(rargv), vargv, "1s")
+ rcfg.Timeout = "1s"
+ fx, s := heldPool(t, rcfg)
+
+ slot, err := s.verifyPool.Acquire(fx.ctx, landVerifyRepoKey(fx.repoProto))
+ require.NoError(t, err)
+ _, err = fx.client.UserLand(fx.ctx, fx.request("Land"))
+ requireMergeConflict(t, err)
+ pkg := landConflictDetail(err)
+ require.NotNil(t, pkg, "the package, not a bare deadline")
+ for _, p := range []string{"a", "b"} {
+ f := packageFile(t, pkg, p)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_NOT_ATTEMPTED, f.GetOutcome(), p)
+ require.Contains(t, f.GetReason(), "could not verify before the deadline of 2s", p)
+ }
+ require.Equal(t, 0, verifyRuns(t, vdir), "the command did not run: the slot of its repository was taken")
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)
+
+ // With the slot free the same land passes the verification and lands, signed by what ran.
+ slot.Release()
+ resp, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ require.NoError(t, err)
+ commit, _ := landedCommit(t, fx, resp)
+ require.Contains(t, string(commit.GetBody()), "Graft-Checked-By: verify:verify.sh\n")
+ require.Equal(t, 1, verifyRuns(t, vdir))
+}
+
+// TestUserLand_anAgentsResolutionIsVerifiedThroughThePool: the same for the resolution of an agent, with no
+// AI resolver at all: its verification runs before the lane with a slot of the pool, and a deadline ends the
+// land with the package and the reason, the command not run.
+func TestUserLand_anAgentsResolutionIsVerifiedThroughThePool(t *testing.T) {
+ t.Parallel()
+
+ vdir, vargv := countingVerify(t, "grep -qx agent-a a")
+ rcfg := withVerify(config.GraftResolver{Timeout: "1s"}, vargv, "1s")
+ fx, s := heldPool(t, rcfg)
+
+ slot, err := s.verifyPool.Acquire(fx.ctx, landVerifyRepoKey(fx.repoProto))
+ require.NoError(t, err)
+ req := fx.request("Land")
+ req.Resolution = resolutionOf(fx, "agent-a\n", "agent-b\n")
+ _, err = fx.client.UserLand(fx.ctx, req)
+ pkg := landConflictDetail(err)
+ require.NotNil(t, pkg)
+ require.Contains(t, pkg.GetReason(), "the resolution could not be verified: could not verify before the deadline of 2s")
+ require.Equal(t, 0, verifyRuns(t, vdir))
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)
+
+ slot.Release()
+ resp, err := fx.client.UserLand(fx.ctx, req)
+ require.NoError(t, err)
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, git.ObjectID(resp.GetLandedCommitId()))
+ require.Equal(t, 1, verifyRuns(t, vdir))
+}
+
+// TestUserLand_aRulesLandWhoseTipMovedIsVerifiedAgain: the tree is what is verified. While the first
+// verification of a rules land runs, a writer adds a file to the branch: the tree to land is another one,
+// the lane hands the land back, the same verdicts are verified on the new tree out of the lane (the resolver
+// is not called again), and that is what lands.
+func TestUserLand_aRulesLandWhoseTipMovedIsVerifiedAgain(t *testing.T) {
+ t.Parallel()
+
+ fx, rdir, vdir := verifiedFixture(t, validAnswer(), moveBranchBody, nil)
+ writer := gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.tip),
+ gittest.WithTreeEntries(entries("a", "ours\n", "b", "ours\n", "c", "c\n", "e", "e\n")...))
+ writeVerifyMoves(t, vdir, fx, writer)
+
+ resp, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ require.NoError(t, err)
+ commit, landed := landedCommit(t, fx, resp)
+ require.Equal(t, []string{writer.String(), fx.commit.String()}, commit.GetParentIds())
+ require.Equal(t, "e\n", fx.blobAt(t, landed, "e"))
+ require.Equal(t, 1, resolverCalls(t, rdir), "no resolver call: the files did not change")
+ require.Equal(t, 2, verifyRuns(t, vdir), "the tree of the new tip is verified, out of the lane")
+}
+
+// TestUserLand_aFailingVerificationRejectsTheRulesFiles: a command that fails on the tree of a rules land
+// rejects its files with the output, before the lane, and the branch stays.
+func TestUserLand_aFailingVerificationRejectsTheRulesFiles(t *testing.T) {
+ t.Parallel()
+
+ fx, _, vdir := verifiedFixture(t, validAnswer(), `echo "2 tests failed"; exit 3`, nil)
+ _, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ requireMergeConflict(t, err)
+ for _, f := range landConflictDetail(err).GetFiles() {
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_REJECTED, f.GetOutcome())
+ require.Equal(t, "verification failed: exit status 3: 2 tests failed", f.GetReason())
+ }
+ require.Equal(t, 1, verifyRuns(t, vdir))
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)
+}
diff --git a/internal/gitaly/service/operations/land_verify_test.go b/internal/gitaly/service/operations/land_verify_test.go
index 49faf12..688f303 100644
--- a/internal/gitaly/service/operations/land_verify_test.go
+++ b/internal/gitaly/service/operations/land_verify_test.go
@@ -5,10 +5,12 @@ import (
"bufio"
"bytes"
"context"
+ "errors"
"fmt"
"io"
"os"
"path/filepath"
+ "runtime"
"strings"
"testing"
"time"
@@ -317,7 +319,7 @@ func TestLandCheckMaterialized_failsClosed(t *testing.T) {
entries, contents := fakeEntries(t)
dir := t.TempDir()
require.NoError(t, landMaterialize(entries, batchOf(contents, entries), dir))
- require.NoError(t, landCheckMaterialized(entries, dir), "an exact checkout passes")
+ require.NoError(t, landCheckMaterialized(context.Background(), entries, dir), "an exact checkout passes")
return entries, dir
}
@@ -361,7 +363,7 @@ func TestLandCheckMaterialized_failsClosed(t *testing.T) {
} {
entries, dir := materialized(t)
damage(t, dir)
- require.ErrorContains(t, landCheckMaterialized(entries, dir), "the checkout differs from the tree", name)
+ require.ErrorContains(t, landCheckMaterialized(context.Background(), entries, dir), "the checkout differs from the tree", name)
}
}
@@ -428,6 +430,19 @@ func TestUserLand_verifySeesTheWholeTree(t *testing.T) {
}
}
+// landMaterialize writes the entries under dir without limits, reading the contents of the files and
+// links from batch in their order (the answer of git-cat-file(1) --batch to the blobs of the listing,
+// gitlinks left out).
+func landMaterialize(entries []landTreeEntry, batch *bufio.Reader, dir string) error {
+ m := newLandMaterializer(context.Background(), dir, landCheckoutLimits{})
+ for i := range entries {
+ if err := m.add(&entries[i], batch); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
// fakeTreeEntry is an entry of a tree that landMaterialize writes, with the content that git-cat-file(1)
// would answer for it.
type fakeTreeEntry struct {
@@ -775,17 +790,19 @@ func TestLandMaterialize_refusesWhatTheFileSystemWouldNot(t *testing.T) {
}
for name, tc := range map[string]struct {
dir func(t *testing.T) string
+ prepare func(t *testing.T, dir string)
entries []fakeTreeEntry
reason string
}{
"a name of more than 255 bytes": {entries: []fakeTreeEntry{{reg, longName, "x"}}, reason: "more than 255 bytes"},
"a path that reaches PATH_MAX": {entries: []fakeTreeEntry{{reg, strings.Join(longPath, "/"), "x"}}, reason: "too long"},
"a NUL in the target of a link": {entries: []fakeTreeEntry{{link, "l", "a\x00b"}}, reason: "NUL"},
- "a system call that fails": {
- dir: func(t *testing.T) string { return filepath.Join(t.TempDir(), "missing") },
- entries: []fakeTreeEntry{{reg, "a/b", "x"}},
- reason: "cannot be written on this file system",
+ "a name that exists already": {
+ prepare: func(t *testing.T, dir string) { require.NoError(t, os.WriteFile(filepath.Join(dir, "a"), nil, 0o644)) },
+ entries: []fakeTreeEntry{{reg, "a", "x"}},
+ reason: "open a: file exists",
},
+ "a link target over 4095 bytes": {entries: []fakeTreeEntry{{link, "l", strings.Repeat("t", 4096)}}, reason: "more than the 4095 that Linux takes"},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
@@ -794,12 +811,38 @@ func TestLandMaterialize_refusesWhatTheFileSystemWouldNot(t *testing.T) {
if tc.dir != nil {
dir = tc.dir(t)
}
+ if tc.prepare != nil {
+ tc.prepare(t, dir)
+ }
var refusal *landCheckoutRefusal
require.ErrorAs(t, materializeFake(t, dir, tc.entries...), &refusal)
require.Contains(t, refusal.reason, tc.reason)
+ require.NotContains(t, refusal.reason, dir, "a reason names the path inside the checkout, not the directory of the server")
})
}
+ // What the server's own file system does is no refusal: a fault, for the log and a metric, never a
+ // proposal or a rejection of the agent's files.
+ t.Run("a failure of the server's file system is a fault", func(t *testing.T) {
+ t.Parallel()
+
+ err := materializeFake(t, filepath.Join(t.TempDir(), "missing"), fakeTreeEntry{reg, "a/b", "x"})
+ var fault *landCheckoutFault
+ var refusal *landCheckoutRefusal
+ require.ErrorAs(t, err, &fault)
+ require.False(t, errors.As(err, &refusal))
+ })
+
+ t.Run("the link target at the limit is written", func(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ require.NoError(t, materializeFake(t, dir, fakeTreeEntry{link, "l", strings.Repeat("t", 4095)}))
+ got, err := os.Readlink(filepath.Join(dir, "l"))
+ require.NoError(t, err)
+ require.Len(t, got, 4095)
+ })
+
t.Run("a context that ended stops the next entry", func(t *testing.T) {
t.Parallel()
@@ -810,3 +853,134 @@ func TestLandMaterialize_refusesWhatTheFileSystemWouldNot(t *testing.T) {
require.ErrorIs(t, m.add(&entries[0], batchOf(contents, entries)), context.Canceled)
})
}
+
+// TestLandMaterialize_theDirectoryClauseOfPathMax: a path that is short itself but with the directory
+// of the checkout reaches PATH_MAX is refused up front with its own reason, not left to fail on the
+// system call.
+func TestLandMaterialize_theDirectoryClauseOfPathMax(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ for i := 0; i < 15; i++ {
+ dir = filepath.Join(dir, strings.Repeat("d", 250))
+ require.NoError(t, os.Mkdir(dir, 0o755))
+ }
+ // The path is 300 bytes in names of at most 100: far from the limits of its own, over PATH_MAX with the directory.
+ long := strings.Join([]string{strings.Repeat("a", 100), strings.Repeat("b", 100), strings.Repeat("c", 98)}, "/")
+ require.Less(t, len(long), landMaxTreePath)
+ require.GreaterOrEqual(t, len(dir)+1+len(long), landPathMax)
+
+ var refusal *landCheckoutRefusal
+ require.ErrorAs(t, materializeFake(t, dir, fakeTreeEntry{0o100644, long, "x"}), &refusal)
+ require.Contains(t, refusal.reason, "too long to write", "refused by the clause of the directory, not by the system call")
+}
+
+// TestLandCheckoutTree_aSymlinkHeavyTreeStaysSmall: the targets of the links are not kept. A tree of
+// 20000 links to one 4095-byte blob would hold 80 MB of targets in the entries; what is kept is the
+// paths, which is a fraction of it. The heap is read while the checkout is complete and not yet
+// checked, when everything it keeps is live. Not parallel: the heap is the process's.
+func TestLandCheckoutTree_aSymlinkHeavyTreeStaysSmall(t *testing.T) {
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})
+ repo := localrepo.NewTestRepo(t, cfg, repoProto)
+
+ blob := gittest.WriteBlob(t, cfg, repoPath, []byte(strings.Repeat("a", 4095)))
+ var links []gittest.TreeEntry
+ for i := 0; i < 1000; i++ {
+ links = append(links, gittest.TreeEntry{Mode: "120000", Path: fmt.Sprintf("l%04d", i), OID: blob})
+ }
+ sub := gittest.WriteTree(t, cfg, repoPath, links)
+ var root []gittest.TreeEntry
+ for i := 0; i < 20; i++ {
+ root = append(root, gittest.TreeEntry{Mode: "040000", Path: fmt.Sprintf("n%02d", i), OID: sub})
+ }
+ tree := gittest.WriteTree(t, cfg, repoPath, root)
+
+ heap := func() uint64 {
+ var m runtime.MemStats
+ runtime.GC()
+ runtime.ReadMemStats(&m)
+ return m.HeapAlloc
+ }
+ before := heap()
+ var during uint64
+ dir := t.TempDir()
+ require.NoError(t, landCheckoutTreeThen(ctx, repo, tree, dir, landCheckoutLimits{entries: 100000, depth: 64, pathBytes: landMaxPathBytes}, func() { during = heap() }))
+ grew := int64(during) - int64(before)
+ t.Logf("heap while the checkout is complete: +%d KiB for 20000 links of 4095 bytes", grew/1024)
+ require.Less(t, grew, int64(24<<20), "the targets of the links are not kept")
+ got, err := os.Readlink(filepath.Join(dir, "n19", "l0999"))
+ require.NoError(t, err)
+ require.Len(t, got, 4095)
+}
+
+// TestLandCheckMaterialized_stopsWithItsContext: the read-back looks at the context at every entry and
+// while it hashes a file, and an ended context is the error, not a difference of the checkout.
+func TestLandCheckMaterialized_stopsWithItsContext(t *testing.T) {
+ t.Parallel()
+
+ entries, contents := fakeEntries(t)
+ dir := t.TempDir()
+ require.NoError(t, landMaterialize(entries, batchOf(contents, entries), dir))
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ err := landCheckMaterialized(ctx, entries, dir)
+ require.ErrorIs(t, err, context.Canceled)
+ require.NotContains(t, err.Error(), "differs from the tree")
+
+ ctx, cancel = context.WithTimeout(context.Background(), time.Nanosecond)
+ defer cancel()
+ time.Sleep(2 * time.Millisecond)
+ require.ErrorIs(t, landCheckMaterialized(ctx, entries, dir), context.DeadlineExceeded)
+
+ // Through the checkout: a context that ends between the writing and the read-back ends it with the
+ // context's error, which the verification turns into the timeout of its verdict.
+ fx := newTreeFixture(t)
+ hookCtx, hookCancel := context.WithCancel(testhelper.Context(t))
+ defer hookCancel()
+ require.ErrorIs(t, landCheckoutTreeThen(hookCtx, fx.repo, fx.tree, t.TempDir(), landCheckoutLimits{}, hookCancel), context.Canceled)
+}
+
+// TestLandVerify_cleansUpWhatTheCommandLeaves: a command that makes directories unreadable and
+// read-only leaves nothing behind, and a directory that cannot be removed is counted.
+func TestLandVerify_cleansUpWhatTheCommandLeaves(t *testing.T) {
+ t.Parallel()
+
+ fx := newTreeFixture(t)
+ vdir, vargv := countingVerify(t, `mkdir -p locked/inner
+echo x > locked/inner/f
+chmod 000 locked/inner
+chmod 555 locked
+chmod 555 dir
+exit 0`)
+ rcfg := withVerify(config.GraftResolver{}, vargv, "")
+ s := newResolveFixture(t, rcfg).server(t, rcfg)
+ s.verifyTmpDir = t.TempDir()
+
+ require.NoError(t, s.landVerify(testhelper.Context(t), fx.repo, fx.tree))
+ require.Equal(t, 1, verifyRuns(t, vdir))
+ left, err := os.ReadDir(s.verifyTmpDir)
+ require.NoError(t, err)
+ require.Empty(t, left, "the verification directory is gone, whatever the command made of its permissions")
+}
+
+// TestLandVerify_aDeadlineAfterTheCheckoutIsTheTimeoutOfTheVerdict: the timeout covers the read-back too. A
+// verification whose deadline passes between the writing of the checkout and its read-back ends as the
+// rejection that says it did not finish (PROPOSED for a model, with the reason), never as an error of
+// the server, and the command does not run.
+func TestLandVerify_aDeadlineAfterTheCheckoutIsTheTimeoutOfTheVerdict(t *testing.T) {
+ t.Parallel()
+
+ fx := newTreeFixture(t)
+ vdir, vargv := countingVerify(t, "exit 0")
+ rcfg := withVerify(config.GraftResolver{}, vargv, "200ms")
+ s := newResolveFixture(t, rcfg).server(t, rcfg)
+ s.verifyAfterWriteHook = func() { time.Sleep(600 * time.Millisecond) }
+
+ err := s.landVerify(testhelper.Context(t), fx.repo, fx.tree)
+ var verifyErr *landVerifyError
+ require.ErrorAs(t, err, &verifyErr)
+ require.Contains(t, verifyErr.reason, "did not finish within 200ms")
+ require.Equal(t, 0, verifyRuns(t, vdir))
+}
diff --git a/internal/gitaly/service/operations/server.go b/internal/gitaly/service/operations/server.go
index 0b39c71..7af1d6e 100644
--- a/internal/gitaly/service/operations/server.go
+++ b/internal/gitaly/service/operations/server.go
@@ -41,12 +41,25 @@ type Server struct {
// landCheckBatchHook, when set, runs in every check batch while it holds its slot, with the
// context of the batch; only tests set it.
landCheckBatchHook func(ctx context.Context)
+ // verifyAfterWriteHook, when set, runs between the writing of a checkout and its read-back; only
+ // tests set it, to let a deadline pass there.
+ verifyAfterWriteHook func()
+ // verifyTmpDir is where the checkouts of the verification are made; empty means the system's
+ // temporary directory (TMPDIR). Tests set it to see what is left behind.
+ verifyTmpDir string
+ // landLaneHook, when set, runs in the lane when it applies the verdicts of a land, holding the
+ // lane for as long as it runs; only tests set it, to keep a lane busy.
+ landLaneHook func()
// landResolveHook, when set, runs in every resolver call while it holds its resolve slot; only
// tests set it.
landResolveHook func()
// landFinalPassHook, when set, runs in every pass of the final pass of a land between its
// pre-resolution and its lane, with the number of the pass; only tests set it.
landFinalPassHook func(pass int)
+ // verifyPool bounds the verifications of the process (PLAN10 15.9): the operator's command runs
+ // on the tree of every land that needs it, out of the lane, whatever made the resolution. It is
+ // set whenever verify.enabled is, with or without an AI resolver.
+ verifyPool *resolver.VerifyPool
// conflictResolver answers the resolve requests of UserLand. Nil when resolution is disabled,
// or when the configured resolver cannot be built: then resolverErr says why, and every land
// that needs the resolver fails closed with it (resolveLandConflicts, landWithResolution).
@@ -79,7 +92,7 @@ func NewServer(deps *service.Dependencies) *Server {
deps.GetLogger().WithError(shared.Err).Error("graft: the conflict resolver of land cannot be built: lands that need it fail")
}
}
- conflictResolver, resolverErr := shared.Client, shared.Err
+ conflictResolver, resolverErr, verifyPool := shared.Client, shared.Err, shared.Verify
if conflictResolver != nil && resolverActorsUnmatchable(deps.GetCfg().Graft) {
deps.GetLogger().Warn("graft: [graft.land.resolver] actors is set but no [[graft.agent]] is " +
"configured, so callers carry no identity and no landing may resolve conflicts")
@@ -99,6 +112,7 @@ func NewServer(deps *service.Dependencies) *Server {
resolveCfg: resolveCfg,
conflictResolver: conflictResolver,
resolverErr: resolverErr,
+ verifyPool: verifyPool,
shutdownCtx: deps.GetShutdownContext(),
storageSync: deps.GetGraftStorageSyncer(),
}
diff --git a/internal/graft/resolver/check.go b/internal/graft/resolver/check.go
index c5f47b8..cc135a7 100644
--- a/internal/graft/resolver/check.go
+++ b/internal/graft/resolver/check.go
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
+ "sync"
"time"
"github.com/prometheus/client_golang/prometheus"
@@ -135,6 +136,31 @@ var verifySlotWait = promauto.NewHistogram(prometheus.HistogramOpts{
Buckets: prometheus.ExponentialBuckets(0.01, 4, 8),
})
+// VerifyPool is the pool of verifications of a process (PLAN10 15.9): the operator's command runs on
+// the exact tree of a land, out of the lane, whatever made the resolution (a rule, an agent, a
+// model), and at most verify_concurrency of them run at once in the whole process, and at most
+// verify_per_repository for one repository. Every verification takes a slot; there is no other way
+// to run the command.
+type VerifyPool struct {
+ slots chan struct{}
+ // repos are the slots of each repository that has a verification running or waiting, taken
+ // before a slot of the pool so that one repository cannot hold the whole pool; an entry lives as
+ // long as it has a holder or a waiter. perRepo is their number.
+ mu sync.Mutex
+ repos map[string]*repoVerifySlots
+ perRepo int
+}
+
+// NewVerifyPool builds the pool of cfg: verify_concurrency slots, verify_per_repository of them for
+// one repository.
+func NewVerifyPool(cfg config.GraftResolver) *VerifyPool {
+ return &VerifyPool{
+ slots: make(chan struct{}, cfg.VerifyConcurrencyOrDefault()),
+ repos: map[string]*repoVerifySlots{},
+ perRepo: cfg.VerifyPerRepositoryOrDefault(),
+ }
+}
+
// repoVerifySlots are the verify slots of one repository: its verifications at once, and the
// number of those holding or waiting for one, which keeps the entry in the map.
type repoVerifySlots struct {
@@ -147,7 +173,7 @@ type repoVerifySlots struct {
// (AcquireVerifySlot), in a pool apart from the checks' and the resolutions'. Release it when the
// verification is done.
type VerifySlot struct {
- c *Client
+ p *VerifyPool
repo string
released bool
}
@@ -160,22 +186,30 @@ func (c *Client) AcquireVerifySlot(ctx context.Context, repo string) (*VerifySlo
if c == nil {
return nil, ErrDisabled
}
+ return c.verify.Acquire(ctx, repo)
+}
+
+// Acquire waits for a verify slot of repo and then for one of the pool; see Client.AcquireVerifySlot.
+func (p *VerifyPool) Acquire(ctx context.Context, repo string) (*VerifySlot, error) {
+ if p == nil {
+ return nil, ErrDisabled
+ }
start := time.Now()
observe := func() { verifySlotWait.Observe(time.Since(start).Seconds()) }
- c.verifyMu.Lock()
- r, ok := c.verifyRepos[repo]
+ p.mu.Lock()
+ r, ok := p.repos[repo]
if !ok {
- r = &repoVerifySlots{slots: make(chan struct{}, c.verifyPerRepo)}
- c.verifyRepos[repo] = r
+ r = &repoVerifySlots{slots: make(chan struct{}, p.perRepo)}
+ p.repos[repo] = r
}
r.users++
- c.verifyMu.Unlock()
+ p.mu.Unlock()
leave := func() {
- c.verifyMu.Lock()
- defer c.verifyMu.Unlock()
+ p.mu.Lock()
+ defer p.mu.Unlock()
if r.users--; r.users == 0 {
- delete(c.verifyRepos, repo)
+ delete(p.repos, repo)
}
}
@@ -187,9 +221,9 @@ func (c *Client) AcquireVerifySlot(ctx context.Context, repo string) (*VerifySlo
return nil, fmt.Errorf("waiting for a verify slot of the repository: %w", ctx.Err())
}
select {
- case c.verifySlots <- struct{}{}:
+ case p.slots <- struct{}{}:
observe()
- return &VerifySlot{c: c, repo: repo}, nil
+ return &VerifySlot{p: p, repo: repo}, nil
case <-ctx.Done():
observe()
<-r.slots
@@ -204,14 +238,14 @@ func (s *VerifySlot) Release() {
return
}
s.released = true
- <-s.c.verifySlots
+ <-s.p.slots
- s.c.verifyMu.Lock()
- defer s.c.verifyMu.Unlock()
- r := s.c.verifyRepos[s.repo]
+ s.p.mu.Lock()
+ defer s.p.mu.Unlock()
+ r := s.p.repos[s.repo]
<-r.slots
if r.users--; r.users == 0 {
- delete(s.c.verifyRepos, s.repo)
+ delete(s.p.repos, s.repo)
}
}
diff --git a/internal/graft/resolver/client.go b/internal/graft/resolver/client.go
index f5b54cc..4bc8b03 100644
--- a/internal/graft/resolver/client.go
+++ b/internal/graft/resolver/client.go
@@ -10,7 +10,6 @@ import (
"os"
"slices"
"strings"
- "sync"
"time"
"unicode/utf8"
@@ -103,15 +102,10 @@ type Client struct {
// the short checks (PLAN10 15.8).
checkSlots chan struct{}
resolveSlots chan struct{}
- // verifySlots bounds the verifications of a model's resolution that run at once (VerifySlot),
- // a third pool, as the operator's command takes minutes (PLAN10 15.9).
- verifySlots chan struct{}
- // verifyRepos are the slots of each repository that has a verification running or waiting, taken
- // before a verify slot so that one repository cannot hold the whole pool; an entry lives as long
- // as it has a holder or a waiter. verifyPerRepo is their number.
- verifyMu sync.Mutex
- verifyRepos map[string]*repoVerifySlots
- verifyPerRepo int
+ // verify bounds the verifications that run at once (VerifySlot), a third pool, as the operator's
+ // command takes minutes (PLAN10 15.9). Shared replaces it with the pool of the process, which
+ // the verification of every land goes through, with or without an AI resolver.
+ verify *VerifyPool
// sendRules decide the files that are never sent (SendRules).
sendRules *SendRules
}
@@ -159,19 +153,17 @@ func New(cfg config.GraftResolver) (*Client, error) {
return nil, err
}
c := &Client{
- command: slices.Clone(cfg.Command),
- timeout: timeout,
- maxFiles: cfg.MaxFilesOrDefault(),
- maxFileBytes: cfg.MaxFileBytesOrDefault(),
- pathsAllow: slices.Clone(cfg.PathsAllow),
- pathsDeny: slices.Clone(cfg.PathsDeny),
- actors: slices.Clone(cfg.Actors),
- sendRules: rules,
- checkSlots: make(chan struct{}, cfg.CheckConcurrencyOrDefault()),
- resolveSlots: make(chan struct{}, cfg.ResolveConcurrencyOrDefault()),
- verifySlots: make(chan struct{}, cfg.VerifyConcurrencyOrDefault()),
- verifyRepos: map[string]*repoVerifySlots{},
- verifyPerRepo: cfg.VerifyPerRepositoryOrDefault(),
+ command: slices.Clone(cfg.Command),
+ timeout: timeout,
+ maxFiles: cfg.MaxFilesOrDefault(),
+ maxFileBytes: cfg.MaxFileBytesOrDefault(),
+ pathsAllow: slices.Clone(cfg.PathsAllow),
+ pathsDeny: slices.Clone(cfg.PathsDeny),
+ actors: slices.Clone(cfg.Actors),
+ sendRules: rules,
+ checkSlots: make(chan struct{}, cfg.CheckConcurrencyOrDefault()),
+ resolveSlots: make(chan struct{}, cfg.ResolveConcurrencyOrDefault()),
+ verify: NewVerifyPool(cfg),
}
if cfg.URL != "" {
diff --git a/internal/graft/resolver/client_test.go b/internal/graft/resolver/client_test.go
index 67126c9..2f7f50f 100644
--- a/internal/graft/resolver/client_test.go
+++ b/internal/graft/resolver/client_test.go
@@ -829,7 +829,7 @@ func TestAcquireVerifySlot_boundsTheVerificationsAtOnce(t *testing.T) {
_, err = disabled.AcquireVerifySlot(ctx, "a")
require.ErrorIs(t, err, ErrDisabled)
- require.Equal(t, config.DefaultGraftResolverVerifyConcurrency, cap(newClient(t, commandConfig([]string{"/bin/true"})).verifySlots))
+ require.Equal(t, config.DefaultGraftResolverVerifyConcurrency, cap(newClient(t, commandConfig([]string{"/bin/true"})).verify.slots))
}
// TestAcquireVerifySlot_oneRepositoryCannotTakeThePool: at most verify_per_repository verifications run
@@ -841,7 +841,7 @@ func TestAcquireVerifySlot_oneRepositoryCannotTakeThePool(t *testing.T) {
cfg.VerifyConcurrency = 2
client := newClient(t, cfg)
ctx := testhelper.Context(t)
- require.Equal(t, config.DefaultGraftResolverVerifyPerRepository, client.verifyPerRepo)
+ require.Equal(t, config.DefaultGraftResolverVerifyPerRepository, client.verify.perRepo)
first, err := client.AcquireVerifySlot(ctx, "busy")
require.NoError(t, err)
@@ -862,9 +862,9 @@ func TestAcquireVerifySlot_oneRepositoryCannotTakeThePool(t *testing.T) {
waiting <- err
}()
require.Eventually(t, func() bool {
- client.verifyMu.Lock()
- defer client.verifyMu.Unlock()
- return client.verifyRepos["busy"] != nil && client.verifyRepos["busy"].users == 2
+ client.verify.mu.Lock()
+ defer client.verify.mu.Unlock()
+ return client.verify.repos["busy"] != nil && client.verify.repos["busy"].users == 2
}, 5*time.Second, 5*time.Millisecond, "the second land never waited")
other, err := client.AcquireVerifySlot(ctx, "other")
require.NoError(t, err, "a land that waits for its repository holds nothing of the pool")
@@ -874,18 +874,18 @@ func TestAcquireVerifySlot_oneRepositoryCannotTakeThePool(t *testing.T) {
require.NoError(t, <-waiting)
other.Release()
- client.verifyMu.Lock()
- require.Empty(t, client.verifyRepos, "nothing is kept of a repository without holder or waiter")
- client.verifyMu.Unlock()
+ client.verify.mu.Lock()
+ require.Empty(t, client.verify.repos, "nothing is kept of a repository without holder or waiter")
+ client.verify.mu.Unlock()
// A waiter that gave up leaves no trace either.
held, err := client.AcquireVerifySlot(ctx, "gone")
require.NoError(t, err)
require.Error(t, timedOut(t, client, "gone"))
held.Release()
- client.verifyMu.Lock()
- require.Empty(t, client.verifyRepos)
- client.verifyMu.Unlock()
+ client.verify.mu.Lock()
+ require.Empty(t, client.verify.repos)
+ client.verify.mu.Unlock()
// Configured, the cap is the number of verifications of one repository at once.
cfg.VerifyPerRepository = 2
diff --git a/internal/graft/resolver/shared.go b/internal/graft/resolver/shared.go
index 8cfcd83..7525ddf 100644
--- a/internal/graft/resolver/shared.go
+++ b/internal/graft/resolver/shared.go
@@ -10,13 +10,24 @@ type Shared struct {
Client *Client
// Err is why the configured resolver cannot be built, if it cannot.
Err error
+ // Verify is the pool of verifications of the process (PLAN10 15.9). It is set whenever
+ // verify.enabled is, with or without an AI resolver: an agent's resolution and the rules' are
+ // verified too, out of the lane, and the pool bounds all of them together.
+ Verify *VerifyPool
}
// NewShared builds the resolver client of the process from cfg.
func NewShared(cfg config.GraftResolver) *Shared {
+ var pool *VerifyPool
+ if cfg.Verify.Enabled {
+ pool = NewVerifyPool(cfg)
+ }
client, err := New(cfg)
if err != nil {
- return &Shared{Err: err}
+ return &Shared{Err: err, Verify: pool}
+ }
+ if client != nil && pool != nil {
+ client.verify = pool
}
- return &Shared{Client: client}
+ return &Shared{Client: client, Verify: pool}
}