Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

PR CI: reuse a passed head's results after Update branch when main didn't touch their inputs (#2186)

0xken15ee8e0cde5965
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ce46e1770..574a2bf3c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -3,7 +3,8 @@ name: CI
# Fast PR checks: each changed component runs its own Linux unit tests. Native,
# macOS, coverage-gate, browser and end-to-end checks are outside PR CI; see
# docs/validation-and-release.md#what-pr-ci-does-not-run. Merge candidates rerun
-# only the plan and process checks; the PR run covered the code.
+# only the plan and process checks; the PR run covered the code. After Update branch,
+# a suite whose inputs main's new commits left untouched reuses the earlier head's result.
# Contributor code runs with a read-only token and no release credentials.
on:
pull_request:
@@ -49,6 +50,12 @@ jobs:
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
runs-on: ubuntu-latest
timeout-minutes: 5
+ # Read-only lookups of an earlier head's ci/required run, whose results an update merge may
+ # reuse (docs/validation-and-release.md#reuse-after-update-branch). No write scope.
+ permissions:
+ contents: read
+ actions: read
+ checks: read
outputs:
plan: ${{ steps.select.outputs.plan }}
ready: ${{ steps.select.outputs.ready }}
@@ -95,6 +102,7 @@ jobs:
- id: select
env:
CI_SCOPE: ${{ inputs.scope || 'full' }}
+ GH_TOKEN: ${{ github.token }}
run: python3 scripts/ci-plan.py --scope "$CI_SCOPE" --output "$RUNNER_TEMP/ci-plan.json"
- uses: actions/upload-artifact@v4
with:
diff --git a/docs/validation-and-release.md b/docs/validation-and-release.md
index 50d61dc04..1d42616f3 100644
--- a/docs/validation-and-release.md
+++ b/docs/validation-and-release.md
@@ -42,12 +42,62 @@ PR CI is deliberately fast. Each changed component runs only its own Linux suite
| anything else (docs, workflows, scripts, store) | `process-checks` only | about 1.5 min |
`plan` (actionlint and suite selection), `process-checks` and `ci/required` always
-run; `ci/required` is the only required check. On a draft PR only those three run
+run (after **Update branch**, untouched components reuse their earlier result: see
+[Reuse after Update branch](#reuse-after-update-branch)); `ci/required` is the only required check. On a draft PR only those three run
and `ci/required` stays blocked; marking the PR ready runs the component checks.
Normal merges directly squash the reviewed head after successful PR CI. If GitHub
requires a merge queue, its runs rerun only the three always-on jobs (about 2 minutes):
the PR run already tested the change. Queue setup is not required to merge or release.
+### Reuse after Update branch
+
+When main moves after a PR's `ci/required` passed, **Update branch** merges main into
+the PR and CI runs again. The new run reuses the earlier result of each selected
+component whose inputs main's new commits left untouched, and runs the rest. `plan`,
+`process-checks` and `ci/required` always run, so an update that touches nothing a
+component reads passes in about two minutes instead of 7–15.
+
+A component's result is reused only when all of these hold; otherwise it runs as before:
+
+- The new head is a merge with exactly two parents: the earlier head, then a commit on
+ the PR's base branch. Any other commit, including one you push by hand, runs everything.
+- The earlier head's latest `ci/required` check passed. The planner follows that check
+ to its exact run and attempt, and reads that run's receipt. A component the run
+ itself reused points at the run where it last ran.
+- Nothing that differs between the tested commit and the new head is a component input:
+ its own directory (by the map above) or a file it reads elsewhere (`DEPENDS` in
+ `scripts/ci-plan.py`). `backend/integration/` and `cli/e2e/` (apart from
+ `cli/e2e/harness/`) are skipped because the PR jobs never read them. Each Swarm UI
+ style allowlist (`scripts/ui-style-allowlist/mobile.json`, `desktop.json`) is an input
+ of the component whose job checks it, and a PR that changes one runs that component.
+- No changed path is a shared input: `.github/`, `scripts/` (the planner and the merge
+ helper; apart from the two style allowlists above), `Makefile`, `.gitattributes`, `.gitignore`, `.gitmodules`, any lockfile or
+ toolchain pin, or any path outside a component other than `docs/` (but `docs/images/`
+ counts) and top-level `.md` files. A shared change reruns every component.
+- The PR itself changes no workflow, script, `Makefile` or Git setting: PR CI runs the
+ PR's own planner.
+- Every GitHub read succeeded within a minute. The plan job has read-only `actions`
+ and `checks` access for these reads.
+
+The `ci/required` receipt lists each reused component with its run id, attempt, link,
+tested commit and tree, and the reason. The run summary also says why each other
+component ran again. `make merge-pr` loads the planner from main's tree, the reviewed
+base, and uses it to re-derive every reuse from Git and from the runs it names. In the
+reviewed checkout it also scans the reused component's sources for reads `DEPENDS` is
+missing: relative paths (`../backend/...`) and quoted paths from the repository root
+(`ROOT / 'cli/src/cable/serial.ts'`), outside comments. A path that could name a file inside
+the component and also a real path elsewhere counts as a read of both, so a look-alike file
+cannot hide a read; every path main changed, deleted ones included, counts as existing, so a
+read of a deleted file is still a read. Strings that only look like paths (test data, scripts
+run by hand) are listed with a reason in `NOT_READS`; a path built from parts must be
+declared by hand. It refuses to merge on any mismatch, or when main's copy cannot verify reuse.
+To run everything again, push a commit to the PR.
+
+`make merge-pr` itself runs from the PR checkout. Any change to the merge helper or the
+planner (`scripts/merge-validated-pr.py`, `scripts/queue-validated-pr.py`,
+`scripts/ci-plan.py`, `scripts/record-ci-validation.py`) needs two independent
+security reviews before it merges.
+
The `plan` and `ci/required` jobs run `scripts/ci-plan.py` from the PR head. A branch
created before #991 still carries the old planner, which expects removed jobs and
fails `ci/required` for CLI, Desktop, workflow and script changes. Update the branch
diff --git a/scripts/ci-plan.py b/scripts/ci-plan.py
index fb6a5af20..97bf6bcd6 100644
--- a/scripts/ci-plan.py
+++ b/scripts/ci-plan.py
@@ -3,14 +3,24 @@
A PR runs the unit suite of each component it changes, and nothing else. Merge
groups run only the plan and process checks: the PR run already covered the code.
+After GitHub merges main into a PR head whose ci/required passed, a component whose
+inputs main's new commits left untouched reuses that run's result instead of running
+again (docs/validation-and-release.md#reuse-after-update-branch).
Checks outside PR CI are listed in docs/validation-and-release.md#what-pr-ci-does-not-run.
"""
import argparse
+import importlib.util
import json
import os
-from pathlib import Path
+from pathlib import Path, PurePosixPath
import re
import subprocess
+import time
+import zipfile
+
+_spec = importlib.util.spec_from_file_location("ci_evidence", Path(__file__).with_name("record-ci-validation.py"))
+ci = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(ci)
CORE = {"cli", "desktop", "tui", "backend", "companions"}
@@ -38,9 +48,97 @@ COMPONENTS = {"cli": "cli", "tests": "cli", "desktop": "desktop", "tui": "tui",
ALSO = {"cli/src/lib/ptyHost/": "pty", "cli/src/lib/ptyHostBackend": "pty",
"cli/src/lib/terminalBackend.contract.ts": "pty", "tui/vendor/": "pty", "term/": "tui",
"cli/src/lib/windows/__fixtures__/pipe-auth-vectors.json": "desktop"}
+# Each Swarm UI style allowlist is checked by its own component's job (check-ui-style.py --only), so
+# it is that component's input: a PR or main commit that changes it runs (never reuses) that suite.
+# Any other file in scripts/ui-style-allowlist/ stays a shared input.
+STYLE_ALLOWLISTS = {"scripts/ui-style-allowlist/desktop.json": "desktop", "scripts/ui-style-allowlist/mobile.json": "mobile"}
+ALSO.update(STYLE_ALLOWLISTS)
# Narrower directories that add a suite on top of their component's (the ESP32-P4 build and golden screens).
SUBCOMPONENTS = {"devices/harness-pro/": "pro"}
+# Reuse after Update branch. A PR's own diff selects suites by the maps above; reusing an
+# earlier result also needs every file a suite reads outside its own directory, because
+# main's commits were never tested together with this PR. "dir/" is a prefix, else a file.
+# test_ci_plan.CrossReadTests scans the components' sources and fails on an undeclared read.
+# The desktop VM tests run in a sparse checkout of exactly these inputs (ci-source-inputs.py).
+DEPENDS = {
+ "backend": ("cli/src/lib/productHealthContract.ts", "cli/src/lib/productUsageContract.ts", "daemons/frames.json",
+ "desktop/lib/measurement/", "desktop/lib/super_tabs/buddies.dart", "mobile/lib/measurement/",
+ "provider/reference-provider/", "tests/fixtures/"),
+ "cli": ("backend/src/lib/deviceAck.vectors.json", "backend/src/lib/deviceAckCore.ts", "desktop/",
+ "cli/e2e/harness/", "devices/harness-device/firmware/test/vectors/", "docs/contracts/", "mobile/", "pty/", "tui/"),
+ "companions": ("cli/", "docs/research/2026-09-30-memory-batch-extraction-cases.json",
+ "docs/research/2026-09-30-memory-extraction-cases.json"),
+ # Its generators write (and its checks may compare) these outputs.
+ "daemons": ("backend/src/lib/daemonRoster.g.ts", "cli/src/", "desktop/assets/daemon-art/",
+ "desktop/lib/daemons/illustrated_alignment.g.dart", "desktop/lib/daemons/plates.g.dart",
+ "desktop/lib/daemons/roster.g.dart", "mobile/lib/daemons/plates.g.dart", "mobile/lib/daemons/roster.g.dart"),
+ "firmware": ("cli/scripts/", "cli/src/", "daemons/plates/tux.mjs", "daemons/review/traits.html",
+ "daemons/tools/illustrated/daemon_art.py", "daemons/tools/plate.mjs", "desktop/assets/model-icons/claude.png"),
+ # Its tools also import the web release scripts; its tests see only the sparse inputs.
+ "desktop": ci.SOURCE_INPUTS["desktop"] + ("website/scripts/harness-web-assets.mjs", "website/scripts/precompress.mjs"),
+ "mobile": ("backend/src/types/productHealthContract.ts", "backend/src/types/productUsage.ts", "cli/src/",
+ "daemons/banner.json", "daemons/frames.json", "daemons/plates.json", "daemons/roster.json", "desktop/",
+ "docs/performance/"),
+ # pty-checks also runs the CLI's ptyhost contract, which imports CLI code.
+ # os-checks reads the CLI's dial ids (cli/src/cable/serial.ts), its version and the TUI it packages;
+ # term/ by hand: the update fixture copies `git ls-files -- tui term`, which the scan cannot see.
+ "os": ("cli/", "devices/harness-device/firmware/README.md", "docs/naming-system.md", "docs/terminology.md", "term/", "tui/"),
+ "pty": ("cli/", "desktop/windows/runner/resources/app_icon.ico"),
+ # THIRD_PARTY_NOTICES.md: tui/src/main.rs includes ../THIRD_PARTY_NOTICES.md, which names a root file too.
+ "tui": ("THIRD_PARTY_NOTICES.md", "cli/package.json", "os/network.py", "os/onboarding.py", "os/root/usr/bin/hn",
+ "tests/fixtures/shared-pane-layouts.json"),
+ "website": ("backend/src/", "cli/packages/harness-mcp/", "cli/src/agentClient/commands.ts", "cli/src/lib/newCommand.ts",
+ "cli/src/messages/command.ts", "desktop/lib/core/phone_app_links.dart",
+ "desktop/macos/Runner/Assets.xcassets/AppIcon.appiconset/app_icon_512.png"),
+}
+# Folders inside a suite's own component that its PR jobs never read: the backend unit suite
+# runs src/**/*.test.ts, and the CLI's typecheck and unit tests cover src/ (end to end runs elsewhere).
+# CrossReadTests fails when a file outside the folder refers into it.
+NOT_READ = {"backend": ("backend/integration/",), "cli": ("cli/e2e/",)}
+# Strings in a suite's files that name another directory's path but are not read by its PR jobs:
+# test data, or a script run only by hand. {source: (why, {path the scan reports})}.
+NOT_READS = {
+ **{source: ("test data: owned-code paths and model names", {"desktop/lib/", "desktop/lib/super_tabs/",
+ "desktop/lib/super_tabs/roster.dart"})
+ for source in ("backend/src/superTabs/commands.test.ts",)},
+ # cli-tests runs vitest on src/ only; these start the real backend with its installed prisma.
+ **{source: ("end-to-end and local stack scripts, run by hand", {"backend/"})
+ for source in ("cli/e2e/harness/realBackend.ts", "cli/scripts/box-claim-e2e.ts", "cli/scripts/local-stack-peer.ts",
+ "cli/scripts/remote-viewers-stack.ts", "cli/scripts/share-harness-e2e.ts")},
+ "cli/src/engines/__fixtures__/other-runtime.golden.json": ("test data: a model name", {"provider/"}),
+ "cli/src/lib/buddyGuard.claude.spec.ts": ("test data: files in a temporary repository", {"docs/"}),
+ "cli/src/lib/handoffChangedFiles.spec.ts": ("test data: changed file names", {"docs/"}),
+ "cli/src/lib/opencodeOneShot.spec.ts": ("test data: a model name", {"provider/"}),
+ "cli/src/lib/runtimeProfile.spec.ts": ("test data: a model name", {"provider/"}),
+ "cli/src/lib/transcriptReader.spec.ts": ("test data: a tool result's text", {"devices/"}),
+ "companions/src/application/modelProfile.spec.ts": ("test data: a model name", {"provider/"}),
+ "companions/src/memory/lessons/signals.spec.ts": ("test data: a test id", {"tests/"}),
+ "desktop/test/p2p/mobile_parity_test.dart": ("reads ../mobile/<file> only for the phone files desktop-tests' sparse "
+ "checkout holds, each declared in ci.SOURCE_INPUTS", {"mobile/"}),
+ "desktop/test/community_publish_test.dart": ("test data: a file in a temporary folder", {"docs/"}),
+ "desktop/test/runtime_model_name_test.dart": ("test data: a model name", {"provider/"}),
+ "desktop/test/super_tab_read_fixtures/first_load.json": ("test data: a task's owned paths", {"docs/design/"}),
+ "mobile/lib/demo/sample_scripts.dart": ("demo transcript text", {"tests/"}),
+ "tui/src/runtime_model.rs": ("test data: a model name", {"provider/"}),
+ # tui-test runs cargo test only (tui/tests/ holds no Rust tests); these read build output and installed packages.
+ "tui/tests/local-first.py": ("a script run by hand against the built CLI (cli/dist)", {"cli/"}),
+ "tui/tests/viewer-live.mjs": ("a script run by hand with the backend's installed prisma", {"backend/"}),
+ "website/src/app/download/page.spec.tsx": ("test data: text the page must not show", {"desktop/"}),
+ "website/src/app/hub/publish/page.spec.tsx": ("test data: a file in an upload", {"docs/"}),
+}
+# Inputs every suite shares: a change here reruns everything, and a PR that changes them
+# never reuses (PR CI runs the PR's own planner and workflow; the merge helper is in scripts/).
+SHARED_PREFIXES = (".github/", "scripts/")
+SHARED_FILES = {"Makefile", ".gitattributes", ".gitignore", ".gitmodules"}
+LOCKS_AND_TOOLCHAINS = {
+ "package-lock.json", "npm-shrinkwrap.json", "yarn.lock", "pnpm-lock.yaml", "pnpm-workspace.yaml", "bun.lockb",
+ "bun.lock", "Cargo.lock", "pubspec.lock", "go.sum", "go.mod", "go.work", "go.work.sum", "uv.lock", "poetry.lock",
+ "Pipfile.lock", "Gemfile.lock", "Podfile.lock", "flake.lock", "gradle.lockfile", "composer.lock",
+ ".nvmrc", ".node-version", ".tool-versions", ".python-version", ".ruby-version", ".go-version",
+ "rust-toolchain", "rust-toolchain.toml", ".fvmrc", ".flutter-version", "global.json",
+}
+
def git(root, *args):
return subprocess.check_output(["git", *args], cwd=root, stderr=subprocess.PIPE, timeout=120).decode().strip()
@@ -66,7 +164,377 @@ def select(paths):
return selected, reasons
-def make_plan(root, event_name, event, scope="full", source=None):
+def covers(entry, path):
+ return path == entry or (entry.endswith("/") and path.startswith(entry))
+
+
+def inert(path):
+ """Paths no suite reads: the docs (except the posters desktop shows) and top-level notes."""
+ return ((path.startswith("docs/") and not path.startswith("docs/images/"))
+ or ("/" not in path and path.endswith(".md")))
+
+
+def shared_input(path):
+ """Why a changed path invalidates every suite, or None."""
+ if path in STYLE_ALLOWLISTS:
+ return None
+ if path.startswith(SHARED_PREFIXES) or path in SHARED_FILES:
+ return "workflow, CI script, merge tool or repository setting"
+ if PurePosixPath(path).name in LOCKS_AND_TOOLCHAINS:
+ return "lockfile or toolchain pin"
+ if not select([path])[0] and not inert(path):
+ return "outside every component"
+ return None
+
+
+def reads(suite, path):
+ own = suite in select([path])[0] and not any(covers(skip, path) for skip in NOT_READ.get(suite, ()))
+ return own or any(covers(entry, path) for entry in DEPENDS.get(suite, ()))
+
+
+def changed_paths(root, old, new):
+ changed = subprocess.check_output(["git", "diff", "--no-renames", "--name-only", "-z", old, new], cwd=root, timeout=120)
+ return sorted(p.decode() for p in changed.split(b"\0") if p)
+
+
+def is_ancestor(root, old, new):
+ return subprocess.run(["git", "merge-base", "--is-ancestor", old, new], cwd=root, stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL, timeout=120).returncode == 0
+
+
+def update_parents(root, head, base):
+ """The passed head and main commit that GitHub's Update branch merged, or ValueError."""
+ parents = git(root, "rev-list", "--parents", "-n", "1", head).split()[1:]
+ if len(parents) != 2:
+ raise Rerun("head is not a merge of main into an earlier head")
+ passed, main = (sha(value) for value in parents)
+ if not is_ancestor(root, main, base):
+ raise Rerun("the merged main commit is not on the base branch")
+ return passed, main
+
+
+def own_shared_change(root, head, base):
+ """A shared input this PR itself changes (its own CI runs its own planner), or None."""
+ own = changed_paths(root, git(root, "merge-base", base, head), head)
+ return next((path for path in own if (path.startswith(SHARED_PREFIXES) or path in SHARED_FILES)
+ and path not in STYLE_ALLOWLISTS), None)
+
+
+def reuse_blocker(root, suite, origin, head):
+ """(scope, reason) when the result tested at origin cannot stand for head; scope "all" or "suite"."""
+ if not is_ancestor(root, origin, head):
+ return "suite", f"its recorded run tested {origin[:12]}, which is not an ancestor of this head"
+ paths = changed_paths(root, origin, head)
+ for path in paths:
+ reason = shared_input(path)
+ if reason:
+ return "all", f"main changed {path} ({reason})"
+ touched = [path for path in paths if reads(suite, path)]
+ if touched:
+ more = f" and {len(touched) - 1} more" if len(touched) > 1 else ""
+ return "suite", f"main changed its input {touched[0]}{more}"
+ return None, "; ".join(paths[:3]) + (f" and {len(paths) - 3} more" if len(paths) > 3 else "")
+
+
+REUSE_LOOKUP_SECONDS = 60
+
+
+class Rerun(ValueError):
+ """A deliberate refusal to reuse: its message is the receipt's reason."""
+
+
+REUSE_ERRORS = (OSError, RuntimeError, ValueError, KeyError, TypeError, AttributeError, subprocess.SubprocessError,
+ zipfile.BadZipFile, json.JSONDecodeError)
+
+
+def plan_reuse(root, head, base, suites, evidence):
+ """Reuse what the passed head's exact ci/required run proved, for suites main's commits left alone.
+
+ Fails closed: any doubt or unreadable evidence leaves every suite to run as before."""
+ reuse = dict(status="none", reason=None, passed_head=None, main_parent=None, rerun={})
+ reused = {}
+ try:
+ if evidence is None:
+ raise Rerun("no CI evidence reader")
+ passed, main = update_parents(root, head, base)
+ reuse.update(passed_head=passed, main_parent=main)
+ changed = own_shared_change(root, head, base)
+ if changed:
+ raise Rerun(f"this PR changes {changed}, which its own CI run reads")
+ gate = evidence.passed_gate(passed)
+ receipt = gate["receipt"]
+ if receipt["source_sha"] != passed or receipt["status"] != "passed":
+ raise ValueError("the passed head's ci/required receipt names another source")
+ earlier = receipt.get("reused") or {}
+ for suite in sorted(suites):
+ if suite not in receipt["suites"]:
+ reuse["rerun"][suite] = "the passed head's run did not run it"
+ continue
+ if suite in earlier:
+ record = earlier[suite]
+ origin, run_id, attempt = sha(record["origin_sha"]), record["run_id"], record["run_attempt"]
+ found = evidence.ran(origin, run_id, attempt, suite)
+ tree, url = found["tree"], found["url"]
+ else:
+ origin, run_id, attempt, tree, url = passed, gate["run_id"], gate["run_attempt"], receipt["source_tree"], gate["url"]
+ scope, reason = reuse_blocker(root, suite, origin, head)
+ if scope == "all":
+ raise Rerun(reason)
+ if scope == "suite":
+ reuse["rerun"][suite] = reason
+ continue
+ reused[suite] = dict(origin_sha=origin, origin_tree=sha(tree), run_id=run_id, run_attempt=attempt, run_url=url,
+ reason=f"passed in run {run_id} on {origin[:12]}; main's new commits change none of its inputs ({reason})")
+ except Rerun as refusal:
+ return {}, dict(reuse, rerun={}, reason=str(refusal))
+ except REUSE_ERRORS as error:
+ return {}, dict(reuse, rerun={}, reason=f"earlier evidence could not be verified: {str(error) or type(error).__name__}")
+ reuse.update(status="applied" if reused else "none",
+ reason=f"reused {len(reused)} of {len(suites)} suites from the passed head's run")
+ return reused, reuse
+
+
+# Literal relative paths into another top-level directory ("../backend/x.ts", "../../tests/fixtures/").
+SOURCE_FILE = re.compile(r"\.(ts|tsx|mts|cts|js|mjs|cjs|dart|rs|py|sh|go|c|h|cc|cpp|kt|swift|toml|yaml|yml|json|gradle|cmake)$"
+ r"|(^|/)(Makefile|CMakeLists\.txt|Dockerfile)$")
+RELATIVE_READ = re.compile(r"(?<![\w/.-])((?:\.\./)+)([A-Za-z0-9_.-]+(?:/[^'\"`\s$(){}<>,;]*)?)")
+# A quoted path from the repository root ("ROOT / 'cli/src/cable/serial.ts'", "join(repo, 'backend/src/')").
+QUOTED_PATH = re.compile(r"""(?<![\w/.-])['"](?!\.\.?/)([A-Za-z0-9_.-]+/[^'"`\s$(){}<>,;*]*)""")
+COMMENT_LINE = ("//", "#", "*", "/*", "<!--")
+
+
+def undeclared_reads(root, suite, known_paths=()):
+ """Files of this suite that name another directory's path without a DEPENDS entry: [(source, target)].
+
+ Run on a full checkout (the merge helper's, before accepting reuse; or locally). A read the scan
+ cannot see (a path built from parts) belongs in DEPENDS by hand. known_paths count as existing:
+ the helper passes every path main changed, deletions included, so a read of a deleted file stays a read."""
+ listed = subprocess.check_output(["git", "ls-files", "-z"], cwd=root, timeout=60).decode().split("\0")
+ sources = {path for path in listed if path}
+ tracked = sources | set(known_paths)
+ folders = {str(PurePosixPath(path).parent) for path in tracked}
+ folders |= {str(parent) for folder in list(folders) for parent in PurePosixPath(folder).parents}
+ tops = {path.split("/", 1)[0] for path in tracked if "/" in path}
+ found = []
+ for source in sorted(sources):
+ if not SOURCE_FILE.search(source) or "/evidence/" in source or suite not in select([source])[0]:
+ continue
+ try:
+ text = (Path(root) / source).read_text(encoding="utf-8")
+ except (OSError, UnicodeDecodeError):
+ continue
+ top = source.split("/", 1)[0]
+ if not reads(suite, source):
+ continue # In a folder this suite skips; what it refers to is read only if a read file says so.
+
+ def existing(target):
+ """The tracked file or folder ("dir/") a path names, trimmed to what exists; or None."""
+ known = target
+ while known not in tracked and known not in folders and "/" in known:
+ known = known.rsplit("/", 1)[0]
+ if known not in tracked and known not in folders:
+ return None
+ return known + "/" if known in folders else known
+
+ data = NOT_READS.get(source, ("", set()))[1]
+ code = "\n".join("" if line.lstrip().startswith(COMMENT_LINE) else line for line in text.splitlines())
+ for match in RELATIVE_READ.finditer(code):
+ relative = match.group(1) + match.group(2)
+ targets = []
+ for start in (str(PurePosixPath(source).parent), top): # beside the file, or from its package root
+ resolved = os.path.normpath(os.path.join(start, relative)).replace(os.sep, "/")
+ if not resolved.startswith("../") and resolved not in {".", ".."}:
+ targets.append(resolved)
+ # Inside its own directory (an existing file or folder there): an internal path, unless it
+ # reaches into a folder this suite's jobs skip.
+ inside = [t for t in targets if t.split("/", 1)[0] == top
+ and (t in tracked or t in folders or str(PurePosixPath(t).parent) in folders)]
+ if inside:
+ known = existing(inside[0])
+ if known and not reads(suite, known + "x" if known.endswith("/") else known):
+ found.append((source, known))
+ # Fail closed when the other reading names a real path too: a look-alike file inside the
+ # component (which the PR controls) must not hide a read of another directory.
+ for target in targets:
+ if target.split("/", 1)[0] != top and (target in tracked or target in folders):
+ known = target + "/" if target in folders else target
+ probe = known + "x" if known.endswith("/") else known
+ if known not in data and not shared_input(probe) and not reads(suite, probe):
+ found.append((source, known))
+ continue
+ for target in targets:
+ if target.split("/", 1)[0] == top:
+ continue
+ known = existing(target)
+ if known is None:
+ continue # Names nothing in this repository (a URL path, a generated file).
+ # A folder read (a path built at run time) needs the whole folder declared.
+ probe = known + "x" if known.endswith("/") else known
+ if not shared_input(probe) and not reads(suite, probe):
+ found.append((source, known))
+ break
+
+ # Paths named from the repository root, on lines that are not comments.
+ for match in QUOTED_PATH.finditer(code):
+ target = os.path.normpath(match.group(1)).replace(os.sep, "/")
+ if target.split("/", 1)[0] not in tops or target.split("/", 1)[0] == top:
+ continue
+ # The same path inside its own directory or beside the file ("OS / 'tests/x.py'") is internal,
+ # but only when the repository root has no such path: an ambiguous path counts as a read.
+ if target not in tracked and target not in folders and any(
+ f"{start}/{target}" in tracked or f"{start}/{target}" in folders
+ for start in (top, str(PurePosixPath(source).parent))):
+ continue
+ known = existing(target)
+ if known is None or known in data:
+ continue
+ probe = known + "x" if known.endswith("/") else known
+ if not shared_input(probe) and not reads(suite, probe):
+ found.append((source, known))
+ return sorted(set(found))
+
+
+def verify_reuse(root, receipt, base, evidence):
+ """Re-derive a receipt's reused results from Git and the exact runs they name, or raise ValueError.
+
+ The merge helper runs this from main's copy of the planner: PR CI ran the PR's own copy."""
+ head = sha(receipt.get("source_sha"))
+ if git(root, "rev-parse", "HEAD") != head or git(root, "status", "--porcelain", "--untracked-files=normal"):
+ raise ValueError("reused results are verified only in a clean checkout of the reviewed head")
+ reused = check_reused_records(receipt.get("reused"), receipt.get("suites") or [], head, receipt.get("event"))
+ if not reused:
+ return {}
+ try:
+ passed, main = update_parents(root, head, base)
+ except Rerun as refusal:
+ raise ValueError(f"reused results refused: {refusal}") from None
+ reuse = receipt.get("reuse")
+ if not isinstance(reuse, dict) or (reuse.get("passed_head"), reuse.get("main_parent")) != (passed, main):
+ raise ValueError("reused results name other parents than the head's update merge")
+ changed = own_shared_change(root, head, base)
+ if changed:
+ raise ValueError(f"reused results refused: this PR changes {changed}, which its own CI run reads")
+ for suite, record in sorted(reused.items()):
+ found = evidence.ran(record["origin_sha"], record["run_id"], record["run_attempt"], suite)
+ if found["tree"] != record["origin_tree"]:
+ raise ValueError(f"reused {suite} names another tree than its run tested")
+ if found["url"] != record["run_url"]:
+ raise ValueError(f"reused {suite} names another link than its run")
+ scope, reason = reuse_blocker(root, suite, record["origin_sha"], head)
+ if scope:
+ raise ValueError(f"reused {suite} refused: {reason}")
+ # The planner ran in a sparse checkout; this full one shows reads DEPENDS is missing.
+ changed = changed_paths(root, record["origin_sha"], head)
+ for source, target in undeclared_reads(root, suite, changed):
+ hit = next((path for path in changed if covers(target, path)), None)
+ if hit:
+ raise ValueError(f"reused {suite} refused: {source} reads {target}, which main changed ({hit}), "
+ "and DEPENDS in scripts/ci-plan.py does not declare it; push a commit to run it again")
+ return reused
+
+
+def main_planner(root, base):
+ """main's copy of this planner, loaded from the reviewed base commit rather than the PR checkout."""
+ import tempfile
+ try:
+ with tempfile.TemporaryDirectory(prefix="main-ci-plan-") as folder:
+ for name in ("ci-plan.py", "record-ci-validation.py"):
+ data = subprocess.check_output(["git", "show", f"{sha(base)}:scripts/{name}"], cwd=root,
+ stderr=subprocess.PIPE, timeout=15)
+ (Path(folder) / name).write_bytes(data)
+ spec = importlib.util.spec_from_file_location("main_ci_plan", Path(folder) / "ci-plan.py")
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ if not callable(getattr(module, "verify_reuse", None)) or not callable(getattr(module, "GitHubEvidence", None)):
+ raise ValueError("no verify_reuse")
+ except Exception as error: # Any failure to load main's verifier refuses reuse; nothing else depends on it.
+ raise ValueError(f"main's planner cannot verify reused results ({type(error).__name__})") from None
+ return module
+
+
+def read_gate(client, run):
+ """The verified ci/required receipt of one passed automatic CI run: (receipt, artifact, archive)."""
+ ci.check_run(run, client.repository, run["id"])
+ if run.get("event") not in {"pull_request", "merge_group"}:
+ raise ValueError("only automatic PR or merge-group CI can authorize queued integration")
+ jobs = client.pages(f"actions/runs/{run['id']}/jobs?filter=latest", "jobs")
+ gates = [j for j in jobs if j["name"] == "ci/required"]
+ if (len(gates) != 1 or gates[0].get("head_sha") != run["head_sha"]
+ or gates[0].get("status") != "completed" or gates[0].get("conclusion") != "success"):
+ raise ValueError("automatic required CI gate did not pass")
+ artifacts = client.pages(f"actions/runs/{run['id']}/artifacts", "artifacts")
+ matches = [a for a in artifacts if a.get("name") == "ci-required"]
+ if len(matches) != 1 or matches[0].get("size_in_bytes", 65537) > 65536:
+ raise ValueError("expected one bounded required-gate receipt")
+ artifact = matches[0]
+ archive = client.command("api", f"repos/{client.repository}/actions/artifacts/{artifact['id']}/zip", binary=True)
+ receipt = ci.summary_contents(artifact, archive, run, "ci-required")
+ commit = client.api(f"git/commits/{run['head_sha']}")
+ if (commit.get("sha") != run["head_sha"] or receipt.get("schema") != 1
+ or receipt.get("kind") != "ci-required" or receipt.get("status") != "passed"
+ or receipt.get("source_sha") != run["head_sha"] or receipt.get("event") != run["event"]
+ or receipt.get("source_tree") != commit.get("tree", {}).get("sha")
+ or receipt.get("run_id") != run["id"] or receipt.get("run_attempt") != gates[0].get("run_attempt")
+ or not receipt.get("required_jobs")
+ or any(receipt.get("results", {}).get(job) != "success" for job in receipt["required_jobs"])
+ or receipt.get("results", {}).get("plan") != "success"):
+ raise ValueError("required-gate receipt has inconsistent source, coverage or run identity")
+ suites = receipt.get("suites")
+ if not isinstance(suites, list) or not set(suites) <= SUITES:
+ raise ValueError("required-gate receipt has invalid suites")
+ reused = check_reused_records(receipt.get("reused", {}), suites, receipt["source_sha"], receipt["event"])
+ if receipt["required_jobs"] != sorted({"process-checks"} | set().union(*(JOBS[name] for name in set(suites) - set(reused)))):
+ raise ValueError("required-gate receipt does not run every suite it did not reuse")
+ after = client.api(f"actions/runs/{run['id']}")
+ ci.check_run(after, client.repository, run["id"])
+ if any(after[key] != run[key] for key in ["head_sha", "run_attempt", "updated_at"]):
+ raise ValueError("CI changed during queue evidence collection")
+ return receipt, artifact, archive
+
+
+class GitHubEvidence:
+ """Passed ci/required runs read from GitHub; every lookup is pinned to an exact run and attempt."""
+
+ def __init__(self, client):
+ self.client = client
+
+ def pr_run(self, sha_value, run_id, attempt):
+ run = self.client.api(f"actions/runs/{run_id}")
+ ci.check_run(run, self.client.repository, run_id)
+ if run.get("event") != "pull_request" or run.get("head_sha") != sha_value:
+ raise ValueError("the recorded run belongs to another source or event")
+ if run.get("run_attempt") != attempt:
+ raise ValueError("the recorded run attempt is no longer its latest attempt")
+ return run
+
+ def passed_gate(self, sha_value):
+ """The run behind the commit's latest ci/required check, which must be a pass."""
+ checks = self.client.pages(f"commits/{sha(sha_value)}/check-runs?check_name=ci%2Frequired&filter=latest", "check_runs")
+ checks = [c for c in checks if c.get("name") == "ci/required" and c.get("head_sha") == sha_value
+ and c.get("app", {}).get("slug") == "github-actions" and positive(c.get("id"))]
+ if not checks:
+ raise ValueError("the passed head has no ci/required check")
+ latest = max(checks, key=lambda c: c["id"])
+ if latest.get("status") != "completed" or latest.get("conclusion") != "success":
+ raise ValueError("the passed head's latest ci/required is not a pass")
+ job = self.client.api(f"actions/jobs/{latest['id']}")
+ if (job.get("id") != latest["id"] or job.get("name") != "ci/required" or job.get("head_sha") != sha_value
+ or job.get("conclusion") != "success" or not positive(job.get("run_id")) or not positive(job.get("run_attempt"))):
+ raise ValueError("the ci/required check does not match its job")
+ run = self.pr_run(sha_value, job["run_id"], job["run_attempt"])
+ receipt = read_gate(self.client, run)[0]
+ return dict(run_id=run["id"], run_attempt=run["run_attempt"], url=run["html_url"], receipt=receipt)
+
+ def ran(self, sha_value, run_id, attempt, suite):
+ """The tested tree and link of a run that itself ran and passed this suite."""
+ run = self.pr_run(sha_value, run_id, attempt)
+ receipt = read_gate(self.client, run)[0]
+ if suite not in receipt["suites"] or suite in receipt.get("reused", {}):
+ raise ValueError("the recorded run did not run this suite")
+ return dict(tree=receipt["source_tree"], url=run["html_url"])
+
+
+def make_plan(root, event_name, event, scope="full", source=None, evidence=None):
head = sha(source or git(root, "rev-parse", "HEAD"))
if git(root, "rev-parse", "HEAD") != head:
raise ValueError("checkout differs from the requested source")
@@ -101,13 +569,20 @@ def make_plan(root, event_name, event, scope="full", source=None):
if scope not in MANUAL:
raise ValueError("invalid manual CI scope")
suites = MANUAL[scope]
- required = {"process-checks"} | set().union(*(JOBS[name] for name in suites))
+ reused, reuse = {}, dict(status="none", reason="only a ready PR's update merge can reuse", rerun={})
+ if event_name == "pull_request" and not draft and suites:
+ reused, reuse = plan_reuse(root, head, base_sha(event), suites, evidence)
+ required = {"process-checks"} | set().union(*(JOBS[name] for name in suites - set(reused)))
return dict(schema=1, kind="ci-plan", event=event_name, head=head, base=base, draft=draft,
tree=git(root, "rev-parse", "HEAD^{tree}"), paths=paths, reasons=reasons,
- suites=sorted(suites), required_jobs=sorted(required),
+ suites=sorted(suites), required_jobs=sorted(required), reused=reused, reuse=reuse,
manual_acceptance="Review still selects relevant native, real-engine, hardware and visual acceptance.")
+def base_sha(event):
+ return sha(event["pull_request"]["base"]["sha"])
+
+
def verify(plan, needs, source):
if plan.get("schema") != 1 or plan.get("kind") != "ci-plan" or plan.get("head") != sha(source):
raise ValueError("CI plan identity does not match this source")
@@ -117,7 +592,8 @@ def verify(plan, needs, source):
suites = plan.get("suites")
if not isinstance(suites, list) or len(set(suites)) != len(suites) or not set(suites) <= SUITES:
raise ValueError("invalid planned suites")
- expected = {"process-checks"} | set().union(*(JOBS[name] for name in suites))
+ reused = check_reused_records(plan.get("reused", {}), suites, plan["head"], plan.get("event"))
+ expected = {"process-checks"} | set().union(*(JOBS[name] for name in set(suites) - set(reused)))
if plan.get("required_jobs") != sorted(expected):
raise ValueError("CI plan has incomplete job coverage")
if needs.get("plan", {}).get("result") != "success":
@@ -125,15 +601,47 @@ def verify(plan, needs, source):
for job in expected:
if needs.get(job, {}).get("result") != "success":
raise ValueError(f"required CI job did not pass: {job}")
+ for job in set().union(*(JOBS[name] for name in reused)):
+ if needs.get(job, {}).get("result") != "skipped":
+ raise ValueError(f"reused CI job ran again: {job}")
for job, result in needs.items():
if result.get("result") not in {"success", "skipped"}:
raise ValueError(f"CI job did not complete successfully: {job}")
return dict(schema=1, kind="ci-required", status="passed", source_sha=plan["head"],
source_tree=plan["tree"], event=plan["event"], base=plan["base"],
suites=suites, required_jobs=sorted(expected), results={k: v["result"] for k, v in needs.items()},
+ reused=reused, reuse=plan.get("reuse"),
run_id=int(os.environ.get("GITHUB_RUN_ID", "0")), run_attempt=int(os.environ.get("GITHUB_RUN_ATTEMPT", "0")))
+RUN_URL = re.compile(r"https://github\.com/[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*/actions/runs/[1-9][0-9]*")
+
+
+def is_sha(value):
+ return isinstance(value, str) and re.fullmatch(r"[0-9a-f]{40}", value) is not None
+
+
+def positive(value):
+ return type(value) is int and value > 0
+
+
+def check_reused_records(reused, suites, head, event):
+ """The shape of a plan's or receipt's reused results; the merge helper re-derives their substance."""
+ if not isinstance(reused, dict) or not set(reused) <= set(suites):
+ raise ValueError("reused suites must be planned suites")
+ if reused and event != "pull_request":
+ raise ValueError("only a pull request's update merge can reuse results")
+ for suite, record in reused.items():
+ if (not isinstance(record, dict) or not is_sha(record.get("origin_sha")) or record["origin_sha"] == head
+ or not is_sha(record.get("origin_tree"))
+ or not positive(record.get("run_id")) or not positive(record.get("run_attempt"))
+ or not isinstance(record.get("run_url"), str) or not RUN_URL.fullmatch(record["run_url"])
+ or not record["run_url"].endswith(f"/actions/runs/{record['run_id']}")
+ or not isinstance(record.get("reason"), str)):
+ raise ValueError(f"malformed reused result: {suite}")
+ return reused
+
+
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--verify", action="store_true")
@@ -147,17 +655,30 @@ def main():
raise ValueError("CI gate checkout differs from the planned source")
record = verify(plan, json.loads(os.environ["CI_NEEDS"]), os.environ["CI_SOURCE_SHA"])
else:
- record = make_plan(Path.cwd(), os.environ["GITHUB_EVENT_NAME"],
- json.loads(Path(os.environ["GITHUB_EVENT_PATH"]).read_text()), args.scope, os.environ["CI_SOURCE_SHA"])
+ event_name = os.environ["GITHUB_EVENT_NAME"]
+ # Read-only Actions lookups with a one-minute budget; any failure reruns as before.
+ evidence = (GitHubEvidence(ci.Client(os.environ["GITHUB_REPOSITORY"], time.monotonic() + REUSE_LOOKUP_SECONDS))
+ if event_name == "pull_request" and os.environ.get("GH_TOKEN") else None)
+ record = make_plan(Path.cwd(), event_name, json.loads(Path(os.environ["GITHUB_EVENT_PATH"]).read_text()),
+ args.scope, os.environ["CI_SOURCE_SHA"], evidence)
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write("plan=" + json.dumps(record, separators=(",", ":")) + "\n")
output.write(f"ready={'false' if record['draft'] else 'true'}\n")
+ # A reused suite's jobs skip; ci/required accepts that skip only for it.
for suite in sorted(SUITES):
- output.write(f"{suite}={'true' if suite in record['suites'] else 'false'}\n")
+ runs = suite in record["suites"] and suite not in record["reused"]
+ output.write(f"{suite}={'true' if runs else 'false'}\n")
args.output.write_text(json.dumps(record, indent=2) + "\n")
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as summary:
summary.write(f"Source: `{record.get('head', record.get('source_sha'))}`\n\n")
summary.write("Automatic suites: " + ", ".join(record["suites"] or ["repository process checks"]) + ".\n")
+ for suite, reused in sorted((record.get("reused") or {}).items()):
+ summary.write(f"\nReused `{suite}` from [run {reused['run_id']}]({reused['run_url']}) "
+ f"(attempt {reused['run_attempt']}, `{reused['origin_sha']}`): {reused['reason']}.\n")
+ if (record.get("reuse") or {}).get("reason"):
+ summary.write(f"\nReuse: {record['reuse']['reason']}.\n")
+ for suite, reason in sorted(((record.get("reuse") or {}).get("rerun") or {}).items()):
+ summary.write(f"- `{suite}` runs again: {reason}.\n")
if record.get("draft"):
summary.write("Draft PR: only workflow/process checks run; mark ready for review to run the complete affected suites. The integration gate remains blocked.\n")
summary.write("Native, real-engine, hardware and visual acceptance remain explicit review requirements.\n")
diff --git a/scripts/merge-validated-pr.py b/scripts/merge-validated-pr.py
index 076b8bfc0..c425f6734 100644
--- a/scripts/merge-validated-pr.py
+++ b/scripts/merge-validated-pr.py
@@ -112,7 +112,7 @@ def finish(client, root, number, run_id, scope, head, base, output, *, merge=Fal
evidence_dir = output / "ci"
evidence_dir.mkdir()
if automatic:
- evidence = phase("collection", lambda: queue.gate_evidence(client, observed, evidence_dir))
+ evidence = phase("collection", lambda: queue.gate_evidence(client, observed, evidence_dir, root, base))
record["pr_evidence"] = evidence
if (evidence["source_sha"] != head
or evidence["source_tree"] != record["reviewed_source"]["tree"]):
@@ -165,6 +165,8 @@ def finish(client, root, number, run_id, scope, head, base, output, *, merge=Fal
if evidence is not None:
if automatic:
lines += ["", f"Verified [automatic PR CI]({evidence['url']}) for `{evidence['source_sha']}`."]
+ lines += [f"- reused {suite} from {record['run_url']} (attempt {record['run_attempt']}, `{record['origin_sha']}`), "
+ f"re-derived with main's planner: {record['reason']}" for suite, record in sorted(evidence.get("reused", {}).items())]
else:
lines += ["", ci.markdown(evidence)]
(output / "merge.md").write_text("\n".join(lines) + "\n")
diff --git a/scripts/queue-validated-pr.py b/scripts/queue-validated-pr.py
index 66c96339e..c5adaa83c 100644
--- a/scripts/queue-validated-pr.py
+++ b/scripts/queue-validated-pr.py
@@ -16,6 +16,10 @@ spec = importlib.util.spec_from_file_location("queue_ci_evidence", Path(__file__
ci = importlib.util.module_from_spec(spec)
spec.loader.exec_module(ci)
+planner_spec = importlib.util.spec_from_file_location("queue_ci_plan", Path(__file__).with_name("ci-plan.py"))
+planner = importlib.util.module_from_spec(planner_spec)
+planner_spec.loader.exec_module(planner)
+
STATE_QUERY = """query($owner:String!,$name:String!,$number:Int!) {
repository(owner:$owner,name:$name) { pullRequest(number:$number) {
id number state isDraft headRefOid baseRefName mergedAt
@@ -49,42 +53,21 @@ def state(client, number, head, branch):
return pr
-def gate_evidence(client, run, output):
- ci.check_run(run, client.repository, run["id"])
- if run.get("event") not in {"pull_request", "merge_group"}:
- raise ValueError("only automatic PR or merge-group CI can authorize queued integration")
- jobs = client.pages(f"actions/runs/{run['id']}/jobs?filter=latest", "jobs")
- gates = [j for j in jobs if j["name"] == "ci/required"]
- if (len(gates) != 1 or gates[0].get("head_sha") != run["head_sha"]
- or gates[0].get("status") != "completed" or gates[0].get("conclusion") != "success"):
- raise ValueError("automatic required CI gate did not pass")
- artifacts = client.pages(f"actions/runs/{run['id']}/artifacts", "artifacts")
- matches = [a for a in artifacts if a.get("name") == "ci-required"]
- if len(matches) != 1 or matches[0].get("size_in_bytes", 65537) > 65536:
- raise ValueError("expected one bounded required-gate receipt")
- artifact = matches[0]
- archive = client.command("api", f"repos/{client.repository}/actions/artifacts/{artifact['id']}/zip", binary=True)
- receipt = ci.summary_contents(artifact, archive, run, "ci-required")
- commit = client.api(f"git/commits/{run['head_sha']}")
- if (commit.get("sha") != run["head_sha"] or receipt.get("schema") != 1
- or receipt.get("kind") != "ci-required" or receipt.get("status") != "passed"
- or receipt.get("source_sha") != run["head_sha"] or receipt.get("event") != run["event"]
- or receipt.get("source_tree") != commit.get("tree", {}).get("sha")
- or receipt.get("run_id") != run["id"] or receipt.get("run_attempt") != gates[0].get("run_attempt")
- or not receipt.get("required_jobs")
- or any(receipt.get("results", {}).get(job) != "success" for job in receipt["required_jobs"])
- or receipt.get("results", {}).get("plan") != "success"):
- raise ValueError("required-gate receipt has inconsistent source, coverage or run identity")
- after = client.api(f"actions/runs/{run['id']}")
- ci.check_run(after, client.repository, run["id"])
- if any(after[key] != run[key] for key in ["head_sha", "run_attempt", "updated_at"]):
- raise ValueError("CI changed during queue evidence collection")
+def gate_evidence(client, run, output, root=None, base=None):
+ receipt, artifact, archive = planner.read_gate(client, run)
+ reused = receipt.get("reused") or {}
+ if reused:
+ # PR CI ran the PR's own planner. Re-derive its reuse with main's copy, from the reviewed checkout.
+ if root is None or base is None or run.get("event") != "pull_request":
+ raise ValueError("reused CI results can only be verified for a PR run against the reviewed checkout")
+ main = planner.main_planner(root, base)
+ main.verify_reuse(root, receipt, base, main.GitHubEvidence(client))
output.mkdir(parents=True, exist_ok=True)
(output / "ci-required.zip").write_bytes(archive)
(output / "receipt.json").write_text(json.dumps(receipt, indent=2) + "\n")
return dict(run_id=run["id"], run_attempt=run["run_attempt"], url=run["html_url"],
source_sha=receipt["source_sha"], source_tree=receipt["source_tree"],
- suites=receipt["suites"], artifact_id=artifact["id"], artifact_digest=artifact["digest"])
+ suites=receipt["suites"], reused=reused, artifact_id=artifact["id"], artifact_digest=artifact["digest"])
def pr_run(client, head):
@@ -96,8 +79,8 @@ def pr_run(client, head):
return candidates[0]
-def pr_gate(client, head, output):
- return gate_evidence(client, pr_run(client, head), output)
+def pr_gate(client, head, output, root=None, base=None):
+ return gate_evidence(client, pr_run(client, head), output, root, base)
def merged_gate(client, tree, branch, started_at, output):
@@ -147,7 +130,7 @@ def finish(client, root, number, head, base, output, *, merge=False, wait_timeou
pr = phase("preflight", lambda: state(client, number, head, branch))
if pr["state"] != "OPEN":
raise ValueError("PR is not open")
- record["pr_evidence"] = phase("pr_validation", lambda: pr_gate(client, head, output / "pr-ci"))
+ record["pr_evidence"] = phase("pr_validation", lambda: pr_gate(client, head, output / "pr-ci", root, base))
pr = state(client, number, head, branch)
if pr["state"] != "OPEN":
raise ValueError("PR state changed before enqueueing")
diff --git a/scripts/tests/test_ci_plan.py b/scripts/tests/test_ci_plan.py
index 7a051a0bf..8b14afd9d 100644
--- a/scripts/tests/test_ci_plan.py
+++ b/scripts/tests/test_ci_plan.py
@@ -1,10 +1,15 @@
import copy
+import hashlib
import importlib.util
+import io
+import itertools
import json
from pathlib import Path
+import re
import subprocess
import tempfile
import unittest
+import zipfile
spec = importlib.util.spec_from_file_location("ci_plan", Path(__file__).resolve().parents[1] / "ci-plan.py")
planner = importlib.util.module_from_spec(spec)
@@ -55,6 +60,30 @@ class SelectionTests(unittest.TestCase):
self.assertEqual(planner.select(["cli/src/lib/windows/__fixtures__/pipe-auth-vectors.json"])[0], {"cli", "desktop"})
self.assertEqual(planner.select(["cli/src/lib/windows/pipeAuth.ts"])[0], {"cli"})
+ def test_a_style_allowlist_selects_the_suite_whose_job_checks_it(self):
+ # mobile-checks and desktop-tests run scripts/check-ui-style.py --only against their area's list.
+ self.assertEqual(planner.select(["scripts/ui-style-allowlist/mobile.json"])[0], {"mobile"})
+ self.assertEqual(planner.select(["scripts/ui-style-allowlist/desktop.json"])[0], {"desktop"})
+ self.assertEqual(planner.select(["scripts/ui-style-allowlist/website.json"])[0], set())
+
+ def test_the_style_allowlist_map_matches_the_style_check_and_the_workflow(self):
+ scripts = Path(__file__).resolve().parents[1]
+ style_spec = importlib.util.spec_from_file_location("check_ui_style", scripts / "check-ui-style.py")
+ style = importlib.util.module_from_spec(style_spec)
+ style_spec.loader.exec_module(style)
+ expected = {f"{style.ALLOWLISTS}/{area}.json": planner.select([folder + "/x.dart"])[0]
+ for folder, area in style.ROOTS.items()}
+ self.assertEqual({path: {suite} for path, suite in planner.STYLE_ALLOWLISTS.items()}, expected)
+ # Each area's check runs inside a job of the suite the list selects.
+ workflow = (scripts.parent / ".github/workflows/ci.yml").read_text()
+ jobs = dict(re.findall(r"(?m)^ ([\w-]+):\n((?:(?: .*)?\n)+)", workflow))
+ for folder, area in style.ROOTS.items():
+ with self.subTest(area=area):
+ suite = planner.STYLE_ALLOWLISTS[f"{style.ALLOWLISTS}/{area}.json"]
+ checking = {name for name, body in jobs.items() if f"check-ui-style.py --only {folder}" in body}
+ self.assertTrue(checking)
+ self.assertLessEqual(checking, planner.JOBS[suite])
+
def test_the_shared_terminal_crate_selects_both_the_pty_host_and_the_tui(self):
# term/ is a dependency of the PTY host and of the shipped TUI (`hn`): both suites run.
for path in ("term/src/capture.rs", "term/Cargo.toml"):
@@ -184,6 +213,281 @@ class GitPlanTests(unittest.TestCase):
planner.make_plan(self.root, "workflow_dispatch", {}, "typo")
+class FakeEvidence:
+ """Passed ci/required gates by commit, as the GitHub reader returns them after its own checks."""
+
+ def __init__(self):
+ self.gates, self.failure, self.lookups = {}, None, []
+
+ def add(self, root, sha, suites, run_id, reused=None):
+ tree = subprocess.check_output(["git", "rev-parse", f"{sha}^{{tree}}"], cwd=root, text=True).strip()
+ receipt = dict(schema=1, kind="ci-required", status="passed", source_sha=sha, source_tree=tree,
+ event="pull_request", suites=sorted(suites), reused=reused or {}, run_id=run_id, run_attempt=1)
+ self.gates[sha] = dict(run_id=run_id, run_attempt=1, url=f"https://github.com/o/r/actions/runs/{run_id}",
+ receipt=receipt)
+
+ def passed_gate(self, sha):
+ self.lookups.append(("gate", sha))
+ if self.failure:
+ raise self.failure
+ if sha not in self.gates:
+ raise ValueError("the passed head's latest ci/required is not a pass")
+ return copy.deepcopy(self.gates[sha])
+
+ def ran(self, sha, run_id, run_attempt, suite):
+ self.lookups.append(("ran", sha, run_id, suite))
+ gate = self.gates.get(sha)
+ if (not gate or gate["run_id"] != run_id or gate["run_attempt"] != run_attempt
+ or suite not in gate["receipt"]["suites"] or suite in gate["receipt"]["reused"]):
+ raise ValueError("the recorded run did not run this suite")
+ return dict(tree=gate["receipt"]["source_tree"], url=gate["url"])
+
+
+class ReuseFixture(unittest.TestCase):
+ """A repository where GitHub's Update branch merges main into a head whose ci/required passed."""
+
+ def setUp(self):
+ tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(tmp.cleanup)
+ self.root = Path(tmp.name)
+ self.git("init", "-q", "-b", "main")
+ self.git("config", "user.email", "[email protected]")
+ self.git("config", "user.name", "Fixture")
+ for path in ["README.md", "docs/guide.md", "docs/images/poster.png", "mobile/lib/app.dart", "backend/src/app.ts",
+ "backend/src/types/productUsage.ts", "cli/src/a.ts", "desktop/lib/a.dart", "scripts/ci-plan.py",
+ ".github/workflows/ci.yml", "Makefile", "store/catalog.json", "cli/package-lock.json",
+ "mobile/pubspec.lock", "tui/rust-toolchain.toml", ".gitmodules"]:
+ self.write(path, "base " + path)
+ self.base = self.commit()
+ self.evidence = FakeEvidence()
+
+ def git(self, *args):
+ return subprocess.check_output(["git", *args], cwd=self.root, stderr=subprocess.PIPE, text=True).strip()
+
+ def write(self, path, value):
+ target = self.root / path
+ target.parent.mkdir(parents=True, exist_ok=True)
+ target.write_text(value)
+
+ def commit(self, message="fixture"):
+ self.git("add", "-A")
+ self.git("commit", "-qm", message)
+ return self.git("rev-parse", "HEAD")
+
+ def branch(self, start, paths, name):
+ self.git("checkout", "-q", "-B", name, start)
+ for path in paths:
+ self.write(path, f"{name} {path}")
+ return self.commit(name)
+
+ def update(self, passed, main):
+ """GitHub's Update branch: merge main into the passed head, first parent the head."""
+ self.git("checkout", "-q", "--detach", passed)
+ self.git("merge", "-q", "--no-ff", "--no-edit", main)
+ return self.git("rev-parse", "HEAD")
+
+ def plan(self, head, base, evidence=True, draft=False):
+ self.git("checkout", "-q", "--detach", head)
+ payload = {"pull_request": {"head": {"sha": head}, "base": {"sha": base}, "draft": draft}}
+ return planner.make_plan(self.root, "pull_request", payload, source=head,
+ evidence=self.evidence if evidence else None)
+
+ def scenario(self, pr_paths, main_paths, passed_suites=None):
+ passed = self.branch(self.base, pr_paths, "pr")
+ suites = passed_suites if passed_suites is not None else planner.select(pr_paths)[0]
+ self.evidence.add(self.root, passed, suites, run_id=11)
+ main = self.branch(self.base, main_paths, "main")
+ return passed, main, self.update(passed, main)
+
+ def assertRerunsAll(self, plan, reason):
+ self.assertEqual(plan["reused"], {})
+ self.assertEqual(plan["reuse"]["status"], "none")
+ self.assertIn(reason, plan["reuse"]["reason"])
+ self.assertEqual(plan["required_jobs"], sorted({"process-checks"} | set().union(*(planner.JOBS[s] for s in plan["suites"]))))
+
+
+
+class ReuseTests(ReuseFixture):
+ """Reuse what main's new commits left untouched; rerun everything on any doubt."""
+
+ def test_an_unrelated_main_change_reuses_the_passed_heads_result(self):
+ passed, main, head = self.scenario(["mobile/lib/app.dart"], ["backend/src/app.ts"])
+ plan = self.plan(head, main)
+ self.assertEqual(plan["suites"], ["mobile"])
+ self.assertEqual(plan["required_jobs"], ["process-checks"])
+ reused = plan["reused"]["mobile"]
+ self.assertEqual(reused["origin_sha"], passed)
+ self.assertEqual(reused["origin_tree"], self.git("rev-parse", f"{passed}^{{tree}}"))
+ self.assertEqual((reused["run_id"], reused["run_attempt"]), (11, 1))
+ self.assertEqual(reused["run_url"], "https://github.com/o/r/actions/runs/11")
+ self.assertIn("backend/src/app.ts", reused["reason"])
+ self.assertEqual(plan["reuse"]["status"], "applied")
+ self.assertEqual((plan["reuse"]["passed_head"], plan["reuse"]["main_parent"]), (passed, main))
+
+ def test_docs_and_top_level_notes_on_main_do_not_force_a_rerun(self):
+ _, main, head = self.scenario(["mobile/lib/app.dart"], ["docs/guide.md", "README.md"])
+ self.assertEqual(list(self.plan(head, main)["reused"]), ["mobile"])
+
+ def test_a_related_main_change_reruns_only_that_component(self):
+ _, main, head = self.scenario(["mobile/lib/app.dart", "backend/src/app.ts"], ["backend/src/other.ts"])
+ plan = self.plan(head, main)
+ self.assertEqual(plan["suites"], ["backend", "mobile"])
+ self.assertEqual(list(plan["reused"]), ["mobile"])
+ self.assertEqual(plan["required_jobs"], sorted({"process-checks"} | planner.JOBS["backend"]))
+ self.assertIn("backend/src/other.ts", plan["reuse"]["rerun"]["backend"])
+
+ def test_tests_outside_the_pr_suite_do_not_rerun_it(self):
+ # backend-desk runs src/**/*.test.ts only; cli-tests and cli-typecheck cover cli/src only.
+ _, main, head = self.scenario(["backend/src/app.ts", "cli/src/a.ts"], ["backend/integration/teams.test.ts", "cli/e2e/run.e2e.ts"])
+ plan = self.plan(head, main)
+ self.assertEqual(list(plan["reused"]), ["backend", "cli"])
+ self.assertEqual(planner.select(["backend/integration/teams.test.ts"])[0], {"backend"})
+
+ def test_a_style_allowlist_main_changed_reruns_the_suite_that_checks_it(self):
+ # mobile-checks runs the mobile style check: a list main shrank can fail with this PR's code.
+ for area, suite, other in (("mobile", "mobile", "backend"), ("desktop", "desktop", "mobile")):
+ with self.subTest(area=area):
+ pr_paths = {"mobile": ["mobile/lib/app.dart", "backend/src/app.ts"],
+ "desktop": ["desktop/lib/a.dart", "mobile/lib/app.dart"]}[area]
+ _, main, head = self.scenario(pr_paths, [f"scripts/ui-style-allowlist/{area}.json"])
+ plan = self.plan(head, main)
+ self.assertEqual(list(plan["reused"]), [other])
+ self.assertIn(f"scripts/ui-style-allowlist/{area}.json", plan["reuse"]["rerun"][suite])
+ self.assertEqual(plan["required_jobs"], sorted({"process-checks"} | planner.JOBS[suite]))
+
+ def test_a_style_allowlist_the_pr_changes_is_its_components_input_not_a_shared_one(self):
+ # The mobile batch that waited behind a backend batch also shrank its style allowlist (#2154).
+ _, main, head = self.scenario(["mobile/lib/app.dart", "scripts/ui-style-allowlist/mobile.json"],
+ ["backend/src/app.ts", "scripts/ui-style-allowlist/desktop.json"])
+ plan = self.plan(head, main)
+ self.assertEqual(plan["suites"], ["mobile"])
+ self.assertEqual(list(plan["reused"]), ["mobile"])
+
+ def test_an_unknown_style_allowlist_on_main_reruns_every_component(self):
+ _, main, head = self.scenario(["mobile/lib/app.dart"], ["scripts/ui-style-allowlist/website.json"])
+ self.assertRerunsAll(self.plan(head, main), "scripts/ui-style-allowlist/website.json")
+
+ def test_the_os_suite_reruns_when_main_changes_the_dial_ids_it_checks(self):
+ # os/tests/test_lock_and_screenshot.py reads cli/src/cable/serial.ts from the repository root.
+ _, main, head = self.scenario(["os/root/usr/lib/udev/rules.d/70-harness-device.rules"], ["cli/src/cable/serial.ts"])
+ plan = self.plan(head, main)
+ self.assertEqual(plan["suites"], ["os"])
+ self.assertEqual(plan["reused"], {})
+ self.assertIn("cli/src/cable/serial.ts", plan["reuse"]["rerun"]["os"])
+ self.assertEqual(plan["required_jobs"], sorted({"process-checks"} | planner.JOBS["os"]))
+
+ def test_a_file_another_component_reads_reruns_the_reader(self):
+ # mobile's measurement tests read the backend's product usage types.
+ _, main, head = self.scenario(["mobile/lib/app.dart"], ["backend/src/types/productUsage.ts"])
+ plan = self.plan(head, main)
+ self.assertEqual(plan["reused"], {})
+ self.assertIn("backend/src/types/productUsage.ts", plan["reuse"]["rerun"]["mobile"])
+ self.assertEqual(plan["required_jobs"], sorted({"process-checks"} | planner.JOBS["mobile"]))
+
+ def test_shared_inputs_on_main_rerun_every_component(self):
+ passed = self.branch(self.base, ["mobile/lib/app.dart", "desktop/lib/a.dart"], "pr")
+ self.evidence.add(self.root, passed, {"desktop", "mobile"}, run_id=11)
+ for index, path in enumerate([".github/workflows/ci.yml", "scripts/ci-plan.py", "Makefile", ".gitmodules",
+ "cli/package-lock.json", "mobile/pubspec.lock", "tui/rust-toolchain.toml",
+ "store/catalog.json", "docs/images/poster.png", "unknown-top-level.txt"]):
+ with self.subTest(path=path):
+ main = self.branch(self.base, [path], f"main-{index}")
+ plan = self.plan(self.update(passed, main), main)
+ self.assertRerunsAll(plan, path)
+ self.assertEqual(plan["suites"], ["desktop", "mobile"])
+
+ def test_a_pr_that_edits_the_planner_or_workflows_never_reuses(self):
+ main = self.branch(self.base, ["backend/src/app.ts"], "main")
+ for index, path in enumerate(["scripts/ci-plan.py", ".github/workflows/ci.yml", "scripts/merge-validated-pr.py", "Makefile"]):
+ with self.subTest(path=path):
+ passed = self.branch(self.base, ["mobile/lib/app.dart", path], f"pr-{index}")
+ self.evidence.add(self.root, passed, {"mobile"}, run_id=11 + index)
+ self.assertRerunsAll(self.plan(self.update(passed, main), main), path)
+
+ def test_a_head_that_is_not_exactly_main_merged_into_the_passed_head_reruns(self):
+ passed, main, merged = self.scenario(["mobile/lib/app.dart"], ["backend/src/app.ts"])
+ # A plain commit on top of the passed head.
+ self.git("checkout", "-q", "--detach", passed)
+ self.write("mobile/lib/app.dart", "edited")
+ plain = self.commit()
+ self.assertRerunsAll(self.plan(plain, main), "not a merge")
+ # An octopus merge.
+ other = self.branch(self.base, ["backend/src/other.ts"], "other")
+ self.git("checkout", "-q", "--detach", passed)
+ self.git("merge", "-q", "--no-ff", "--no-edit", main, other)
+ self.assertRerunsAll(self.plan(self.git("rev-parse", "HEAD"), main), "not a merge")
+ # A second parent that is not on the PR's base branch.
+ side = self.branch(self.base, ["backend/src/side.ts"], "side")
+ self.assertRerunsAll(self.plan(self.update(passed, side), main), "not on the base branch")
+ # Main merged into a commit with no passed gate.
+ unpassed = self.branch(passed, ["mobile/lib/app.dart"], "later")
+ self.assertRerunsAll(self.plan(self.update(unpassed, main), main), "not a pass")
+ # Parents reversed: main first.
+ self.git("checkout", "-q", "--detach", main)
+ self.git("merge", "-q", "--no-ff", "--no-edit", passed)
+ self.assertRerunsAll(self.plan(self.git("rev-parse", "HEAD"), main), "not on the base branch")
+
+ def test_unreadable_evidence_or_no_reader_reruns(self):
+ _, main, head = self.scenario(["mobile/lib/app.dart"], ["backend/src/app.ts"])
+ self.assertRerunsAll(self.plan(head, main, evidence=False), "no CI evidence reader")
+ for error in [RuntimeError("GitHub lookup failed (1): api"), TimeoutError("deadline"), KeyError("jobs"),
+ ValueError("summary archive digest mismatch"), OSError("gh missing")]:
+ with self.subTest(error=error):
+ self.evidence.failure = error
+ self.assertRerunsAll(self.plan(head, main), "could not be verified")
+
+ def test_a_component_the_passed_run_did_not_cover_reruns(self):
+ _, main, head = self.scenario(["mobile/lib/app.dart", "cli/src/a.ts"], ["backend/src/app.ts"], passed_suites={"mobile"})
+ plan = self.plan(head, main)
+ self.assertEqual(list(plan["reused"]), ["mobile"])
+ self.assertIn("did not run", plan["reuse"]["rerun"]["cli"])
+
+ def test_a_merge_resolution_inside_a_component_reruns_it(self):
+ passed, main, _ = self.scenario(["mobile/lib/app.dart"], ["backend/src/app.ts"])
+ self.git("checkout", "-q", "--detach", passed)
+ self.git("merge", "-q", "--no-ff", "--no-commit", main)
+ self.write("mobile/lib/app.dart", "resolved by hand")
+ self.git("add", "-A")
+ self.git("commit", "-qm", "merge main")
+ plan = self.plan(self.git("rev-parse", "HEAD"), main)
+ self.assertEqual(plan["reused"], {})
+ self.assertIn("mobile/lib/app.dart", plan["reuse"]["rerun"]["mobile"])
+
+ def test_a_second_update_reuses_the_run_where_the_component_ran(self):
+ passed, main, first = self.scenario(["mobile/lib/app.dart"], ["backend/src/app.ts"])
+ reused = self.plan(first, main)["reused"]
+ self.evidence.add(self.root, first, {"mobile"}, run_id=12, reused=reused)
+ later_main = self.branch(main, ["backend/src/other.ts"], "main")
+ second = self.update(first, later_main)
+ plan = self.plan(second, later_main)
+ self.assertEqual(plan["reused"]["mobile"]["origin_sha"], passed)
+ self.assertEqual(plan["reused"]["mobile"]["run_id"], 11)
+ self.assertIn(("ran", passed, 11, "mobile"), self.evidence.lookups)
+ # The origin run can no longer be verified: rerun.
+ del self.evidence.gates[passed]
+ self.assertRerunsAll(self.plan(second, later_main), "could not be verified")
+
+ def test_a_recorded_origin_that_is_not_an_ancestor_reruns(self):
+ passed, main, first = self.scenario(["mobile/lib/app.dart"], ["backend/src/app.ts"])
+ stranger = self.branch(self.base, ["mobile/lib/app.dart"], "stranger")
+ self.evidence.add(self.root, stranger, {"mobile"}, run_id=99)
+ forged = {"mobile": dict(origin_sha=stranger, origin_tree=self.git("rev-parse", f"{stranger}^{{tree}}"),
+ run_id=99, run_attempt=1, run_url="https://github.com/o/r/actions/runs/99", reason="x")}
+ self.evidence.add(self.root, first, {"mobile"}, run_id=12, reused=forged)
+ later_main = self.branch(main, ["backend/src/other.ts"], "main")
+ plan = self.plan(self.update(first, later_main), later_main)
+ self.assertEqual(plan["reused"], {})
+ self.assertIn("not an ancestor", plan["reuse"]["rerun"]["mobile"])
+
+ def test_draft_and_merge_group_plans_never_reuse(self):
+ _, main, head = self.scenario(["mobile/lib/app.dart"], ["backend/src/app.ts"])
+ self.assertEqual(self.plan(head, main, draft=True)["reused"], {})
+ self.assertEqual(self.evidence.lookups, [])
+ payload = {"action": "checks_requested", "merge_group": {"head_sha": head, "base_sha": main}}
+ self.assertEqual(planner.make_plan(self.root, "merge_group", payload, source=head, evidence=self.evidence)["reused"], {})
+ self.assertEqual(self.evidence.lookups, [])
+
+
class GateTests(unittest.TestCase):
def setUp(self):
self.head = "a" * 40
@@ -231,5 +535,449 @@ class GateTests(unittest.TestCase):
planner.verify(dict(self.plan, **patch), self.needs, self.head)
+class VerifyReuseTests(ReuseFixture):
+ """The merge helper re-derives a receipt's reuse from Git and the exact runs it names."""
+
+ def receipt(self, plan):
+ return dict(source_sha=plan["head"], suites=plan["suites"], reused=plan["reused"], reuse=plan["reuse"],
+ event="pull_request")
+
+ def reusing(self, pr_paths=("mobile/lib/app.dart",), main_paths=("backend/src/app.ts",)):
+ passed, main, head = self.scenario(list(pr_paths), list(main_paths))
+ plan = self.plan(head, main)
+ self.assertTrue(plan["reused"])
+ return passed, main, head, self.receipt(plan)
+
+ def test_a_planned_reuse_verifies(self):
+ _, main, head, receipt = self.reusing()
+ self.assertEqual(planner.verify_reuse(self.root, receipt, main, self.evidence), {"mobile": receipt["reused"]["mobile"]})
+ later = self.branch(main, ["docs/guide.md"], "main")
+ self.git("checkout", "-q", "--detach", head)
+ planner.verify_reuse(self.root, receipt, later, self.evidence)
+
+ def test_a_claimed_reuse_main_invalidated_is_refused(self):
+ passed, main, head = self.scenario(["mobile/lib/app.dart"], ["backend/src/types/productUsage.ts"])
+ tree = self.git("rev-parse", f"{passed}^{{tree}}")
+ forged = dict(source_sha=head, suites=["mobile"], event="pull_request",
+ reuse=dict(passed_head=passed, main_parent=main),
+ reused={"mobile": dict(origin_sha=passed, origin_tree=tree, run_id=11, run_attempt=1,
+ run_url="https://github.com/o/r/actions/runs/11", reason="trust me")})
+ with self.assertRaisesRegex(ValueError, "productUsage"):
+ planner.verify_reuse(self.root, forged, main, self.evidence)
+
+ def test_a_claimed_reuse_past_a_shrunk_style_allowlist_is_refused(self):
+ passed, main, head = self.scenario(["mobile/lib/app.dart"], ["scripts/ui-style-allowlist/mobile.json"])
+ tree = self.git("rev-parse", f"{passed}^{{tree}}")
+ forged = dict(source_sha=head, suites=["mobile"], event="pull_request",
+ reuse=dict(passed_head=passed, main_parent=main),
+ reused={"mobile": dict(origin_sha=passed, origin_tree=tree, run_id=11, run_attempt=1,
+ run_url="https://github.com/o/r/actions/runs/11", reason="trust me")})
+ with self.assertRaisesRegex(ValueError, "ui-style-allowlist/mobile.json"):
+ planner.verify_reuse(self.root, forged, main, self.evidence)
+
+ def test_a_pr_that_edits_the_planner_cannot_have_reuse_accepted(self):
+ _, main, head, receipt = self.reusing()
+ self.git("checkout", "-q", "--detach", head)
+ self.write("scripts/ci-plan.py", "def plan_reuse(*a): return everything")
+ edited = self.commit("PR edits the planner")
+ later = self.branch(main, ["backend/src/other.ts"], "main")
+ merged = self.update(edited, later)
+ self.evidence.add(self.root, edited, {"mobile"}, run_id=21)
+ forged = dict(receipt, source_sha=merged, reuse=dict(receipt["reuse"], passed_head=edited))
+ with self.assertRaisesRegex(ValueError, "scripts/ci-plan.py"):
+ planner.verify_reuse(self.root, dict(forged, reuse=dict(forged["reuse"], main_parent=later)), later, self.evidence)
+
+ def test_an_undeclared_read_of_a_file_main_changed_is_refused(self):
+ # CI's sparse planner cannot see the new read; the helper's full checkout can.
+ _, main, head, receipt = self.reusing(pr_paths=("mobile/lib/app.dart", "mobile/test/new_test.dart"))
+ self.git("checkout", "-q", "--detach", head)
+ (self.root / "mobile/test/new_test.dart").write_text("File('../backend/src/app.ts').readAsStringSync();")
+ self.git("commit", "-qam", "reads the backend")
+ reader = self.git("rev-parse", "HEAD")
+ self.evidence.add(self.root, reader, {"mobile"}, run_id=31)
+ for name, main_paths, refused in [("main-a", ["backend/src/other.ts"], False), ("main-b", ["backend/src/app.ts"], True)]:
+ with self.subTest(main=main_paths):
+ later = self.branch(main, main_paths, name)
+ head = self.update(reader, later)
+ plan = self.plan(head, later)
+ self.assertEqual(list(plan["reused"]), ["mobile"])
+ if refused:
+ with self.assertRaisesRegex(ValueError, "mobile/test/new_test.dart reads backend/src/app.ts"):
+ planner.verify_reuse(self.root, self.receipt(plan), later, self.evidence)
+ else:
+ planner.verify_reuse(self.root, self.receipt(plan), later, self.evidence)
+
+ def test_an_undeclared_read_from_the_repository_root_is_refused(self):
+ _, main, head, receipt = self.reusing(pr_paths=("mobile/lib/app.dart", "mobile/test/root_test.dart"))
+ self.git("checkout", "-q", "--detach", head)
+ (self.root / "mobile/test/root_test.dart").write_text("File(p.join(repo, 'backend/src/app.ts')).readAsStringSync();")
+ self.git("commit", "-qam", "reads the backend from the root")
+ reader = self.git("rev-parse", "HEAD")
+ self.evidence.add(self.root, reader, {"mobile"}, run_id=41)
+ later = self.branch(main, ["backend/src/app.ts"], "main-root")
+ head = self.update(reader, later)
+ plan = self.plan(head, later)
+ self.assertEqual(list(plan["reused"]), ["mobile"])
+ with self.assertRaisesRegex(ValueError, "mobile/test/root_test.dart reads backend/src/app.ts"):
+ planner.verify_reuse(self.root, self.receipt(plan), later, self.evidence)
+
+ def test_a_look_alike_inside_the_component_cannot_hide_a_root_read_from_the_helper(self):
+ for (name, line), deleted in itertools.product(
+ (("root", "File(p.join(repo, 'backend/src/app.ts')).readAsStringSync();"),
+ ("relative", "File('../backend/src/app.ts').readAsStringSync();")), (False, True)):
+ with self.subTest(form=name, main_deletes_it=deleted):
+ _, main, head, receipt = self.reusing(pr_paths=("mobile/lib/app.dart", "mobile/test/root_test.dart"))
+ self.git("checkout", "-q", "--detach", head)
+ (self.root / "mobile/test/root_test.dart").write_text(line)
+ self.write("mobile/backend/src/app.ts", "an unused look-alike")
+ reader = self.commit(f"reads the backend past a look-alike ({name})")
+ self.evidence.add(self.root, reader, {"mobile"}, run_id=51)
+ if deleted: # The real read now fails; the look-alike must not make it look internal.
+ self.git("checkout", "-q", "-B", f"main-delete-{name}", main)
+ self.git("rm", "-q", "backend/src/app.ts")
+ later = self.commit("main deletes the file")
+ else:
+ later = self.branch(main, ["backend/src/app.ts"], f"main-look-alike-{name}")
+ merged = self.update(reader, later)
+ plan = self.plan(merged, later)
+ self.assertEqual(list(plan["reused"]), ["mobile"])
+ with self.assertRaisesRegex(ValueError, "mobile/test/root_test.dart reads backend/src/app.ts"):
+ planner.verify_reuse(self.root, self.receipt(plan), later, self.evidence)
+
+ def test_mismatched_parents_run_tree_or_link_are_refused(self):
+ passed, main, head, receipt = self.reusing()
+ record = receipt["reused"]["mobile"]
+ for patch, reason in [(dict(reuse=dict(receipt["reuse"], passed_head=main)), "parents"),
+ (dict(reuse=dict(receipt["reuse"], main_parent=passed)), "parents"),
+ (dict(reused={"mobile": dict(record, origin_tree="f" * 40)}), "tree"),
+ (dict(reused={"mobile": dict(record, run_url="https://github.com/o/other/actions/runs/11")}), "link"),
+ (dict(reused={"mobile": dict(record, run_id=12, run_url="https://github.com/o/r/actions/runs/12")}),
+ "did not run")]:
+ with self.subTest(reason=reason, patch=list(patch)), self.assertRaisesRegex(ValueError, reason):
+ planner.verify_reuse(self.root, dict(receipt, **patch), main, self.evidence)
+ self.git("checkout", "-q", "--detach", head)
+ self.write("docs/guide.md", "a plain commit after the update")
+ plain = self.commit()
+ with self.assertRaisesRegex(ValueError, "not a merge"):
+ planner.verify_reuse(self.root, dict(receipt, source_sha=plain), main, self.evidence)
+
+
+class MainPlannerTests(unittest.TestCase):
+ def setUp(self):
+ tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(tmp.cleanup)
+ self.root = Path(tmp.name)
+ self.git("init", "-q")
+ self.git("config", "user.email", "[email protected]")
+ self.git("config", "user.name", "Fixture")
+
+ def git(self, *args):
+ return subprocess.check_output(["git", *args], cwd=self.root, stderr=subprocess.PIPE, text=True).strip()
+
+ def commit_scripts(self, planner_source):
+ scripts = Path(__file__).resolve().parents[1]
+ (self.root / "scripts").mkdir(exist_ok=True)
+ (self.root / "scripts/ci-plan.py").write_text(planner_source)
+ (self.root / "scripts/record-ci-validation.py").write_text((scripts / "record-ci-validation.py").read_text())
+ self.git("add", "-A")
+ self.git("commit", "-qm", "main")
+ return self.git("rev-parse", "HEAD")
+
+ def test_main_planner_is_read_from_the_base_commit_not_the_checkout(self):
+ real = (Path(__file__).resolve().parents[1] / "ci-plan.py").read_text()
+ base = self.commit_scripts(real)
+ (self.root / "scripts/ci-plan.py").write_text(real.replace("def verify_reuse(", "def verify_reuse_disabled("))
+ module = planner.main_planner(self.root, base)
+ self.assertTrue(callable(module.verify_reuse))
+ self.assertIsNot(module, planner)
+
+ def test_a_base_without_a_reuse_verifier_or_a_broken_planner_refuses(self):
+ for source in ["def make_plan(): pass\n", "this is not python", ""]:
+ with self.subTest(source=source[:12]):
+ base = self.commit_scripts(source) if source else self.git("rev-parse", "HEAD")
+ with self.assertRaisesRegex(ValueError, "main's planner cannot verify reused results"):
+ planner.main_planner(self.root, base)
+ with self.assertRaisesRegex(ValueError, "main's planner cannot verify reused results"):
+ planner.main_planner(self.root, "f" * 40)
+
+
+class CrossReadTests(unittest.TestCase):
+ """A suite's reads outside its own directory must be declared before its results may be reused."""
+
+ def setUp(self):
+ tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(tmp.cleanup)
+ self.root = Path(tmp.name)
+ self.git("init", "-q")
+ for path in ["backend/src/types/productUsage.ts", "backend/src/app.ts", "cli/src/lib/a.ts", "desktop/lib/devices/x.dart",
+ "docs/contracts/store/v1.json", "docs/guide.md", "store/catalog.json"]:
+ self.write(path, "data")
+
+ def git(self, *args):
+ return subprocess.check_output(["git", *args], cwd=self.root, stderr=subprocess.PIPE, text=True).strip()
+
+ def write(self, path, value):
+ (self.root / path).parent.mkdir(parents=True, exist_ok=True)
+ (self.root / path).write_text(value)
+ self.git("add", path)
+
+ def test_an_undeclared_read_of_another_component_is_found(self):
+ self.write("mobile/test/a_test.dart", "File('../backend/src/app.ts'); File('../backend/src/types/productUsage.ts');")
+ self.write("mobile/test/b_test.dart", "final f = File('../cli/src/$path'); // declared: cli/src/")
+ self.write("mobile/test/c_test.dart", "File('../docs/contracts/store/v1.json'); File('../docs/guide.md');")
+ self.assertEqual(planner.undeclared_reads(self.root, "mobile"), [
+ ("mobile/test/a_test.dart", "backend/src/app.ts"),
+ ("mobile/test/c_test.dart", "docs/contracts/store/v1.json"), ("mobile/test/c_test.dart", "docs/guide.md")])
+
+ def test_internal_paths_urls_shared_inputs_and_missing_targets_are_not_reads(self):
+ self.write("desktop/lib/screens/s.dart", "import '../devices/x.dart';")
+ self.write("cli/src/u.ts", "const a = 'https://cdn.example/harness/desktop/../backend/x.zip'")
+ self.write("cli/src/v.ts", "readFileSync(new URL('../../store/catalog.json', import.meta.url)); '../../nothing/here.ts'")
+ self.write("cli/src/w.ts", "import '../../mobile/missing.dart'")
+ self.assertEqual(planner.undeclared_reads(self.root, "desktop"), [])
+ self.assertEqual(planner.undeclared_reads(self.root, "cli"), [])
+
+ def test_a_read_into_a_folder_the_suite_skips_is_found(self):
+ self.write("backend/integration/helpers.ts", "export const x = 1")
+ self.write("backend/src/a.test.ts", "import { x } from '../integration/helpers'")
+ self.write("backend/integration/b.test.ts", "import { y } from './helpers'; import '../src/a'")
+ self.assertEqual(planner.undeclared_reads(self.root, "backend"), [("backend/src/a.test.ts", "backend/integration/")])
+
+ def test_a_read_built_at_run_time_needs_its_whole_folder_declared(self):
+ self.write("tui/src/a.rs", 'include_str!("../../backend/src/types/productUsage.ts"); let p = "../../backend/src/";')
+ self.assertEqual(planner.undeclared_reads(self.root, "tui"), [
+ ("tui/src/a.rs", "backend/src/"), ("tui/src/a.rs", "backend/src/types/productUsage.ts")])
+ self.write("mobile/test/a_test.dart", "File('../backend/src/$name');")
+ self.assertEqual(planner.undeclared_reads(self.root, "mobile"), [("mobile/test/a_test.dart", "backend/src/")])
+
+ def test_a_path_named_from_the_repository_root_is_a_read(self):
+ # As os/tests/test_lock_and_screenshot.py reads the dial ids: (ROOT.parents[1] / 'cli/src/cable/serial.ts').read_text()
+ self.write("website/src/dial.ts", "DIAL_VENDOR_ID = 0x303a")
+ self.write("os/tests/session_vm.py", "pass")
+ self.write("tests/fixture.ts", "data") # a top-level tests/ too: the internal path must not resolve there
+ self.write("os/tests/test_a.py", "\n".join([
+ "serial = (ROOT.parents[1] / 'website/src/dial.ts').read_text()",
+ 'usage = Path(REPO, "backend/src/types/productUsage.ts")',
+ "# a comment naming 'backend/src/app.ts' is not a read",
+ "vm = (OS / 'tests/session_vm.py').read_text() # os/tests/session_vm.py: inside its own directory",
+ "url = 'https://example.test/backend/src/app.ts'",
+ "missing = ROOT / 'backend/src/nothing.ts'",
+ "from './backend/src/app.ts' import x # relative imports are the other scan's",
+ ]))
+ self.assertEqual(planner.undeclared_reads(self.root, "os"), [
+ ("os/tests/test_a.py", "backend/src/"), ("os/tests/test_a.py", "backend/src/types/productUsage.ts"),
+ ("os/tests/test_a.py", "website/src/dial.ts")])
+
+ def test_a_look_alike_inside_the_component_does_not_hide_a_real_read(self):
+ # A PR could add an unused mobile/backend/src/app.ts so that a read of backend/src/app.ts looks internal.
+ self.write("mobile/backend/src/app.ts", "unused")
+ self.write("mobile/test/root_test.dart", "File(p.join(repo, 'backend/src/app.ts'));")
+ self.write("mobile/test/relative_test.dart", "File('../backend/src/app.ts');")
+ self.write("mobile/lib/only_inside.dart", "File(p.join(here, 'backend/src/inner.ts'));")
+ self.write("mobile/backend/src/inner.ts", "an internal file with no root twin")
+ self.write("mobile/lib/notes.dart", "/// `../backend/src/app.ts` named in a doc comment is not a read")
+ self.assertEqual(planner.undeclared_reads(self.root, "mobile"), [
+ ("mobile/test/relative_test.dart", "backend/src/app.ts"), ("mobile/test/root_test.dart", "backend/src/app.ts")])
+
+ def test_a_read_of_a_path_main_deleted_counts_when_the_helper_names_it(self):
+ self.write("mobile/backend/gone/app.ts", "an unused look-alike")
+ self.write("mobile/test/root_test.dart", "File(p.join(repo, 'backend/gone/app.ts'));")
+ self.write("mobile/test/relative_test.dart", "File('../backend/gone/app.ts');")
+ self.assertEqual(planner.undeclared_reads(self.root, "mobile"), [])
+ self.assertEqual(planner.undeclared_reads(self.root, "mobile", ["backend/gone/app.ts"]), [
+ ("mobile/test/relative_test.dart", "backend/gone/app.ts"), ("mobile/test/root_test.dart", "backend/gone/app.ts")])
+
+ def test_strings_that_are_only_data_are_listed_with_a_reason(self):
+ for source, (reason, targets) in planner.NOT_READS.items():
+ with self.subTest(source=source):
+ self.assertTrue(reason.strip() and targets)
+ self.assertTrue(all(isinstance(target, str) and target for target in targets))
+ self.write("backend/src/fixture.test.ts", "work('lead', 'Own', ['cli/src/lib'])")
+ self.assertEqual(planner.undeclared_reads(self.root, "backend"), [("backend/src/fixture.test.ts", "cli/src/lib/")])
+ original = dict(planner.NOT_READS)
+ self.addCleanup(lambda: (planner.NOT_READS.clear(), planner.NOT_READS.update(original)))
+ planner.NOT_READS["backend/src/fixture.test.ts"] = ("owned-code strings in a fixture", {"cli/src/lib/"})
+ self.assertEqual(planner.undeclared_reads(self.root, "backend"), [])
+
+ def test_every_cross_read_in_this_repository_is_declared(self):
+ repository = Path(__file__).resolve().parents[2]
+ if not all((repository / top).is_dir() for top in ["cli/src", "desktop/lib", "mobile/lib", "backend/src"]):
+ self.skipTest("needs a full checkout; the merge helper runs this scan before accepting reuse")
+ for suite in sorted(planner.SUITES):
+ with self.subTest(suite=suite):
+ self.assertEqual(planner.undeclared_reads(repository, suite), [])
+
+
+class ReusedGateTests(unittest.TestCase):
+ def setUp(self):
+ self.head = "a" * 40
+ self.record = dict(origin_sha="e" * 40, origin_tree="f" * 40, run_id=11, run_attempt=1,
+ run_url="https://github.com/o/r/actions/runs/11", reason="passed in run 11")
+ self.plan = dict(schema=1, kind="ci-plan", head=self.head, tree="b" * 40, event="pull_request", base="c" * 40,
+ draft=False, suites=["backend", "mobile"], reused={"mobile": self.record},
+ reuse=dict(status="applied", passed_head="e" * 40, main_parent="c" * 40, rerun={}, reason="x"),
+ required_jobs=sorted({"process-checks"} | planner.JOBS["backend"]))
+ self.needs = {name: {"result": "success"} for name in self.plan["required_jobs"] + ["plan"]}
+ self.needs["mobile-checks"] = {"result": "skipped"}
+
+ def test_a_reused_suite_may_skip_and_the_receipt_names_its_run(self):
+ result = planner.verify(self.plan, self.needs, self.head)
+ self.assertEqual(result["status"], "passed")
+ self.assertEqual(result["suites"], ["backend", "mobile"])
+ self.assertEqual(result["required_jobs"], sorted({"process-checks"} | planner.JOBS["backend"]))
+ self.assertEqual(result["reused"], {"mobile": self.record})
+ self.assertEqual(result["reuse"]["passed_head"], "e" * 40)
+
+ def test_a_suite_that_was_not_reused_must_still_pass(self):
+ needs = copy.deepcopy(self.needs)
+ needs["backend-desk"]["result"] = "skipped"
+ with self.assertRaisesRegex(ValueError, "backend-desk"):
+ planner.verify(self.plan, needs, self.head)
+ with self.assertRaisesRegex(ValueError, "incomplete job coverage"):
+ planner.verify(dict(self.plan, required_jobs=["process-checks"]), self.needs, self.head)
+
+ def test_a_reused_suite_whose_job_ran_or_failed_is_rejected(self):
+ for result in ["success", "failure", "cancelled"]:
+ needs = copy.deepcopy(self.needs)
+ needs["mobile-checks"]["result"] = result
+ with self.subTest(result=result), self.assertRaises(ValueError):
+ planner.verify(self.plan, needs, self.head)
+
+ def test_malformed_or_unplanned_reuse_is_rejected(self):
+ bad = [dict(origin_sha="short"), dict(origin_tree=None), dict(run_id=0), dict(run_id="11"), dict(run_id=True),
+ dict(run_attempt=0), dict(run_url="http://evil.example/runs/11"), dict(run_url=None), dict(origin_sha=self.head),
+ dict(origin_sha=5), dict(origin_tree=["f" * 40]), dict(reason=None)]
+ for patch in bad:
+ with self.subTest(patch=patch), self.assertRaisesRegex(ValueError, "malformed reused result: mobile"):
+ planner.verify(dict(self.plan, reused={"mobile": dict(self.record, **patch)}), self.needs, self.head)
+ for plan in [dict(self.plan, reused={"cli": self.record}), dict(self.plan, reused=[]),
+ dict(self.plan, event="merge_group"), dict(self.plan, event="workflow_dispatch")]:
+ with self.subTest(plan=plan["event"]), self.assertRaisesRegex(ValueError, "reuse"):
+ planner.verify(plan, self.needs, self.head)
+
+
+class FakeGitHub:
+ """GitHub's check-run, job, run and artifact reads for passed ci/required runs."""
+ repository = "o/r"
+
+ def __init__(self):
+ self.runs, self.checks, self.receipts, self.reads = {}, {}, {}, []
+
+ def add_run(self, run_id, head, tree, suites, attempt=1, reused=None, event="pull_request", conclusion="success"):
+ required = sorted({"process-checks"} | set().union(*(planner.JOBS[s] for s in set(suites) - set(reused or {}))))
+ self.runs[run_id] = dict(id=run_id, repository=dict(full_name=self.repository), path=".github/workflows/ci.yml",
+ event=event, head_sha=head, run_attempt=attempt, status="completed",
+ conclusion=conclusion, html_url=f"https://github.com/o/r/actions/runs/{run_id}",
+ updated_at="2026-10-29T00:00:00Z")
+ self.receipts[run_id] = dict(schema=1, kind="ci-required", status="passed", source_sha=head, source_tree=tree,
+ event=event, suites=sorted(suites), required_jobs=required, reused=reused or {},
+ results={job: "success" for job in required + ["plan"]},
+ run_id=run_id, run_attempt=attempt)
+ self.checks.setdefault(head, []).append(dict(id=run_id * 10, name="ci/required", head_sha=head, status="completed",
+ conclusion=conclusion, app=dict(slug="github-actions")))
+
+ def archive(self, run_id):
+ buf = io.BytesIO()
+ with zipfile.ZipFile(buf, "w") as bundle:
+ bundle.writestr(zipfile.ZipInfo("ci-required.json", (2026, 10, 29, 0, 0, 0)), json.dumps(self.receipts[run_id]))
+ return buf.getvalue()
+
+ def api(self, path):
+ self.reads.append(path)
+ if path.startswith("actions/jobs/"):
+ job_id = int(path.split("/")[2])
+ run = self.runs[job_id // 10]
+ return dict(id=job_id, name="ci/required", run_id=run["id"], run_attempt=run["run_attempt"],
+ head_sha=run["head_sha"], status="completed", conclusion=run["conclusion"])
+ if path.startswith("actions/runs/"):
+ return copy.deepcopy(self.runs[int(path.split("/")[2])])
+ if path.startswith("git/commits/"):
+ head = path.split("/")[2]
+ return dict(sha=head, tree=dict(sha=next(r["source_tree"] for r in self.receipts.values() if r["source_sha"] == head)))
+ raise AssertionError(path)
+
+ def pages(self, path, key):
+ self.reads.append(path)
+ if key == "check_runs":
+ head = path.split("/")[1]
+ return copy.deepcopy(self.checks.get(head, []))
+ run_id = int(path.split("/")[2])
+ run = self.runs[run_id]
+ if key == "jobs":
+ return [dict(name="ci/required", status="completed", conclusion=run["conclusion"], head_sha=run["head_sha"],
+ run_attempt=run["run_attempt"])]
+ if key == "artifacts":
+ archive = self.archive(run_id)
+ return [dict(id=run_id, name="ci-required", expired=False, size_in_bytes=len(archive),
+ digest="sha256:" + hashlib.sha256(archive).hexdigest(),
+ workflow_run=dict(id=run_id, head_sha=run["head_sha"]))]
+ raise AssertionError(path)
+
+ def command(self, *args, binary=False):
+ assert binary and args[1].endswith("/zip"), args
+ return self.archive(int(args[1].split("/")[-2]))
+
+
+class GitHubEvidenceTests(unittest.TestCase):
+ def setUp(self):
+ self.github = FakeGitHub()
+ self.head, self.tree = "a" * 40, "b" * 40
+ self.github.add_run(11, self.head, self.tree, ["mobile"])
+ self.evidence = planner.GitHubEvidence(self.github)
+
+ def test_the_passed_gate_is_the_run_behind_the_latest_ci_required_check(self):
+ self.github.add_run(12, self.head, self.tree, ["mobile", "backend"])
+ gate = self.evidence.passed_gate(self.head)
+ self.assertEqual((gate["run_id"], gate["run_attempt"]), (12, 1))
+ self.assertEqual(gate["url"], "https://github.com/o/r/actions/runs/12")
+ self.assertEqual(gate["receipt"]["suites"], ["backend", "mobile"])
+ self.assertIn("actions/jobs/120", self.github.reads)
+
+ def test_a_failed_latest_check_is_not_rescued_by_an_older_pass(self):
+ self.github.add_run(12, self.head, self.tree, ["mobile"], conclusion="failure")
+ with self.assertRaisesRegex(ValueError, "not a pass"):
+ self.evidence.passed_gate(self.head)
+
+ def test_no_check_a_rerun_attempt_another_event_workflow_or_repository_fails(self):
+ with self.assertRaisesRegex(ValueError, "no ci/required"):
+ self.evidence.passed_gate("c" * 40)
+ for patch in [dict(run_attempt=2), dict(event="workflow_dispatch"), dict(path=".github/workflows/other.yml"),
+ dict(repository=dict(full_name="someone/else")), dict(head_sha="d" * 40), dict(status="in_progress")]:
+ with self.subTest(patch=patch):
+ github = FakeGitHub()
+ github.add_run(11, self.head, self.tree, ["mobile"])
+ original = github.api
+
+ def api(path, original=original, patch=patch):
+ value = original(path)
+ return dict(value, **patch) if path == "actions/runs/11" else value
+ github.api = api
+ with self.assertRaises(ValueError):
+ planner.GitHubEvidence(github).passed_gate(self.head)
+
+ def test_a_check_from_another_app_or_commit_is_ignored(self):
+ self.github.checks[self.head] = [dict(c, app=dict(slug="someone-else")) for c in self.github.checks[self.head]]
+ with self.assertRaisesRegex(ValueError, "no ci/required"):
+ self.evidence.passed_gate(self.head)
+
+ def test_a_tampered_receipt_archive_fails(self):
+ self.github.archive = lambda run_id: b"not the archive GitHub hashed"
+ with self.assertRaises((ValueError, zipfile.BadZipFile)):
+ self.evidence.passed_gate(self.head)
+
+ def test_ran_requires_that_exact_run_to_have_run_the_suite(self):
+ self.assertEqual(self.evidence.ran(self.head, 11, 1, "mobile"),
+ dict(tree=self.tree, url="https://github.com/o/r/actions/runs/11"))
+ for args, reason in [((self.head, 11, 2, "mobile"), "attempt"), (("c" * 40, 11, 1, "mobile"), "another source"),
+ ((self.head, 11, 1, "backend"), "did not run")]:
+ with self.subTest(args=args), self.assertRaisesRegex(ValueError, reason):
+ self.evidence.ran(*args)
+ record = dict(origin_sha="e" * 40, origin_tree="f" * 40, run_id=9, run_attempt=1,
+ run_url="https://github.com/o/r/actions/runs/9", reason="x")
+ self.github.add_run(13, "c" * 40, self.tree, ["mobile"], reused={"mobile": record})
+ with self.assertRaisesRegex(ValueError, "did not run"):
+ self.evidence.ran("c" * 40, 13, 1, "mobile")
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/scripts/tests/test_merge_validated_pr.py b/scripts/tests/test_merge_validated_pr.py
index 263bd2797..e0776be39 100644
--- a/scripts/tests/test_merge_validated_pr.py
+++ b/scripts/tests/test_merge_validated_pr.py
@@ -6,6 +6,7 @@ import json
from pathlib import Path
import subprocess
import tempfile
+import types
import unittest
from unittest import mock
import zipfile
@@ -105,6 +106,55 @@ class MergeTests(unittest.TestCase):
self.assertIn("CI changed during verification", result["error"])
self.assertEqual(self.client.writes, [])
+ def reused_receipt(self):
+ record = dict(origin_sha="e" * 40, origin_tree="f" * 40, run_id=99, run_attempt=1,
+ run_url="https://github.com/owner/repo/actions/runs/99", reason="passed in run 99")
+ self.client.receipt_patch = dict(suites=["mobile"], reused={"mobile": record},
+ reuse=dict(status="applied", passed_head="e" * 40, main_parent=self.base))
+ return record
+
+ def main_planner(self, verify):
+ calls = []
+
+ def verify_reuse(root, receipt, base, evidence):
+ calls.append((root, receipt["reused"], base, evidence))
+ return verify(receipt)
+ main = types.SimpleNamespace(GitHubEvidence=lambda client: ("main's reader", client), verify_reuse=verify_reuse)
+ loader = self.patch(merger.queue.planner, "main_planner", return_value=main)
+ return loader, calls
+
+ def test_reused_results_are_rederived_by_mains_planner_before_merging(self):
+ record = self.reused_receipt()
+ loader, calls = self.main_planner(lambda receipt: receipt["reused"])
+ result = self.finish(automatic=True)
+ self.assertEqual(result["status"], "merged")
+ loader.assert_called_once_with(self.root, self.base)
+ self.assertEqual(calls, [(self.root, {"mobile": record}, self.base, ("main's reader", self.client))])
+ self.assertEqual(result["pr_evidence"]["reused"], {"mobile": record})
+ self.assertIn("reused mobile from https://github.com/owner/repo/actions/runs/99", (self.output / "merge.md").read_text())
+
+ def test_reuse_that_mains_planner_refuses_or_cannot_check_blocks_the_merge(self):
+ self.reused_receipt()
+
+ def refuse(receipt):
+ raise ValueError("reused mobile refused: main changed its input backend/src/types/productUsage.ts")
+ self.main_planner(refuse)
+ result = self.finish(automatic=True)
+ self.assertEqual(result["status"], "not_merged")
+ self.assertIn("productUsage", result["error"])
+ self.assertEqual(self.client.writes, [])
+ self.output = self.output.with_name("no-verifier")
+ self.patch(merger.queue.planner, "main_planner", side_effect=ValueError("main's planner cannot verify reused results (X)"))
+ result = self.finish(automatic=True)
+ self.assertIn("cannot verify reused results", result["error"])
+ self.assertEqual(self.client.writes, [])
+
+ def test_a_receipt_that_skips_a_suite_it_did_not_reuse_blocks_the_merge(self):
+ self.client.receipt_patch = dict(suites=["mobile"])
+ result = self.finish(automatic=True)
+ self.assertIn("does not run every suite", result["error"])
+ self.assertEqual(self.client.writes, [])
+
def test_default_command_uses_direct_merge_unless_github_requires_queue(self):
direct = self.patch(merger, "finish", return_value={"status": "ready"})
queued = self.patch(merger.queue, "finish", return_value={"status": "ready_for_queue"})
@@ -283,7 +333,7 @@ class FakeClient:
self.merged_sha, self.merged_tree = "c" * 40, fixture.tree
self.tested_tree = fixture.tree
self.required_gate, self.run_list_reads, self.superseding_run = False, 0, None
- self.rules = []
+ self.rules, self.receipt_patch = [], {}
self.pr = dict(number=5, state="open", draft=False, merged=False, mergeable=True, mergeable_state="clean",
html_url="https://github.com/owner/repo/pull/5", head=dict(sha=fixture.head),
base=dict(sha=fixture.base, ref="main", repo=dict(full_name=self.repository)))
@@ -325,6 +375,7 @@ class FakeClient:
source_tree=self.tested_tree, event="pull_request", run_id=123, run_attempt=1,
suites=[], required_jobs=["process-checks"],
results={"plan": "success", "process-checks": "success"})
+ receipt.update(self.receipt_patch)
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as bundle:
bundle.writestr("ci-required.json", json.dumps(receipt))
diff --git a/scripts/tests/test_queue_validated_pr.py b/scripts/tests/test_queue_validated_pr.py
index 4f9470792..e7e2fc9ac 100644
--- a/scripts/tests/test_queue_validated_pr.py
+++ b/scripts/tests/test_queue_validated_pr.py
@@ -48,6 +48,14 @@ class QueueTests(unittest.TestCase):
self.assertEqual(json.loads((self.output / "receipt.json").read_text()), result)
return result
+ def test_reused_results_without_the_reviewed_checkout_are_refused(self):
+ # Merge-group candidates never reuse; only a PR run's receipt, checked against the reviewed checkout, can.
+ record = dict(origin_sha="e" * 40, origin_tree="f" * 40, run_id=99, run_attempt=1,
+ run_url="https://github.com/owner/repo/actions/runs/99", reason="x")
+ self.client.receipt_patch = dict(suites=["mobile"], reused={"mobile": record})
+ with self.assertRaisesRegex(ValueError, "reviewed checkout"):
+ queue.gate_evidence(self.client, self.client.runs[123], self.output / "gate")
+
def test_preview_collects_automatic_evidence_without_writes(self):
self.assertEqual(self.finish(False)["status"], "ready_for_queue")
self.assertEqual(self.client.writes, [])