The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
Reuse verified CI when selected test inputs are unchanged (#671)
Deeaf5db0c754a7
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 71a9240d9..f9377afa4 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -55,8 +55,29 @@ jobs:timeout-minutes: 3steps:- uses: actions/checkout@v7+ with:+ sparse-checkout: |+ /.github/+ /scripts/+ /desktop/scripts/+ /.gitattributes+ /.gitignore+ /.gitmodules+ /Makefile+ sparse-checkout-cone-mode: false+ - name: Verify declared source checkout+ run: python3 scripts/ci-source-inputs.py process- name: Validation deadlines, cleanup, receipts and release version guardsrun: python3 -m unittest discover -s scripts/tests -p 'test_*.py' -v+ - name: Record source inputs for CI evidence reuse+ run: python3 scripts/ci-source-inputs.py process --record "$RUNNER_TEMP/ci-source-inputs.json"+ - uses: actions/upload-artifact@v4+ with:+ name: ci-source-inputs+ path: ${{ runner.temp }}/ci-source-inputs.json+ if-no-files-found: error+ overwrite: true+ retention-days: 7# workflow_dispatch restricts the choice; reusable callers can pass any string.# Fail invalid calls rather than reporting success after skipping every check.@@ -78,6 +99,24 @@ jobs:shard: [1, 2, 3, 4]steps:- uses: actions/checkout@v7+ with:+ sparse-checkout: |+ /.github/+ /scripts/+ /desktop/+ /cli/+ /tests/+ /daemons/+ /store/+ /docs/images/+ /mobile/pubspec.lock+ /.gitattributes+ /.gitignore+ /.gitmodules+ /Makefile+ sparse-checkout-cone-mode: false+ - name: Verify declared source checkout+ run: python3 scripts/ci-source-inputs.py desktop- id: toolchainrun: |version="$(sed -n "s/^ FLUTTER_VERSION: '\([^']*\)'.*/\1/p" .github/workflows/release-desktop.yml)"@@ -112,7 +151,10 @@ jobs:sparse-checkout: |/desktop/test//scripts/+ /.gitignoresparse-checkout-cone-mode: false+ - name: Verify declared source checkout+ run: python3 scripts/ci-source-inputs.py desktop --subset- uses: actions/download-artifact@v4with:pattern: desktop-shard-*diff --git a/AGENTS.md b/AGENTS.mdindex cb1be7c50..87a1ab9e5 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -29,6 +29,11 @@ validation and shipping, use [docs/validation-and-release.md](docs/validation-anuse `scripts/record-ci-validation.py RUN_ID --scope SCOPE --pr PR_NUMBER --wait`.Retain routine results in the ignored receipt and PR body instead of anotherdocumentation commit. Resolve source differences explicitly before reusing evidence.+ Process and Desktop VM CI can retain the original run across unrelated changes+ when their verified source-input contracts match; pass that run to the same+ collector/merge helper. Check its changed-path list and validate other affected+ scopes separately. Changes to included tests, dependencies or workflows require+ new evidence; see the validation guide for the complete input boundaries.- For an authorized Desktop release, start `make release-desktop ARGS="--prepare"`from the final pushed PR branch alongside validation and review. It preparesverified packages without publishing; merge and release only after checks pass.diff --git a/docs/validation-and-release.md b/docs/validation-and-release.mdindex 0d9445589..3b0ed67c0 100644--- a/docs/validation-and-release.md+++ b/docs/validation-and-release.md@@ -288,16 +288,47 @@ file/case totals. Desktop scope similarly verifies the summary for both platformand preserves explicit startup-recovery counts. Network calls are bounded; thedefault collection budget is 90s.-Exit 0 means the requested CI scope passed and its committed source tree matches-the selected target (`HEAD` by default). Exit 3 still saves the record but flags-different source, a dirty working tree or a PR head mismatch for review. It lists-changed paths without deciding that they are harmless. Review their impact and-reuse only applicable evidence; a different SHA with the same tree is accepted.+Exit 0 means the requested CI scope passed and covers the selected target (`HEAD`+by default): either the full source tree matches or the verified input contract+below matches. Exit 3 still saves the record but flags uncovered source changes,+a dirty working tree or a PR head mismatch for review. A clean squash with the+same tree is accepted. The receipt always retains both actual commits/trees.`--target COMMIT` supports historical audits and labels them as covering thatcommit rather than the current working copy. Missing objects need a fetch beforecomparison; failed/pending runs, missing/skipped required jobs, corrupt/expiredartifacts, and changing run/PR state cannot become a successful record.+For `process` and `desktop`, CI records a `ci-source-inputs` artifact and verifies+each required job's sparse checkout before running its checks. The collector+checks that artifact's immutable digest, run/source/attempt identity and recorded+Git objects, then independently compares the selected inputs with the target:++| CI scope | Complete input boundary |+| --- | --- |+| `process` | `.github/`, `scripts/`, `desktop/scripts/`, root Git configuration files and `Makefile` |+| `desktop` | `.github/`, `scripts/`, `desktop/`, `cli/`, `tests/`, `daemons/`, `store/`, `docs/images/`, `mobile/pubspec.lock`, root Git configuration files and `Makefile` |++The root Git files are `.gitattributes`, `.gitignore` and `.gitmodules`. Directory+objects include every tracked descendant; file modes, additions and removals are+covered. Desktop's tests read CLI protocol definitions, shared layout fixtures,+daemon metadata and catalog artwork, so those are inputs too. The workflow/action+trees include toolchain pins and cache recipes. Extend both the checkout and+`SOURCE_INPUTS` in `scripts/record-ci-validation.py` before using a new dependency.+CI verifies that the checkout agrees with that contract and exposes no tracked+files outside it; the Desktop aggregate uses a smaller subset of the same inputs.++A documentation edit outside those boundaries or an unrelated firmware merge can+therefore preserve a completed run. Pass its original run ID to the collector or+merge helper; no extra option or replacement CI run is needed. The resulting+receipt says which scope was reused, records the matching fingerprint and lists+changed paths outside that scope. Review those changes and run any checks they+require separately. This is a source-input comparison, not a file-extension rule:+Desktop documentation inside `desktop/` and artwork inside `docs/images/` remain+inputs. Other CI scopes and older runs without an input receipt still require+whole-tree equality. A malformed receipt cannot authorize reuse. After a partial+rerun, the receipt must match the actual successful process job's attempt; a+stale receipt from a replaced process job is rejected.+Keep routine validation evidence in these receipts, CI artifacts and the PR body.Do not add a documentation commit or recreate all raw logs just to record anotherordinary check. Commit a performance report when the comparison itself is useful@@ -388,11 +419,13 @@ inspects the same PR without replaying the merge request.A changed head, dirty checkout, moved main, failed CI or blocked merge stops thecommand. Review new source differences and reuse only applicable evidence, as above;the command does not decide that impact or replace native checks and code review.-After a merge, it verifies the actual Git tree against the tested tree. A different-tree is recorded as `merged_source_review_required` (exit 3), so resolve that source-review before release. An unconfirmed write is recorded as `merge_not_confirmed`;-inspect the same PR before further action. This command does not create release tags-or dispatch release workflows.+After a merge, it verifies the actual Git tree against the reviewed target tree.+The receipt separately reports whether the full tested tree matches; scoped CI+reuse may preserve test inputs while other reviewed files differ. A merge tree+different from the reviewed target is recorded as `merged_source_review_required`+(exit 3), so resolve that source review before release. An unconfirmed write is+recorded as `merge_not_confirmed`; inspect the same PR before further action. This+command does not create release tags or dispatch release workflows.## Complete the authorized releasediff --git a/scripts/ci-source-inputs.py b/scripts/ci-source-inputs.pynew file mode 100644index 000000000..7130a643e--- /dev/null+++ b/scripts/ci-source-inputs.py@@ -0,0 +1,75 @@+#!/usr/bin/env python3+"""Enforce CI's declared source checkout and record its immutable input objects."""+import argparse+import importlib.util+import json+import os+from pathlib import Path+import subprocess++spec = importlib.util.spec_from_file_location("ci_evidence", Path(__file__).with_name("record-ci-validation.py"))+ci = importlib.util.module_from_spec(spec)+spec.loader.exec_module(ci)+++def git(root, *args):+ return subprocess.check_output(["git", *args], cwd=root, stderr=subprocess.PIPE, timeout=15).decode().strip()+++def verify_checkout(root, scope, subset=False):+ if git(root, "config", "--bool", "--default=false", "core.sparseCheckout") != "true":+ raise ValueError("CI source inputs require a sparse checkout")+ # actions/checkout leaves this unset in non-cone mode (Git defaults false).+ if git(root, "config", "--bool", "--default=false", "core.sparseCheckoutCone") != "false":+ raise ValueError("CI source inputs require literal non-cone patterns")+ declared = set(ci.SOURCE_INPUTS[scope])+ patterns = set(git(root, "sparse-checkout", "list").splitlines())+ def covered(path):+ return any(path == item.rstrip("/") or (item.endswith("/") and path.startswith(item)) for item in declared)+ if not patterns or any(not value.startswith("/") or any(c in value for c in "!*?[]\\")+ or ".." in value.split("/") or not covered(value[1:].rstrip("/")) for value in patterns):+ raise ValueError("checkout includes source outside the declared CI inputs")+ if not subset and patterns != {"/" + path for path in declared}:+ raise ValueError("checkout does not match the complete declared CI inputs")+ entries = subprocess.check_output(["git", "ls-files", "-t", "-z"], cwd=root, timeout=15).decode().split("\0")+ for entry in filter(None, entries):+ flag, path = entry[:1], entry[2:]+ # Catch a populated file outside the sparse patterns, including when Git+ # still marks it skip-worktree. Directories are covered by their children.+ present = os.path.lexists(root / path)+ if not covered(path) and (flag != "S" or present):+ raise ValueError(f"undeclared tracked source is available: {path}")+ if git(root, "status", "--porcelain", "--untracked-files=normal"):+ raise ValueError("CI source checkout must be clean")+++def capture(root):+ verify_checkout(root, "process")+ sha = git(root, "rev-parse", "HEAD")+ if os.environ.get("GITHUB_SHA") != sha:+ raise ValueError("checkout differs from the CI source")+ run, attempt = (int(os.environ[key]) for key in ("GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT"))+ if run < 1 or attempt < 1:+ raise ValueError("invalid CI run identity")+ return dict(schema=1, kind="ci-source-inputs", status="recorded", source_sha=sha,+ source_tree=git(root, "rev-parse", "HEAD^{tree}"), dirty=False, run_id=run, run_attempt=attempt,+ scopes={scope: ci.input_snapshot(root, sha, scope) for scope in ci.SOURCE_INPUTS})+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("scope", choices=ci.SOURCE_INPUTS)+ parser.add_argument("--subset", action="store_true", help="allow an aggregate's smaller checkout within the same contract")+ parser.add_argument("--record", type=Path, help="after process checks, record both contracts for this exact CI run")+ args = parser.parse_args()+ if args.record and (args.scope != "process" or args.subset):+ parser.error("recording requires the complete process checkout")+ root = Path(git(Path.cwd(), "rev-parse", "--show-toplevel"))+ verify_checkout(root, args.scope, args.subset)+ if args.record:+ args.record.write_text(json.dumps(capture(root), indent=2) + "\n")+ print(f"Verified {args.scope} CI source checkout")+++if __name__ == "__main__":+ main()diff --git a/scripts/merge-validated-pr.py b/scripts/merge-validated-pr.pyindex 14c142580..b78845fa6 100644--- a/scripts/merge-validated-pr.py+++ b/scripts/merge-validated-pr.py@@ -44,7 +44,7 @@ def preflight(client, root, number, head, base, ready=False):"url": pr["html_url"]}-def confirm_merge(client, number, head, expected_tree, response):+def confirm_merge(client, number, head, expected_tree, response, tested_tree=None):# A successful write can outlive an HTTP timeout. Observe the same PR;# never submit the mutation twice or assume that a timeout rejected it.for attempt in range(5):@@ -64,7 +64,8 @@ def confirm_merge(client, number, head, expected_tree, response):tree = commit.get("tree", {}).get("sha")if not re.fullmatch(r"[0-9a-f]{40}", tree or ""):raise ValueError("merge commit is missing its source tree")- return dict(commit=sha, tree=tree, same_tested_tree=tree == expected_tree,+ return dict(commit=sha, tree=tree, same_reviewed_tree=tree == expected_tree,+ same_tested_tree=tree == (tested_tree or expected_tree),merged_at=pr.get("merged_at"), parents=[p["sha"] for p in commit.get("parents", [])])if pr.get("state") == "closed" or attempt == 4:break@@ -126,8 +127,9 @@ def finish(client, root, number, run_id, scope, head, base, output, *, merge=Fal# Inspection gets a fresh bounded budget even if the write timed out.client.deadline = time.monotonic() + min(timeout, 30)record["merge"] = phase("verification", lambda: confirm_merge(- client, number, head, evidence["source"]["tested_tree"], response))- record["status"] = "merged" if record["merge"]["same_tested_tree"] else "merged_source_review_required"+ client, number, head, evidence["source"]["target_tree"], response,+ tested_tree=evidence["source"]["tested_tree"]))+ record["status"] = "merged" if record["merge"]["same_reviewed_tree"] else "merged_source_review_required"except (OSError, RuntimeError, ValueError, KeyError, TypeError, subprocess.SubprocessError, ci.zipfile.BadZipFile) as error:record["error"] = str(error)record["status"] = "merge_not_confirmed" if "merge_requested_at" in record else "not_merged"@@ -139,7 +141,8 @@ def finish(client, root, number, run_id, scope, head, base, output, *, merge=Fallines.append(record["error"])if "merge" in record:result = record["merge"]- lines.append(f"Merged commit `{result['commit']}`; tested tree matches: `{result['same_tested_tree']}`.")+ lines.append(f"Merged commit `{result['commit']}`; reviewed target tree matches: `{result['same_reviewed_tree']}`; "+ f"full tested tree matches: `{result['same_tested_tree']}`.")if evidence is not None:lines += ["", ci.markdown(evidence)](output / "merge.md").write_text("\n".join(lines) + "\n")diff --git a/scripts/record-ci-validation.py b/scripts/record-ci-validation.pyindex 45fad92e6..3fce8676f 100644--- a/scripts/record-ci-validation.py+++ b/scripts/record-ci-validation.py@@ -27,6 +27,71 @@ DESKTOP_JOBS = {"desktop-test-summary"} | {f"desktop-tests ({platform}, {index})SCOPES = {"cli": CLI_JOBS, "tui": TUI_JOBS, "backend": {"backend-desk"}, "desktop": DESKTOP_JOBS, "process": set()}SCOPES["full"] = CLI_JOBS | TUI_JOBS | SCOPES["backend"] | DESKTOP_JOBS+# These are the source boundaries enforced by CI's sparse checkouts. Keep the+# whole workflow/action and helper trees: a changed test, toolchain pin, cache+# recipe or source contract must invalidate earlier evidence too.+SOURCE_INPUTS = {+ "process": (".github/", "scripts/", "desktop/scripts/", ".gitattributes", ".gitignore", ".gitmodules", "Makefile"),+ # VM tests also read CLI protocol definitions, shared layout fixtures,+ # daemon metadata and store catalog/artwork, including docs/images posters.+ "desktop": (".github/", "scripts/", "desktop/", "cli/", "tests/", "daemons/", "store/", "docs/images/",+ "mobile/pubspec.lock", ".gitattributes", ".gitignore", ".gitmodules", "Makefile"),+}+CHECKOUT_STEP = "Verify declared source checkout"+++def input_snapshot(root, commit, scope):+ """Hash immutable Git objects, including missing optional files and modes."""+ objects = []+ for path in sorted(SOURCE_INPUTS[scope]):+ name = path.rstrip("/")+ data = subprocess.check_output(["git", "ls-tree", "-z", commit, "--", name],+ cwd=root, stderr=subprocess.PIPE, timeout=10).decode()+ entries = [entry for entry in data.split("\0") if entry]+ if not entries:+ if path.endswith("/"):+ raise ValueError(f"required CI input directory is missing: {path}")+ objects.append(dict(path=path, object=None))+ continue+ if len(entries) != 1:+ raise ValueError(f"ambiguous CI input: {path}")+ metadata, actual_path = entries[0].split("\t", 1)+ mode, kind, oid = metadata.split()+ if (actual_path != name or not re.fullmatch(r"[0-9a-f]{40}", oid)+ or (path.endswith("/") and (mode, kind) != ("040000", "tree"))+ or (not path.endswith("/") and (kind != "blob" or mode not in {"100644", "100755"}))):+ raise ValueError(f"unexpected CI input object: {path}")+ objects.append(dict(path=path, object=dict(mode=mode, kind=kind, oid=oid)))+ payload = dict(scope=scope, objects=objects)+ digest = hashlib.sha256(json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()).hexdigest()+ return dict(payload, sha256=digest)+++def check_input_jobs(jobs, scope):+ required = SCOPES[scope] | {"process-checks"}+ for job in jobs:+ if job["name"] in required:+ matches = [step for step in job.get("steps", []) if step.get("name") == CHECKOUT_STEP]+ if len(matches) != 1 or matches[0].get("conclusion") != "success":+ raise ValueError(f"CI input checkout was not verified: {job['name']}")+++def read_input_summary(artifact, archive, run, root, scope, process_attempt):+ summary = summary_contents(artifact, archive, run, "ci-source-inputs")+ if (type(summary.get("schema")) is not int or summary["schema"] != 1+ or summary.get("kind") != "ci-source-inputs" or summary.get("status") != "recorded"+ or summary.get("source_sha") != run["head_sha"] or summary.get("dirty") is not False+ or not isinstance(summary.get("scopes"), dict)+ or type(summary.get("run_id")) is not int or summary["run_id"] != run["id"]+ or type(summary.get("run_attempt")) is not int or summary["run_attempt"] != process_attempt):+ raise ValueError("CI input receipt has another source/run/attempt")+ tree = subprocess.check_output(["git", "rev-parse", f"{run['head_sha']}^{{tree}}"],+ cwd=root, stderr=subprocess.PIPE, timeout=10).decode().strip()+ expected = input_snapshot(root, run["head_sha"], scope)+ if summary.get("source_tree") != tree or summary.get("scopes", {}).get(scope) != expected:+ raise ValueError("CI input receipt differs from the tested Git objects or input contract")+ return expected+def utc_now():return datetime.now(timezone.utc).isoformat()@@ -220,7 +285,7 @@ def collect(client, root, run_id, scope, target, output, pr_number=None, expecteraise ValueError("CI run changed between waiting and evidence collection")with ThreadPoolExecutor(max_workers=3) as pool:job_future = pool.submit(client.pages, f"actions/runs/{run_id}/jobs?filter=latest", "jobs")- artifact_future = pool.submit(client.pages, f"actions/runs/{run_id}/artifacts", "artifacts") if scope in {"cli", "desktop", "full"} else None+ artifact_future = pool.submit(client.pages, f"actions/runs/{run_id}/artifacts", "artifacts") if scope in {"cli", "desktop", "full", "process"} else Nonepr_future = pool.submit(client.api, f"pulls/{pr_number}") if pr_number else Nonejobs = job_future.result()artifacts = artifact_future.result() if artifact_future else []@@ -258,6 +323,26 @@ def collect(client, root, run_id, scope, target, output, pr_number=None, expecteraise ValueError("Desktop summary Git tree differs from the tested source")record["desktop_artifact"] = {key: artifact[key] for key in ["id", "name", "digest", "size_in_bytes"]}(output / "desktop-test-summary.zip").write_bytes(archive)+ if scope in SOURCE_INPUTS:+ matches = [artifact for artifact in artifacts if artifact["name"] == "ci-source-inputs"]+ if len(matches) > 1:+ raise ValueError("duplicate CI input receipts")+ if matches:+ artifact = matches[0]+ if artifact.get("size_in_bytes", 65537) > 65536:+ raise ValueError("CI input artifact is unexpectedly large")+ archive = client.command("api", f"repos/{client.repository}/actions/artifacts/{artifact['id']}/zip", binary=True)+ check_input_jobs(jobs, scope)+ # Rerunning failed jobs can retain an earlier successful process job.+ # Bind the artifact to that job's actual attempt, not a later shard's.+ process_attempt = next(job["run_attempt"] for job in jobs if job["name"] == "process-checks")+ tested_inputs = read_input_summary(artifact, archive, run, root, scope, process_attempt)+ target_inputs = input_snapshot(root, source["target_sha"], scope)+ record["scope_reuse"] = dict(scope=scope, paths=sorted(SOURCE_INPUTS[scope]),+ tested_sha256=tested_inputs["sha256"], target_sha256=target_inputs["sha256"],+ same_inputs=tested_inputs == target_inputs)+ record["input_artifact"] = {key: artifact[key] for key in ["id", "name", "digest", "size_in_bytes"]}+ (output / "ci-source-inputs.zip").write_bytes(archive)if pr:record["pr"] = {"number": pr_number, "url": pr["html_url"], "head": pr["head"]["sha"], "base": pr["base"]["sha"]}# Detect reruns and moving PR heads after the independent downloads. Never@@ -274,7 +359,8 @@ def collect(client, root, run_id, scope, target, output, pr_number=None, expecteraise ValueError("PR head/base changed during collection; review the new state")if source != source_comparison(root, run["head_sha"], target):raise ValueError("local source changed during collection")- if not source["same_tree"] or source["working_tree_dirty"] or (pr and pr["head"]["sha"] != source["target_sha"]):+ covered = source["same_tree"] or record.get("scope_reuse", {}).get("same_inputs") is True+ if not covered or source["working_tree_dirty"] or (pr and pr["head"]["sha"] != source["target_sha"]):record["status"] = "source_review_required"return record@@ -310,6 +396,12 @@ def markdown(record):lines.append("The current working tree also contains changes outside this committed CI result.")if "pr" in record and record["pr"]["head"] != source["target_sha"]:lines.append(f"PR head `{record['pr']['head']}` differs from the selected target.")+ elif not source["same_tree"]:+ reuse = record["scope_reuse"]+ lines.append(f"CI evidence reused for `{record['scope']}`: the declared source inputs match "+ f"(`{reuse['tested_sha256']}`). Receipt digest, Git objects and bounded checkouts verified.")+ lines.append("Changed paths outside that CI scope: " + ", ".join(f"`{path.replace('`', '')}`" for path in source["changed_files"]) + ".")+ lines.append("This covers the selected CI checks, not review or validation required by those other changes.")else:lines.append("The tested and target source trees match. Scope selection and code/merge review remain explicit.")if source["working_tree_dirty"] is None:diff --git a/scripts/tests/test_ci_source_inputs.py b/scripts/tests/test_ci_source_inputs.pynew file mode 100644index 000000000..3e6d18ce3--- /dev/null+++ b/scripts/tests/test_ci_source_inputs.py@@ -0,0 +1,99 @@+import importlib.util+import os+from pathlib import Path+import subprocess+import tempfile+import unittest+from unittest import mock++spec = importlib.util.spec_from_file_location("ci_inputs", Path(__file__).resolve().parents[1] / "ci-source-inputs.py")+inputs = importlib.util.module_from_spec(spec)+spec.loader.exec_module(inputs)+++class SourceInputTests(unittest.TestCase):+ def setUp(self):+ self.temp = tempfile.TemporaryDirectory()+ self.addCleanup(self.temp.cleanup)+ self.root = Path(self.temp.name)+ for name in (".github/workflows/ci.yml", "scripts/check.py", "desktop/scripts/upload.sh",+ "desktop/lib/main.dart", "desktop/test/app_test.dart", "cli/main.ts", "docs/guide.md",+ "tests/fixtures/layout.json", "daemons/frames.json", "store/catalog.json", "docs/images/poster.png"):+ path = self.root / name+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text("fixture\n")+ self.git("init", "-q")+ self.git("add", ".")+ self.git("-c", "user.name=Fixture", "-c", "[email protected]", "commit", "-qm", "source")+ self.sha = self.git("rev-parse", "HEAD")++ def git(self, *args):+ return subprocess.check_output(["git", *args], cwd=self.root, stderr=subprocess.PIPE, text=True).strip()++ def sparse(self, paths):+ subprocess.run(["git", "sparse-checkout", "set", "--no-cone", "--stdin"], cwd=self.root,+ input="\n".join("/" + path for path in paths) + "\n", text=True,+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)++ def test_process_checkout_removes_unrelated_source_but_keeps_every_declared_input(self):+ self.sparse(inputs.ci.SOURCE_INPUTS["process"])+ inputs.verify_checkout(self.root, "process")+ self.assertTrue((self.root / "desktop/scripts/upload.sh").exists())+ self.assertFalse((self.root / "desktop/lib/main.dart").exists())+ self.assertFalse((self.root / "cli/main.ts").exists())+ self.assertFalse((self.root / "docs/guide.md").exists())++ def test_desktop_checkout_and_aggregate_subset_are_bounded_by_the_same_contract(self):+ self.sparse(inputs.ci.SOURCE_INPUTS["desktop"])+ inputs.verify_checkout(self.root, "desktop")+ self.assertTrue((self.root / "desktop/lib/main.dart").exists())+ for path in ("cli/main.ts", "tests/fixtures/layout.json", "daemons/frames.json", "store/catalog.json", "docs/images/poster.png"):+ self.assertTrue((self.root / path).exists(), path)+ self.sparse(["scripts/", "desktop/test/"])+ inputs.verify_checkout(self.root, "desktop", subset=True)+ with self.assertRaisesRegex(ValueError, "complete declared"):+ inputs.verify_checkout(self.root, "desktop")++ def test_full_wildcard_or_expanded_checkout_cannot_claim_a_narrow_scope(self):+ with self.assertRaises((ValueError, subprocess.CalledProcessError)):+ inputs.verify_checkout(self.root, "process")+ for paths in ([*inputs.ci.SOURCE_INPUTS["process"], "cli/"], ["*"]):+ self.sparse(paths)+ with self.assertRaises(ValueError):+ inputs.verify_checkout(self.root, "process")++ def test_checkout_actions_default_non_cone_mode_is_supported(self):+ self.sparse(inputs.ci.SOURCE_INPUTS["process"])+ self.git("config", "--worktree", "--unset", "core.sparseCheckoutCone")+ inputs.verify_checkout(self.root, "process")+ self.git("config", "--worktree", "core.sparseCheckoutCone", "true")+ with self.assertRaisesRegex(ValueError, "non-cone"):+ inputs.verify_checkout(self.root, "process")++ def test_populated_undeclared_file_and_dirty_source_are_rejected(self):+ self.sparse(inputs.ci.SOURCE_INPUTS["process"])+ path = self.root / "cli/main.ts"+ path.parent.mkdir(parents=True)+ path.write_text("fixture\n")+ with self.assertRaisesRegex(ValueError, "undeclared tracked"):+ inputs.verify_checkout(self.root, "process")+ path.unlink()+ (self.root / "scripts/check.py").write_text("dirty\n")+ with self.assertRaisesRegex(ValueError, "clean"):+ inputs.verify_checkout(self.root, "process")++ def test_capture_binds_git_objects_to_the_real_source_run_and_attempt(self):+ self.sparse(inputs.ci.SOURCE_INPUTS["process"])+ with mock.patch.dict(os.environ, GITHUB_SHA=self.sha, GITHUB_RUN_ID="123", GITHUB_RUN_ATTEMPT="2"):+ result = inputs.capture(self.root)+ self.assertEqual(result["source_sha"], self.sha)+ self.assertEqual(result["run_attempt"], 2)+ self.assertEqual(set(result["scopes"]), {"process", "desktop"})+ self.assertEqual(result["scopes"]["desktop"], inputs.ci.input_snapshot(self.root, self.sha, "desktop"))+ with mock.patch.dict(os.environ, GITHUB_SHA="b" * 40):+ with self.assertRaisesRegex(ValueError, "differs from the CI source"):+ inputs.capture(self.root)+++if __name__ == "__main__":+ unittest.main()diff --git a/scripts/tests/test_merge_validated_pr.py b/scripts/tests/test_merge_validated_pr.pyindex a09ba5202..dd599df63 100644--- a/scripts/tests/test_merge_validated_pr.py+++ b/scripts/tests/test_merge_validated_pr.py@@ -67,6 +67,21 @@ class MergeTests(unittest.TestCase):self.assertIn("merge_request", result["phases"])self.assertIn("verification", result["phases"])+ def test_scoped_ci_reuse_verifies_the_reviewed_target_without_claiming_equal_tested_trees(self):+ collect = merger.ci.collect+ def reused(*args, **kwargs):+ evidence = collect(*args, **kwargs)+ evidence["source"].update(tested_sha=self.base, tested_tree=self.git("rev-parse", f"{self.base}^{{tree}}"),+ same_tree=False, changed_files=["docs/guide.md"])+ evidence["scope_reuse"] = dict(same_inputs=True, tested_sha256="a" * 64)+ return evidence+ self.patch(merger.ci, "collect", side_effect=reused)+ result = self.finish()+ self.assertEqual(result["status"], "merged")+ self.assertTrue(result["merge"]["same_reviewed_tree"])+ self.assertFalse(result["merge"]["same_tested_tree"])+ self.assertIn("CI evidence reused", (self.output / "merge.md").read_text())+def test_wrong_pr_draft_target_or_head_stops_before_ci_or_mutation(self):original = copy.deepcopy(self.client.pr)for change in [dict(number=6), dict(state="closed"), dict(draft=True), dict(merged=True),@@ -217,8 +232,8 @@ class FakeClient:raise AssertionError(path)def pages(self, path, key):- assert key == "jobs", key- return copy.deepcopy(self.jobs)+ assert key in {"jobs", "artifacts"}, key+ return copy.deepcopy(self.jobs) if key == "jobs" else []def command(self, *args, binary=False):self.writes.append(args)diff --git a/scripts/tests/test_record_ci_validation.py b/scripts/tests/test_record_ci_validation.pyindex 079aec51f..241c96c47 100644--- a/scripts/tests/test_record_ci_validation.py+++ b/scripts/tests/test_record_ci_validation.py@@ -99,6 +99,150 @@ class EvidenceTests(unittest.TestCase):self.assertNotIn("cli_summary", result)self.assertFalse((self.output / "cli-test-summary.zip").exists())+ def input_fixture(self, scope="process"):+ for path in (".github/workflows/ci.yml", "scripts/check.py", "desktop/scripts/upload.sh", "desktop/scripts/keep.py",+ "cli/core.ts", "tests/fixtures/layout.json", "daemons/frames.json", "store/catalog.json", "docs/images/poster.png"):+ target = self.root / path+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_text("tested input\n")+ self.git("add", ".")+ self.git("-c", "user.name=Fixture", "-c", "[email protected]", "commit", "-qm", "bounded inputs")+ self.sha = self.git("rev-parse", "HEAD")+ self.run["head_sha"] = self.pr["head"]["sha"] = self.pr["base"]["sha"] = self.sha+ self.artifact["workflow_run"]["head_sha"] = self.sha+ for job in self.jobs:+ job["head_sha"] = self.sha+ if scope == "desktop":+ self.desktop_fixture()+ else:+ self.jobs = [job for job in self.jobs if job["name"] == "process-checks"]+ for job in self.jobs:+ job["steps"] = [dict(name=recorder.CHECKOUT_STEP, conclusion="success")]+ self.input_summary = dict(schema=1, kind="ci-source-inputs", status="recorded", source_sha=self.sha,+ source_tree=self.git("rev-parse", "HEAD^{tree}"), dirty=False, run_id=123, run_attempt=1,+ scopes={name: recorder.input_snapshot(self.root, self.sha, name) for name in recorder.SOURCE_INPUTS})+ self.input_artifact = dict(id=989, name="ci-source-inputs", expired=False,+ workflow_run=dict(id=123, head_sha=self.sha))+ self.artifacts = [self.artifact, self.input_artifact] if scope == "desktop" else [self.input_artifact]+ self.archives = {"repos/owner/repo/actions/artifacts/987/zip": self.archive}+ self.input_archive()++ def input_archive(self):+ archive = self.zip_summary(self.input_summary, "ci-source-inputs.json")+ self.archives["repos/owner/repo/actions/artifacts/989/zip"] = archive+ self.input_artifact.update(size_in_bytes=len(archive), digest="sha256:" + hashlib.sha256(archive).hexdigest())++ def change_source(self, path, content="changed\n"):+ target = self.root / path+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_text(content)+ self.commit_change()++ def commit_change(self):+ self.git("add", "-A")+ self.git("-c", "user.name=Fixture", "-c", "[email protected]", "commit", "-qm", "target change")+ self.pr["head"]["sha"] = self.git("rev-parse", "HEAD")++ def test_verified_process_inputs_allow_documentation_and_unrelated_component_changes(self):+ self.input_fixture()+ self.change_source("desktop/RELEASE.md")+ self.change_source("cli/src/memory/store.ts")+ result = self.collect(scope="process", with_pr=True)+ self.assertEqual(result["status"], "passed")+ self.assertFalse(result["source"]["same_tree"])+ self.assertTrue(result["scope_reuse"]["same_inputs"])+ self.assertIn("CI evidence reused", recorder.markdown(result))+ self.assertIn("not review or validation", recorder.markdown(result))+ self.assertTrue((self.output / "ci-source-inputs.zip").exists())++ def test_desktop_summary_and_inputs_both_cover_reuse_across_a_firmware_change(self):+ self.input_fixture("desktop")+ self.change_source("devices/firmware/main.c")+ result = self.collect(scope="desktop", with_pr=True)+ self.assertEqual(result["status"], "passed")+ self.assertEqual(result["desktop_summary"]["source"]["commit"], self.sha)+ self.assertTrue(result["scope_reuse"]["same_inputs"])+ self.change_source("desktop/test/new_test.dart")+ self.assertEqual(self.collect(scope="desktop")["status"], "source_review_required")++ def test_desktop_shared_fixtures_protocols_and_catalogs_are_inputs(self):+ self.input_fixture("desktop")+ for path in ("cli/core.ts", "tests/fixtures/layout.json", "daemons/frames.json", "store/catalog.json",+ "docs/images/poster.png", "mobile/pubspec.lock"):+ with self.subTest(path=path):+ self.git("reset", "--hard", self.sha)+ self.change_source(path)+ result = self.collect(scope="desktop")+ self.assertEqual(result["status"], "source_review_required")+ self.assertFalse(result["scope_reuse"]["same_inputs"])++ def test_changed_input_contents_additions_deletions_and_modes_invalidate_reuse(self):+ self.input_fixture()+ for path in ("scripts/check.py", ".github/workflows/ci.yml", "desktop/scripts/new.sh", ".gitattributes"):+ with self.subTest(path=path):+ self.git("reset", "--hard", self.sha)+ self.change_source(path)+ result = self.collect(scope="process")+ self.assertEqual(result["status"], "source_review_required")+ self.assertFalse(result["scope_reuse"]["same_inputs"])+ for mode in ("delete", "executable"):+ with self.subTest(mode=mode):+ self.git("reset", "--hard", self.sha)+ path = self.root / "desktop/scripts/upload.sh"+ if mode == "delete":+ path.unlink()+ else:+ path.chmod(0o755)+ self.commit_change()+ self.assertEqual(self.collect(scope="process")["status"], "source_review_required")++ def test_missing_input_receipt_keeps_different_source_unverified(self):+ self.input_fixture()+ self.artifacts = []+ self.change_source("docs/guide.md")+ self.assertEqual(self.collect(scope="process")["status"], "source_review_required")++ def test_source_receipt_cannot_lie_about_inputs_identity_or_checkout(self):+ self.input_fixture()+ original = copy.deepcopy(self.input_summary)+ for change in (dict(run_attempt=2), dict(run_id=True), dict(schema=True), dict(dirty=True),+ dict(source_tree="b" * 40), dict(scopes=None), dict(scopes={}), dict(source_sha="b" * 40)):+ with self.subTest(change=change):+ self.input_summary = dict(original, **change)+ self.input_archive()+ with self.assertRaises(ValueError):+ self.collect(scope="process")+ self.input_summary = original+ self.input_archive()+ self.jobs[0]["steps"][0]["conclusion"] = "skipped"+ with self.assertRaisesRegex(ValueError, "checkout was not verified"):+ self.collect(scope="process")++ def test_scoped_reuse_does_not_accept_dirty_work_or_a_different_pr_head(self):+ self.input_fixture()+ self.change_source("docs/guide.md")+ self.pr["head"]["sha"] = self.sha+ self.assertEqual(self.collect(scope="process", with_pr=True)["status"], "source_review_required")+ self.pr["head"]["sha"] = self.git("rev-parse", "HEAD")+ (self.root / "uncommitted.txt").write_text("unreviewed")+ self.assertEqual(self.collect(scope="process", with_pr=True)["status"], "source_review_required")++ def test_partial_rerun_keeps_the_receipt_bound_to_its_successful_process_job(self):+ self.input_fixture("desktop")+ self.run["run_attempt"] = 2+ for job in self.jobs:+ if job["name"] != "process-checks":+ job["run_attempt"] = 2+ self.change_source("docs/guide.md")+ self.assertEqual(self.collect(scope="desktop")["status"], "passed")+ for job in self.jobs:+ job["run_attempt"] = 2+ with self.assertRaisesRegex(ValueError, "another source/run/attempt"):+ self.collect(scope="desktop")+ self.input_summary["run_attempt"] = 2+ self.input_archive()+ self.assertEqual(self.collect(scope="desktop")["status"], "passed")+def desktop_fixture(self):self.jobs = [dict(self.jobs[0], id=i, name=name) for i, name inenumerate(sorted(recorder.DESKTOP_JOBS | {"process-checks"}), 1)]