Skip to content

Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.

fix(validation): reclaim local and SSH runs by default (#215)

澄潭d9fe4c04ae2c
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.md
index f6928bf9..5ac66989 100644
--- a/docs/code-map/testing.md
+++ b/docs/code-map/testing.md
@@ -5,7 +5,15 @@
[Execution locations](../validation-routing.md) owns caller HOME routing,
per-origin SSH workbenches, shared compiler capacity and separate candidate/driver
-revision identities. A runner setup failure is not a product test result.
+revision identities. A runner setup failure is not a product test result. Ordinary local and SSH runs
+save results and [private evidence](../../scripts/validation-report.py) before
+reclaiming their managed checkout/TMPDIR, including failed or blocked runs.
+Explicit retention covers diagnosis and consumers that still need build artifacts;
+[local lifecycle regressions](../../scripts/test-local-rust.py) and
+[SSH worker regressions](../../scripts/test-remote-rust.py) protect both paths.
+[Scheduler preparation regressions](../../scripts/test-validation-schedule.py)
+verify complete smoke receipt export without preserving executable snapshots;
+the shared exporter keeps Pilot's existing target-artifact consumer paths.
The [console artifact owner](../../scripts/console-assets.py) checks selected SSH
installed-package prerequisites before Rust execution and verifies the same-candidate
frontend from an execution-host staging directory under the runner's checkout lease.
diff --git a/docs/local-rust-validation.md b/docs/local-rust-validation.md
index f15a7160..e84ee59c 100644
--- a/docs/local-rust-validation.md
+++ b/docs/local-rust-validation.md
@@ -71,25 +71,50 @@ parsed test counts, scenario status and before/after sccache statistics are unde
`~/.cache/hiroute/local-rust/runs/ID/`. Compilation failure, timeout and missing
scenario assessment are not green. A zero-match test selection is not success.
-`--cargo-only` explicitly asserts this is a pure Cargo/component check, with no
-separate E2E verdict and no unique evidence inside `target/debug`. Successful checks
-then export all non-debug target artifacts before removing the disposable checkout.
-Failure retains it. Use `--keep` for investigations or binaries you still need.
-
-For E2E, omit `--cargo-only`: even exit=0 stays `unassessed`, and the wrapper exits
-nonzero rather than claiming product success. After assessing actual scenarios and
-copying any unique debug evidence elsewhere:
+Every ordinary run defaults to immediate reclamation after execution, including
+failed commands, timeouts, interrupted waits and preparation blocks. The runner
+keeps its acquired compiler/checkout leases through report publication, evidence
+export and cleanup. It saves results, logs and cache diagnostics in the run record,
+exports non-compiler target artifacts to `runs/ID/artifacts/`, preserving Pilot's
+existing consumer paths. Checkout `artifacts/` and surviving private TMPDIR contents
+go to `runs/ID/evidence/`. The `evidence.json` manifest records each saved path and
+file hash, including smoke build receipts with complete recipes.
+Debug/release/custom Cargo profiles, cross-target profiles, generated documentation,
+package outputs, smoke binaries and `target/validation-schedule` executable snapshots
+are rebuildable and are not copied. Inherited Cargo targets are recorded; Cargo's
+profile lock also identifies configured and previously used custom profiles. Keep unique evidence outside compiler profiles; use
+`--keep` when arbitrary debug files or built binaries are still needed.
+
+`--cargo-only` asserts a pure Cargo/component check with no separate product
+assessment. It controls the scenario verdict, independently of cleanup. Omitting
+it leaves exit-zero execution `unassessed` and the wrapper returns nonzero; the
+saved record can still be assessed after its checkout is removed. Failure and
+cleanup never manufacture a green scenario.
+
+`--keep` is the explicit exception for investigations, subsequent binary use and
+Desktop/Pilot acceptance. Native Pilot acquisition already requests retention;
+its final registered owner releases it through `pilot-build finish`. For a manually
+kept E2E run, assess actual scenarios and export any unique debug evidence before:
```sh
python3 scripts/local-rust.py finish RUN_ID --verdict green --evidence-saved
+python3 scripts/local-rust.py release RUN_ID
python3 scripts/local-rust.py preview
python3 scripts/local-rust.py apply RUN_ID --token TOKEN_FROM_PREVIEW
```
+Final cleanup of new explicitly kept runs uses the same evidence exporter before
+removing the checkout and TMPDIR. Unknown source/untracked/ignored material or an
+export failure retains the originals with a reason; save or remove the material
+and retry cleanup. Existing records keep their earlier evidence-confirmation rules.
+
Use `red` or `expected_red` honestly; neither proves product completion. `finish`
-records assessment but does not itself delete anything. Missing artifact exports,
-links in artifacts, source changes or untracked/ignored files outside target cause
-cleanup to retain the checkout. Logs/results remain even after successful cleanup.
+records assessment but does not itself delete anything. Failed report/export,
+links or special files in evidence, source changes, unknown generated files or
+changed resource identities retain the checkout with an explicit cleanup reason.
+Private runtime evidence is exported before its TMPDIR is removed. Developer
+source worktrees, other owners and shared compiler caches remain separately owned.
+Existing retained records keep their original evidence-confirmation requirements.
## Retire a developer target explicitly
@@ -125,9 +150,8 @@ non-destructive; apply rechecks the snapshot and holds the same locks until done
Before local validation, less than 30 GiB free triggers this bounded managed GC,
aiming for 60 GiB. If still below 30 GiB, compilation is refused. Unknown or kept
-directories are not sacrificed to meet the target. Remote retention remains the
-existing `remote-rust.py` policy and commands; this patch does not alter remote
-execution or introduce a scheduler.
+directories are not sacrificed to meet the target. The SSH runner uses the same immediate-cleanup default through `--retention never`;
+`--retention always` and `on-failure` are explicit retention exceptions.
## Owner cleanup during build/fix/retest
@@ -203,7 +227,7 @@ exact pushed candidate and its platform environment.
An opt-in integration test uses a tiny dependency-free crate pushed to a disposable
local bare origin, real Cargo and an isolated sccache endpoint/cache under the shared host lock. It tests this runner's
-fetch/build/evidence/reclamation path, two compilations across run cleanup, stale-server refusal and failed-test retention without compiling the HiRoute workspace:
+fetch/build/evidence/reclamation path, two compilations across run cleanup, stale-server refusal and default failed-test reclamation and explicit retention without compiling the HiRoute workspace:
```sh
HIROUTE_REAL_CARGO_TEST=1 python3 scripts/test-local-rust.py Cleanup.test_real_cargo_pushed_fixture_lifecycle
diff --git a/docs/remote-rust-validation.md b/docs/remote-rust-validation.md
index 82c17558..47469d1f 100644
--- a/docs/remote-rust-validation.md
+++ b/docs/remote-rust-validation.md
@@ -183,7 +183,8 @@ needs the appropriate platform.
For local macOS/platform checks, use the separate [local runner and cleanup
commands](local-rust-validation.md). They preserve the same default-target boundary;
-the HOME configuration selects the execution location and this workbench's retention is unchanged.
+the HOME configuration selects the execution location. Both runners reclaim ordinary
+terminal runs after saving evidence; explicit retention supports subsequent use.
Every run has an isolated default `target/`; it is deliberately never shared with a
different checkout. sccache reuses compiler objects safely, but Cargo must still write
@@ -191,19 +192,34 @@ link outputs, test binaries, fingerprints and build-script products into that ru
own target. Keeping every successful target therefore consumes disk without being
required for normal incremental development.
-`run` defaults to `--retention on-failure`:
+`run` defaults to `--retention never`:
| Policy | Completed run | Failed or blocked run | Appropriate use |
| --- | --- | --- | --- |
-| `on-failure` (default) | Automatically removes checkout, private target, private TMPDIR and validation ref after evidence is written | Retains them for diagnosis | Normal focused development |
+| `on-failure` | Automatically removes checkout, private target, private TMPDIR and validation ref after evidence is written | Retains them for diagnosis | Explicit failure diagnosis |
| `always` | Retains them | Retains them | E2E artifacts, benchmark investigation or an intentional inspection checkpoint |
-| `never` | Automatically removes them | Automatically removes them | Fully disposable repeatable checks |
-
-Run logs, result JSON, cache statistics and worker diagnostics are always retained.
+| `never` (default) | Automatically removes them | Automatically removes them | Ordinary builds and tests |
+
+Run logs, result JSON, cache statistics and worker diagnostics are retained in the
+run record. Before automatic cleanup, the worker holds its checkout lease while
+exporting non-compiler target artifacts to `runs/ID/artifacts/`, and checkout
+`artifacts/` and surviving private TMPDIR contents to `runs/ID/evidence/`. The
+`evidence.json` manifest records the source revision, saved paths and file hashes.
+Smoke build receipts preserve complete recipes; adjacent binaries and scheduler
+executable snapshots remain rebuildable cache and are not copied. Cargo
+debug/release/custom profiles, cross-target profiles, generated documentation and
+package outputs are also excluded. Inherited Cargo targets are recorded and Cargo
+profile locks identify configured and previously used custom profiles. Put unique evidence outside compiler profiles;
+select explicit retention for binaries or arbitrary compiler-profile files needed
+later. Report/export failure retains the originals and records the cleanup reason;
+cleanup does not change the command's exit or scenario verdict.
Automatic cleanup first verifies a terminal result, the recorded worker lifecycle, the
-direct managed-worktree path and Git registration. If a generated test checkout has
-untracked artifacts such as Python `__pycache__`, the runner may use Git's force
-removal **only after** those ownership checks. It never force-removes an arbitrary
+direct managed-worktree path and Git registration. New records also reject modified
+source and unknown untracked/ignored files, preserving the checkout and TMPDIR until
+the owner saves or removes that material. Explicit cleanup re-exports evidence under
+the checkout lease. Known generated roots such as target, exported artifacts and
+Python `__pycache__` may use Git's force removal **only after** those checks.
+It never force-removes an arbitrary
path, an active run, a shared cache or a user checkout.
Inspect and explicitly reclaim one completed run when its retained artifacts are no
diff --git a/docs/validation-routing.md b/docs/validation-routing.md
index 8ba25148..0836e02f 100644
--- a/docs/validation-routing.md
+++ b/docs/validation-routing.md
@@ -93,6 +93,11 @@ Help does not contact the execution host or require its checkout to be updated.
Arguments after `backend`/`desktop` otherwise follow the selected runner's CLI.
SSH backend returns a queued run ID and supports `wait/status/logs/cleanup/gc`.
+Both local and SSH ordinary runs save reports and private evidence, then reclaim
+their managed checkout and temporary directory after success, failure, timeout or
+pre-command block. Use local `--keep` or SSH `--retention always` when subsequent
+steps need those binaries; SSH `--retention on-failure` explicitly retains failures.
+Desktop/Pilot acquisition retains its build until the registered owners finish.
Use `backend wait RUN_ID --timeout 50` for one compact terminal or timeout
observation; repeat a bounded wait when necessary, and inspect full status/logs
for failures or specific evidence. A wait timeout does not change the run state.
@@ -614,7 +619,6 @@ ssh "$backend_host" "mkdir -p '$staging_dir'"
scp -r apps/desktop/dist "$backend_host:$staging_dir/"
python3 scripts/validation.py --frontend-dist "~/$staging_dir/dist" backend run \
--ref refs/heads/codex/my-change --sha CANDIDATE_SHA \
- --reuse-checkout BLOCKED_RUN_ID --retention always \
--plan my-change --phase final -- \
cargo test --locked -p hiroute-daemon --test publication_process installed_ -- --nocapture
```
@@ -637,8 +641,8 @@ executed related runs and Cargo arguments. For an integration checkout retained
advances; the original owner remains responsible for cleanup. An early blocked owner
retained by `on-failure` uses exact-candidate reuse. Both paths recheck stopped-worker,
current-state, clean-source and driver identity under the checkout lease before
-checking the supplied assets again. `never` removes a fresh checkout, so its command
-starts a new one. Old results remain unchanged. The checkout state records resource use;
+checking the supplied assets again. The default `never` removes a fresh checkout,
+so its recovery command starts a new one and keeps the immediate-cleanup policy. Old results remain unchanged. The checkout state records resource use;
`--related-run` accepts executed reports, so do not pass a setup-blocked run there.
Do not replace production assets with a dummy page
or weaken package assertions. Hosted CI retains its existing frontend artifact
diff --git a/issue-spec/specs/product-smoke-verification/spec.md b/issue-spec/specs/product-smoke-verification/spec.md
index 833a93d2..11575cde 100644
--- a/issue-spec/specs/product-smoke-verification/spec.md
+++ b/issue-spec/specs/product-smoke-verification/spec.md
@@ -214,6 +214,32 @@ attributable terminal results and counts, the preparation failure, and explicit
unexecuted target/doctest coverage. Completed independent targets cannot establish
complete validation when dependent targets were not admitted.
+#### Scenario: Managed execution reclaims disposable artifacts after saving evidence
+
+Ordinary local and SSH build/test runs MUST default to reclaiming their managed
+checkout, compiler outputs and private temporary directory after terminal success,
+failure, timeout or preparation block. Before reclamation, they MUST save the exact
+source/command, actual process exit, tests/scenario verdict, logs and required
+private artifact/runtime evidence outside the disposable checkout. Default cleanup
+MUST NOT copy rebuildable compiler profiles or smoke build caches wholesale into
+the evidence store. Selected smoke build receipts MUST preserve complete build
+recipes, while scheduler executable snapshots remain disposable. New explicitly
+retained runs MUST use the same evidence export when their owner releases them.
+Commands that never create a target directory MUST still reclaim their resources
+when no recorded target identity or unknown material prevents cleanup.
+Evidence export and deletion MUST remain protected by ownership
+and execution leases; an export failure, active use, changed identity or unknown
+material MUST preserve the affected original resources with an actionable reason.
+Cleanup MUST NOT turn an unassessed/failed run into successful product acceptance.
+
+Explicit retention MUST remain available for investigation and subsequent artifact
+consumers. Native Pilot builds retain their registered ownership until the final
+consumer finishes. A default console-preparation retry MUST use a fresh checkout
+with the same immediate-cleanup policy after its previous checkout is reclaimed.
+Developer source worktrees, shared compiler caches and other owners' resources
+MUST NOT be reclaimed by this terminal-run policy. Existing retained records keep
+their original retention and evidence-confirmation meaning.
+
### Requirement: Native Pilot App reopening preserves its isolated owner
An explicitly requested macOS native Pilot App entry MUST belong to one registered
diff --git a/scripts/local-rust.py b/scripts/local-rust.py
index 90c2dafe..5944ebff 100644
--- a/scripts/local-rust.py
+++ b/scripts/local-rust.py
@@ -129,17 +129,20 @@ class Store:
if row["kind"] == "managed":
if checkout.parent != self.root / "checkouts" or checkout.name != row["id"]:
raise ValueError("Not a managed checkout")
- if git(checkout, "status", "--porcelain", "--untracked-files=all"):
- raise ValueError("Source changes/untracked files retained")
- ignored = git(checkout, "ls-files", "--others", "--ignored", "--exclude-standard")
- if any(not p.startswith("target/") for p in ignored.splitlines()):
- raise ValueError("Ignored files outside target retained")
+ if "retention" in row:
+ remote.verify_disposable_source(checkout)
+ else:
+ if git(checkout, "status", "--porcelain", "--untracked-files=all"):
+ raise ValueError("Source changes/untracked files retained")
+ ignored = git(checkout, "ls-files", "--others", "--ignored", "--exclude-standard")
+ if any(not p.startswith("target/") for p in ignored.splitlines()):
+ raise ValueError("Ignored files outside target retained")
target = checkout / "target"
- # Lock admission can fail before Cargo creates a target. Only a managed,
- # never-started run without recorded target identities may use this path.
+ # Some commands never create a target, including failed package selection
+ # and cargo fmt. New managed records attest that absence at completion.
# A later-created target (including a dangling link) still fails closed.
- if (row["kind"] == "managed" and row.get("process_exit") is None
- and not row.get("command_started_at")
+ if (row["kind"] == "managed" and ("retention" in row or
+ row.get("process_exit") is None and not row.get("command_started_at"))
and "target_identity" not in row and "debug_identity" not in row
and not os.path.lexists(target)):
if git(checkout, "ls-files", "target"):
@@ -149,27 +152,30 @@ class Store:
if target.resolve() != target or identity(target) != row["target_identity"]:
raise ValueError("Target was replaced")
debug = target / "debug"
+ if "retention" in row and "debug_identity" not in row and not os.path.lexists(debug):
+ return checkout, debug
if identity(debug) != row["debug_identity"]:
raise ValueError("Debug directory was replaced")
if git(checkout, "ls-files", "target"):
raise ValueError("Target contains tracked files")
return checkout, debug
- def preview_one(self, row):
+ def preview_one(self, row, *, lease=None):
report = {"id": row["id"], "checkout": row["checkout"], "state": "skipped"}
try:
- with self.locked(row["checkout"]):
+ with self.locked(row["checkout"]) if lease is None else contextlib.nullcontext(lease):
if row.get("removed") or row.get("keep") or row["status"] not in ("terminal", "retired"):
raise ValueError("Active, kept, removed, or unknown lifecycle")
if row.get("pilot_reuse") and not row.get("pilot_cleanup_ready"):
raise ValueError("Pilot owners have not finished their registered sessions")
if not row.get("evidence_saved"):
raise ValueError("Evidence not confirmed/exported")
- _, debug = self.verify(row)
- stamp, size = fingerprint(debug)
+ checkout, debug = self.verify(row)
+ artifact = checkout / "target" if row["kind"] == "managed" and "retention" in row else debug
+ stamp, size = fingerprint(artifact)
token = hashlib.sha256((json.dumps(row, sort_keys=True) + stamp).encode()).hexdigest()
report.update(state="candidate", token=token, bytes=size,
- target=str(debug), sha=row["sha"], kind=row["kind"])
+ target=str(artifact), sha=row["sha"], kind=row["kind"])
except (OSError, ValueError, KeyError, subprocess.SubprocessError) as error:
report["reason"] = str(error)
return report
@@ -178,10 +184,10 @@ class Store:
return [self.preview_one(self.load(p.parent.name))
for p in sorted((self.root / "runs").glob("*/result.json"))]
- def apply(self, key, token):
+ def apply(self, key, token, *, lease=None):
row = self.load(key)
# Hold locks throughout revalidation and deletion, including snapshot comparison.
- with self.locked(row["checkout"]):
+ with self.locked(row["checkout"]) if lease is None else contextlib.nullcontext(lease):
current_row = self.load(key)
if current_row["checkout"] != row["checkout"]:
raise ValueError("Record checkout changed")
@@ -193,24 +199,30 @@ class Store:
if not row.get("evidence_saved"):
raise ValueError("Evidence not saved")
checkout, debug = self.verify(row)
- stamp, size = fingerprint(debug)
+ artifact = checkout / "target" if row["kind"] == "managed" and "retention" in row else debug
+ stamp, size = fingerprint(artifact)
current = hashlib.sha256((json.dumps(row, sort_keys=True) + stamp).encode()).hexdigest()
if token != current:
raise ValueError("Candidate changed; preview again")
if row["kind"] == "managed":
- # Preserve all non-debug target artifacts before removing this disposable tree.
- if os.path.lexists(checkout / "target"):
+ # New records export runtime evidence even without a target tree.
+ if "retention" in row or os.path.lexists(checkout / "target"):
self.export_target(row)
remote.command(["git", "-C", row["repo"], "worktree", "remove", "--force", str(checkout)])
else:
shutil.rmtree(debug) # Never remove a developer worktree or its release/evidence.
row.update(removed=True, cleaned_at=time.time(), reclaimed_debug_bytes=size)
+ if "retention" in row:
+ row["cleanup"] = dict(state="removed", bytes=size)
if row.get("temporary_directory"):
row["temporary_cleanup"] = remote.cleanup_temp_directory(row["temporary_directory"])
self.save(row)
return {"id": key, "state": "removed", "bytes": size}
def export_target(self, row):
+ if "retention" in row:
+ row["evidence_export"] = remote.reporting().preserve_evidence(self.record_path(row["id"]).parent, row)
+ return
target = Path(row["checkout"]) / "target"
destination = self.record_path(row["id"]).parent / "artifacts"
for entry in target.iterdir():
@@ -293,8 +305,10 @@ def run(store, args, prepare=None, on_record=None):
checkout = store.root / "checkouts" / key
row = dict(id=key, kind="managed", repo=str(repo), checkout=str(checkout), sha=args.sha,
command=cargo, status="preparing", keep=args.keep, evidence_saved=False,
+ retention="always" if args.keep else "never",
process_exit=None, scenario="unassessed", candidate_source=source,
build_jobs=jobs, wait_for_lock=getattr(args, "wait_for_lock", False),
+ cargo_build_target=os.environ.get("CARGO_BUILD_TARGET"),
platform=sys.platform, architecture=platform.machine(),
submitted_at=time.time(), **feedback)
row["executor_sha256"] = {
@@ -326,16 +340,18 @@ def run(store, args, prepare=None, on_record=None):
store.save(row)
# Cache admission and Cargo share the existing exclusive compiler lease.
env = dict(os.environ, CARGO_BUILD_JOBS=str(jobs), CARGO_INCREMENTAL="0", RUSTC_WRAPPER="sccache")
- process = None
- try:
- server_env = remote.cache_server_environment(store.root)
- row["cache_server_environment"] = {
- "temporary_directory": server_env["TMPDIR"],
- "idle_timeout": "0",
- }
- row["queued_at"] = time.time()
- store.save(row)
- with store.locked(checkout, wait=row["wait_for_lock"]) as handles:
+ with contextlib.ExitStack() as leases:
+ process = None
+ handles = None
+ try:
+ server_env = remote.cache_server_environment(store.root)
+ row["cache_server_environment"] = {
+ "temporary_directory": server_env["TMPDIR"],
+ "idle_timeout": "0",
+ }
+ row["queued_at"] = time.time()
+ store.save(row)
+ handles = leases.enter_context(store.locked(checkout, wait=row["wait_for_lock"]))
row["capacity_acquired_at"] = time.time()
if row["wait_for_lock"] and shutil.disk_usage(store.root).free < 30 * GIB:
raise ValueError("Less than 30 GiB free after waiting; build not started")
@@ -375,21 +391,37 @@ def run(store, args, prepare=None, on_record=None):
"cache-after",
server_env,
)
- except (Exception, KeyboardInterrupt) as error:
- row["error"] = str(error)
- row["scenario"] = "red"
- finally:
- row.update(status="terminal", finished_at=time.time())
- reporter.write_report(store.record_path(key).parent, row, related)
- for name, path in (("target_identity", checkout / "target"), ("debug_identity", checkout / "target/debug")):
- if path.is_dir() and not path.is_symlink():
- row[name] = identity(path)
- store.save(row)
- if row["scenario"] == "green" and not row["keep"]:
- candidate = store.preview_one(row)
- if candidate["state"] == "candidate":
- store.apply(key, candidate["token"])
- return store.load(key)
+ except (Exception, KeyboardInterrupt) as error:
+ row["error"] = str(error)
+ row["scenario"] = "red"
+ finally:
+ row.update(status="terminal", finished_at=time.time())
+ reporter.write_report(store.record_path(key).parent, row, related)
+ for name, path in (("target_identity", checkout / "target"), ("debug_identity", checkout / "target/debug")):
+ if path.is_dir() and not path.is_symlink():
+ row[name] = identity(path)
+ store.save(row)
+ if not row["keep"]:
+ try:
+ with store.locked(checkout) if handles is None else contextlib.nullcontext(handles):
+ store.verify(row)
+ store.export_target(row)
+ row["evidence_saved"] = True
+ store.save(row)
+ candidate = store.preview_one(row, lease=handles)
+ if candidate["state"] == "candidate":
+ store.apply(key, candidate["token"], lease=handles)
+ else:
+ row = store.load(key)
+ row["cleanup"] = dict(state="retained", reason=candidate["reason"])
+ store.save(row)
+ except (OSError, ValueError, KeyError, subprocess.SubprocessError) as error:
+ row = store.load(key)
+ row["evidence_saved"] = False
+ row["evidence_export"] = dict(state="failed", error=str(error))
+ row["cleanup"] = dict(state="retained", reason=str(error))
+ store.save(row)
+ return store.load(key)
def argument_parser():
@@ -403,7 +435,7 @@ def argument_parser():
p.add_argument("--jobs", type=int, default=2)
p.add_argument("--wait-for-lock", action="store_true",
help="Wait on the existing exclusive host/checkout locks before starting Cargo")
- p.add_argument("--keep", action="store_true")
+ p.add_argument("--keep", action="store_true", help="Explicitly retain the checkout and artifacts after this run")
p.add_argument("--cargo-only", action="store_true", help="Assert pure Cargo check: no external scenario verdict or unique debug artifacts")
p.add_argument("--timeout", type=int, default=3600)
remote.reporting().add_arguments(p)
diff --git a/scripts/remote-rust.py b/scripts/remote-rust.py
index 756b0c79..8623faf3 100644
--- a/scripts/remote-rust.py
+++ b/scripts/remote-rust.py
@@ -416,7 +416,8 @@ def checkout_recovery_command(root, result, frontend_dist=None):
modes = checkout_recovery_modes(first, result)
rerun = ["python3", "scripts/remote-rust.py", "--host", result["ssh_host"],
"--workbench", result.get("workbench", ROOT), "run", "--ref", result["ref"],
- "--sha", result["sha"], "--retention", "always", "--timeout", str(result["timeout"])]
+ "--sha", result["sha"], "--retention",
+ "never" if result.get("retention") == "never" else "always", "--timeout", str(result["timeout"])]
if frontend_dist:
rerun += ["--frontend-dist", str(frontend_dist)]
if modes:
@@ -456,6 +457,24 @@ def cleanup_temp_directory(info):
return {"state": "retained", "path": str(directory), "error": str(error)}
+def verify_disposable_source(checkout):
+ """Keep unknown source/material; only registered generated roots are disposable."""
+ allowed = ("target/", "artifacts/", "apps/desktop/node_modules/",
+ "apps/desktop/dist/", "apps/desktop/src-tauri/gen/")
+
+ def generated(path):
+ return path.startswith(allowed) or "__pycache__" in Path(path).parts
+
+ changes = subprocess.check_output(
+ ["git", "-C", str(checkout), "status", "--porcelain", "-z", "--untracked-files=all"], text=True)
+ if any(entry and (entry[:2] != "??" or not generated(entry[3:])) for entry in changes.split("\0")):
+ raise ValueError("Source changes/untracked files retained; save or remove them before cleanup")
+ ignored = subprocess.check_output(
+ ["git", "-C", str(checkout), "ls-files", "--others", "--ignored", "--exclude-standard", "-z"], text=True)
+ if any(path and not generated(path) for path in ignored.split("\0")):
+ raise ValueError("Unknown ignored files retained; save or remove them before cleanup")
+
+
def cleanup_checkout(root, result):
"""Remove one terminal, generated checkout without ever accepting an arbitrary path."""
checkout_text = result.get("checkout")
@@ -481,6 +500,17 @@ def cleanup_checkout(root, result):
raise RuntimeError("checkout is not this run's managed direct child")
if resolved not in worktree_paths(repo):
raise RuntimeError("checkout is not registered by the managed bare repository")
+ if "evidence_export" in result:
+ try:
+ verify_disposable_source(resolved)
+ # Re-export under the lease, including later evidence from retained runs.
+ # A retained owner may have advanced; bind exported bytes to its actual source.
+ revision = command(["git", "-C", str(resolved), "rev-parse", "HEAD"])
+ result["evidence_export"] = reporting().preserve_evidence(
+ root / "runs" / result["id"], dict(result, sha=revision))
+ except Exception as error:
+ result["evidence_export"] = dict(state="failed", error=str(error))
+ raise
status = subprocess.run(["git", "-C", str(resolved), "status", "--porcelain"],
text=True, capture_output=True, check=True).stdout.splitlines()
command_args = ["git", "--git-dir", str(repo), "worktree", "remove"]
@@ -533,7 +563,9 @@ def cleanup_terminal_run(root, run_id, current_worker=False):
checkout = cleanup_checkout(root, result)
validation_ref = cleanup_validation_ref(root, run_id,
"removed" if result.get("checkout_owner") and result["checkout_owner"] != run_id else checkout["state"])
- temporary = cleanup_temp_directory(result.get("temporary_directory"))
+ temporary = (cleanup_temp_directory(result.get("temporary_directory"))
+ if result.get("evidence_export", {}).get("state") != "failed"
+ else {"state": "retained", "reason": "Evidence export failed"})
result["cleanup"] = {
"policy": result.get("retention", "always"),
"requested_at": time.time(),
@@ -742,6 +774,7 @@ def worker(request_path):
check_target(checkout, metadata["target_directory"])
result["rustc"] = command(["rustc", "--version"], checkout)
result["cargo"] = command(["cargo", "--version"], checkout)
+ result["cargo_build_target"] = os.environ.get("CARGO_BUILD_TARGET")
result["build_jobs"] = int(os.environ["CARGO_BUILD_JOBS"])
save(result_path, result)
with (run / "command.log").open("w") as output:
@@ -820,13 +853,18 @@ def worker(request_path):
except Exception as error:
result["validation_report"] = dict(complete=False, error=str(error))
require_dag_report(result)
+ try:
+ result["evidence_export"] = reporting().preserve_evidence(run, result)
+ except Exception as error:
+ result["evidence_export"] = dict(state="failed", error=str(error))
save(result_path, result)
if reused_lease is not None:
reused_lease.close()
- if should_auto_cleanup(result):
+ if should_auto_cleanup(result) and result["evidence_export"]["state"] == "saved":
cleanup_terminal_run(root, request["id"], current_worker=True)
else:
- retain_cleanup_facts(result, "retention policy preserves this terminal run")
+ retain_cleanup_facts(result, "Evidence export failed" if result["evidence_export"]["state"] == "failed"
+ else "retention policy preserves this terminal run")
save(result_path, result)
@@ -961,8 +999,8 @@ def main():
run.add_argument("--frontend-dist", help="Execution-host staging directory from validation.py frontend build --console-assets")
run.add_argument("--exclusive", action="store_true", help="Reserve all 8 slots, 64 compiler jobs (bench/Gates)")
run.add_argument("--timeout", type=int, default=3600)
- run.add_argument("--retention", choices=RETENTION_POLICIES, default="on-failure",
- help="Checkout/TMPDIR retention: on-failure (default), always, or never")
+ run.add_argument("--retention", choices=RETENTION_POLICIES, default="never",
+ help="Checkout/TMPDIR retention: never (default), on-failure, or always")
run.add_argument("--reuse-checkout", metavar="RUN_ID",
help="Reuse a retained owner's current exact checkout and target under an exclusive lease")
run.add_argument("--continue-checkout", metavar="PREVIOUS_RUN_ID",
diff --git a/scripts/test-console-assets.py b/scripts/test-console-assets.py
index fc6e7e5c..00916fb2 100644
--- a/scripts/test-console-assets.py
+++ b/scripts/test-console-assets.py
@@ -422,7 +422,7 @@ sys.exit(101 if bad else 0)
initial = self.generated_request(shlex.join([
'python3', 'scripts/remote-rust.py', '--host', 'fixture-backend',
'--workbench', '.local/share/fixture-workbench', 'run', '--ref', self.git('symbolic-ref', 'HEAD'),
- '--sha', self.sha, '--plan', 'console-retry', '--',
+ '--sha', self.sha, '--retention', 'on-failure', '--plan', 'console-retry', '--',
'cargo', 'test', '--test', 'publication_process', 'installed_']), owner)
self.assertEqual(initial['retention'], 'on-failure')
first, stats = self.execute_worker(root, initial)
diff --git a/scripts/test-local-rust.py b/scripts/test-local-rust.py
index 44885341..ca745c60 100644
--- a/scripts/test-local-rust.py
+++ b/scripts/test-local-rust.py
@@ -1,6 +1,8 @@
#!/usr/bin/env python3
"""Cleanup safety tests: real temporary Git worktrees, no Rust compilation."""
import importlib.util
+import contextlib
+import io
import json
import os
from pathlib import Path
@@ -30,7 +32,7 @@ class Cleanup(unittest.TestCase):
m.git(self.repo, "config", "user.name", "Fixture")
m.git(self.repo, "config", "user.email", "[email protected]")
(self.repo / "source").write_text("preserve source")
- (self.repo / ".gitignore").write_text("target/\n")
+ (self.repo / ".gitignore").write_text("target/\nartifacts/\n")
m.git(self.repo, "add", ".")
m.git(self.repo, "commit", "-qm", "fixture")
self.store = m.Store(self.base / "state", self.base / "global.lock")
@@ -525,7 +527,8 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
self.assertEqual(m.toolchain_environment(),
os.pathsep.join([str(loaded), str(fallback)]))
- def run_fixture(self, exit_code=0, cargo_only=True, delay=0, source="origin", jobs=2, command=None):
+ def run_fixture(self, exit_code=0, cargo_only=True, delay=0, source="origin", jobs=2, command=None,
+ keep=False, profile='debug', cache_error=None, no_target=False, prepare=None):
# Real git origin/worktree and subprocess lifecycle; fake compiler means NO Rust proof.
m.git(self.repo, 'remote', 'add', 'origin', str(self.base / 'unpublished') if source == 'local' else str(self.repo))
branch = m.git(self.repo, 'symbolic-ref', 'HEAD')
@@ -533,15 +536,21 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
binary.mkdir()
cargo = binary / 'cargo'
cargo.write_text('#!' + m.sys.executable + '\n'
- 'import sys,json,pathlib\n'
+ 'import sys,json,pathlib,os\n'
'p=pathlib.Path.cwd()/"target"\n'
'if sys.argv[1]=="metadata":\n'
' print(json.dumps({"target_directory":str(p)}))\n'
'elif sys.argv[1]=="--version":\n'
' print("cargo fixture")\n'
'else:\n'
- ' (p/"debug").mkdir(parents=True)\n'
- ' (p/"debug/binary").write_text("fixture")\n'
+ + ('' if no_target else
+ ' (p/' + repr(profile) + ').mkdir(parents=True)\n'
+ ' (p/' + repr(profile) + '/"binary").write_text("rebuildable fixture")\n'
+ ' (p/"e2e").mkdir()\n'
+ ' (p/"e2e/result.json").write_text(\'{"observed":"fixture"}\')\n') +
+ ' (pathlib.Path.cwd()/"artifacts").mkdir()\n'
+ ' (pathlib.Path.cwd()/"artifacts/unique.json").write_text("checkout evidence")\n'
+ ' (pathlib.Path(os.environ["TMPDIR"])/"diagnostic.log").write_text("private runtime evidence")\n'
' import time; time.sleep(' + str(delay) + ')\n'
' print("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)")\n'
' print("running 1 test")\n'
@@ -554,14 +563,14 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
tool.chmod(0o700)
args = types.SimpleNamespace(repo=str(self.repo), ref=branch,
sha=m.git(self.repo, 'rev-parse', 'HEAD'),
- command=command or ['cargo', 'test'], keep=False,
+ command=command or ['cargo', 'test'], keep=keep,
cargo_only=cargo_only, timeout=1 if delay else 10,
source=source, jobs=jobs)
with patch.dict(os.environ, {'PATH': str(binary) + os.pathsep + os.environ['PATH']}), \
patch.object(m.remote, 'stats', return_value={'cache_hits': 1}), \
- patch.object(m.remote, 'admit_cache_server', return_value={'state': 'ready'}), \
+ patch.object(m.remote, 'admit_cache_server', side_effect=cache_error, return_value={'state': 'ready'}), \
patch.object(self.store, 'ensure_space'):
- result = m.run(self.store, args)
+ result = m.run(self.store, args, prepare=prepare)
if result.get('temporary_directory'):
self.addCleanup(m.remote.cleanup_temp_directory, result['temporary_directory'])
return result
@@ -585,7 +594,7 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
self.assertEqual(result['scenario'], 'unassessed')
self.assertEqual(result['candidate_source'], 'local')
self.assertEqual(result['build_jobs'], 3)
- self.assertEqual((Path(result['checkout']) / 'source').read_text(), 'preserve source')
+ self.assertFalse(Path(result['checkout']).exists())
self.assertEqual((self.repo / 'source').read_text(), 'uncommitted source must stay outside build')
def test_managed_cargo_run_keeps_feedback_after_automatic_cleanup(self):
@@ -612,28 +621,152 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
self.assertTrue((self.store.record_path(result['id']).parent / 'command.log').exists())
self.assertFalse(Path(result['checkout']).exists())
- def test_managed_run_failure_retains_checkout_and_temp(self):
+ def test_managed_run_failure_exports_evidence_then_removes_checkout_and_temp(self):
result = self.run_fixture(exit_code=4)
self.assertEqual(result['process_exit'], 4)
self.assertEqual(result['scenario'], 'red')
- self.assertTrue(Path(result['checkout']).exists())
- self.assertTrue(Path(result['temporary_directory']['path']).exists())
-
- def test_zero_exit_without_scenario_assessment_retains(self):
+ self.assertFalse(Path(result['checkout']).exists())
+ self.assertFalse(Path(result['temporary_directory']['path']).exists())
+ run = self.store.record_path(result['id']).parent
+ self.assertEqual((run / 'artifacts/e2e/result.json').read_text(), '{"observed":"fixture"}')
+ self.assertEqual((run / 'evidence/temporary/diagnostic.log').read_text(), 'private runtime evidence')
+ self.assertFalse((run / 'artifacts/debug').exists())
+ self.assertEqual(json.loads((run / 'validation-report.json').read_text())['process_exit'], 4)
+ self.assertEqual(result['evidence_export']['state'], 'saved')
+
+ def test_zero_exit_without_scenario_assessment_cleans_without_claiming_green(self):
result = self.run_fixture(cargo_only=False)
self.assertEqual(result['process_exit'], 0)
self.assertEqual(result['scenario'], 'unassessed')
- self.assertFalse(result['evidence_saved'])
- self.assertTrue(Path(result['checkout']).exists())
+ self.assertTrue(result['evidence_saved'])
+ self.assertFalse(Path(result['checkout']).exists())
- def test_timeout_reaps_child_and_releases_locks_without_cleanup(self):
+ def test_timeout_reaps_child_before_cleanup_and_preserves_failure_evidence(self):
result = self.run_fixture(delay=30)
self.assertEqual(result['process_exit'], -9)
self.assertEqual(result['scenario'], 'red')
- self.assertTrue(Path(result['checkout']).exists())
+ self.assertFalse(Path(result['checkout']).exists())
+ self.assertFalse(Path(result['temporary_directory']['path']).exists())
+ self.assertEqual((self.store.record_path(result['id']).parent / 'evidence/temporary/diagnostic.log').read_text(),
+ 'private runtime evidence')
with self.store.locked(result['checkout']):
pass
+ def test_explicit_keep_retains_failed_run_for_diagnosis(self):
+ result = self.run_fixture(exit_code=101, keep=True)
+ self.assertEqual((result['scenario'], result['process_exit']), ('red', 101))
+ self.assertTrue(Path(result['checkout']).exists())
+ self.assertTrue(Path(result['temporary_directory']['path']).exists())
+
+ def test_kept_release_run_exports_all_evidence_when_the_owner_finishes_and_releases(self):
+ result = self.run_fixture(exit_code=101, keep=True, profile='release',
+ command=['cargo', 'test', '--release'])
+ for arguments in (['finish', result['id'], '--verdict', 'expected_red', '--evidence-saved'],
+ ['release', result['id']]):
+ with patch.object(m.sys, 'argv', ['local-rust.py', *arguments]), \
+ patch.object(m, 'Store', return_value=self.store), contextlib.redirect_stdout(io.StringIO()):
+ self.assertEqual(m.main(), 0)
+ row = self.store.load(result['id'])
+ candidate = self.store.preview_one(row)
+ self.assertEqual(candidate['state'], 'candidate', candidate)
+ self.store.apply(row['id'], candidate['token'])
+ saved = self.store.load(row['id'])
+ run = self.store.record_path(row['id']).parent
+ self.assertEqual((saved['scenario'], saved['process_exit']), ('expected_red', 101))
+ self.assertFalse(Path(row['checkout']).exists())
+ self.assertFalse(Path(row['temporary_directory']['path']).exists())
+ self.assertEqual(saved.get('evidence_export', {}).get('state'), 'saved')
+ self.assertTrue((run / 'evidence.json').is_file())
+ self.assertEqual((run / 'artifacts/e2e/result.json').read_text(), '{"observed":"fixture"}')
+ self.assertEqual((run / 'evidence/artifacts/unique.json').read_text(), 'checkout evidence')
+ self.assertEqual((run / 'evidence/temporary/diagnostic.log').read_text(), 'private runtime evidence')
+ self.assertFalse((run / 'artifacts/release').exists())
+
+ def assert_targetless_cleanup(self, command, exit_code):
+ result = self.run_fixture(command=command, exit_code=exit_code, no_target=True)
+ self.assertEqual(result['process_exit'], exit_code)
+ self.assertIn('command_started_at', result)
+ self.assertTrue(result.get('removed', False), result.get('cleanup'))
+ self.assertFalse(Path(result['checkout']).exists())
+ self.assertFalse(Path(result['temporary_directory']['path']).exists())
+ run = self.store.record_path(result['id']).parent
+ self.assertEqual((run / 'evidence/temporary/diagnostic.log').read_text(), 'private runtime evidence')
+
+ def test_targetless_format_check_is_reclaimed(self):
+ self.assert_targetless_cleanup(['cargo', 'fmt', '--check'], 0)
+
+ def test_targetless_package_selection_failure_is_reclaimed(self):
+ self.assert_targetless_cleanup(['cargo', 'test', '-p', 'no-such-package'], 101)
+
+ def test_inherited_wasi_target_is_recorded_and_compiler_output_is_reclaimed(self):
+ with patch.dict(os.environ, CARGO_BUILD_TARGET='wasm32-wasip1'):
+ result = self.run_fixture(profile='wasm32-wasip1/debug', command=['cargo', 'check'])
+ self.assertEqual(result.get('cargo_build_target'), 'wasm32-wasip1')
+ self.assertTrue(result.get('removed', False))
+ run = self.store.record_path(result['id']).parent
+ self.assertFalse((run / 'artifacts/wasm32-wasip1/debug').exists())
+ self.assertTrue((run / 'artifacts/e2e/result.json').is_file())
+
+ def test_report_and_cleanup_finish_before_releasing_the_original_compiler_lease(self):
+ reporter = m.remote.reporting()
+ original = reporter.write_report
+ probes = []
+ def write_report(*args, **kwargs):
+ probe = subprocess.run([m.sys.executable, '-c',
+ 'import fcntl,sys\nf=open(sys.argv[1],"a")\n'
+ 'try: fcntl.flock(f,fcntl.LOCK_EX|fcntl.LOCK_NB)\n'
+ 'except BlockingIOError: sys.exit(11)\n', str(self.store.global_lock)])
+ probes.append(probe.returncode)
+ return original(*args, **kwargs)
+ with patch.object(m.remote, 'reporting', return_value=reporter), \
+ patch.object(reporter, 'write_report', side_effect=write_report):
+ result = self.run_fixture()
+ self.assertTrue(result['removed'])
+ self.assertEqual(probes, [11])
+
+ def test_precommand_cache_block_is_cleaned_without_fabricating_execution(self):
+ result = self.run_fixture(cache_error=ValueError('fixture cache admission failed'))
+ self.assertIsNone(result['process_exit'])
+ self.assertNotIn('command_started_at', result)
+ self.assertEqual(result['scenario'], 'red')
+ self.assertFalse(Path(result['checkout']).exists())
+ self.assertEqual(result['evidence_export']['state'], 'saved')
+
+ def test_export_failure_retains_resources_and_the_actual_test_failure(self):
+ with patch.object(m.tempfile, 'NamedTemporaryFile', side_effect=OSError('fixture export failed')):
+ result = self.run_fixture(exit_code=101)
+ self.assertEqual((result['scenario'], result['process_exit']), ('red', 101))
+ self.assertTrue(Path(result['checkout']).exists())
+ self.assertTrue(Path(result['temporary_directory']['path']).exists())
+ self.assertFalse(result['evidence_saved'])
+ self.assertEqual(result['cleanup']['state'], 'retained')
+ self.assertIn('fixture export failed', result['cleanup']['reason'])
+
+ def test_link_profile_marker_retains_the_checkout_instead_of_losing_unique_evidence(self):
+ def prepare(checkout, *_args):
+ evidence = checkout / 'target/unique-evidence'
+ evidence.mkdir(parents=True)
+ (evidence / 'diagnostic.json').write_text('unique runtime evidence')
+ (evidence / '.cargo-lock').symlink_to(self.repo / 'source')
+ result = self.run_fixture(command=['cargo', 'check'], prepare=prepare)
+ self.assertEqual((result['scenario'], result['process_exit']), ('green', 0))
+ self.assertEqual(result['cleanup']['state'], 'retained')
+ self.assertEqual(result['evidence_export']['state'], 'failed')
+ self.assertIn('link or special file', result['evidence_export']['error'])
+ self.assertTrue(Path(result['temporary_directory']['path']).is_dir())
+ self.assertEqual((Path(result['checkout']) / 'target/unique-evidence/diagnostic.json').read_text(),
+ 'unique runtime evidence')
+ self.assertEqual((self.repo / 'source').read_text(), 'preserve source')
+
+ def test_release_only_target_is_reclaimed_without_copying_compiler_outputs(self):
+ result = self.run_fixture(profile='release', command=['cargo', 'build', '--release'])
+ self.assertEqual(result['process_exit'], 0)
+ self.assertTrue(result.get('removed', False))
+ self.assertFalse(Path(result['checkout']).exists())
+ artifacts = self.store.record_path(result['id']).parent / 'artifacts'
+ self.assertFalse((artifacts / 'release').exists())
+ self.assertTrue((artifacts / 'e2e/result.json').is_file())
+
def test_short_canonical_temporary_directory(self):
info = m.private_temp()
self.addCleanup(m.remote.cleanup_temp_directory, info)
@@ -715,6 +848,7 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
self.assertEqual(json.loads((evidence / 'cache-admission.json').read_text())['state'], 'blocked')
self.assertTrue((evidence / 'cache-probe.log').stat().st_size)
self.assertFalse(obsolete.exists())
+ self.assertFalse(Path(blocked['checkout']).exists())
# Explicit owner recovery is only for this test's isolated, idle endpoint.
subprocess.run(['sccache', '--stop-server'], env=server_env, check=True, capture_output=True)
(self.repo / 'src/lib.rs').write_text('#[test] fn intended_failure() { panic!("fixture"); }\n')
@@ -726,9 +860,17 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
self.addCleanup(m.remote.cleanup_temp_directory, failed['temporary_directory'])
self.assertEqual(failed['process_exit'], 101)
self.assertEqual(failed['scenario'], 'red')
- self.assertTrue(Path(failed['checkout']).exists())
- self.assertTrue(Path(failed['temporary_directory']['path']).exists())
+ self.assertFalse(Path(failed['checkout']).exists())
+ self.assertFalse(Path(failed['temporary_directory']['path']).exists())
report(failed)
+ args.keep = True
+ retained = m.run(self.store, args)
+ self.addCleanup(m.remote.cleanup_temp_directory, retained['temporary_directory'])
+ self.assertEqual(retained['process_exit'], 101)
+ self.assertEqual(retained['scenario'], 'red')
+ self.assertTrue(Path(retained['checkout']).is_dir())
+ self.assertTrue(Path(retained['temporary_directory']['path']).is_dir())
+ report(retained)
if __name__ == '__main__':
diff --git a/scripts/test-remote-rust.py b/scripts/test-remote-rust.py
index 2dd7d6ff..441679ca 100644
--- a/scripts/test-remote-rust.py
+++ b/scripts/test-remote-rust.py
@@ -65,6 +65,193 @@ class Contract(unittest.TestCase):
self.assertEqual(argv[argv.index('--') + 1:], row['command'])
self.assertEqual(argv.count(key), 1)
+ def test_terminal_workers_export_evidence_before_default_cleanup_and_honor_explicit_retention(self):
+ # Real worktrees, checkout leases and command processes; no Rust compilation.
+ for mode, retention in (('success', 'never'), ('failure', 'never'), ('blocked', 'never'),
+ ('timeout', 'never'), ('export-failure', 'never'),
+ ('unknown-untracked', 'never'), ('unknown-modified', 'never'),
+ ('unknown-ignored', 'never'), ('unknown-ignored-space', 'never'),
+ ('success', 'always'), ('failure', 'on-failure')):
+ with self.subTest(mode=mode, retention=retention), tempfile.TemporaryDirectory(dir='/tmp') as directory:
+ base = Path(directory).resolve()
+ source, root, home = base / 'source', base / 'workbench', base / 'home'
+ subprocess.run(['git', 'init', '-q', str(source)], check=True)
+ (source / '.gitignore').write_text('target/\n*.private\n')
+ (source / 'source.txt').write_text('tracked source')
+ git = ['git', '-C', str(source)]
+ subprocess.run([*git, 'add', '.'], check=True)
+ subprocess.run([*git, '-c', 'user.name=Fixture', '-c', '[email protected]',
+ '-c', 'commit.gpgsign=false', 'commit', '-qm', 'fixture'], check=True)
+ sha = subprocess.check_output([*git, 'rev-parse', 'HEAD'], text=True).strip()
+ ref = subprocess.check_output([*git, 'symbolic-ref', 'HEAD'], text=True).strip()
+ binary = home / '.cargo/bin'
+ binary.mkdir(parents=True)
+ compiler = '''import json,os,pathlib,sys,time
+if '--version' in sys.argv:
+ print('fixture compiler');sys.exit(0)
+p=pathlib.Path.cwd()/'target'
+if 'metadata' in sys.argv:
+ print(json.dumps({'target_directory':str(p)}));sys.exit(0)
+for profile in ('debug','release','x86_64-pc-windows-msvc/debug','smoke/builds'):
+ (p/profile).mkdir(parents=True,exist_ok=True)
+ (p/profile/'binary').write_bytes(b'rebuildable compiler fixture')
+(p/'e2e').mkdir()
+(p/'e2e/report.json').write_text('{"observed":"fixture"}')
+if MODE=='export-failure':
+ (p/'e2e/link').symlink_to(pathlib.Path.cwd()/'.gitignore')
+if MODE.startswith('unknown-'):
+ name={'unknown-untracked':'unknown-diagnostic.txt','unknown-modified':'source.txt','unknown-ignored':'unknown.private',
+ 'unknown-ignored-space':' target/unique.private'}[MODE]
+ (pathlib.Path.cwd()/name).parent.mkdir(parents=True,exist_ok=True)
+ (pathlib.Path.cwd()/name).write_text('unique material')
+(pathlib.Path(os.environ['TMPDIR'])/'diagnostic.log').write_text('private runtime evidence')
+time.sleep(30 if MODE=='timeout' else 0)
+bad=MODE in ('failure','export-failure') or MODE.startswith('unknown-')
+print('Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)')
+print('running 1 test')
+print('test component ... '+('FAILED' if bad else 'ok'))
+print('test result: '+('FAILED' if bad else 'ok')+'. '+str(int(not bad))+' passed; '+str(int(bad))+' failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s')
+sys.exit(101 if bad else 0)
+'''
+ for name in ('cargo', 'rustc'):
+ path = binary / name
+ path.write_text('#!' + sys.executable + '\nMODE=' + repr(mode) + '\n' + compiler)
+ path.chmod(0o700)
+ key = '20261024-050001-abcdef12'
+ run = root / 'runs' / key
+ run.mkdir(parents=True)
+ request = dict(id=key, ref=ref, sha=sha, executor_sha=sha, origin=str(source), retention=retention,
+ command=['cargo', 'test', '--test', 'publication_process', 'installed_'] if mode == 'blocked'
+ else ['cargo', 'test', '--lib'], timeout=1 if mode == 'timeout' else 10,
+ exclusive=False, ssh_host='fixture-host')
+ runner.save(run / 'request.json', request)
+ with patch.object(runner.Path, 'home', return_value=home), patch.dict(os.environ, {}, clear=False), \
+ patch.object(runner.sys, 'platform', 'linux'), \
+ patch.object(runner.tempfile, 'gettempdir', return_value=str(base)), \
+ patch.object(runner, 'MIN_FREE_BYTES', 0), \
+ patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \
+ patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}):
+ runner.worker(run / 'request.json')
+ result = json.loads((run / 'result.json').read_text())
+ self.assertEqual(result['status'], {'success': 'completed', 'failure': 'failed',
+ 'timeout': 'failed', 'blocked': 'blocked', 'export-failure': 'failed',
+ 'unknown-untracked': 'failed', 'unknown-modified': 'failed', 'unknown-ignored': 'failed',
+ 'unknown-ignored-space': 'failed'}[mode], result.get('error'))
+ self.assertEqual(result['process_exit'], {'success': 0, 'failure': 101,
+ 'timeout': -9, 'blocked': None, 'export-failure': 101,
+ 'unknown-untracked': 101, 'unknown-modified': 101, 'unknown-ignored': 101,
+ 'unknown-ignored-space': 101}[mode])
+ if mode.startswith('unknown-'):
+ name = {'unknown-untracked': 'unknown-diagnostic.txt', 'unknown-modified': 'source.txt',
+ 'unknown-ignored': 'unknown.private', 'unknown-ignored-space': ' target/unique.private'}[mode]
+ self.assertTrue(Path(result['checkout']).is_dir(), result.get('cleanup'))
+ self.assertEqual((Path(result['checkout']) / name).read_text(), 'unique material')
+ self.assertEqual(result['cleanup']['checkout']['state'], 'retained')
+ self.assertIn('retained', result['cleanup']['checkout']['error'])
+ self.assertTrue(Path(result['temporary_directory']['path']).is_dir())
+ self.assertEqual(result['cleanup']['validation_ref']['state'], 'retained')
+ continue
+ if mode == 'export-failure':
+ self.assertEqual(result['evidence_export']['state'], 'failed')
+ self.assertIn('link', result['evidence_export']['error'])
+ self.assertTrue(Path(result['checkout']).is_dir())
+ self.assertTrue(Path(result['temporary_directory']['path']).is_dir())
+ self.assertTrue((source / '.gitignore').is_file())
+ (Path(result['checkout']) / 'target/e2e/link').unlink()
+ with patch.object(runner, 'worker_alive', return_value=False):
+ cleanup = runner.cleanup_terminal_run(root, key)
+ self.assertEqual(cleanup['checkout']['state'], 'removed')
+ self.assertFalse(Path(result['temporary_directory']['path']).exists())
+ saved = json.loads((run / 'result.json').read_text())
+ self.assertEqual((saved['status'], saved['process_exit']), ('failed', 101))
+ self.assertEqual(saved['evidence_export']['state'], 'saved')
+ self.assertEqual((run / 'artifacts/e2e/report.json').read_text(), '{"observed":"fixture"}')
+ continue
+ if mode != 'blocked':
+ self.assertTrue((run / 'artifacts/e2e/report.json').is_file())
+ self.assertEqual(result['evidence_export']['state'], 'saved')
+ self.assertEqual(Path(result['checkout']).exists(), retention != 'never')
+ self.assertTrue((run / 'validation-report.json').is_file())
+ self.assertTrue(source.is_dir())
+ if mode == 'blocked':
+ self.assertNotIn('command_started_at', result)
+ recovery = result['console_assets']['preparation'][-1]
+ self.assertIn('--retention never', recovery)
+ self.assertNotIn('--reuse-checkout', recovery)
+ else:
+ self.assertEqual(Path(result['temporary_directory']['path']).exists(), retention != 'never')
+ self.assertEqual((run / 'artifacts/e2e/report.json').read_text(), '{"observed":"fixture"}')
+ self.assertEqual((run / 'evidence/temporary/diagnostic.log').read_text(), 'private runtime evidence')
+ for path in ('debug', 'release', 'x86_64-pc-windows-msvc/debug', 'smoke/builds/binary'):
+ self.assertFalse((run / 'artifacts' / path).exists())
+ if retention == 'never':
+ self.assertEqual(result['cleanup']['checkout']['state'], 'removed')
+ self.assertNotIn(key, subprocess.check_output(
+ ['git', '--git-dir', str(root / 'repository.git'), 'worktree', 'list', '--porcelain'], text=True))
+
+ def test_reusing_a_retained_checkout_with_another_command_does_not_export_previous_profiles(self):
+ # Two production worker calls and real Git/leases; compiler outputs are fixtures.
+ with tempfile.TemporaryDirectory(dir='/tmp') as directory:
+ base = Path(directory).resolve()
+ source, root, home = base / 'source', base / 'state', base / 'home'
+ subprocess.run(['git', 'init', '-q', str(source)], check=True)
+ (source / '.gitignore').write_text('target/\n')
+ git = ['git', '-C', str(source)]
+ subprocess.run([*git, 'add', '.'], check=True)
+ subprocess.run([*git, '-c', 'user.name=Fixture', '-c', '[email protected]',
+ '-c', 'commit.gpgsign=false', 'commit', '-qm', 'fixture'], check=True)
+ sha = subprocess.check_output([*git, 'rev-parse', 'HEAD'], text=True).strip()
+ ref = subprocess.check_output([*git, 'symbolic-ref', 'HEAD'], text=True).strip()
+ binary = home / '.cargo/bin'
+ binary.mkdir(parents=True)
+ compiler = '''import json,pathlib,sys
+if '--version' in sys.argv:
+ print('fixture compiler');sys.exit(0)
+p=pathlib.Path.cwd()/'target'
+if 'metadata' in sys.argv:
+ print(json.dumps({'target_directory':str(p)}));sys.exit(0)
+profile=sys.argv[sys.argv.index('--profile')+1] if '--profile' in sys.argv else 'debug'
+folder=p/profile
+folder.mkdir(parents=True,exist_ok=True)
+(folder/'.cargo-lock').touch()
+(folder/'binary').write_bytes(b'rebuildable custom compiler output')
+(p/'e2e/debug').mkdir(parents=True,exist_ok=True)
+(p/'e2e/debug/diagnostic.log').write_text('unique runtime evidence')
+'''
+ for name in ('cargo', 'rustc'):
+ path = binary / name
+ path.write_text('#!' + sys.executable + '\n' + compiler)
+ path.chmod(0o700)
+ first = '20261024-190001-abcdef01'
+ with patch.object(runner.Path, 'home', return_value=home), patch.dict(os.environ, {}, clear=False), \
+ patch.object(runner.sys, 'platform', 'linux'), \
+ patch.object(runner.tempfile, 'gettempdir', return_value=str(base)), \
+ patch.object(runner, 'MIN_FREE_BYTES', 0), \
+ patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \
+ patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}), \
+ patch.object(runner, 'worker_alive', return_value=False):
+ for index, command in enumerate((['cargo', 'check', '--profile', 'ci'], ['cargo', 'check'])):
+ key = first if index == 0 else '20261024-190002-abcdef02'
+ run = root / 'runs' / key
+ run.mkdir(parents=True)
+ request = dict(id=key, ref=ref, sha=sha, executor_sha=sha, origin=str(source),
+ retention='always' if index == 0 else 'never', command=command,
+ timeout=10, exclusive=False, ssh_host='fixture-host')
+ if index:
+ request['reuse_checkout'] = first
+ runner.save(run / 'request.json', request)
+ runner.worker(run / 'request.json')
+ result = json.loads((run / 'result.json').read_text())
+ self.assertEqual((result['status'], result['process_exit']), ('completed', 0), result.get('error'))
+ self.assertEqual(result['checkout_owner'], first)
+ self.assertFalse((run / 'artifacts/ci').exists())
+ self.assertFalse((run / 'artifacts/debug').exists())
+ self.assertEqual((run / 'artifacts/e2e/debug/diagnostic.log').read_text(), 'unique runtime evidence')
+ self.assertTrue((root / 'worktrees' / first / 'target/ci/binary').is_file())
+ if index:
+ self.assertEqual(result['cleanup']['checkout']['state'], 'retained')
+ self.assertFalse(Path(result['temporary_directory']['path']).exists())
+
def test_separate_workbench_submission_readback_and_maintenance_keep_the_selected_root(self):
with tempfile.TemporaryDirectory() as directory:
config = Path(directory) / 'remote-rust.json'
@@ -332,6 +519,7 @@ sys.exit(101 if bad else 0)
payload = json.loads(submit.call_args.kwargs['input'])
self.assertEqual(payload['command'], command)
self.assertTrue(payload['exclusive'])
+ self.assertEqual(payload['retention'], 'never')
def test_bootstrap_preserves_report_helper_for_immutable_worker(self):
with tempfile.TemporaryDirectory() as directory:
diff --git a/scripts/test-validation-report.py b/scripts/test-validation-report.py
index 74ece80c..e392dc62 100644
--- a/scripts/test-validation-report.py
+++ b/scripts/test-validation-report.py
@@ -4,6 +4,7 @@ import importlib.util
import contextlib
import io
import json
+import os
from pathlib import Path
import tempfile
import types
@@ -216,6 +217,139 @@ class Feedback(unittest.TestCase):
class Storage(unittest.TestCase):
+ def test_evidence_export_preserves_reports_and_runtime_without_copying_compiler_caches(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder)
+ checkout, run = root / 'checkout', root / 'run'
+ run.mkdir()
+ expected = {'target/e2e/result.json': 'actual failure',
+ 'target/smoke/run-a/private/diagnostic.log': 'private evidence',
+ 'target/e2e/debug/details.json': 'nested evidence, not a Cargo profile',
+ 'artifacts/screenshot.png': 'captured pixels',
+ 'target/smoke/builds/' + 'a' * 64 + '/hiroute-cli/receipt.json': '{"recipe":"selected"}'}
+ skipped = ('target/debug', 'target/release', 'target/custom',
+ 'target/x86_64-pc-windows-msvc/debug', 'target/wasm32-wasip1/debug',
+ 'target/validation-schedule', 'target/smoke/builds/' + 'a' * 64 + '/hiroute-cli/binary')
+ for name, value in expected.items():
+ path = checkout / name
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(value)
+ for name in skipped:
+ path = checkout / name / 'large-binary'
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text('rebuildable')
+ record = dict(checkout=str(checkout), sha='a' * 40,
+ command=['cargo', 'test', '--profile=custom', '--target=wasm32-wasip1'],
+ process_exit=101, scenario='red')
+ first = report.preserve_evidence(run, record)
+ self.assertEqual(first, report.preserve_evidence(run, record))
+ self.assertEqual((record['process_exit'], record['scenario']), (101, 'red'))
+ manifest = json.loads(Path(first['manifest']).read_text())
+ self.assertEqual(manifest['source_revision'], record['sha'])
+ self.assertEqual(set(manifest['files']), set(expected))
+ for name, value in expected.items():
+ self.assertEqual((run / manifest['files'][name]['path']).read_text(), value)
+ for name in skipped:
+ self.assertFalse((run / 'artifacts' / Path(name).relative_to('target')).exists())
+ (checkout / 'target/e2e/result.json').write_text('changed evidence')
+ with self.assertRaisesRegex(ValueError, 'Previously exported evidence differs'):
+ report.preserve_evidence(run, record)
+ self.assertEqual((run / 'artifacts/e2e/result.json').read_text(), 'actual failure')
+
+ def test_inherited_and_configured_cargo_targets_do_not_export_compiler_profiles(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder)
+ checkout, run = root / 'checkout', root / 'run'
+ run.mkdir()
+ for target in ('wasm32-wasip1', 'private_json_target'):
+ profile = checkout / 'target' / target / 'debug'
+ profile.mkdir(parents=True)
+ (profile / 'binary').write_bytes(b'rebuildable compiler output')
+ if target == 'private_json_target':
+ (profile / '.cargo-lock').touch()
+ nested = checkout / 'target/e2e/debug/details.json'
+ nested.parent.mkdir(parents=True)
+ nested.write_text('unique runtime evidence')
+ record = dict(checkout=str(checkout), command=['cargo', 'check'],
+ cargo_build_target='wasm32-wasip1')
+ report.preserve_evidence(run, record)
+ manifest = json.loads((run / 'evidence.json').read_text())
+ self.assertEqual(set(manifest['files']), {'target/e2e/debug/details.json'})
+ self.assertEqual((run / 'artifacts/e2e/debug/details.json').read_text(), 'unique runtime evidence')
+ self.assertFalse((run / 'artifacts/wasm32-wasip1/debug').exists())
+ self.assertFalse((run / 'artifacts/private_json_target/debug').exists())
+
+ def test_evidence_export_refuses_links_and_missing_reports_before_any_resource_deletion(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder)
+ checkout, run = root / 'checkout', root / 'run'
+ (checkout / 'target/e2e').mkdir(parents=True)
+ run.mkdir()
+ outside = root / 'outside'
+ outside.write_text('borrowed bytes')
+ link = checkout / 'target/e2e/link'
+ link.symlink_to(outside)
+ record = dict(checkout=str(checkout), command=['cargo', 'test'])
+ with self.assertRaisesRegex(ValueError, 'link or special file'):
+ report.preserve_evidence(run, record)
+ self.assertEqual(outside.read_text(), 'borrowed bytes')
+ self.assertTrue(link.is_symlink())
+ link.unlink()
+ record['validation_report'] = dict(error='fixture report write failure')
+ with self.assertRaisesRegex(ValueError, 'report was not saved'):
+ report.preserve_evidence(run, record)
+ self.assertTrue(checkout.is_dir())
+ record.pop('validation_report')
+ (run / 'evidence.json').symlink_to(outside)
+ with self.assertRaisesRegex(ValueError, 'manifest is a link'):
+ report.preserve_evidence(run, record)
+ self.assertEqual(outside.read_text(), 'borrowed bytes')
+
+ def test_previous_custom_profiles_stay_disposable_when_the_command_changes(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder)
+ checkout = root / 'checkout'
+ for profile in ('ci', 'private_json_target/ci'):
+ directory = checkout / 'target' / profile
+ directory.mkdir(parents=True)
+ (directory / '.cargo-lock').touch()
+ (directory / 'binary').write_bytes(b'rebuildable custom compiler output')
+ evidence = checkout / 'target/e2e/debug/details.json'
+ evidence.parent.mkdir(parents=True)
+ evidence.write_text('unique runtime evidence')
+ for index, command in enumerate((['cargo', 'check', '--profile', 'ci'], ['cargo', 'check'])):
+ run = root / str(index)
+ run.mkdir()
+ report.preserve_evidence(run, dict(checkout=str(checkout), command=command))
+ manifest = json.loads((run / 'evidence.json').read_text())
+ self.assertEqual(set(manifest['files']), {'target/e2e/debug/details.json'})
+ self.assertFalse((run / 'artifacts/ci').exists())
+ self.assertFalse((run / 'artifacts/private_json_target/ci').exists())
+
+ def test_link_and_special_profile_markers_cannot_hide_unique_evidence(self):
+ for kind in ('link', 'dangling-link', *(['fifo'] if hasattr(os, 'mkfifo') else [])):
+ with self.subTest(kind=kind), tempfile.TemporaryDirectory() as folder:
+ root = Path(folder)
+ checkout, run = root / 'checkout', root / 'run'
+ evidence = checkout / 'target/unique-evidence'
+ evidence.mkdir(parents=True)
+ (evidence / 'diagnostic.json').write_text('unique runtime evidence')
+ outside = root / 'outside'
+ if kind == 'link':
+ outside.write_text('external bytes')
+ marker = evidence / '.cargo-lock'
+ if kind == 'fifo':
+ os.mkfifo(marker)
+ else:
+ marker.symlink_to(outside)
+ run.mkdir()
+ with self.assertRaisesRegex(ValueError, 'link or special file'):
+ report.preserve_evidence(run, dict(checkout=str(checkout), command=['cargo', 'check']))
+ self.assertEqual((evidence / 'diagnostic.json').read_text(), 'unique runtime evidence')
+ self.assertFalse((run / 'evidence.json').exists())
+ if kind == 'link':
+ self.assertEqual(outside.read_text(), 'external bytes')
+
def test_report_survives_checkout_removal_and_never_changes_verdict(self):
with tempfile.TemporaryDirectory() as folder:
root = Path(folder)
diff --git a/scripts/test-validation-schedule.py b/scripts/test-validation-schedule.py
index 05099029..42d70c49 100644
--- a/scripts/test-validation-schedule.py
+++ b/scripts/test-validation-schedule.py
@@ -662,6 +662,30 @@ class Preparation(unittest.TestCase):
return scheduler.dag_products('build preparation=' + str(report) + '\n',
self.root, self.sha, 'fixture')
+ def test_cleanup_evidence_keeps_selected_recipes_without_scheduler_or_smoke_binaries(self):
+ preparation = self.prepare(32)
+ environment = self.execute(preparation)
+ data = json.loads(preparation.read_text())
+ receipts = [item['receipt'] for item in data['artifacts']] + [data['gateway']['receipt']]
+ run = self.root / 'run'
+ run.mkdir()
+ (run / 'dag-targets.json').write_text('{"complete":true}')
+ reporting.preserve_evidence(run, dict(checkout=str(self.root), sha=self.sha, command=COMMAND))
+ manifest = json.loads((run / 'evidence.json').read_text())
+ for binding in receipts:
+ source = Path(binding['path'])
+ key = source.relative_to(self.root).as_posix()
+ self.assertIn(key, manifest['files'])
+ saved = run / manifest['files'][key]['path']
+ self.assertEqual(saved.read_bytes(), source.read_bytes())
+ self.assertEqual('sha256:' + scheduler.file_digest(saved), binding['sha256'])
+ self.assertIn('recipe', json.loads(saved.read_text()))
+ self.assertTrue((run / 'artifacts/smoke/prepare-32/preparation.json').is_file())
+ self.assertFalse((run / 'artifacts/validation-schedule').exists())
+ self.assertTrue((Path(environment['HIROUTE_VALIDATION_PRODUCT_BIN_DIR']) / 'hiroute').is_file())
+ self.assertFalse(any((run / 'artifacts/smoke/builds').glob('*/*/hiroute*')))
+ self.assertEqual((run / 'dag-targets.json').read_text(), '{"complete":true}')
+
def test_focused_then_dag_selects_exact_receipts_without_removing_other_recipes(self):
focused = self.prepare(8)
full = self.prepare(32)
diff --git a/scripts/validation-report.py b/scripts/validation-report.py
index 1bb57c70..81109b84 100644
--- a/scripts/validation-report.py
+++ b/scripts/validation-report.py
@@ -2,10 +2,14 @@
"""Derived validation diagnostics; never an attestation or an automatic test waiver."""
import argparse
from collections import Counter
+import hashlib
import json
+import os
from pathlib import Path
import re
+import stat
import sys
+import tempfile
VERSION = 1
SAFE_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}\Z")
@@ -15,6 +19,136 @@ CASE = re.compile(r"^test (\S+) \.\.\. (ok|FAILED|ignored)(?:\s.*)?$")
ANSI = re.compile(r"\x1b\[[0-9;]*m")
+def preserve_evidence(run, record):
+ """Export private run evidence before deleting a disposable build or TMPDIR.
+
+ Compiler profiles and executable snapshots are reproducible. Other target
+ artifacts and surviving test scratch may contain unique failure evidence.
+ Unknown links/special files or failed copies retain the original resource.
+ """
+ if record.get("validation_report", {}).get("error"):
+ raise ValueError("Validation report was not saved")
+ run = Path(run)
+ destination = run / "evidence"
+ if destination.is_symlink():
+ raise ValueError("Evidence destination is a link")
+ destination.mkdir(exist_ok=True, mode=0o700)
+ profiles = {"debug", "release", "doc", "package"}
+ targets = set()
+ if record.get("cargo_build_target"):
+ targets.add(Path(record["cargo_build_target"]).stem)
+ args = record.get("command", [])
+ args = args[:args.index("--")] if "--" in args else args
+ for index, arg in enumerate(args):
+ if arg == "--profile" and index + 1 < len(args):
+ profiles.add({"dev": "debug", "test": "debug", "bench": "release"}.get(args[index + 1], args[index + 1]))
+ elif arg.startswith("--profile="):
+ value = arg.split("=", 1)[1]
+ profiles.add({"dev": "debug", "test": "debug", "bench": "release"}.get(value, value))
+ elif arg == "--target" and index + 1 < len(args):
+ targets.add(Path(args[index + 1]).stem)
+ elif arg.startswith("--target="):
+ targets.add(Path(arg.split("=", 1)[1]).stem)
+ files, excluded = {}, []
+
+ def cargo_profile_marker(directory):
+ try:
+ return stat.S_ISREG((directory / ".cargo-lock").lstat().st_mode)
+ except FileNotFoundError:
+ return False
+
+ def copy(source, relative, target_tree=False):
+ info = source.lstat()
+ parts = relative.parts
+ # Cargo profiles can also be under target/<target-triple>/PROFILE.
+ if target_tree and stat.S_ISDIR(info.st_mode):
+ profile = len(parts) in (2, 3) and (cargo_profile_marker(source)
+ or source.name in profiles and (len(parts) == 2 or parts[1] in targets
+ or re.fullmatch(r"\w+(?:-[\w.]+){2,}|wasm\d+-[\w.-]+", parts[1])))
+ if profile or parts == ("target", "validation-schedule"):
+ excluded.append(relative.as_posix())
+ return
+ if stat.S_ISLNK(info.st_mode) or not (stat.S_ISREG(info.st_mode) or stat.S_ISDIR(info.st_mode)):
+ raise ValueError("Evidence contains a link or special file: " + relative.as_posix())
+ # Smoke preparation binds these small receipts by path/hash. Keep the
+ # complete recipe, but not the adjacent executable snapshots.
+ if (target_tree and parts[:3] == ("target", "smoke", "builds") and len(parts) > 3
+ and not (len(parts) in (4, 5) and stat.S_ISDIR(info.st_mode)
+ or len(parts) == 6 and parts[-1] == "receipt.json")):
+ excluded.append(relative.as_posix())
+ return
+ # Keep local/Pilot's existing target-artifact consumer paths. Runtime
+ # and checkout artifacts have a separate namespace to prevent collisions.
+ output = run / "artifacts" / Path(*parts[1:]) if parts[0] == "target" else destination / relative
+ if output.is_symlink():
+ raise ValueError("Evidence destination contains a link")
+ if stat.S_ISDIR(info.st_mode):
+ output.mkdir(exist_ok=True, mode=0o700)
+ for child in sorted(source.iterdir()):
+ copy(child, relative / child.name, target_tree)
+ return
+ digest = hashlib.sha256()
+ with source.open("rb") as incoming, tempfile.NamedTemporaryFile(dir=output.parent, delete=False) as outgoing:
+ temporary = Path(outgoing.name)
+ try:
+ while True:
+ data = incoming.read(1024 * 1024)
+ if not data:
+ break
+ digest.update(data)
+ outgoing.write(data)
+ outgoing.flush()
+ after = source.lstat()
+ if (info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns) != (
+ after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns):
+ raise ValueError("Evidence changed during export: " + relative.as_posix())
+ if output.exists():
+ with output.open("rb") as saved:
+ existing = hashlib.sha256()
+ for data in iter(lambda: saved.read(1024 * 1024), b""):
+ existing.update(data)
+ if existing.digest() != digest.digest():
+ raise ValueError("Previously exported evidence differs: " + relative.as_posix())
+ else:
+ os.replace(temporary, output)
+ finally:
+ temporary.unlink(missing_ok=True)
+ files[relative.as_posix()] = dict(path=output.relative_to(run).as_posix(),
+ bytes=info.st_size, sha256=digest.hexdigest())
+
+ checkout = Path(record["checkout"]) if record.get("checkout") else None
+ if checkout is not None and checkout.exists():
+ if checkout.is_symlink():
+ raise ValueError("Evidence checkout is a link")
+ for name in ("target", "artifacts"):
+ source = checkout / name
+ if os.path.lexists(source):
+ copy(source, Path(name), name == "target")
+ scratch = record.get("temporary_directory")
+ if scratch and os.path.lexists(scratch["path"]):
+ source = Path(scratch["path"])
+ info = source.lstat()
+ if (source.is_symlink() or not source.is_dir() or info.st_uid != os.geteuid()
+ or info.st_mode & 0o077 or not source.name.startswith("hr-")
+ or source.parent.resolve() != Path(scratch["system_directory"]).resolve()):
+ raise ValueError("Temporary evidence directory identity is unsafe")
+ copy(source, Path("temporary"))
+ manifest = run / "evidence.json"
+ value = dict(source_revision=record.get("sha"), files=files, excluded=excluded)
+ if manifest.is_symlink():
+ raise ValueError("Evidence manifest is a link")
+ with tempfile.NamedTemporaryFile(dir=run, mode="w", delete=False) as output:
+ temporary = Path(output.name)
+ try:
+ output.write(json.dumps(value, indent=2) + "\n")
+ output.flush()
+ os.replace(temporary, manifest)
+ finally:
+ temporary.unlink(missing_ok=True)
+ return dict(state="saved", manifest=str(manifest), files=len(files),
+ bytes=sum(value["bytes"] for value in files.values()))
+
+
def add_arguments(parser):
parser.add_argument("--plan", help="This feature's validation group (not a global latest-run lookup)")
parser.add_argument("--phase", choices=("diagnostic", "focused", "final"), default="diagnostic")