Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.
Rust ai-agentsai-gatewayclaude-codecodexcoding-agentcost-optimizationdeepseekllm-gatewayllm-routerlocal-firstmodel-routingrusttask-routingtauri
fix(validation): recover continued checkouts after setup blocks (#214)
澄潭feae7f3f75f7
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.mdindex 4b71009a..f6928bf9 100644--- a/docs/code-map/testing.md+++ b/docs/code-map/testing.md@@ -10,6 +10,9 @@ The [console artifact owner](../../scripts/console-assets.py) checks selected SSinstalled-package prerequisites before Rust execution and verifies the same-candidatefrontend from an execution-host staging directory under the runner's checkout lease.Standard file tools supply that directory; the runner does not transport frontend bytes.+The [SSH checkout owner](../../scripts/remote-rust.py) shares retained-state admission+with recovery-command selection; [continuation regressions](../../scripts/test-console-assets.py)+cover repeated source advances, preparation blocks and execution of the generated retry.Windows development execution is owned by [windows-dev.py](../../scripts/windows-dev.py)and its [native executor](../../scripts/windows-dev-host.py), reached through`validation.py windows`. HOME configuration owns machine/session/client locations;diff --git a/docs/validation-routing.md b/docs/validation-routing.mdindex c6fa5a1f..8ba25148 100644--- a/docs/validation-routing.md+++ b/docs/validation-routing.md@@ -489,13 +489,18 @@ acceptance remains a separate obligation.### Reuse one retained backend checkout for product smokeFor repeated validation on the same pushed SHA, first run with `--retention always`.-After it completes, submit another command with `--reuse-checkout FIRST_RUN_ID`, the+After it stops, submit another command with `--reuse-checkout FIRST_RUN_ID`, thesame `--sha` and branch, and the same build-slot class. The workbench checks the ownerrun, SHA, origin, committed executor, clean worktree and Git registration before usingits default `target/`. A checkout lease serializes reuse and prevents owner cleanupwhile a borrower runs. Borrower cleanup keeps the owner checkout; after borrowersfinish, `backend cleanup FIRST_RUN_ID` releases it.+Same-candidate reuse remains available after a non-timeout test failure, including+when no `--plan` was supplied or the original `on-failure` owner stopped at a+recoverable preparation block. Each retry creates a new result; the failed result+keeps its original verdict.+```shpython3 scripts/validation.py backend run --ref refs/heads/codex/my-change \--retention always --plan my-change --phase focused -- \@@ -522,6 +527,10 @@ and the same plan/branch. The source may advance to a descendant committed SHA;the runner retains the checkout's own target while validating current source andrecording a new result. Continue only after the previous worker stops. Stale runIDs, dirty source, changed executor implementation and a different plan are rejected.+The exact latest run may also be a recoverable pre-command console-assets or cache+preflight block. Prepare the missing input or repair the cache using its diagnostic,+then execute the generated recovery command from the same committed driver. Recovery+creates a new run; the blocked result and all earlier source revisions remain intact.Cleanup belongs to the first checkout owner, after all commands finish. Keep eachrun's original SHA when citing evidence; cached artifacts are not scenario evidence.@@ -621,10 +630,15 @@ a backend frontend toolchain. Do not copy files directly into a managed checkoutThe supplied staging directory remains caller-owned and may be removed after therun finishes. A valid retained dist is rechecked on subsequent installed runswithout supplying `--frontend-dist` again.-An early console-blocked checkout retained by `always` or `on-failure` can be reused-at the same SHA/driver; its original blocked result remains unchanged. `never`-removes the checkout, so omit reuse for the next run. Usual completed-run reuse and-cleanup ownership still apply. The checkout owner records this setup association;+The result's `recovery.command` uses the existing `remote-rust.py` SSH entry and+preserves the selected host/workbench, candidate, timeout, schedule, plan, phase,+executed related runs and Cargo arguments. For an integration checkout retained by+`always`, it continues the latest blocked run, including after multiple source+advances; the original owner remains responsible for cleanup. An early blocked owner+retained by `on-failure` uses exact-candidate reuse. Both paths recheck stopped-worker,+current-state, clean-source and driver identity under the checkout lease before+checking the supplied assets again. `never` removes a fresh checkout, so its command+starts a new one. Old results remain unchanged. The checkout state records resource use;`--related-run` accepts executed reports, so do not pass a setup-blocked run there.Do not replace production assets with a dummy pageor weaken package assertions. Hosted CI retains its existing frontend artifactdiff --git a/issue-spec/specs/product-smoke-verification/spec.md b/issue-spec/specs/product-smoke-verification/spec.mdindex 85b295a6..833a93d2 100644--- a/issue-spec/specs/product-smoke-verification/spec.md+++ b/issue-spec/specs/product-smoke-verification/spec.md@@ -68,6 +68,33 @@ that exclude the installed scenarios MUST NOT require a frontend build. Real insmanagement and Quickstart assertions remain required after preparation; artifact verification alone does notestablish their outcomes.+#### Scenario: Resume a retained checkout after a recoverable preparation block++When a retained integration checkout has advanced through one or more candidate+revisions, a missing or mismatched console input, or failed cache admission before+the selected command starts, MUST leave its exact latest stopped run recoverable.+The fixed checkout owner controls retention and cleanup; current resource state+and the latest run control recovery. A prior owner's SHA or verdict MUST NOT stand+in for the current candidate. Recovery instructions and admission MUST use the same+explicit preparation-state policy and preserve the selected host/workbench,+candidate, command, timeout, schedule, plan, phase and executed related-run links.++After preparation, the generated command MUST create a new run in the same retained+checkout and recheck its lease, stopped worker, latest-state identity, clean source,+origin, plan, driver/helpers, descendant revision and required artifact provenance.+Unknown or executed blockers, stale borrowers and invalid resource state MUST NOT+gain preparation-recovery admission. All earlier results retain their source and+verdict; an unexecuted blocked run MUST remain ineligible as related execution+evidence. The selected tests MUST actually execute after recovery; valid artifacts+alone do not establish acceptance. Fresh blocked `on-failure` owners retain their+exact-candidate reuse path; `never` retention requires a new checkout.++After an executed, non-timeout failure, an otherwise eligible retained checkout+MUST remain reusable for the exact same candidate, even without a plan or when+its original `on-failure` owner had a recoverable preparation block. Reuse MUST+recheck the latest resource state and preserve earlier results; an unchanged+owner result alone MUST NOT establish admission.+### Requirement: Evidence binds actual revision, artifact, role, and pathReport MUST include checked source revision, build/executable identity, platform,diff --git a/scripts/remote-rust.py b/scripts/remote-rust.pyindex 2f1b91f7..756b0c79 100644--- a/scripts/remote-rust.py+++ b/scripts/remote-rust.py@@ -26,6 +26,7 @@ MAX_TMPDIR_BYTES = 40 # Leave at least 66 bytes for nested Linux Unix socket naMIN_FREE_BYTES = 30 * 1024**3RETENTION_POLICIES = ("on-failure", "always", "never")TERMINAL_STATUSES = ("completed", "failed", "blocked")+RECOVERABLE_SETUP_BLOCKERS = ("console_assets", "cache_server_unhealthy")WORKBENCH_SLOTS = 8SHARED_BUILD_JOBS = 8EXCLUSIVE_BUILD_JOBS = WORKBENCH_SLOTS * SHARED_BUILD_JOBS@@ -310,71 +311,129 @@ def checkout_lease(root, run_id):return handle-def verify_reused_checkout(root, request):- owner_id = request["reuse_checkout"]- validate_run_id(owner_id)- previous_path = root / "runs" / owner_id / "result.json"- if not previous_path.is_file():- raise RunBlocked("checkout_missing", "Reusable checkout owner run is missing")- previous = json.loads(previous_path.read_text())- checkout = root / "worktrees" / owner_id- console_blocked = (previous.get("status") == "blocked" and previous.get("blocker") == "console_assets"- and previous.get("process_exit") is None and not previous.get("command_started_at"))- ready = previous.get("status") == "completed" or console_blocked- if (previous.get("id") != owner_id or not ready- or (previous.get("retention") != "always"- and not (console_blocked and previous.get("retention") == "on-failure"))- or previous.get("sha") != request["sha"]- or previous.get("origin") != request["origin"]- or previous.get("executor_sha") != request["executor_sha"]- or previous.get("checkout") != str(checkout)- or previous.get("cleanup", {}).get("checkout", {}).get("state") in ("removed", "absent")- or not checkout.is_dir() or checkout.is_symlink()- or checkout.resolve(strict=True) not in worktree_paths(root / "repository.git")- or command(["git", "-C", str(checkout), "rev-parse", "HEAD"]) != request["sha"]- or command(["git", "-C", str(checkout), "status", "--porcelain", "--untracked-files=all"])):- raise RunBlocked("checkout_changed", "Reusable checkout or build inputs differ from its owner run")- return checkout+def recoverable_setup(result):+ return (result.get("status") == "blocked"+ and result.get("blocker") in RECOVERABLE_SETUP_BLOCKERS+ and result.get("process_exit") is None+ and result.get("command_started_at") is None+ and result.get("command_finished_at") is None+ and not result.get("timed_out"))-def continuation_owner(root, request):- previous_id = request["continue_checkout"]- validate_run_id(previous_id)- previous = json.loads((root / "runs" / previous_id / "result.json").read_text())- owner = previous.get("checkout_owner", previous_id)+def checkout_recovery_modes(first, latest):+ """Resource admission and command generation share the same stopped-state policy.++ Liveness, leases and current source are checked separately at acquisition.+ A recoverable setup result remains ineligible as executed test evidence.+ """+ if latest.get("timed_out"):+ return ()+ setup = recoverable_setup(latest)+ modes = []+ if (first.get("retention") == "always" and latest.get("plan")+ and latest.get("executor_digest")+ and (latest.get("status") in ("completed", "failed") or setup)):+ modes.append("continue")+ if ((first.get("retention") == "always"+ or (first.get("retention") == "on-failure" and recoverable_setup(first)))+ and (latest.get("status") in ("completed", "failed") or setup)):+ modes.append("reuse")+ return tuple(modes)+++def checkout_run(root, run_id):+ validate_run_id(run_id)+ path = root / "runs" / run_id / "result.json"+ if not path.is_file():+ raise RunBlocked("checkout_missing", "Retained checkout run is missing")+ result = json.loads(path.read_text())+ if result.get("id") != run_id:+ raise RunBlocked("checkout_changed", "Retained checkout run identity differs")+ return result+++def checkout_owner(root, run_id):+ previous = checkout_run(root, run_id)+ owner = previous.get("checkout_owner", run_id)validate_run_id(owner)return owner-def continue_checkout(root, request, owner):- """Called with repository and checkout leases held; never rewrites old evidence."""- previous_id = request["continue_checkout"]- previous = json.loads((root / "runs" / previous_id / "result.json").read_text())- first = json.loads((root / "runs" / owner / "result.json").read_text())- state_path = root / ("checkout-" + owner + ".json")- state = json.loads(state_path.read_text())+def retained_checkout(root, request, owner, mode):+ """Resolve mutable checkout state independently of its immutable owner result."""+ selected = request[mode + "_checkout"]+ first = checkout_run(root, owner)+ state = json.loads((root / ("checkout-" + owner + ".json")).read_text())+ previous = checkout_run(root, state["run_id"])checkout = root / "worktrees" / owner- if (previous.get("status") not in ("completed", "failed")- or previous.get("timed_out") or worker_alive(previous.get("worker_pid"))- or first.get("retention") != "always"- or not request.get("plan") or previous.get("plan") != request["plan"]- or any(previous.get(key) != request.get(key)- for key in ("origin", "ref", "executor_digest", "schedule_digest"))- or not request.get("executor_digest")- or state.get("run_id") != previous_id or state.get("sha") != previous.get("sha")- or previous.get("checkout") != str(checkout)+ keys = ("origin", "ref", "plan", "executor_digest", "schedule_digest", "report_digest", "console_digest")+ if mode == "reuse":+ keys += ("executor_sha",)+ if (mode not in checkout_recovery_modes(first, previous)+ or (selected != previous["id"] and (mode == "continue" or selected != owner))+ or first.get("checkout_owner", owner) != owner+ or previous.get("checkout_owner", previous["id"]) != owner+ or worker_alive(previous.get("worker_pid"))+ or state.get("sha") != previous.get("sha")+ or (mode == "reuse" and previous.get("sha") != request["sha"])+ or any(previous.get(key) != request.get(key) for key in keys)+ or any(row.get("checkout") != str(checkout)+ or row.get("cleanup", {}).get("checkout", {}).get("state") in ("removed", "absent")+ for row in (first, previous))or not checkout.is_dir() or checkout.is_symlink()or checkout.resolve(strict=True) not in worktree_paths(root / "repository.git")or command(["git", "-C", str(checkout), "rev-parse", "HEAD"]) != previous["sha"]or command(["git", "-C", str(checkout), "status", "--porcelain", "--untracked-files=all"])):- raise RunBlocked("continuation_changed", "Continuation requires the latest stopped run, same plan/driver, and clean retained source")+ raise RunBlocked("continuation_changed" if mode == "continue" else "checkout_changed",+ "Recovery requires the current stopped checkout, same plan/driver, and clean source")+ return checkout, previous+++def verify_reused_checkout(root, request):+ owner = checkout_owner(root, request["reuse_checkout"])+ return retained_checkout(root, request, owner, "reuse")[0]+++def continuation_owner(root, request):+ return checkout_owner(root, request["continue_checkout"])+++def continue_checkout(root, request, owner):+ """Called with repository and checkout leases held; never rewrites old evidence."""+ checkout, previous = retained_checkout(root, request, owner, "continue")+ state_path = root / ("checkout-" + owner + ".json")command(["git", "-C", str(checkout), "merge-base", "--is-ancestor", previous["sha"], request["sha"]])# Record intent first: interrupted transitions cannot be mistaken for the previous run.- save(state_path, dict(run_id=request["id"], sha=request["sha"], previous_run=previous_id))+ save(state_path, dict(run_id=request["id"], sha=request["sha"], previous_run=previous["id"]))command(["git", "-C", str(checkout), "checkout", "--detach", request["sha"]])return checkout+def checkout_recovery_command(root, result, frontend_dist=None):+ """Use the existing SSH entry so recovery retains the selected host/workbench."""+ owner = result["checkout_owner"]+ first = result if owner == result["id"] else checkout_run(root, owner)+ modes = checkout_recovery_modes(first, result)+ rerun = ["python3", "scripts/remote-rust.py", "--host", result["ssh_host"],+ "--workbench", result.get("workbench", ROOT), "run", "--ref", result["ref"],+ "--sha", result["sha"], "--retention", "always", "--timeout", str(result["timeout"])]+ if frontend_dist:+ rerun += ["--frontend-dist", str(frontend_dist)]+ if modes:+ mode = modes[0]+ rerun += ["--" + mode + "-checkout", result["id"] if mode == "continue" else owner]+ if result.get("schedule"):+ rerun += ["--schedule", result["schedule"]]+ if result.get("exclusive"):+ rerun += ["--exclusive"]+ if result.get("plan"):+ rerun += ["--plan", result["plan"]]+ rerun += ["--phase", result.get("phase", "focused")]+ for related in result.get("related_runs", []):+ rerun += ["--related-run", related]+ return shlex.join(rerun + ["--", *result["command"]])++def cleanup_temp_directory(info):"""Remove only a runner-created private TMPDIR; return diagnostics instead of raising."""if not isinstance(info, dict) or not info.get("path"):@@ -614,8 +673,8 @@ def worker(request_path):if shutil.disk_usage(root).free < MIN_FREE_BYTES:raise RunBlocked("insufficient_disk", "Less than 30 GiB free; validation did not start")repo = root / "repository.git"- owner_id = (continuation_owner(root, request) if request.get("continue_checkout")- else request.get("reuse_checkout"))+ selected_checkout = request.get("continue_checkout") or request.get("reuse_checkout")+ owner_id = checkout_owner(root, selected_checkout) if selected_checkout else Nonereused_lease = checkout_lease(root, owner_id or request["id"])checkout = root / "worktrees" / (owner_id or request["id"])checkout.parent.mkdir(exist_ok=True)@@ -650,26 +709,13 @@ def worker(request_path):result["console_assets"] = dict(state="blocked", reason=str(error))staging = Path.home() / ".cache/hiroute/console-assets" / request["id"]host = request["ssh_host"]- rerun = ["python3", "scripts/validation.py", "--frontend-dist", str(staging / "dist"),- "backend", "run", "--ref", request["ref"], "--sha", request["sha"],- "--retention", "always"]- if owner_id or request["retention"] != "never":- rerun += ["--reuse-checkout", owner_id or request["id"]]- if request.get("schedule"):- rerun += ["--schedule", request["schedule"]]- if request.get("exclusive"):- rerun += ["--exclusive"]- if request.get("plan"):- rerun += ["--plan", request["plan"], "--phase", request.get("phase", "focused")]- rerun += ["--", *request["command"]]result["console_assets"]["preparation"] = ["Use clean candidate " + request["sha"] + " on the configured frontend target:","python3 scripts/validation.py frontend ci","python3 scripts/validation.py frontend build --console-assets","From the machine with that dist, copy it to a separate backend staging directory:",shlex.join(["ssh", host, shlex.join(["mkdir", "-p", str(staging)])]),- shlex.join(["scp", "-r", "apps/desktop/dist", host + ":" + str(staging) + "/"]),- "Rerun from the caller; --frontend-dist is a path on the backend host:", shlex.join(rerun)]+ shlex.join(["scp", "-r", "apps/desktop/dist", host + ":" + str(staging) + "/"])]raise RunBlocked("console_assets", str(error) +"; prepare with validation.py frontend build --console-assets, then pass ""--frontend-dist to SSH backend run; see result.console_assets.preparation") from error@@ -740,6 +786,15 @@ def worker(request_path):result["status"] = "failed" if result["process_exit"] != 0 or bad_tests else "completed"except RunBlocked as error:result.update(status="blocked", blocker=error.blocker, error=str(error))+ if recoverable_setup(result) and result.get("checkout_owner"):+ frontend = (Path.home() / ".cache/hiroute/console-assets" / request["id"] / "dist"+ if error.blocker == "console_assets" else request.get("frontend_dist"))+ result["recovery"] = dict(command=checkout_recovery_command(root, result, frontend),+ after_worker_stops=True)+ if error.blocker == "console_assets":+ result["console_assets"]["preparation"] += [+ "After this worker stops, rerun from the same committed driver; "+ "--frontend-dist is a path on the backend host:", result["recovery"]["command"]]except Exception as error:result.update(status="failed", error=str(error))finally:@@ -792,7 +847,8 @@ def require_dag_report(result):BOOTSTRAP = '''import hashlib,json,os,pathlib,re,subprocess,sys,uuidos.umask(0o077)p=json.load(sys.stdin)-relative=p.pop("workbench",".local/share/hiroute-rust")+relative=p.get("workbench",".local/share/hiroute-rust")+p["workbench"]=relativeparts=relative.split("/")if not re.fullmatch(r"[A-Za-z0-9_.-]+(?:/[A-Za-z0-9_.-]+)*",relative) or any(part in (".","..") for part in parts): raise RuntimeError("invalid workbench path")r=pathlib.Path.home()/relative@@ -908,9 +964,9 @@ def main():run.add_argument("--retention", choices=RETENTION_POLICIES, default="on-failure",help="Checkout/TMPDIR retention: on-failure (default), always, or never")run.add_argument("--reuse-checkout", metavar="RUN_ID",- help="Reuse a completed --retention always run's exact checkout and target under an exclusive lease")+ help="Reuse a retained owner's current exact checkout and target under an exclusive lease")run.add_argument("--continue-checkout", metavar="PREVIOUS_RUN_ID",- help="Advance a retained integration checkout to a descendant SHA on the same plan")+ help="Continue the latest stopped or recoverable setup run to a descendant SHA on the same plan")run.add_argument("--schedule", choices=("workspace-smoke", "workspace-dag"),help="Prepare backend builds, then run smoke and remaining tests in separate shared slots")reporting().add_arguments(run)diff --git a/scripts/test-console-assets.py b/scripts/test-console-assets.pyindex 4d7e3aca..fc6e7e5c 100644--- a/scripts/test-console-assets.py+++ b/scripts/test-console-assets.py@@ -1,6 +1,7 @@#!/usr/bin/env python3"""Console selection, provenance and real managed-checkout admission regressions."""import importlib.util+import hashlibimport jsonimport osfrom pathlib import Path@@ -94,6 +95,64 @@ class Artifact(unittest.TestCase):def git(self, *args):return subprocess.check_output(['git', '-C', str(self.repo), *args], text=True).strip()+ def execute_worker(self, root, row, source=None, cache_error=None):+ home = self.root / 'home'+ binary = home / '.cargo/bin'+ binary.mkdir(parents=True, exist_ok=True)+ fake = '''import sys,json,pathlib+if '--version' in sys.argv:+ print(pathlib.Path(sys.argv[0]).name+' fixture');sys.exit(0)+if 'metadata' in sys.argv:+ print(json.dumps({'target_directory':str(pathlib.Path.cwd()/'target')}));sys.exit(0)+bad=(pathlib.Path(sys.argv[0]).resolve().parents[2]/'fail-command').exists()+print('Running tests/publication_process.rs (target/debug/deps/publication_process-abcdef0123456789)')+print('running 1 test')+print('test component_fixture ... '+('FAILED' if bad else 'ok'))+print('test result: '+('FAILED' if bad else 'ok')+'. '+str(int(not bad))+' passed; '+str(int(bad))+' failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s')+sys.exit(101 if bad else 0)+'''+ for name in ('cargo', 'rustc'):+ path = binary / name+ path.write_text('#!' + sys.executable + '\n' + fake)+ path.chmod(0o700)+ run = root / 'runs' / row['id']+ run.mkdir(parents=True)+ if source is not None:+ row = dict(row, frontend_dist=str(source))+ runner.save(run / 'request.json', row)+ with patch.object(runner.Path, 'home', return_value=home), patch.dict(os.environ, {}, clear=False), \+ patch.object(runner.sys, 'platform', 'linux'), \+ patch.object(runner.tempfile, 'gettempdir', return_value=str(self.root)), \+ patch.object(runner, 'worker_alive', return_value=False), \+ patch.object(runner, 'MIN_FREE_BYTES', 0), \+ patch.object(runner, 'admit_cache_server', side_effect=cache_error, return_value={'state': 'ready'}), \+ patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}) as stats:+ runner.worker(run / 'request.json')+ result = json.loads((run / 'result.json').read_text())+ if result.get('temporary_directory'):+ self.addCleanup(runner.cleanup_temp_directory, result['temporary_directory'])+ return result, stats++ def generated_request(self, command, run_id, executor_sha=None):+ argv = shlex.split(command)+ self.assertEqual(argv[:2], ['python3', 'scripts/remote-rust.py'])+ revision = executor_sha or self.git('rev-parse', 'HEAD')+ def committed(arguments):+ if arguments == ['git', 'rev-parse', 'HEAD']:+ return revision+ if arguments == ['git', 'remote', 'get-url', 'origin']:+ return str(self.repo)+ if arguments[:2] == ['git', 'show']:+ return (Path(__file__).parent / arguments[2].split(':scripts/', 1)[1]).read_text().rstrip('\n')+ self.fail('Unexpected submission read: ' + repr(arguments))+ with patch.object(runner, 'command', side_effect=committed), \+ patch.object(runner.sys, 'argv', argv[1:]), \+ patch.object(runner.subprocess, 'run', return_value=subprocess.CompletedProcess([], 0)) as ssh:+ runner.main()+ request = json.loads(ssh.call_args.kwargs['input'])+ request['id'] = run_id+ return request+def test_build_stamps_only_successful_unchanged_source(self):with patch.object(assets.subprocess, 'call', return_value=0) as npm:self.assertEqual(assets.build(self.repo, {}), 0)@@ -165,38 +224,8 @@ class Artifact(unittest.TestCase):'installed_', '--', '--nocapture'],retention='on-failure', exclusive=False, timeout=10, plan='console-assets', phase='focused',ssh_host='fixture-backend')- home = self.root / 'home'- binary = home / '.cargo/bin'- binary.mkdir(parents=True)- fake = '''import sys,json,pathlib-if '--version' in sys.argv:- print(pathlib.Path(sys.argv[0]).name+' fixture');sys.exit(0)-if 'metadata' in sys.argv:- print(json.dumps({'target_directory':str(pathlib.Path.cwd()/'target')}));sys.exit(0)-print('Running tests/publication_process.rs (target/debug/deps/publication_process-abcdef0123456789)')-print('running 1 test')-print('test component_fixture ... ok')-print('test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s')-'''- for name in ('cargo', 'rustc'):- path = binary / name- path.write_text('#!' + sys.executable + '\n' + fake)- path.chmod(0o700)-def execute(row, source=None):- run = root / 'runs' / row['id']- run.mkdir(parents=True)- if source is not None:- row = dict(row, frontend_dist=str(source))- runner.save(run / 'request.json', row)- with patch.object(runner.Path, 'home', return_value=home), patch.dict(os.environ, {}, clear=False), \- patch.object(runner.sys, 'platform', 'linux'), \- patch.object(runner.tempfile, 'gettempdir', return_value=str(self.root)), \- patch.object(runner, 'MIN_FREE_BYTES', 0), \- patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \- patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}) as stats:- runner.worker(run / 'request.json')- return json.loads((run / 'result.json').read_text()), stats+ return self.execute_worker(root, row, source)before, stats = execute(request)self.assertEqual((before['status'], before['blocker'], before['process_exit']),@@ -205,10 +234,11 @@ print('test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered oustats.assert_not_called()self.assertFalse((root / 'runs' / first / 'command.log').exists())self.assertIn('--reuse-checkout ' + first, before['console_assets']['preparation'][-1])- staging = home / '.cache/hiroute/console-assets' / first+ staging = self.root / 'home/.cache/hiroute/console-assets' / firstpreparation = before['console_assets']['preparation']self.assertIn(shlex.join(['scp', '-r', 'apps/desktop/dist', 'fixture-backend:' + str(staging) + '/']), preparation)- self.assertEqual(shlex.split(preparation[-1])[2:4], ['--frontend-dist', str(staging / 'dist')])+ arguments = shlex.split(preparation[-1])+ self.assertEqual(arguments[arguments.index('--frontend-dist') + 1], str(staging / 'dist'))row = dict(request, id=second, reuse_checkout=first, retention='always')lease = runner.checkout_lease(root, first)try:@@ -239,6 +269,218 @@ print('test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered ouself.assertEqual(json.loads((root / 'runs' / first / 'result.json').read_text())['status'], 'blocked')runner.cleanup_temp_directory(after['temporary_directory'])+ def test_multiple_advances_block_then_execute_the_generated_recovery_in_the_same_checkout(self):+ # Real Git/leases/worker/parser and command processes; compiler output is a fixture.+ for kind in ('missing', 'stale'):+ with self.subTest(kind=kind):+ root = self.root / ('workbench-' + kind)+ owner, middle, latest, blocked, retry, recovered = [+ '20261024-0' + str(int(kind == 'stale')) + '000' + str(i) + '-abcdef12'+ for i in range(1, 7)]+ sha = self.git('rev-parse', 'HEAD')+ common = dict(ref=self.git('symbolic-ref', 'HEAD'), sha=sha, executor_sha=sha,+ origin=str(self.repo), retention='always',+ exclusive=False, timeout=37, plan='console-recovery', phase='focused',+ ssh_host='fixture-backend', workbench='.local/share/fixture-workbench', related_runs=[])+ for key, name in (('executor', 'remote-rust'), ('schedule', 'validation-schedule'),+ ('report', 'validation-report'), ('console', 'console-assets')):+ contents = (Path(__file__).parent / (name + '.py')).read_text().rstrip('\n') + '\n'+ common[key + '_digest'] = hashlib.sha256(contents.encode()).hexdigest()+ home = self.root / 'home'+ home.mkdir(exist_ok=True)+ (home / 'fail-command').touch()+ # The stale case leaves an admitted older receipt through the runner itself.+ (self.dist / assets.RECEIPT).write_text(json.dumps(assets.manifest(sha, assets.files(self.dist))))+ first, _ = self.execute_worker(root, dict(common, id=owner,+ command=['cargo', 'test', '--test', 'publication_process', 'installed_'] if kind == 'stale'+ else ['cargo', 'test', '--lib']), self.dist if kind == 'stale' else None)+ (home / 'fail-command').unlink()+ self.assertEqual(first['status'], 'failed', first)+ checkout = Path(first['checkout'])+ (checkout / 'target').mkdir(exist_ok=True)+ sentinel = checkout / 'target/retained-compiler-objects'+ sentinel.write_bytes(b'retained compilation')+ previous = owner+ history = [owner]+ for index, key in enumerate((middle, latest), 1):+ (self.repo / 'advance').write_text(kind + str(index))+ self.git('add', 'advance')+ self.git('-c', 'user.name=Fixture', '-c', '[email protected]',+ '-c', 'commit.gpgsign=false', 'commit', '-qm', 'advance fixture')+ common['sha'] = self.git('rev-parse', 'HEAD')+ result, _ = self.execute_worker(root, dict(common, id=key, continue_checkout=previous,+ command=['cargo', 'test', '--lib']))+ self.assertEqual(result['status'], 'completed', result)+ previous = key+ history.append(key)+ command = ['cargo', 'test', '-p', 'hiroute-daemon', '--test', 'publication_process', 'installed_']+ blocked_request = dict(common, id=blocked, continue_checkout=latest,+ phase='final', related_runs=[middle, latest], command=command)+ before, stats = self.execute_worker(root, blocked_request)+ self.assertEqual((before['status'], before['blocker'], before['process_exit']),+ ('blocked', 'console_assets', None), before)+ self.assertNotIn('command_started_at', before)+ stats.assert_not_called()+ history.append(blocked)+ snapshots = {key: (root / 'runs' / key / 'result.json').read_bytes() for key in history}+ generated = before['console_assets']['preparation'][-1]+ self.assertIn('--continue-checkout ' + blocked, generated)+ self.assertEqual(generated, before['recovery']['command'])+ self.assertNotIn('--reuse-checkout', generated)+ # Resource safety remains independent of the recoverable result class.+ recovery = dict(blocked_request, id=retry, continue_checkout=blocked)+ with runner.checkout_lease(root, owner), patch.object(runner, 'worker_alive', return_value=False):+ for key in ('plan', 'origin', 'ref', 'executor_digest', 'schedule_digest', 'report_digest', 'console_digest'):+ with self.subTest(changed=key), self.assertRaises(runner.RunBlocked):+ runner.continue_checkout(root, dict(recovery, **{key: 'changed'}), owner)+ for key in (owner, middle, latest):+ with self.subTest(stale=key), self.assertRaises(runner.RunBlocked):+ runner.continue_checkout(root, dict(recovery, continue_checkout=key), owner)+ with self.assertRaises(runner.RunBlocked):+ runner.verify_reused_checkout(root, dict(recovery, reuse_checkout=middle))+ with patch.object(runner, 'worker_alive', return_value=True), self.assertRaises(runner.RunBlocked):+ runner.continue_checkout(root, recovery, owner)+ (checkout / 'dirty-source').touch()+ with self.assertRaises(runner.RunBlocked):+ runner.continue_checkout(root, recovery, owner)+ (checkout / 'dirty-source').unlink()+ state_path = root / ('checkout-' + owner + '.json')+ saved_state = state_path.read_bytes()+ runner.save(state_path, dict(run_id=middle, sha=common['sha']))+ with self.assertRaises(runner.RunBlocked):+ runner.continue_checkout(root, recovery, owner)+ state_path.write_bytes(saved_state)+ with self.assertRaises(subprocess.CalledProcessError):+ runner.continue_checkout(root, dict(recovery, sha=first['sha']), owner)+ # Keep hashes/source checks active on repeated preparation failures.+ again, _ = self.execute_worker(root, recovery, self.dist)+ self.assertEqual(again.get('blocker'), 'console_assets', again)+ snapshots[retry] = (root / 'runs' / retry / 'result.json').read_bytes()+ (self.dist / assets.RECEIPT).write_text(json.dumps(assets.manifest(common['sha'], assets.files(self.dist))))+ argv = shlex.split(again['recovery']['command'])+ staging = Path(argv[argv.index('--frontend-dist') + 1])+ staging.parent.mkdir(parents=True, exist_ok=True)+ shutil.copytree(self.dist, staging)+ request = self.generated_request(again['recovery']['command'], recovered, common['executor_sha'])+ for key in ('sha', 'executor_sha', 'executor_digest', 'schedule_digest', 'report_digest',+ 'console_digest', 'timeout', 'plan', 'workbench', 'ssh_host'):+ self.assertEqual(request[key], common[key], key)+ self.assertEqual((request['phase'], request['related_runs'], request['command']),+ ('final', [middle, latest], command))+ after, _ = self.execute_worker(root, request)+ self.assertEqual(after['status'], 'completed', after)+ self.assertEqual(after['process_exit'], 0)+ self.assertIn('command_started_at', after)+ self.assertEqual(after['tests']['counts']['passed'], 1)+ self.assertEqual((after['checkout'], after['checkout_owner']), (str(checkout), owner))+ self.assertEqual(after['console_assets']['source_revision'], common['sha'])+ self.assertEqual(sentinel.read_bytes(), b'retained compilation')+ for key, snapshot in snapshots.items():+ self.assertEqual((root / 'runs' / key / 'result.json').read_bytes(), snapshot)+ metadata = dict(plan=common['plan'], related_runs=[blocked])+ with self.assertRaisesRegex(ValueError, 'has not executed'):+ runner.reporting().related_reports(root / 'runs', metadata)++ def test_successful_owner_without_plan_reuses_checkout_after_a_failed_borrower(self):+ root = self.root / 'workbench'+ owner, failed, retry = ['20261024-03000' + str(i) + '-abcdef12' for i in range(1, 4)]+ arguments = ['python3', 'scripts/remote-rust.py', '--host', 'fixture-backend',+ '--workbench', '.local/share/fixture-workbench', 'run',+ '--ref', self.git('symbolic-ref', 'HEAD'), '--sha', self.sha,+ '--retention', 'always']+ command = ['--', 'cargo', 'test', '--lib']+ initial = self.generated_request(shlex.join(arguments + command), owner)+ self.assertIsNone(initial.get('plan'))+ first, _ = self.execute_worker(root, initial)+ self.assertEqual((first['status'], first['process_exit']), ('completed', 0), first)+ checkout = Path(first['checkout'])+ (checkout / 'target').mkdir(exist_ok=True)+ sentinel = checkout / 'target/retained-compiler-objects'+ sentinel.write_bytes(b'retained compilation')+ reuse = shlex.join(arguments + ['--reuse-checkout', owner] + command)+ failure = self.root / 'home/fail-command'+ failure.touch()+ before, _ = self.execute_worker(root, self.generated_request(reuse, failed))+ failure.unlink()+ self.assertEqual((before['status'], before['process_exit']), ('failed', 101), before)+ self.assertFalse(before.get('timed_out'))+ self.assertIn('command_finished_at', before)+ snapshots = {key: (root / 'runs' / key / 'result.json').read_bytes() for key in (owner, failed)}+ after, _ = self.execute_worker(root, self.generated_request(reuse, retry))+ self.assertEqual((after['status'], after['process_exit']), ('completed', 0), after.get('error'))+ self.assertEqual(after['tests']['counts']['passed'], 1)+ self.assertEqual((after['checkout'], after['checkout_owner']), (str(checkout), owner))+ self.assertEqual((after['sha'], after['executor_digest']), (before['sha'], before['executor_digest']))+ self.assertEqual(sentinel.read_bytes(), b'retained compilation')+ self.assertEqual(json.loads((root / ('checkout-' + owner + '.json')).read_text())['run_id'], retry)+ for key, snapshot in snapshots.items():+ self.assertEqual((root / 'runs' / key / 'result.json').read_bytes(), snapshot)++ def test_on_failure_setup_owner_reuses_checkout_after_generated_recovery_test_fails(self):+ root = self.root / 'workbench'+ owner, failed, retry = ['20261024-04000' + str(i) + '-abcdef12' for i in range(1, 4)]+ initial = self.generated_request(shlex.join([+ 'python3', 'scripts/remote-rust.py', '--host', 'fixture-backend',+ '--workbench', '.local/share/fixture-workbench', 'run', '--ref', self.git('symbolic-ref', 'HEAD'),+ '--sha', self.sha, '--plan', 'console-retry', '--',+ 'cargo', 'test', '--test', 'publication_process', 'installed_']), owner)+ self.assertEqual(initial['retention'], 'on-failure')+ first, stats = self.execute_worker(root, initial)+ self.assertEqual((first['status'], first['blocker'], first['process_exit']),+ ('blocked', 'console_assets', None), first)+ stats.assert_not_called()+ checkout = Path(first['checkout'])+ (checkout / 'target').mkdir(exist_ok=True)+ sentinel = checkout / 'target/retained-compiler-objects'+ sentinel.write_bytes(b'retained compilation')+ recovery = first['recovery']['command']+ argv = shlex.split(recovery)+ self.assertEqual(argv[argv.index('--reuse-checkout') + 1], owner)+ staging = Path(argv[argv.index('--frontend-dist') + 1])+ staging.parent.mkdir(parents=True, exist_ok=True)+ shutil.copytree(self.dist, staging)+ borrower = self.generated_request(recovery, failed)+ self.assertEqual(borrower['retention'], 'always')+ failure = self.root / 'home/fail-command'+ failure.touch()+ before, _ = self.execute_worker(root, borrower)+ failure.unlink()+ self.assertEqual((before['status'], before['process_exit']), ('failed', 101), before)+ self.assertFalse(before.get('timed_out'))+ self.assertIn('command_finished_at', before)+ self.assertEqual(before['console_assets']['state'], 'verified')+ snapshots = {key: (root / 'runs' / key / 'result.json').read_bytes() for key in (owner, failed)}+ after, _ = self.execute_worker(root, self.generated_request(recovery, retry))+ self.assertEqual((after['status'], after['process_exit']), ('completed', 0), after.get('error'))+ self.assertEqual(after['tests']['counts']['passed'], 1)+ self.assertEqual((after['checkout'], after['checkout_owner']), (str(checkout), owner))+ self.assertEqual((after['sha'], after['executor_digest']), (before['sha'], before['executor_digest']))+ self.assertEqual(after['console_assets']['source_revision'], self.sha)+ self.assertEqual(sentinel.read_bytes(), b'retained compilation')+ self.assertEqual(json.loads((root / ('checkout-' + owner + '.json')).read_text())['run_id'], retry)+ for key, snapshot in snapshots.items():+ self.assertEqual((root / 'runs' / key / 'result.json').read_bytes(), snapshot)++ def test_cache_preflight_uses_the_same_generated_recovery_and_rechecks_setup(self):+ root = self.root / 'workbench'+ owner, retry = '20261024-020001-abcdef12', '20261024-020002-abcdef12'+ initial = self.generated_request(shlex.join([+ 'python3', 'scripts/remote-rust.py', '--host', 'fixture-backend',+ '--workbench', '.local/share/fixture-workbench', 'run', '--ref', self.git('symbolic-ref', 'HEAD'),+ '--sha', self.sha, '--retention', 'always', '--plan', 'cache-recovery', '--', 'cargo', 'test', '--lib']), owner)+ blocked, stats = self.execute_worker(root, initial,+ cache_error=runner.RunBlocked('cache_server_unhealthy', 'fixture cache admission failure'))+ self.assertEqual((blocked['status'], blocked['blocker'], blocked['process_exit']),+ ('blocked', 'cache_server_unhealthy', None))+ self.assertNotIn('command_started_at', blocked)+ stats.assert_not_called()+ self.assertIn('--continue-checkout ' + owner, blocked['recovery']['command'])+ old_result = (root / 'runs' / owner / 'result.json').read_bytes()+ after, _ = self.execute_worker(root, self.generated_request(blocked['recovery']['command'], retry))+ self.assertEqual(after['status'], 'completed', after)+ self.assertEqual(after['checkout'], blocked['checkout'])+ self.assertEqual((root / 'runs' / owner / 'result.json').read_bytes(), old_result)+if __name__ == '__main__':unittest.main()diff --git a/scripts/test-remote-rust.py b/scripts/test-remote-rust.pyindex 90540fde..2dd7d6ff 100644--- a/scripts/test-remote-rust.py+++ b/scripts/test-remote-rust.py@@ -23,6 +23,48 @@ spec.loader.exec_module(runner)class Contract(unittest.TestCase):+ def test_checkout_recovery_requires_a_known_unexecuted_setup_blocker(self):+ first = dict(retention='always')+ latest = dict(status='blocked', blocker='console_assets', process_exit=None,+ plan='recovery', executor_digest='driver')+ for blocker in runner.RECOVERABLE_SETUP_BLOCKERS:+ with self.subTest(blocker=blocker):+ row = dict(latest, blocker=blocker)+ self.assertEqual(runner.checkout_recovery_modes(first, row), ('continue', 'reuse'))+ for changes in (dict(blocker='checkout_busy'), dict(blocker='continuation_changed'),+ dict(blocker='unknown'), dict(process_exit=0), dict(command_started_at=0),+ dict(command_finished_at=1), dict(timed_out=True), dict(status='running')):+ with self.subTest(changes=changes):+ self.assertEqual(runner.checkout_recovery_modes(first, dict(latest, **changes)), ())+ self.assertEqual(runner.checkout_recovery_modes(dict(latest, retention='on-failure'), latest), ('reuse',))+ self.assertEqual(runner.checkout_recovery_modes(dict(retention='never'), latest), ())+ failed = dict(latest, status='failed', process_exit=101, command_started_at=1, command_finished_at=2)+ self.assertEqual(runner.checkout_recovery_modes(first, failed), ('continue', 'reuse'))+ self.assertEqual(runner.checkout_recovery_modes(first, dict(failed, plan=None)), ('reuse',))+ self.assertEqual(runner.checkout_recovery_modes(dict(latest, retention='on-failure'), failed), ('reuse',))+ self.assertEqual(runner.checkout_recovery_modes(first, dict(failed, timed_out=True)), ())+ self.assertEqual(runner.checkout_recovery_modes(dict(latest, retention='on-failure'),+ dict(failed, timed_out=True)), ())++ def test_recovery_command_keeps_execution_scope_and_does_not_link_the_blocked_run(self):+ key = '20261024-010203-abcdef12'+ row = dict(id=key, checkout_owner=key, retention='always', status='blocked', blocker='console_assets',+ process_exit=None, plan='recovery', executor_digest='driver', phase='final',+ ssh_host='fixture-host', workbench='.local/share/fixture-workbench',+ ref='refs/heads/fixture', sha='a' * 40, timeout=123, exclusive=True,+ schedule='workspace-dag', related_runs=['20261024-010201-abcdef12'],+ command=['cargo', 'test', '--locked', '--workspace', '--exclude', 'hiroute-desktop', '--all-features'])+ import shlex+ argv = shlex.split(runner.checkout_recovery_command(Path('/unused'), row, '/staging with spaces/dist'))+ for flag, value in (('--host', row['ssh_host']), ('--workbench', row['workbench']),+ ('--continue-checkout', key), ('--timeout', '123'), ('--schedule', 'workspace-dag'),+ ('--frontend-dist', '/staging with spaces/dist'), ('--phase', 'final')):+ self.assertEqual(argv[argv.index(flag) + 1], value)+ self.assertIn('--exclusive', argv)+ self.assertEqual(argv[argv.index('--related-run') + 1], row['related_runs'][0])+ self.assertEqual(argv[argv.index('--') + 1:], row['command'])+ self.assertEqual(argv.count(key), 1)+def test_separate_workbench_submission_readback_and_maintenance_keep_the_selected_root(self):with tempfile.TemporaryDirectory() as directory:config = Path(directory) / 'remote-rust.json'@@ -89,7 +131,9 @@ class Contract(unittest.TestCase):patch('os.umask'), patch('subprocess.Popen'):exec(runner.BOOTSTRAP, {})run = Path(directory) / payload['workbench'] / 'runs/run-1'- self.assertEqual(json.loads((run / 'request.json').read_text())['frontend_dist'], '~/remote-only/dist')+ saved = json.loads((run / 'request.json').read_text())+ self.assertEqual(saved['frontend_dist'], '~/remote-only/dist')+ self.assertEqual(saved['workbench'], payload['workbench'])self.assertFalse((run / 'console-assets.json').exists())self.assertFalse((Path(directory) / runner.ROOT / 'runs').exists())@@ -249,6 +293,7 @@ sys.exit(101 if bad else 0)patch.object(runner.tempfile, 'gettempdir', return_value=str(base.resolve())), \patch.object(runner, 'console_assets') as console, \patch.object(runner, 'MIN_FREE_BYTES', 0), \+ patch.object(runner, 'worker_alive', return_value=False), \patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}):console.return_value.preflight.return_value = dict(state='not_required', cases=[])@@ -582,6 +627,7 @@ runner.admit_cache_server(root,env,handles=[handle])runner.save(owner_run / 'result.json', dict(id=owner, status='completed',retention='always', sha=sha, origin=str(source), executor_sha=sha,checkout=str(checkout), worker_pid=None))+ runner.save(root / ('checkout-' + owner + '.json'), dict(run_id=owner, sha=sha))request = dict(reuse_checkout=owner, sha=sha, origin=str(source), executor_sha=sha)self.assertEqual(runner.verify_reused_checkout(root, request), checkout)borrowed = root / 'runs' / borrower