Skip to content

Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.

fix(validation): recover continued checkouts after setup blocks (#214)

澄潭feae7f3f75f7
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.md
index 4b71009a..f6928bf9 100644
--- a/docs/code-map/testing.md
+++ b/docs/code-map/testing.md
@@ -10,6 +10,9 @@ The [console artifact owner](../../scripts/console-assets.py) checks selected SS
installed-package prerequisites before Rust execution and verifies the same-candidate
frontend from an execution-host staging directory under the runner's checkout lease.
Standard file tools supply that directory; the runner does not transport frontend bytes.
+The [SSH checkout owner](../../scripts/remote-rust.py) shares retained-state admission
+with recovery-command selection; [continuation regressions](../../scripts/test-console-assets.py)
+cover repeated source advances, preparation blocks and execution of the generated retry.
Windows development execution is owned by [windows-dev.py](../../scripts/windows-dev.py)
and its [native executor](../../scripts/windows-dev-host.py), reached through
`validation.py windows`. HOME configuration owns machine/session/client locations;
diff --git a/docs/validation-routing.md b/docs/validation-routing.md
index c6fa5a1f..8ba25148 100644
--- a/docs/validation-routing.md
+++ b/docs/validation-routing.md
@@ -489,13 +489,18 @@ acceptance remains a separate obligation.
### Reuse one retained backend checkout for product smoke
For repeated validation on the same pushed SHA, first run with `--retention always`.
-After it completes, submit another command with `--reuse-checkout FIRST_RUN_ID`, the
+After it stops, submit another command with `--reuse-checkout FIRST_RUN_ID`, the
same `--sha` and branch, and the same build-slot class. The workbench checks the owner
run, SHA, origin, committed executor, clean worktree and Git registration before using
its default `target/`. A checkout lease serializes reuse and prevents owner cleanup
while a borrower runs. Borrower cleanup keeps the owner checkout; after borrowers
finish, `backend cleanup FIRST_RUN_ID` releases it.
+Same-candidate reuse remains available after a non-timeout test failure, including
+when no `--plan` was supplied or the original `on-failure` owner stopped at a
+recoverable preparation block. Each retry creates a new result; the failed result
+keeps its original verdict.
+
```sh
python3 scripts/validation.py backend run --ref refs/heads/codex/my-change \
--retention always --plan my-change --phase focused -- \
@@ -522,6 +527,10 @@ and the same plan/branch. The source may advance to a descendant committed SHA;
the runner retains the checkout's own target while validating current source and
recording a new result. Continue only after the previous worker stops. Stale run
IDs, dirty source, changed executor implementation and a different plan are rejected.
+The exact latest run may also be a recoverable pre-command console-assets or cache
+preflight block. Prepare the missing input or repair the cache using its diagnostic,
+then execute the generated recovery command from the same committed driver. Recovery
+creates a new run; the blocked result and all earlier source revisions remain intact.
Cleanup belongs to the first checkout owner, after all commands finish. Keep each
run's original SHA when citing evidence; cached artifacts are not scenario evidence.
@@ -621,10 +630,15 @@ a backend frontend toolchain. Do not copy files directly into a managed checkout
The supplied staging directory remains caller-owned and may be removed after the
run finishes. A valid retained dist is rechecked on subsequent installed runs
without supplying `--frontend-dist` again.
-An early console-blocked checkout retained by `always` or `on-failure` can be reused
-at the same SHA/driver; its original blocked result remains unchanged. `never`
-removes the checkout, so omit reuse for the next run. Usual completed-run reuse and
-cleanup ownership still apply. The checkout owner records this setup association;
+The result's `recovery.command` uses the existing `remote-rust.py` SSH entry and
+preserves the selected host/workbench, candidate, timeout, schedule, plan, phase,
+executed related runs and Cargo arguments. For an integration checkout retained by
+`always`, it continues the latest blocked run, including after multiple source
+advances; the original owner remains responsible for cleanup. An early blocked owner
+retained by `on-failure` uses exact-candidate reuse. Both paths recheck stopped-worker,
+current-state, clean-source and driver identity under the checkout lease before
+checking the supplied assets again. `never` removes a fresh checkout, so its command
+starts a new one. Old results remain unchanged. The checkout state records resource use;
`--related-run` accepts executed reports, so do not pass a setup-blocked run there.
Do not replace production assets with a dummy page
or weaken package assertions. Hosted CI retains its existing frontend artifact
diff --git a/issue-spec/specs/product-smoke-verification/spec.md b/issue-spec/specs/product-smoke-verification/spec.md
index 85b295a6..833a93d2 100644
--- a/issue-spec/specs/product-smoke-verification/spec.md
+++ b/issue-spec/specs/product-smoke-verification/spec.md
@@ -68,6 +68,33 @@ that exclude the installed scenarios MUST NOT require a frontend build. Real ins
management and Quickstart assertions remain required after preparation; artifact verification alone does not
establish their outcomes.
+#### Scenario: Resume a retained checkout after a recoverable preparation block
+
+When a retained integration checkout has advanced through one or more candidate
+revisions, a missing or mismatched console input, or failed cache admission before
+the selected command starts, MUST leave its exact latest stopped run recoverable.
+The fixed checkout owner controls retention and cleanup; current resource state
+and the latest run control recovery. A prior owner's SHA or verdict MUST NOT stand
+in for the current candidate. Recovery instructions and admission MUST use the same
+explicit preparation-state policy and preserve the selected host/workbench,
+candidate, command, timeout, schedule, plan, phase and executed related-run links.
+
+After preparation, the generated command MUST create a new run in the same retained
+checkout and recheck its lease, stopped worker, latest-state identity, clean source,
+origin, plan, driver/helpers, descendant revision and required artifact provenance.
+Unknown or executed blockers, stale borrowers and invalid resource state MUST NOT
+gain preparation-recovery admission. All earlier results retain their source and
+verdict; an unexecuted blocked run MUST remain ineligible as related execution
+evidence. The selected tests MUST actually execute after recovery; valid artifacts
+alone do not establish acceptance. Fresh blocked `on-failure` owners retain their
+exact-candidate reuse path; `never` retention requires a new checkout.
+
+After an executed, non-timeout failure, an otherwise eligible retained checkout
+MUST remain reusable for the exact same candidate, even without a plan or when
+its original `on-failure` owner had a recoverable preparation block. Reuse MUST
+recheck the latest resource state and preserve earlier results; an unchanged
+owner result alone MUST NOT establish admission.
+
### Requirement: Evidence binds actual revision, artifact, role, and path
Report MUST include checked source revision, build/executable identity, platform,
diff --git a/scripts/remote-rust.py b/scripts/remote-rust.py
index 2f1b91f7..756b0c79 100644
--- a/scripts/remote-rust.py
+++ b/scripts/remote-rust.py
@@ -26,6 +26,7 @@ MAX_TMPDIR_BYTES = 40 # Leave at least 66 bytes for nested Linux Unix socket na
MIN_FREE_BYTES = 30 * 1024**3
RETENTION_POLICIES = ("on-failure", "always", "never")
TERMINAL_STATUSES = ("completed", "failed", "blocked")
+RECOVERABLE_SETUP_BLOCKERS = ("console_assets", "cache_server_unhealthy")
WORKBENCH_SLOTS = 8
SHARED_BUILD_JOBS = 8
EXCLUSIVE_BUILD_JOBS = WORKBENCH_SLOTS * SHARED_BUILD_JOBS
@@ -310,71 +311,129 @@ def checkout_lease(root, run_id):
return handle
-def verify_reused_checkout(root, request):
- owner_id = request["reuse_checkout"]
- validate_run_id(owner_id)
- previous_path = root / "runs" / owner_id / "result.json"
- if not previous_path.is_file():
- raise RunBlocked("checkout_missing", "Reusable checkout owner run is missing")
- previous = json.loads(previous_path.read_text())
- checkout = root / "worktrees" / owner_id
- console_blocked = (previous.get("status") == "blocked" and previous.get("blocker") == "console_assets"
- and previous.get("process_exit") is None and not previous.get("command_started_at"))
- ready = previous.get("status") == "completed" or console_blocked
- if (previous.get("id") != owner_id or not ready
- or (previous.get("retention") != "always"
- and not (console_blocked and previous.get("retention") == "on-failure"))
- or previous.get("sha") != request["sha"]
- or previous.get("origin") != request["origin"]
- or previous.get("executor_sha") != request["executor_sha"]
- or previous.get("checkout") != str(checkout)
- or previous.get("cleanup", {}).get("checkout", {}).get("state") in ("removed", "absent")
- or not checkout.is_dir() or checkout.is_symlink()
- or checkout.resolve(strict=True) not in worktree_paths(root / "repository.git")
- or command(["git", "-C", str(checkout), "rev-parse", "HEAD"]) != request["sha"]
- or command(["git", "-C", str(checkout), "status", "--porcelain", "--untracked-files=all"])):
- raise RunBlocked("checkout_changed", "Reusable checkout or build inputs differ from its owner run")
- return checkout
+def recoverable_setup(result):
+ return (result.get("status") == "blocked"
+ and result.get("blocker") in RECOVERABLE_SETUP_BLOCKERS
+ and result.get("process_exit") is None
+ and result.get("command_started_at") is None
+ and result.get("command_finished_at") is None
+ and not result.get("timed_out"))
-def continuation_owner(root, request):
- previous_id = request["continue_checkout"]
- validate_run_id(previous_id)
- previous = json.loads((root / "runs" / previous_id / "result.json").read_text())
- owner = previous.get("checkout_owner", previous_id)
+def checkout_recovery_modes(first, latest):
+ """Resource admission and command generation share the same stopped-state policy.
+
+ Liveness, leases and current source are checked separately at acquisition.
+ A recoverable setup result remains ineligible as executed test evidence.
+ """
+ if latest.get("timed_out"):
+ return ()
+ setup = recoverable_setup(latest)
+ modes = []
+ if (first.get("retention") == "always" and latest.get("plan")
+ and latest.get("executor_digest")
+ and (latest.get("status") in ("completed", "failed") or setup)):
+ modes.append("continue")
+ if ((first.get("retention") == "always"
+ or (first.get("retention") == "on-failure" and recoverable_setup(first)))
+ and (latest.get("status") in ("completed", "failed") or setup)):
+ modes.append("reuse")
+ return tuple(modes)
+
+
+def checkout_run(root, run_id):
+ validate_run_id(run_id)
+ path = root / "runs" / run_id / "result.json"
+ if not path.is_file():
+ raise RunBlocked("checkout_missing", "Retained checkout run is missing")
+ result = json.loads(path.read_text())
+ if result.get("id") != run_id:
+ raise RunBlocked("checkout_changed", "Retained checkout run identity differs")
+ return result
+
+
+def checkout_owner(root, run_id):
+ previous = checkout_run(root, run_id)
+ owner = previous.get("checkout_owner", run_id)
validate_run_id(owner)
return owner
-def continue_checkout(root, request, owner):
- """Called with repository and checkout leases held; never rewrites old evidence."""
- previous_id = request["continue_checkout"]
- previous = json.loads((root / "runs" / previous_id / "result.json").read_text())
- first = json.loads((root / "runs" / owner / "result.json").read_text())
- state_path = root / ("checkout-" + owner + ".json")
- state = json.loads(state_path.read_text())
+def retained_checkout(root, request, owner, mode):
+ """Resolve mutable checkout state independently of its immutable owner result."""
+ selected = request[mode + "_checkout"]
+ first = checkout_run(root, owner)
+ state = json.loads((root / ("checkout-" + owner + ".json")).read_text())
+ previous = checkout_run(root, state["run_id"])
checkout = root / "worktrees" / owner
- if (previous.get("status") not in ("completed", "failed")
- or previous.get("timed_out") or worker_alive(previous.get("worker_pid"))
- or first.get("retention") != "always"
- or not request.get("plan") or previous.get("plan") != request["plan"]
- or any(previous.get(key) != request.get(key)
- for key in ("origin", "ref", "executor_digest", "schedule_digest"))
- or not request.get("executor_digest")
- or state.get("run_id") != previous_id or state.get("sha") != previous.get("sha")
- or previous.get("checkout") != str(checkout)
+ keys = ("origin", "ref", "plan", "executor_digest", "schedule_digest", "report_digest", "console_digest")
+ if mode == "reuse":
+ keys += ("executor_sha",)
+ if (mode not in checkout_recovery_modes(first, previous)
+ or (selected != previous["id"] and (mode == "continue" or selected != owner))
+ or first.get("checkout_owner", owner) != owner
+ or previous.get("checkout_owner", previous["id"]) != owner
+ or worker_alive(previous.get("worker_pid"))
+ or state.get("sha") != previous.get("sha")
+ or (mode == "reuse" and previous.get("sha") != request["sha"])
+ or any(previous.get(key) != request.get(key) for key in keys)
+ or any(row.get("checkout") != str(checkout)
+ or row.get("cleanup", {}).get("checkout", {}).get("state") in ("removed", "absent")
+ for row in (first, previous))
or not checkout.is_dir() or checkout.is_symlink()
or checkout.resolve(strict=True) not in worktree_paths(root / "repository.git")
or command(["git", "-C", str(checkout), "rev-parse", "HEAD"]) != previous["sha"]
or command(["git", "-C", str(checkout), "status", "--porcelain", "--untracked-files=all"])):
- raise RunBlocked("continuation_changed", "Continuation requires the latest stopped run, same plan/driver, and clean retained source")
+ raise RunBlocked("continuation_changed" if mode == "continue" else "checkout_changed",
+ "Recovery requires the current stopped checkout, same plan/driver, and clean source")
+ return checkout, previous
+
+
+def verify_reused_checkout(root, request):
+ owner = checkout_owner(root, request["reuse_checkout"])
+ return retained_checkout(root, request, owner, "reuse")[0]
+
+
+def continuation_owner(root, request):
+ return checkout_owner(root, request["continue_checkout"])
+
+
+def continue_checkout(root, request, owner):
+ """Called with repository and checkout leases held; never rewrites old evidence."""
+ checkout, previous = retained_checkout(root, request, owner, "continue")
+ state_path = root / ("checkout-" + owner + ".json")
command(["git", "-C", str(checkout), "merge-base", "--is-ancestor", previous["sha"], request["sha"]])
# Record intent first: interrupted transitions cannot be mistaken for the previous run.
- save(state_path, dict(run_id=request["id"], sha=request["sha"], previous_run=previous_id))
+ save(state_path, dict(run_id=request["id"], sha=request["sha"], previous_run=previous["id"]))
command(["git", "-C", str(checkout), "checkout", "--detach", request["sha"]])
return checkout
+def checkout_recovery_command(root, result, frontend_dist=None):
+ """Use the existing SSH entry so recovery retains the selected host/workbench."""
+ owner = result["checkout_owner"]
+ first = result if owner == result["id"] else checkout_run(root, owner)
+ modes = checkout_recovery_modes(first, result)
+ rerun = ["python3", "scripts/remote-rust.py", "--host", result["ssh_host"],
+ "--workbench", result.get("workbench", ROOT), "run", "--ref", result["ref"],
+ "--sha", result["sha"], "--retention", "always", "--timeout", str(result["timeout"])]
+ if frontend_dist:
+ rerun += ["--frontend-dist", str(frontend_dist)]
+ if modes:
+ mode = modes[0]
+ rerun += ["--" + mode + "-checkout", result["id"] if mode == "continue" else owner]
+ if result.get("schedule"):
+ rerun += ["--schedule", result["schedule"]]
+ if result.get("exclusive"):
+ rerun += ["--exclusive"]
+ if result.get("plan"):
+ rerun += ["--plan", result["plan"]]
+ rerun += ["--phase", result.get("phase", "focused")]
+ for related in result.get("related_runs", []):
+ rerun += ["--related-run", related]
+ return shlex.join(rerun + ["--", *result["command"]])
+
+
def cleanup_temp_directory(info):
"""Remove only a runner-created private TMPDIR; return diagnostics instead of raising."""
if not isinstance(info, dict) or not info.get("path"):
@@ -614,8 +673,8 @@ def worker(request_path):
if shutil.disk_usage(root).free < MIN_FREE_BYTES:
raise RunBlocked("insufficient_disk", "Less than 30 GiB free; validation did not start")
repo = root / "repository.git"
- owner_id = (continuation_owner(root, request) if request.get("continue_checkout")
- else request.get("reuse_checkout"))
+ selected_checkout = request.get("continue_checkout") or request.get("reuse_checkout")
+ owner_id = checkout_owner(root, selected_checkout) if selected_checkout else None
reused_lease = checkout_lease(root, owner_id or request["id"])
checkout = root / "worktrees" / (owner_id or request["id"])
checkout.parent.mkdir(exist_ok=True)
@@ -650,26 +709,13 @@ def worker(request_path):
result["console_assets"] = dict(state="blocked", reason=str(error))
staging = Path.home() / ".cache/hiroute/console-assets" / request["id"]
host = request["ssh_host"]
- rerun = ["python3", "scripts/validation.py", "--frontend-dist", str(staging / "dist"),
- "backend", "run", "--ref", request["ref"], "--sha", request["sha"],
- "--retention", "always"]
- if owner_id or request["retention"] != "never":
- rerun += ["--reuse-checkout", owner_id or request["id"]]
- if request.get("schedule"):
- rerun += ["--schedule", request["schedule"]]
- if request.get("exclusive"):
- rerun += ["--exclusive"]
- if request.get("plan"):
- rerun += ["--plan", request["plan"], "--phase", request.get("phase", "focused")]
- rerun += ["--", *request["command"]]
result["console_assets"]["preparation"] = [
"Use clean candidate " + request["sha"] + " on the configured frontend target:",
"python3 scripts/validation.py frontend ci",
"python3 scripts/validation.py frontend build --console-assets",
"From the machine with that dist, copy it to a separate backend staging directory:",
shlex.join(["ssh", host, shlex.join(["mkdir", "-p", str(staging)])]),
- shlex.join(["scp", "-r", "apps/desktop/dist", host + ":" + str(staging) + "/"]),
- "Rerun from the caller; --frontend-dist is a path on the backend host:", shlex.join(rerun)]
+ shlex.join(["scp", "-r", "apps/desktop/dist", host + ":" + str(staging) + "/"])]
raise RunBlocked("console_assets", str(error) +
"; prepare with validation.py frontend build --console-assets, then pass "
"--frontend-dist to SSH backend run; see result.console_assets.preparation") from error
@@ -740,6 +786,15 @@ def worker(request_path):
result["status"] = "failed" if result["process_exit"] != 0 or bad_tests else "completed"
except RunBlocked as error:
result.update(status="blocked", blocker=error.blocker, error=str(error))
+ if recoverable_setup(result) and result.get("checkout_owner"):
+ frontend = (Path.home() / ".cache/hiroute/console-assets" / request["id"] / "dist"
+ if error.blocker == "console_assets" else request.get("frontend_dist"))
+ result["recovery"] = dict(command=checkout_recovery_command(root, result, frontend),
+ after_worker_stops=True)
+ if error.blocker == "console_assets":
+ result["console_assets"]["preparation"] += [
+ "After this worker stops, rerun from the same committed driver; "
+ "--frontend-dist is a path on the backend host:", result["recovery"]["command"]]
except Exception as error:
result.update(status="failed", error=str(error))
finally:
@@ -792,7 +847,8 @@ def require_dag_report(result):
BOOTSTRAP = '''import hashlib,json,os,pathlib,re,subprocess,sys,uuid
os.umask(0o077)
p=json.load(sys.stdin)
-relative=p.pop("workbench",".local/share/hiroute-rust")
+relative=p.get("workbench",".local/share/hiroute-rust")
+p["workbench"]=relative
parts=relative.split("/")
if not re.fullmatch(r"[A-Za-z0-9_.-]+(?:/[A-Za-z0-9_.-]+)*",relative) or any(part in (".","..") for part in parts): raise RuntimeError("invalid workbench path")
r=pathlib.Path.home()/relative
@@ -908,9 +964,9 @@ def main():
run.add_argument("--retention", choices=RETENTION_POLICIES, default="on-failure",
help="Checkout/TMPDIR retention: on-failure (default), always, or never")
run.add_argument("--reuse-checkout", metavar="RUN_ID",
- help="Reuse a completed --retention always run's exact checkout and target under an exclusive lease")
+ help="Reuse a retained owner's current exact checkout and target under an exclusive lease")
run.add_argument("--continue-checkout", metavar="PREVIOUS_RUN_ID",
- help="Advance a retained integration checkout to a descendant SHA on the same plan")
+ help="Continue the latest stopped or recoverable setup run to a descendant SHA on the same plan")
run.add_argument("--schedule", choices=("workspace-smoke", "workspace-dag"),
help="Prepare backend builds, then run smoke and remaining tests in separate shared slots")
reporting().add_arguments(run)
diff --git a/scripts/test-console-assets.py b/scripts/test-console-assets.py
index 4d7e3aca..fc6e7e5c 100644
--- a/scripts/test-console-assets.py
+++ b/scripts/test-console-assets.py
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
"""Console selection, provenance and real managed-checkout admission regressions."""
import importlib.util
+import hashlib
import json
import os
from pathlib import Path
@@ -94,6 +95,64 @@ class Artifact(unittest.TestCase):
def git(self, *args):
return subprocess.check_output(['git', '-C', str(self.repo), *args], text=True).strip()
+ def execute_worker(self, root, row, source=None, cache_error=None):
+ home = self.root / 'home'
+ binary = home / '.cargo/bin'
+ binary.mkdir(parents=True, exist_ok=True)
+ fake = '''import sys,json,pathlib
+if '--version' in sys.argv:
+ print(pathlib.Path(sys.argv[0]).name+' fixture');sys.exit(0)
+if 'metadata' in sys.argv:
+ print(json.dumps({'target_directory':str(pathlib.Path.cwd()/'target')}));sys.exit(0)
+bad=(pathlib.Path(sys.argv[0]).resolve().parents[2]/'fail-command').exists()
+print('Running tests/publication_process.rs (target/debug/deps/publication_process-abcdef0123456789)')
+print('running 1 test')
+print('test component_fixture ... '+('FAILED' if bad else 'ok'))
+print('test result: '+('FAILED' if bad else 'ok')+'. '+str(int(not bad))+' passed; '+str(int(bad))+' failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s')
+sys.exit(101 if bad else 0)
+'''
+ for name in ('cargo', 'rustc'):
+ path = binary / name
+ path.write_text('#!' + sys.executable + '\n' + fake)
+ path.chmod(0o700)
+ run = root / 'runs' / row['id']
+ run.mkdir(parents=True)
+ if source is not None:
+ row = dict(row, frontend_dist=str(source))
+ runner.save(run / 'request.json', row)
+ with patch.object(runner.Path, 'home', return_value=home), patch.dict(os.environ, {}, clear=False), \
+ patch.object(runner.sys, 'platform', 'linux'), \
+ patch.object(runner.tempfile, 'gettempdir', return_value=str(self.root)), \
+ patch.object(runner, 'worker_alive', return_value=False), \
+ patch.object(runner, 'MIN_FREE_BYTES', 0), \
+ patch.object(runner, 'admit_cache_server', side_effect=cache_error, return_value={'state': 'ready'}), \
+ patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}) as stats:
+ runner.worker(run / 'request.json')
+ result = json.loads((run / 'result.json').read_text())
+ if result.get('temporary_directory'):
+ self.addCleanup(runner.cleanup_temp_directory, result['temporary_directory'])
+ return result, stats
+
+ def generated_request(self, command, run_id, executor_sha=None):
+ argv = shlex.split(command)
+ self.assertEqual(argv[:2], ['python3', 'scripts/remote-rust.py'])
+ revision = executor_sha or self.git('rev-parse', 'HEAD')
+ def committed(arguments):
+ if arguments == ['git', 'rev-parse', 'HEAD']:
+ return revision
+ if arguments == ['git', 'remote', 'get-url', 'origin']:
+ return str(self.repo)
+ if arguments[:2] == ['git', 'show']:
+ return (Path(__file__).parent / arguments[2].split(':scripts/', 1)[1]).read_text().rstrip('\n')
+ self.fail('Unexpected submission read: ' + repr(arguments))
+ with patch.object(runner, 'command', side_effect=committed), \
+ patch.object(runner.sys, 'argv', argv[1:]), \
+ patch.object(runner.subprocess, 'run', return_value=subprocess.CompletedProcess([], 0)) as ssh:
+ runner.main()
+ request = json.loads(ssh.call_args.kwargs['input'])
+ request['id'] = run_id
+ return request
+
def test_build_stamps_only_successful_unchanged_source(self):
with patch.object(assets.subprocess, 'call', return_value=0) as npm:
self.assertEqual(assets.build(self.repo, {}), 0)
@@ -165,38 +224,8 @@ class Artifact(unittest.TestCase):
'installed_', '--', '--nocapture'],
retention='on-failure', exclusive=False, timeout=10, plan='console-assets', phase='focused',
ssh_host='fixture-backend')
- home = self.root / 'home'
- binary = home / '.cargo/bin'
- binary.mkdir(parents=True)
- fake = '''import sys,json,pathlib
-if '--version' in sys.argv:
- print(pathlib.Path(sys.argv[0]).name+' fixture');sys.exit(0)
-if 'metadata' in sys.argv:
- print(json.dumps({'target_directory':str(pathlib.Path.cwd()/'target')}));sys.exit(0)
-print('Running tests/publication_process.rs (target/debug/deps/publication_process-abcdef0123456789)')
-print('running 1 test')
-print('test component_fixture ... ok')
-print('test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s')
-'''
- for name in ('cargo', 'rustc'):
- path = binary / name
- path.write_text('#!' + sys.executable + '\n' + fake)
- path.chmod(0o700)
-
def execute(row, source=None):
- run = root / 'runs' / row['id']
- run.mkdir(parents=True)
- if source is not None:
- row = dict(row, frontend_dist=str(source))
- runner.save(run / 'request.json', row)
- with patch.object(runner.Path, 'home', return_value=home), patch.dict(os.environ, {}, clear=False), \
- patch.object(runner.sys, 'platform', 'linux'), \
- patch.object(runner.tempfile, 'gettempdir', return_value=str(self.root)), \
- patch.object(runner, 'MIN_FREE_BYTES', 0), \
- patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \
- patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}) as stats:
- runner.worker(run / 'request.json')
- return json.loads((run / 'result.json').read_text()), stats
+ return self.execute_worker(root, row, source)
before, stats = execute(request)
self.assertEqual((before['status'], before['blocker'], before['process_exit']),
@@ -205,10 +234,11 @@ print('test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered ou
stats.assert_not_called()
self.assertFalse((root / 'runs' / first / 'command.log').exists())
self.assertIn('--reuse-checkout ' + first, before['console_assets']['preparation'][-1])
- staging = home / '.cache/hiroute/console-assets' / first
+ staging = self.root / 'home/.cache/hiroute/console-assets' / first
preparation = before['console_assets']['preparation']
self.assertIn(shlex.join(['scp', '-r', 'apps/desktop/dist', 'fixture-backend:' + str(staging) + '/']), preparation)
- self.assertEqual(shlex.split(preparation[-1])[2:4], ['--frontend-dist', str(staging / 'dist')])
+ arguments = shlex.split(preparation[-1])
+ self.assertEqual(arguments[arguments.index('--frontend-dist') + 1], str(staging / 'dist'))
row = dict(request, id=second, reuse_checkout=first, retention='always')
lease = runner.checkout_lease(root, first)
try:
@@ -239,6 +269,218 @@ print('test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered ou
self.assertEqual(json.loads((root / 'runs' / first / 'result.json').read_text())['status'], 'blocked')
runner.cleanup_temp_directory(after['temporary_directory'])
+ def test_multiple_advances_block_then_execute_the_generated_recovery_in_the_same_checkout(self):
+ # Real Git/leases/worker/parser and command processes; compiler output is a fixture.
+ for kind in ('missing', 'stale'):
+ with self.subTest(kind=kind):
+ root = self.root / ('workbench-' + kind)
+ owner, middle, latest, blocked, retry, recovered = [
+ '20261024-0' + str(int(kind == 'stale')) + '000' + str(i) + '-abcdef12'
+ for i in range(1, 7)]
+ sha = self.git('rev-parse', 'HEAD')
+ common = dict(ref=self.git('symbolic-ref', 'HEAD'), sha=sha, executor_sha=sha,
+ origin=str(self.repo), retention='always',
+ exclusive=False, timeout=37, plan='console-recovery', phase='focused',
+ ssh_host='fixture-backend', workbench='.local/share/fixture-workbench', related_runs=[])
+ for key, name in (('executor', 'remote-rust'), ('schedule', 'validation-schedule'),
+ ('report', 'validation-report'), ('console', 'console-assets')):
+ contents = (Path(__file__).parent / (name + '.py')).read_text().rstrip('\n') + '\n'
+ common[key + '_digest'] = hashlib.sha256(contents.encode()).hexdigest()
+ home = self.root / 'home'
+ home.mkdir(exist_ok=True)
+ (home / 'fail-command').touch()
+ # The stale case leaves an admitted older receipt through the runner itself.
+ (self.dist / assets.RECEIPT).write_text(json.dumps(assets.manifest(sha, assets.files(self.dist))))
+ first, _ = self.execute_worker(root, dict(common, id=owner,
+ command=['cargo', 'test', '--test', 'publication_process', 'installed_'] if kind == 'stale'
+ else ['cargo', 'test', '--lib']), self.dist if kind == 'stale' else None)
+ (home / 'fail-command').unlink()
+ self.assertEqual(first['status'], 'failed', first)
+ checkout = Path(first['checkout'])
+ (checkout / 'target').mkdir(exist_ok=True)
+ sentinel = checkout / 'target/retained-compiler-objects'
+ sentinel.write_bytes(b'retained compilation')
+ previous = owner
+ history = [owner]
+ for index, key in enumerate((middle, latest), 1):
+ (self.repo / 'advance').write_text(kind + str(index))
+ self.git('add', 'advance')
+ self.git('-c', 'user.name=Fixture', '-c', '[email protected]',
+ '-c', 'commit.gpgsign=false', 'commit', '-qm', 'advance fixture')
+ common['sha'] = self.git('rev-parse', 'HEAD')
+ result, _ = self.execute_worker(root, dict(common, id=key, continue_checkout=previous,
+ command=['cargo', 'test', '--lib']))
+ self.assertEqual(result['status'], 'completed', result)
+ previous = key
+ history.append(key)
+ command = ['cargo', 'test', '-p', 'hiroute-daemon', '--test', 'publication_process', 'installed_']
+ blocked_request = dict(common, id=blocked, continue_checkout=latest,
+ phase='final', related_runs=[middle, latest], command=command)
+ before, stats = self.execute_worker(root, blocked_request)
+ self.assertEqual((before['status'], before['blocker'], before['process_exit']),
+ ('blocked', 'console_assets', None), before)
+ self.assertNotIn('command_started_at', before)
+ stats.assert_not_called()
+ history.append(blocked)
+ snapshots = {key: (root / 'runs' / key / 'result.json').read_bytes() for key in history}
+ generated = before['console_assets']['preparation'][-1]
+ self.assertIn('--continue-checkout ' + blocked, generated)
+ self.assertEqual(generated, before['recovery']['command'])
+ self.assertNotIn('--reuse-checkout', generated)
+ # Resource safety remains independent of the recoverable result class.
+ recovery = dict(blocked_request, id=retry, continue_checkout=blocked)
+ with runner.checkout_lease(root, owner), patch.object(runner, 'worker_alive', return_value=False):
+ for key in ('plan', 'origin', 'ref', 'executor_digest', 'schedule_digest', 'report_digest', 'console_digest'):
+ with self.subTest(changed=key), self.assertRaises(runner.RunBlocked):
+ runner.continue_checkout(root, dict(recovery, **{key: 'changed'}), owner)
+ for key in (owner, middle, latest):
+ with self.subTest(stale=key), self.assertRaises(runner.RunBlocked):
+ runner.continue_checkout(root, dict(recovery, continue_checkout=key), owner)
+ with self.assertRaises(runner.RunBlocked):
+ runner.verify_reused_checkout(root, dict(recovery, reuse_checkout=middle))
+ with patch.object(runner, 'worker_alive', return_value=True), self.assertRaises(runner.RunBlocked):
+ runner.continue_checkout(root, recovery, owner)
+ (checkout / 'dirty-source').touch()
+ with self.assertRaises(runner.RunBlocked):
+ runner.continue_checkout(root, recovery, owner)
+ (checkout / 'dirty-source').unlink()
+ state_path = root / ('checkout-' + owner + '.json')
+ saved_state = state_path.read_bytes()
+ runner.save(state_path, dict(run_id=middle, sha=common['sha']))
+ with self.assertRaises(runner.RunBlocked):
+ runner.continue_checkout(root, recovery, owner)
+ state_path.write_bytes(saved_state)
+ with self.assertRaises(subprocess.CalledProcessError):
+ runner.continue_checkout(root, dict(recovery, sha=first['sha']), owner)
+ # Keep hashes/source checks active on repeated preparation failures.
+ again, _ = self.execute_worker(root, recovery, self.dist)
+ self.assertEqual(again.get('blocker'), 'console_assets', again)
+ snapshots[retry] = (root / 'runs' / retry / 'result.json').read_bytes()
+ (self.dist / assets.RECEIPT).write_text(json.dumps(assets.manifest(common['sha'], assets.files(self.dist))))
+ argv = shlex.split(again['recovery']['command'])
+ staging = Path(argv[argv.index('--frontend-dist') + 1])
+ staging.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copytree(self.dist, staging)
+ request = self.generated_request(again['recovery']['command'], recovered, common['executor_sha'])
+ for key in ('sha', 'executor_sha', 'executor_digest', 'schedule_digest', 'report_digest',
+ 'console_digest', 'timeout', 'plan', 'workbench', 'ssh_host'):
+ self.assertEqual(request[key], common[key], key)
+ self.assertEqual((request['phase'], request['related_runs'], request['command']),
+ ('final', [middle, latest], command))
+ after, _ = self.execute_worker(root, request)
+ self.assertEqual(after['status'], 'completed', after)
+ self.assertEqual(after['process_exit'], 0)
+ self.assertIn('command_started_at', after)
+ self.assertEqual(after['tests']['counts']['passed'], 1)
+ self.assertEqual((after['checkout'], after['checkout_owner']), (str(checkout), owner))
+ self.assertEqual(after['console_assets']['source_revision'], common['sha'])
+ self.assertEqual(sentinel.read_bytes(), b'retained compilation')
+ for key, snapshot in snapshots.items():
+ self.assertEqual((root / 'runs' / key / 'result.json').read_bytes(), snapshot)
+ metadata = dict(plan=common['plan'], related_runs=[blocked])
+ with self.assertRaisesRegex(ValueError, 'has not executed'):
+ runner.reporting().related_reports(root / 'runs', metadata)
+
+ def test_successful_owner_without_plan_reuses_checkout_after_a_failed_borrower(self):
+ root = self.root / 'workbench'
+ owner, failed, retry = ['20261024-03000' + str(i) + '-abcdef12' for i in range(1, 4)]
+ arguments = ['python3', 'scripts/remote-rust.py', '--host', 'fixture-backend',
+ '--workbench', '.local/share/fixture-workbench', 'run',
+ '--ref', self.git('symbolic-ref', 'HEAD'), '--sha', self.sha,
+ '--retention', 'always']
+ command = ['--', 'cargo', 'test', '--lib']
+ initial = self.generated_request(shlex.join(arguments + command), owner)
+ self.assertIsNone(initial.get('plan'))
+ first, _ = self.execute_worker(root, initial)
+ self.assertEqual((first['status'], first['process_exit']), ('completed', 0), first)
+ checkout = Path(first['checkout'])
+ (checkout / 'target').mkdir(exist_ok=True)
+ sentinel = checkout / 'target/retained-compiler-objects'
+ sentinel.write_bytes(b'retained compilation')
+ reuse = shlex.join(arguments + ['--reuse-checkout', owner] + command)
+ failure = self.root / 'home/fail-command'
+ failure.touch()
+ before, _ = self.execute_worker(root, self.generated_request(reuse, failed))
+ failure.unlink()
+ self.assertEqual((before['status'], before['process_exit']), ('failed', 101), before)
+ self.assertFalse(before.get('timed_out'))
+ self.assertIn('command_finished_at', before)
+ snapshots = {key: (root / 'runs' / key / 'result.json').read_bytes() for key in (owner, failed)}
+ after, _ = self.execute_worker(root, self.generated_request(reuse, retry))
+ self.assertEqual((after['status'], after['process_exit']), ('completed', 0), after.get('error'))
+ self.assertEqual(after['tests']['counts']['passed'], 1)
+ self.assertEqual((after['checkout'], after['checkout_owner']), (str(checkout), owner))
+ self.assertEqual((after['sha'], after['executor_digest']), (before['sha'], before['executor_digest']))
+ self.assertEqual(sentinel.read_bytes(), b'retained compilation')
+ self.assertEqual(json.loads((root / ('checkout-' + owner + '.json')).read_text())['run_id'], retry)
+ for key, snapshot in snapshots.items():
+ self.assertEqual((root / 'runs' / key / 'result.json').read_bytes(), snapshot)
+
+ def test_on_failure_setup_owner_reuses_checkout_after_generated_recovery_test_fails(self):
+ root = self.root / 'workbench'
+ owner, failed, retry = ['20261024-04000' + str(i) + '-abcdef12' for i in range(1, 4)]
+ initial = self.generated_request(shlex.join([
+ 'python3', 'scripts/remote-rust.py', '--host', 'fixture-backend',
+ '--workbench', '.local/share/fixture-workbench', 'run', '--ref', self.git('symbolic-ref', 'HEAD'),
+ '--sha', self.sha, '--plan', 'console-retry', '--',
+ 'cargo', 'test', '--test', 'publication_process', 'installed_']), owner)
+ self.assertEqual(initial['retention'], 'on-failure')
+ first, stats = self.execute_worker(root, initial)
+ self.assertEqual((first['status'], first['blocker'], first['process_exit']),
+ ('blocked', 'console_assets', None), first)
+ stats.assert_not_called()
+ checkout = Path(first['checkout'])
+ (checkout / 'target').mkdir(exist_ok=True)
+ sentinel = checkout / 'target/retained-compiler-objects'
+ sentinel.write_bytes(b'retained compilation')
+ recovery = first['recovery']['command']
+ argv = shlex.split(recovery)
+ self.assertEqual(argv[argv.index('--reuse-checkout') + 1], owner)
+ staging = Path(argv[argv.index('--frontend-dist') + 1])
+ staging.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copytree(self.dist, staging)
+ borrower = self.generated_request(recovery, failed)
+ self.assertEqual(borrower['retention'], 'always')
+ failure = self.root / 'home/fail-command'
+ failure.touch()
+ before, _ = self.execute_worker(root, borrower)
+ failure.unlink()
+ self.assertEqual((before['status'], before['process_exit']), ('failed', 101), before)
+ self.assertFalse(before.get('timed_out'))
+ self.assertIn('command_finished_at', before)
+ self.assertEqual(before['console_assets']['state'], 'verified')
+ snapshots = {key: (root / 'runs' / key / 'result.json').read_bytes() for key in (owner, failed)}
+ after, _ = self.execute_worker(root, self.generated_request(recovery, retry))
+ self.assertEqual((after['status'], after['process_exit']), ('completed', 0), after.get('error'))
+ self.assertEqual(after['tests']['counts']['passed'], 1)
+ self.assertEqual((after['checkout'], after['checkout_owner']), (str(checkout), owner))
+ self.assertEqual((after['sha'], after['executor_digest']), (before['sha'], before['executor_digest']))
+ self.assertEqual(after['console_assets']['source_revision'], self.sha)
+ self.assertEqual(sentinel.read_bytes(), b'retained compilation')
+ self.assertEqual(json.loads((root / ('checkout-' + owner + '.json')).read_text())['run_id'], retry)
+ for key, snapshot in snapshots.items():
+ self.assertEqual((root / 'runs' / key / 'result.json').read_bytes(), snapshot)
+
+ def test_cache_preflight_uses_the_same_generated_recovery_and_rechecks_setup(self):
+ root = self.root / 'workbench'
+ owner, retry = '20261024-020001-abcdef12', '20261024-020002-abcdef12'
+ initial = self.generated_request(shlex.join([
+ 'python3', 'scripts/remote-rust.py', '--host', 'fixture-backend',
+ '--workbench', '.local/share/fixture-workbench', 'run', '--ref', self.git('symbolic-ref', 'HEAD'),
+ '--sha', self.sha, '--retention', 'always', '--plan', 'cache-recovery', '--', 'cargo', 'test', '--lib']), owner)
+ blocked, stats = self.execute_worker(root, initial,
+ cache_error=runner.RunBlocked('cache_server_unhealthy', 'fixture cache admission failure'))
+ self.assertEqual((blocked['status'], blocked['blocker'], blocked['process_exit']),
+ ('blocked', 'cache_server_unhealthy', None))
+ self.assertNotIn('command_started_at', blocked)
+ stats.assert_not_called()
+ self.assertIn('--continue-checkout ' + owner, blocked['recovery']['command'])
+ old_result = (root / 'runs' / owner / 'result.json').read_bytes()
+ after, _ = self.execute_worker(root, self.generated_request(blocked['recovery']['command'], retry))
+ self.assertEqual(after['status'], 'completed', after)
+ self.assertEqual(after['checkout'], blocked['checkout'])
+ self.assertEqual((root / 'runs' / owner / 'result.json').read_bytes(), old_result)
+
if __name__ == '__main__':
unittest.main()
diff --git a/scripts/test-remote-rust.py b/scripts/test-remote-rust.py
index 90540fde..2dd7d6ff 100644
--- a/scripts/test-remote-rust.py
+++ b/scripts/test-remote-rust.py
@@ -23,6 +23,48 @@ spec.loader.exec_module(runner)
class Contract(unittest.TestCase):
+ def test_checkout_recovery_requires_a_known_unexecuted_setup_blocker(self):
+ first = dict(retention='always')
+ latest = dict(status='blocked', blocker='console_assets', process_exit=None,
+ plan='recovery', executor_digest='driver')
+ for blocker in runner.RECOVERABLE_SETUP_BLOCKERS:
+ with self.subTest(blocker=blocker):
+ row = dict(latest, blocker=blocker)
+ self.assertEqual(runner.checkout_recovery_modes(first, row), ('continue', 'reuse'))
+ for changes in (dict(blocker='checkout_busy'), dict(blocker='continuation_changed'),
+ dict(blocker='unknown'), dict(process_exit=0), dict(command_started_at=0),
+ dict(command_finished_at=1), dict(timed_out=True), dict(status='running')):
+ with self.subTest(changes=changes):
+ self.assertEqual(runner.checkout_recovery_modes(first, dict(latest, **changes)), ())
+ self.assertEqual(runner.checkout_recovery_modes(dict(latest, retention='on-failure'), latest), ('reuse',))
+ self.assertEqual(runner.checkout_recovery_modes(dict(retention='never'), latest), ())
+ failed = dict(latest, status='failed', process_exit=101, command_started_at=1, command_finished_at=2)
+ self.assertEqual(runner.checkout_recovery_modes(first, failed), ('continue', 'reuse'))
+ self.assertEqual(runner.checkout_recovery_modes(first, dict(failed, plan=None)), ('reuse',))
+ self.assertEqual(runner.checkout_recovery_modes(dict(latest, retention='on-failure'), failed), ('reuse',))
+ self.assertEqual(runner.checkout_recovery_modes(first, dict(failed, timed_out=True)), ())
+ self.assertEqual(runner.checkout_recovery_modes(dict(latest, retention='on-failure'),
+ dict(failed, timed_out=True)), ())
+
+ def test_recovery_command_keeps_execution_scope_and_does_not_link_the_blocked_run(self):
+ key = '20261024-010203-abcdef12'
+ row = dict(id=key, checkout_owner=key, retention='always', status='blocked', blocker='console_assets',
+ process_exit=None, plan='recovery', executor_digest='driver', phase='final',
+ ssh_host='fixture-host', workbench='.local/share/fixture-workbench',
+ ref='refs/heads/fixture', sha='a' * 40, timeout=123, exclusive=True,
+ schedule='workspace-dag', related_runs=['20261024-010201-abcdef12'],
+ command=['cargo', 'test', '--locked', '--workspace', '--exclude', 'hiroute-desktop', '--all-features'])
+ import shlex
+ argv = shlex.split(runner.checkout_recovery_command(Path('/unused'), row, '/staging with spaces/dist'))
+ for flag, value in (('--host', row['ssh_host']), ('--workbench', row['workbench']),
+ ('--continue-checkout', key), ('--timeout', '123'), ('--schedule', 'workspace-dag'),
+ ('--frontend-dist', '/staging with spaces/dist'), ('--phase', 'final')):
+ self.assertEqual(argv[argv.index(flag) + 1], value)
+ self.assertIn('--exclusive', argv)
+ self.assertEqual(argv[argv.index('--related-run') + 1], row['related_runs'][0])
+ self.assertEqual(argv[argv.index('--') + 1:], row['command'])
+ self.assertEqual(argv.count(key), 1)
+
def test_separate_workbench_submission_readback_and_maintenance_keep_the_selected_root(self):
with tempfile.TemporaryDirectory() as directory:
config = Path(directory) / 'remote-rust.json'
@@ -89,7 +131,9 @@ class Contract(unittest.TestCase):
patch('os.umask'), patch('subprocess.Popen'):
exec(runner.BOOTSTRAP, {})
run = Path(directory) / payload['workbench'] / 'runs/run-1'
- self.assertEqual(json.loads((run / 'request.json').read_text())['frontend_dist'], '~/remote-only/dist')
+ saved = json.loads((run / 'request.json').read_text())
+ self.assertEqual(saved['frontend_dist'], '~/remote-only/dist')
+ self.assertEqual(saved['workbench'], payload['workbench'])
self.assertFalse((run / 'console-assets.json').exists())
self.assertFalse((Path(directory) / runner.ROOT / 'runs').exists())
@@ -249,6 +293,7 @@ sys.exit(101 if bad else 0)
patch.object(runner.tempfile, 'gettempdir', return_value=str(base.resolve())), \
patch.object(runner, 'console_assets') as console, \
patch.object(runner, 'MIN_FREE_BYTES', 0), \
+ patch.object(runner, 'worker_alive', return_value=False), \
patch.object(runner, 'admit_cache_server', return_value={'state': 'ready'}), \
patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}):
console.return_value.preflight.return_value = dict(state='not_required', cases=[])
@@ -582,6 +627,7 @@ runner.admit_cache_server(root,env,handles=[handle])
runner.save(owner_run / 'result.json', dict(id=owner, status='completed',
retention='always', sha=sha, origin=str(source), executor_sha=sha,
checkout=str(checkout), worker_pid=None))
+ runner.save(root / ('checkout-' + owner + '.json'), dict(run_id=owner, sha=sha))
request = dict(reuse_checkout=owner, sha=sha, origin=str(source), executor_sha=sha)
self.assertEqual(runner.verify_reused_checkout(root, request), checkout)
borrowed = root / 'runs' / borrower