Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.
Rust ai-agentsai-gatewayclaude-codecodexcoding-agentcost-optimizationdeepseekllm-gatewayllm-routerlocal-firstmodel-routingrusttask-routingtauri
fix(validation): select exact DAG preparation receipts (#197)
澄潭f9f577d8f1cc
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.mdindex 94d9cc94..619674ef 100644--- a/docs/code-map/testing.md+++ b/docs/code-map/testing.md@@ -216,7 +216,10 @@ Final backend DAG admission and preparation are owned by[validation-schedule.py](../../scripts/validation-schedule.py), with[regressions](../../scripts/test-validation-schedule.py). CPA bridge process fixturesrun in an isolated lane before other listeners; Gateway runtime consumers wait for-prepared product roles.+prepared product roles. [Smoke preparation](../../tools/product-e2e/src/smoke/preparation.rs)+and [Gateway preparation](../../tools/e2e-harness/src/p0/production/candidate.rs) bind+the exact receipts consumed by admission; same-byte caches from another recipe do+not replace them. Failed admission retains independent results and missing coverage.Ordinary local/SSH totals and module reports share Cargo-target attribution in[validation-report.py](../../scripts/validation-report.py). Thediff --git a/docs/validation-routing.md b/docs/validation-routing.mdindex 8fe11965..dec980d8 100644--- a/docs/validation-routing.md+++ b/docs/validation-routing.md@@ -547,6 +547,15 @@ finish. Missing or changed artifacts and missing listed cases fail rather thanfalling back to a runtime Cargo build. Default failure handling stops admittingnew targets; `--no-fail-fast` explicitly collects the whole selected scope.+The preparation report binds exact product and Gateway receipt paths and their+file digests. Admission verifies only those selected receipts, preserving their+attested candidate, executable identity, checkout containment and Cargo runtime+environment. Different recipes (including different compiler job counts) can+retain identical executable bytes in the same checkout; cache-wide digest+uniqueness is not required. Missing or changed selected evidence still fails.+If preparation fails, completed independent-target results remain in the aggregate,+with the unexecuted target inventory and doctest admission explicitly recorded.+The run result distinguishes correctness from the five-minute performance target.`queue_seconds`, `execution_seconds` and `end_to_end_seconds` use the workbench'smonotonic clock; execution starts when capacity is acquired and includes checkout,diff --git a/issue-spec/specs/product-smoke-verification/spec.md b/issue-spec/specs/product-smoke-verification/spec.mdindex e1ef6c2f..c4ee0f3c 100644--- a/issue-spec/specs/product-smoke-verification/spec.md+++ b/issue-spec/specs/product-smoke-verification/spec.md@@ -71,6 +71,23 @@ relevant oracle. Evidence-generation integrity is verified before/after launch;large artifact verification time is runner overhead, not consumed from product-stepbudgets. Build and scenario time are recorded separately.+Build preparation MUST bind each selected product role to its exact build receipt+and receipt digest, including the recipe/environment identity verified by its+producer. Different valid recipes MAY retain identical executable bytes in the+same checkout. Unrelated receipts MUST NOT make the selected preparation ambiguous+or substitute a different runtime environment. Admission still verifies the source,+receipt and executable digests, checkout path containment and consistent selected+environment. Missing, changed or conflicting selected evidence MUST fail admission.++#### Scenario: Continue focused smoke with a full workspace preparation++- **WHEN** focused smoke and full workspace preparation use one candidate checkout+ with different build environments and identical resulting executable bytes+- **THEN** each preparation selects its own exact product and Gateway receipts and+ runtime environment without removing or rewriting the other receipts+- **AND** a missing, corrupt, escaping or conflicting selected receipt remains a+ failure even when another valid receipt has the same executable digest.+### Requirement: Validation tracks have independent evidence limits| Track | Can establish | Cannot substitute for |@@ -129,6 +146,11 @@ worktree's normal target, and artifact cleanup preserves active use and uniqueevidence. A leak scan failure blocks acceptance. Concrete measured timing belongsto the runner/run record, not an unmeasured duration promise here.+If preparation fails after independent targets execute, the run MUST retain their+attributable terminal results and counts, the preparation failure, and explicit+unexecuted target/doctest coverage. Completed independent targets cannot establish+complete validation when dependent targets were not admitted.+## Implementation references- [Product control journey](../../../tools/product-e2e/tests/control_shell.rs),@@ -136,3 +158,6 @@ to the runner/run record, not an unmeasured duration promise here.[controlled Gateway runtime](../../../tools/e2e-harness/tests/p0_gateway_runtime.rs).- [Evidence/fixture map](../../../docs/code-map/testing.md) and[validation selection](../../../docs/test-selection.md).+- [Smoke preparation](../../../tools/product-e2e/src/smoke/preparation.rs),+ [Gateway preparation](../../../tools/e2e-harness/src/p0/production/candidate.rs), and+ [DAG admission and regressions](../../../scripts/test-validation-schedule.py).diff --git a/scripts/test-validation-schedule.py b/scripts/test-validation-schedule.pyindex 3122723a..05099029 100644--- a/scripts/test-validation-schedule.py+++ b/scripts/test-validation-schedule.py@@ -1,6 +1,7 @@#!/usr/bin/env python3"""Prove phase isolation, exact coverage and shared-slot execution without Rust."""import contextlib+import hashlibimport importlib.utilimport ioimport json@@ -50,7 +51,7 @@ sys.exit(101 if bad else 0)class Schedule(unittest.TestCase):def run_dag_isolation_fixture(self, fail_cpa=False, fail_fast=True, omit_cpa=False,fail_observation=False, omit_observation=False,- duplicate_observation=False):+ duplicate_observation=False, preparation_failure=None):"""Exercise the real DAG controller with bounded fake build/target processes."""with tempfile.TemporaryDirectory() as directory:root = Path(directory)@@ -71,6 +72,7 @@ class Schedule(unittest.TestCase):cpa_done = threading.Event()observation_done = threading.Event()admitted = []+ ordinary_done = threading.Event()class FixtureRunner:@staticmethod@@ -126,15 +128,23 @@ class Schedule(unittest.TestCase):or (fail_observation and name == 'hiroute_observation'))path = run / ('fixture-' + name + '.log')path.write_text(name + '\n')- results.append(dict(name=name, log=str(path), complete=not failed,+ results.append(dict(name=name, index=item['index'], log=str(path), complete=not failed,process_exit=int(failed),counts=dict(passed=int(not failed), failed=int(failed), ignored=0)))stop_event = kwargs.get('stop_event', _args[-1] if _args else None)if failed and fail_fast and stop_event is not None:stop_event.set()break+ if any(item['name'] == 'hiroute_gateway' for item in selected):+ ordinary_done.set()return results+ def prepare(*_args):+ if preparation_failure:+ self.assertTrue(ordinary_done.wait(5), 'Independent target must finish first')+ raise ValueError(preparation_failure)+ return {}+mocks = dict(dag_catalog=lambda *_args: targets,dag_list=lambda source, *_args: ['default_two_domain_smoke' if source == 'smoke_cli' else source],dag_snapshot=lambda *_args: None,@@ -142,7 +152,7 @@ class Schedule(unittest.TestCase):dag_bind_daemon_fixture=lambda *_args: None,dag_bind_metadata=lambda *_args: None,dag_toolchain_context=lambda *_args: {'path': '/fixture/toolchain'},- dag_products=lambda *_args: {},+ dag_products=prepare,dag_publication_daemon=lambda *_args: root,dag_run_targets=run_targets)output = io.StringIO()@@ -164,6 +174,27 @@ class Schedule(unittest.TestCase):self.assertEqual(result['executed_targets'], 4)self.assertEqual(result['counts'], dict(passed=5, failed=0, ignored=0))+ def test_preparation_failure_preserves_completed_targets_and_unexecuted_coverage(self):+ for fail_fast in (True, False):+ with self.subTest(fail_fast=fail_fast):+ result, admitted, log = self.run_dag_isolation_fixture(+ preparation_failure='selected receipt changed', fail_fast=fail_fast)+ self.assertFalse(result['complete'])+ self.assertEqual(result['process_exit'], 1)+ self.assertEqual(result['error'], 'selected receipt changed')+ self.assertEqual(admitted, ['hiroute_cpa_bridge', 'hiroute_observation',+ 'hiroute_gateway'])+ self.assertEqual(result['executed_targets'], 3)+ self.assertEqual(len(result['target_results']), 3)+ self.assertEqual(result['counts'], dict(passed=3, failed=0, ignored=0))+ self.assertEqual([row['name'] for row in result['unexecuted_targets']], ['smoke_cli'])+ self.assertEqual(result['unexecuted_targets'][0]['cases'], ['default_two_domain_smoke'])+ self.assertFalse(result['doctests_executed'])+ self.assertIn('hiroute_gateway', log)+ summary = runner.scheduled_test_summary(log, COMMAND, result)+ self.assertEqual(summary['counts']['passed'], 3)+ self.assertNotEqual(summary['state'], 'rust_tests_passed')+def test_observation_deadline_isolation_preserves_parallelism_and_all_targets(self):result, admitted, _ = self.run_dag_isolation_fixture()self.assertTrue(result['complete'], result)@@ -499,7 +530,7 @@ class Schedule(unittest.TestCase):test_binary.chmod(0o700)(root / 'tools/product-e2e').mkdir(parents=True)cargo = binary / 'cargo'- cargo.write_text('#!' + sys.executable + '\n' + '''import os,sys,time,pathlib,json+ cargo.write_text('#!' + sys.executable + '\n' + '''import os,sys,time,pathlib,json,hashliba=sys.argv[1:]; root=pathlib.Path.cwd(); mode=os.environ['FIXTURE_MODE']if '--no-run' in a:assert os.environ['CARGO_BUILD_JOBS']=='64'@@ -516,12 +547,18 @@ if 'prepare_default_smoke_builds' in a:env={'CARGO_BUILD_JOBS':'8','CARGO_PKG_NAME':'hiroute-product-e2e','CARGO_BIN_EXE_hiroute-smoke':'/fixture/hiroute-smoke','RUST_RECURSION_COUNT':'1'}artifacts=[]for package in ('hiroute-e2e','hiroute-cli','hiroute-daemon'):- sha='sha256:'+package- artifacts.append({'package':package,'sha256':sha})+ binary={'hiroute-e2e':'hiroute-e2e','hiroute-cli':'hiroute','hiroute-daemon':'hirouted'}[package]+ cached=root/'target/smoke/builds'/('f'*64)/package/binary+ cached.parent.mkdir(parents=True)+ cached.write_text(package)+ sha='sha256:'+hashlib.sha256(cached.read_bytes()).hexdigest()+ artifact={'package':package,'binary':binary,'sha256':sha,'source_revision':'a'*40}+ receipt=cached.with_name('receipt.json')+ value={'artifact':artifact,'path':str(cached),'recipe':{'source_revision':'a'*40,'package':package,'binary':binary,'environment':env}}+ body=json.dumps(value)+ artifacts.append(dict(artifact,receipt={'path':str(receipt),'sha256':'sha256:'+hashlib.sha256(body.encode()).hexdigest()}))if mode=='missing-receipt' and package=='hiroute-cli': continue- receipt=root/'target/smoke/builds/fixture'/package/'receipt.json'- receipt.parent.mkdir(parents=True)- receipt.write_text(json.dumps({'artifact':{'sha256':sha},'recipe':{'source_revision':'a'*40,'environment':env}}))+ receipt.write_text(body)report=root/'target/smoke/prepare-fixture/preparation.json'report.parent.mkdir(parents=True)report.write_text(json.dumps({'schema':'hiroute.smoke.preparation/v1','source_revision':'a'*40,'scenarios_executed':0,'artifacts':artifacts}))@@ -568,7 +605,7 @@ print(f'test result: ok. {passed} passed; 0 failed; 1 ignored; 0 measured; 0 fildef test_unattested_preparation_never_starts_runtime(self):result, _ = self.run_fixture('missing-receipt')self.assertNotEqual(result['process_exit'], 0)- self.assertIn('unique matching receipt', result['error'])+ self.assertIn('receipt.json', result['error'])def test_red_or_omitted_tests_cannot_produce_aggregate_green(self):for mode in ('red-smoke', 'missing-remainder'):@@ -578,5 +615,180 @@ print(f'test result: ok. {passed} passed; 0 failed; 1 ignored; 0 measured; 0 filself.assertFalse(result['complete'])+class Preparation(unittest.TestCase):+ def setUp(self):+ temporary = tempfile.TemporaryDirectory()+ self.addCleanup(temporary.cleanup)+ self.root = Path(temporary.name).resolve()+ self.sha = 'a' * 40++ def bind(self, path, value):+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(json.dumps(value))+ return dict(path=str(path), sha256='sha256:' + scheduler.file_digest(path))++ def prepare(self, jobs):+ environment = dict(CARGO_BUILD_JOBS=str(jobs), CARGO_PKG_NAME='hiroute-product-e2e',+ **{'CARGO_BIN_EXE_hiroute-smoke': '/fixture/hiroute-smoke',+ 'RUST_RECURSION_COUNT': '1'})+ artifacts = []+ for package, binary in (('hiroute-e2e', 'hiroute-e2e'), ('hiroute-cli', 'hiroute'),+ ('hiroute-daemon', 'hirouted'), ('hiroute-gateway', 'hirouted')):+ key = hashlib.sha256(f'{package}-{jobs}'.encode()).hexdigest()+ path = self.root / 'target/smoke/builds' / key / package / binary+ path.parent.mkdir(parents=True)+ path.write_bytes(package.encode()) # Deliberately identical bytes across recipes.+ digest = 'sha256:' + scheduler.file_digest(path)+ if package == 'hiroute-gateway':+ attestation = dict(schema_version='hiroute.e2e.sut-build-attestation/v3',+ source_revision=self.sha, executable_path=str(path),+ executable_sha256=digest, build_nonce=str(jobs))+ receipt = dict(recipe='sha256:' + key, attestation=attestation)+ gateway = dict(schema='hiroute.e2e.current-preparation/v1', prepared=True,+ source_revision=self.sha, executable_sha256=digest,+ receipt=self.bind(path.with_name('receipt.json'), receipt))+ else:+ artifact = dict(package=package, binary=binary, source_revision=self.sha, sha256=digest)+ recipe = dict(package=package, binary=binary, source_revision=self.sha,+ environment=environment)+ receipt = dict(recipe=recipe, artifact=artifact, path=str(path))+ artifacts.append(dict(artifact, receipt=self.bind(path.with_name('receipt.json'), receipt)))+ report = self.root / f'target/smoke/prepare-{jobs}/preparation.json'+ self.bind(report, dict(schema='hiroute.smoke.preparation/v1', source_revision=self.sha,+ scenarios_executed=0, artifacts=artifacts, gateway=gateway))+ return report++ def execute(self, report):+ return scheduler.dag_products('build preparation=' + str(report) + '\n',+ self.root, self.sha, 'fixture')++ def test_focused_then_dag_selects_exact_receipts_without_removing_other_recipes(self):+ focused = self.prepare(8)+ full = self.prepare(32)+ before = {path: path.read_bytes() for path in self.root.glob('target/smoke/builds/*/*/receipt.json')}+ self.assertEqual(len(before), 8)+ env8 = scheduler.prepared_environment('build preparation=' + str(focused), self.root, self.sha)+ env32 = self.execute(full)+ self.assertEqual(env8['CARGO_BUILD_JOBS'], '8')+ self.assertEqual(env32['CARGO_BUILD_JOBS'], '32')+ report = json.loads(full.read_text())+ selected = Path(report['gateway']['receipt']['path']).with_name('hirouted')+ self.assertEqual(env32['HIROUTE_VALIDATION_GATEWAY_BIN'], str(selected))+ directory = Path(env32['HIROUTE_VALIDATION_PRODUCT_BIN_DIR'])+ self.assertEqual((directory / 'hiroute').read_bytes(), b'hiroute-cli')+ self.assertEqual((directory / 'hirouted').read_bytes(), b'hiroute-daemon')+ self.assertEqual(before, {path: path.read_bytes() for path in before})++ def test_ambiguous_role_selection_is_rejected_even_with_matching_bytes(self):+ report = self.prepare(32)+ data = json.loads(report.read_text())+ data['artifacts'].append(data['artifacts'][0])+ report.write_text(json.dumps(data))+ with self.assertRaisesRegex(ValueError, 'incomplete artifacts'):+ self.execute(report)++ def test_multiple_selected_receipts_cannot_stand_in_for_one_exact_identity(self):+ report = self.prepare(32)+ data = json.loads(report.read_text())+ for item in (data['artifacts'][0], data['gateway']):+ binding = item['receipt']+ item['receipt'] = [binding, binding]+ report.write_text(json.dumps(data))+ with self.assertRaisesRegex(ValueError, 'no exact receipt binding'):+ self.execute(report)+ item['receipt'] = binding++ def test_selected_receipt_integrity_source_paths_and_environment_remain_required(self):+ # Every mutation leaves unrelated same-byte receipts available. They must+ # never rescue invalid or missing selected preparation evidence.+ mutations = ('missing-binding', 'missing-receipt', 'corrupt-receipt', 'wrong-receipt-hash',+ 'wrong-source', 'wrong-artifact-source', 'changed-binary', 'wrong-binary-hash',+ 'escaping-receipt', 'escaping-binary', 'other-role-binary',+ 'wrong-jobs', 'different-environment')+ for package in ('hiroute-cli', 'hiroute-gateway'):+ for mutation in mutations:+ if package == 'hiroute-gateway' and mutation in (+ 'wrong-artifact-source', 'wrong-jobs', 'different-environment'):+ continue+ with self.subTest(package=package, mutation=mutation), tempfile.TemporaryDirectory() as temp:+ self.root = Path(temp).resolve()+ self.prepare(8)+ report = self.prepare(32)+ data = json.loads(report.read_text())+ item = (data['gateway'] if package == 'hiroute-gateway'+ else next(row for row in data['artifacts'] if row['package'] == package))+ binding = item['receipt']+ receipt_path = Path(binding['path'])+ receipt = json.loads(receipt_path.read_text())+ facts = receipt['attestation'] if package == 'hiroute-gateway' else receipt['recipe']+ binary_key = 'executable_path' if package == 'hiroute-gateway' else 'path'+ binary_facts = receipt['attestation'] if package == 'hiroute-gateway' else receipt+ binary = Path(binary_facts[binary_key])+ if mutation == 'missing-binding':+ del item['receipt']+ elif mutation == 'missing-receipt':+ receipt_path.unlink()+ elif mutation == 'corrupt-receipt':+ receipt_path.write_text('{')+ elif mutation == 'wrong-receipt-hash':+ binding['sha256'] = 'sha256:' + '0' * 64+ elif mutation == 'wrong-source':+ facts['source_revision'] = 'b' * 40+ elif mutation == 'wrong-artifact-source':+ receipt['artifact']['source_revision'] = 'b' * 40+ elif mutation == 'changed-binary':+ binary.write_bytes(b'changed')+ elif mutation == 'wrong-binary-hash':+ if package == 'hiroute-gateway':+ receipt['attestation']['executable_sha256'] = item['executable_sha256'] = 'sha256:wrong'+ else:+ receipt['artifact']['sha256'] = item['sha256'] = 'sha256:wrong'+ elif mutation == 'escaping-receipt':+ escaped = self.root / 'outside-receipt.json'+ escaped.write_bytes(receipt_path.read_bytes())+ receipt_path.unlink()+ receipt_path.symlink_to(escaped)+ elif mutation == 'escaping-binary':+ escaped = self.root / 'outside-binary'+ escaped.write_bytes(binary.read_bytes())+ binary.unlink()+ binary.symlink_to(escaped)+ elif mutation == 'other-role-binary':+ other = binary.parent / 'other-binary'+ other.write_bytes(binary.read_bytes())+ binary_facts[binary_key] = str(other)+ elif mutation == 'wrong-jobs':+ for row in data['artifacts']:+ path = Path(row['receipt']['path'])+ value = json.loads(path.read_text())+ value['recipe']['environment']['CARGO_BUILD_JOBS'] = '8'+ row['receipt'] = self.bind(path, value)+ receipt = json.loads(receipt_path.read_text())+ elif mutation == 'different-environment':+ facts['environment']['RUST_RECURSION_COUNT'] = '2'+ if mutation in ('wrong-source', 'wrong-artifact-source', 'wrong-binary-hash',+ 'other-role-binary', 'different-environment'):+ item['receipt'] = self.bind(receipt_path, receipt)+ report.write_text(json.dumps(data))+ with self.assertRaises((ValueError, OSError)):+ self.execute(report)+ self.assertFalse((self.root / 'target/validation-schedule').exists())++ def test_report_source_containment_and_unique_identity_are_required(self):+ report = self.prepare(32)+ log = 'build preparation=' + str(report) + '\n'+ for value in ('b' * 40,):+ with self.assertRaises(ValueError):+ scheduler.prepared_environment(log, self.root, value, jobs=32)+ with self.assertRaisesRegex(ValueError, 'exactly one report'):+ scheduler.prepared_environment(log + log, self.root, self.sha, jobs=32)+ escaped = self.root / 'preparation.json'+ escaped.write_bytes(report.read_bytes())+ report.unlink()+ report.symlink_to(escaped)+ with self.assertRaisesRegex(ValueError, 'outside this checkout'):+ self.execute(report)++if __name__ == '__main__':unittest.main()diff --git a/scripts/validation-schedule.py b/scripts/validation-schedule.pyindex 164ed831..a3083a52 100644--- a/scripts/validation-schedule.py+++ b/scripts/validation-schedule.py@@ -76,12 +76,44 @@ def snapshot_copy(source, destination):shutil.copy2(source, destination)-def prepared_environment(log, checkout, sha, jobs=8):- """Recover Cargo's test runtime variables from the attested preparation."""+def preparation_receipt(binding, target, package):+ """Read only the receipt bound by this preparation, never a digest cache search."""+ if (not isinstance(binding, dict) or set(binding) != {"path", "sha256"}+ or not isinstance(binding["path"], str)):+ raise ValueError("Build preparation has no exact receipt binding: " + package)+ path = Path(binding["path"])+ resolved = path.resolve(strict=True)+ build_root = target / "smoke/builds"+ try:+ relative = resolved.relative_to(build_root)+ except ValueError:+ raise ValueError("Prepared receipt escapes this checkout: " + package) from None+ if (not path.is_absolute() or path != resolved or len(relative.parts) != 3+ or not re.fullmatch(r"[0-9a-f]{64}", relative.parts[0])+ or relative.parts[1:] != (package, "receipt.json")):+ raise ValueError("Prepared receipt has an invalid cache path: " + package)+ if "sha256:" + file_digest(resolved) != binding["sha256"]:+ raise ValueError("Prepared receipt changed: " + package)+ return json.loads(resolved.read_text()), resolved.parent+++def prepared_binary(value, directory, binary, digest):+ path = Path(value).resolve(strict=True)+ if path != directory / binary:+ raise ValueError("Product artifact escapes selected receipt directory")+ if "sha256:" + file_digest(path) != digest:+ raise ValueError("Prepared product binary changed")+ return path+++def prepared_products(log, checkout, sha, jobs):+ """Verify the exact product receipts and their captured Cargo environment."""paths = re.findall(r"^build preparation=(\S+)$", log, re.M)if len(paths) != 1:raise ValueError("Build preparation did not identify exactly one report")target = (Path(checkout) / "target").resolve(strict=True)+ if target != Path(checkout).resolve(strict=True) / "target":+ raise ValueError("Prepared target is outside this checkout")report = Path(paths[0]).resolve(strict=True)if (report.parent.parent != target / "smoke" or report.name != "preparation.json"or not report.parent.name.startswith("prepare-")):@@ -97,19 +129,21 @@ def prepared_environment(log, checkout, sha, jobs=8):!= {"hiroute-e2e", "hiroute-cli", "hiroute-daemon"}):raise ValueError("Build preparation has incomplete artifacts")receipts = []+ products = {}for artifact in artifacts:package = artifact["package"]- if package not in {"hiroute-e2e", "hiroute-cli", "hiroute-daemon"}:- raise ValueError("Build preparation contains an unexpected package")- matches = []- for path in (target / "smoke/builds").glob("*/" + package + "/receipt.json"):- receipt = json.loads(path.read_text())- if (receipt.get("artifact", {}).get("sha256") == artifact["sha256"]- and receipt.get("recipe", {}).get("source_revision") == sha):- matches.append(receipt)- if len(matches) != 1:- raise ValueError("Build preparation artifact has no unique matching receipt: " + package)- receipts.append(matches[0])+ receipt, directory = preparation_receipt(artifact.get("receipt"), target, package)+ binary = {"hiroute-e2e": "hiroute-e2e", "hiroute-cli": "hiroute",+ "hiroute-daemon": "hirouted"}[package]+ facts = {key: value for key, value in artifact.items() if key != "receipt"}+ recipe = receipt["recipe"]+ if (receipt["artifact"] != facts or facts.get("source_revision") != sha+ or facts.get("binary") != binary+ or recipe.get("source_revision") != sha or recipe.get("package") != package+ or recipe.get("binary") != binary):+ raise ValueError("Prepared product receipt does not match the candidate: " + package)+ products[package] = prepared_binary(receipt["path"], directory, binary, artifact["sha256"])+ receipts.append(receipt)environments = [item["recipe"]["environment"] for item in receipts]if not all(env == environments[0] for env in environments):raise ValueError("Build preparation receipts disagree on runtime environment")@@ -120,7 +154,11 @@ def prepared_environment(log, checkout, sha, jobs=8):or environment.get("CARGO_PKG_NAME") != "hiroute-product-e2e"or not environment.get("CARGO_BIN_EXE_hiroute-smoke")):raise ValueError("Build preparation has invalid Cargo test environment")- return environment+ return data, environment, products+++def prepared_environment(log, checkout, sha, jobs=8):+ return prepared_products(log, checkout, sha, jobs)[1]def lane_summary(parser, log):@@ -489,48 +527,20 @@ def dag_toolchain_context(runner, checkout, sha, run):def dag_products(log, checkout, sha, run_id):- environment = prepared_environment(log, checkout, sha, jobs=32)- report_path = re.findall(r"^build preparation=(\S+)$", log, re.M)[0]- report = json.loads(Path(report_path).read_text())- build_root = Path(checkout) / "target/smoke/builds"- def within(path):- try:- path.relative_to(build_root.resolve(strict=True))- return True- except ValueError:- return False- artifacts = {}- for item in report["artifacts"]:- package = item["package"]- matches = []- for receipt_path in build_root.glob("*/" + package + "/receipt.json"):- receipt = json.loads(receipt_path.read_text())- artifact = receipt["artifact"]- if (receipt["recipe"]["source_revision"] == sha- and artifact["sha256"] == item["sha256"]):- path = Path(receipt["path"]).resolve(strict=True)- if not within(path):- raise ValueError("Product artifact escapes prepared target")- if "sha256:" + file_digest(path) != artifact["sha256"]:- raise ValueError("Prepared product binary changed")- matches.append(path)- if len(matches) != 1:- raise ValueError("Prepared product receipt is not unique: " + package)- artifacts[package] = matches[0]- gateway = []- for receipt_path in build_root.glob("*/hiroute-gateway/receipt.json"):- receipt = json.loads(receipt_path.read_text())- attestation = receipt["attestation"]- if (attestation["schema_version"] == "hiroute.e2e.sut-build-attestation/v3"- and attestation["source_revision"] == sha- and attestation["executable_sha256"] == report["gateway"]["executable_sha256"]):- path = Path(attestation["executable_path"]).resolve(strict=True)- if (not within(path)- or "sha256:" + file_digest(path) != attestation["executable_sha256"]):- raise ValueError("Prepared Gateway binary changed")- gateway.append(path)- if len(gateway) != 1:- raise ValueError("Prepared Gateway receipt is not unique")+ report, environment, artifacts = prepared_products(log, checkout, sha, jobs=32)+ gateway = report["gateway"]+ receipt, directory = preparation_receipt(gateway.get("receipt"),+ Path(checkout).resolve() / "target", "hiroute-gateway")+ attestation = receipt["attestation"]+ if (gateway.get("schema") != "hiroute.e2e.current-preparation/v1"+ or gateway.get("prepared") is not True or gateway.get("source_revision") != sha+ or attestation["schema_version"] != "hiroute.e2e.sut-build-attestation/v3"+ or attestation["source_revision"] != sha+ or attestation["executable_sha256"] != gateway["executable_sha256"]+ or receipt["recipe"] != "sha256:" + directory.parent.name):+ raise ValueError("Prepared Gateway receipt does not match the candidate")+ gateway_path = prepared_binary(attestation["executable_path"], directory, "hirouted",+ gateway["executable_sha256"])directory = Path(checkout) / "target/validation-schedule" / run_id / "products"directory.mkdir(parents=True, mode=0o700, exist_ok=False)for package, name in (("hiroute-cli", "hiroute"), ("hiroute-daemon", "hirouted")):@@ -540,7 +550,7 @@ def dag_products(log, checkout, sha, run_id):raise ValueError("Prepared product copy changed")environment.update(HIROUTE_VALIDATION_EXECUTION="1",HIROUTE_VALIDATION_PRODUCT_BIN_DIR=str(directory),- HIROUTE_VALIDATION_GATEWAY_BIN=str(gateway[0]),+ HIROUTE_VALIDATION_GATEWAY_BIN=str(gateway_path),HIROUTE_SMOKE_REQUIRE_PREPARED="1")return environment@@ -649,7 +659,7 @@ def dag_run_targets(runner, root, run, targets, checkout, lease, environment, deor sum(summary["counts"].values()) != len(item["cases"])or dag_observed_cases(text) != set(item["cases"])):raise ValueError("Test target did not complete its listed cases: " + name)- return dict(name=name, log=str(path), process_exit=code,+ return dict(name=name, index=item["index"], log=str(path), process_exit=code,complete=summary["complete"] and code == 0,counts=summary["counts"], exec_audit=audit,started_at=started_at, finished_at=time.time(),@@ -676,7 +686,9 @@ def dag_run_targets(runner, root, run, targets, checkout, lease, environment, detry:result = future.result()except Exception as error:- result = dict(name=item["name"], complete=False, process_exit=1,+ result = dict(name=item["name"], index=item["index"],+ log=str(run / ("dag-test-" + str(item["index"]) + ".log")),+ complete=False, process_exit=1,error=str(error), counts=dict(passed=0, failed=0, ignored=0))results.append(result)if not result["complete"] and fail_fast and stop_event is not None:@@ -693,7 +705,10 @@ def execute_dag(runner, root, run, checkout, request, lease, output, reserved_habase = [arg for arg in request["command"] if arg != "--no-fail-fast"]fail_fast = "--no-fail-fast" not in request["command"]phase_records = []- outcome = dict(process_exit=1, timed_out=False, phases=phase_records, complete=False)+ outcome = dict(process_exit=1, timed_out=False, phases=phase_records, complete=False,+ doctests_executed=False)+ targets, results = [], []+ expected = Nonedef cargo_phase(name, argv, slots):path = run / ("dag-" + name + ".log")@@ -777,9 +792,6 @@ def execute_dag(runner, root, run, checkout, request, lease, output, reserved_hafail_fast=fail_fast, reserved_handles=reserved_handles,stop_event=stop_event)if stop_event is not None and stop_event.is_set():- for result in results:- if result.get("log"):- output.write(Path(result["log"]).read_text(errors="replace") + "\n")outcome.update(expected_tests=expected, targets=len(targets),executed_targets=len(results), target_results=results,error="Isolated test target failed; remaining targets and doctests were not admitted")@@ -789,37 +801,42 @@ def execute_dag(runner, root, run, checkout, request, lease, output, reserved_haordinary = pool.submit(dag_run_targets, runner, root, run, independent, checkout,lease, execution_env,deadline, 4, 4, fail_fast, reserved_handles, stop_event)- preparation_log = prepared.result()- summary = runner.test_summary(preparation_log, commands(base)["prepare"])- if summary["counts"] != dict(passed=1, failed=0, ignored=0):- raise ValueError("Build preparation did not execute exactly once")- prepared_env = dag_products(preparation_log, checkout, request["sha"], run.name)- prepared_env["HIROUTE_VALIDATION_TOOLCHAIN_JSON"] = toolchain_artifact["path"]- hooks_command = ["cargo", "build", "--locked", "-p", "hiroute-daemon",- "--bin", "hirouted", "--features", "integration-test-hooks",- "--profile", "dev", "--message-format=json"]- hooks_log = cargo_phase("publication-daemon", hooks_command, 4)- hooks_dir = dag_publication_daemon(hooks_log, checkout, metadata, run.name,- prepared_env, request["sha"], run)- for item in dependent:- if item["name"] == "publication_process":- item["environment"] = {- "HIROUTE_VALIDATION_PRODUCT_BIN_DIR": str(hooks_dir)}- remaining = dag_run_targets(runner, root, run, dependent, checkout, lease,- dict(os.environ, **prepared_env), deadline,- slots=4, workers=4, fail_fast=fail_fast,- reserved_handles=reserved_handles,- stop_event=stop_event)- first = ordinary.result()- results += first + remaining- for result in sorted(results, key=lambda item: item.get("log", "")):- if result.get("log"):- output.write(Path(result["log"]).read_text(errors="replace") + "\n")+ try:+ preparation_log = prepared.result()+ summary = runner.test_summary(preparation_log, commands(base)["prepare"])+ if summary["counts"] != dict(passed=1, failed=0, ignored=0):+ raise ValueError("Build preparation did not execute exactly once")+ prepared_env = dag_products(preparation_log, checkout, request["sha"], run.name)+ prepared_env["HIROUTE_VALIDATION_TOOLCHAIN_JSON"] = toolchain_artifact["path"]+ hooks_command = ["cargo", "build", "--locked", "-p", "hiroute-daemon",+ "--bin", "hirouted", "--features", "integration-test-hooks",+ "--profile", "dev", "--message-format=json"]+ hooks_log = cargo_phase("publication-daemon", hooks_command, 4)+ hooks_dir = dag_publication_daemon(hooks_log, checkout, metadata, run.name,+ prepared_env, request["sha"], run)+ for item in dependent:+ if item["name"] == "publication_process":+ item["environment"] = {+ "HIROUTE_VALIDATION_PRODUCT_BIN_DIR": str(hooks_dir)}+ results.extend(dag_run_targets(runner, root, run, dependent, checkout, lease,+ dict(os.environ, **prepared_env), deadline,+ slots=4, workers=4, fail_fast=fail_fast,+ reserved_handles=reserved_handles,+ stop_event=stop_event))+ except Exception:+ if stop_event is not None:+ stop_event.set()+ raise+ finally:+ # Preparation can fail after independent targets already finished.+ # Retain those results even when dependent admission never begins.+ results.extend(ordinary.result())if stop_event is not None and stop_event.is_set():outcome.update(expected_tests=expected, targets=len(targets),executed_targets=len(results), target_results=results,error="Test target failed; remaining targets and doctests were not admitted")return outcome+ outcome["doctests_executed"] = Truedoc_log = cargo_phase("doctest", [*base, "--doc"], 8)output.write(doc_log + "\n")doc_summary = lane_summary(runner.reporting(), doc_log)@@ -843,4 +860,17 @@ def execute_dag(runner, root, run, checkout, request, lease, output, reserved_haexcept Exception as error:outcome["error"] = str(error)outcome["timed_out"] = time.monotonic() >= deadline+ finally:+ for result in sorted(results, key=lambda item: item.get("log", "")):+ if result.get("log") and Path(result["log"]).is_file():+ output.write(Path(result["log"]).read_text(errors="replace") + "\n")+ if expected is not None:+ executed = {row["index"] for row in results}+ outcome.update(expected_tests=expected, targets=len(targets),+ executed_targets=len(results), target_results=results,+ unexecuted_targets=[dict(index=item["index"], name=item["name"],+ package=item["package_name"], cases=item["cases"])+ for item in targets if item["index"] not in executed])+ outcome.setdefault("counts", {key: sum(row["counts"][key] for row in results)+ for key in ("passed", "failed", "ignored")})return outcomediff --git a/tools/e2e-harness/src/p0/production/candidate.rs b/tools/e2e-harness/src/p0/production/candidate.rsindex 025d70dc..809297e6 100644--- a/tools/e2e-harness/src/p0/production/candidate.rs+++ b/tools/e2e-harness/src/p0/production/candidate.rs@@ -475,9 +475,11 @@ pub fn prepare_current_candidate(root: &Path) -> Result<Value, ProductionError>.as_ref().expect("current bundle has identity");identity.verify_live_sut(&sut)?;+ let receipt = sut.canonical_path.with_file_name("receipt.json");Ok(json!({"schema":"hiroute.e2e.current-preparation/v1", "prepared":true,- "source_revision":identity.revision, "executable_sha256":sut.executable_sha256}),+ "source_revision":identity.revision, "executable_sha256":sut.executable_sha256,+ "receipt":{"path":receipt, "sha256":sha256_hex(&fs::read(&receipt)?)}}),)}diff --git a/tools/product-e2e/src/smoke/preparation.rs b/tools/product-e2e/src/smoke/preparation.rsindex af5586ba..56348010 100644--- a/tools/product-e2e/src/smoke/preparation.rs+++ b/tools/product-e2e/src/smoke/preparation.rs@@ -46,12 +46,23 @@ pub fn prepare(root: &Path, cancel: Arc<AtomicBool>) -> Result<PathBuf> {"gateway_preparation_identity_changed",)?;tool.verify(&source)?;+ // Bind the exact receipts selected by get_or_build, including their recipes.+ // Distinct environments may legitimately produce identical executable bytes.+ let mut records = Vec::new();+ for artifact in &artifacts {+ let receipt = artifact.path.with_file_name("receipt.json");+ let mut record = serde_json::to_value(artifact)?;+ record["receipt"] = json!({+ "path":receipt, "sha256":super::digest(&std::fs::read(&receipt)?),+ });+ records.push(record);+ }let report = directory.join("preparation.json");std::fs::write(&report,serde_json::to_vec_pretty(&json!({"schema":"hiroute.smoke.preparation/v1", "source_revision":source.revision,- "build_ms":started.elapsed().as_millis(), "artifacts":artifacts, "gateway":gateway,+ "build_ms":started.elapsed().as_millis(), "artifacts":records, "gateway":gateway,"scenarios_executed":0,}))?,)?;