Skip to content

Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.

fix(validation): select exact DAG preparation receipts (#197)

澄潭f9f577d8f1cc
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.md
index 94d9cc94..619674ef 100644
--- a/docs/code-map/testing.md
+++ b/docs/code-map/testing.md
@@ -216,7 +216,10 @@ Final backend DAG admission and preparation are owned by
[validation-schedule.py](../../scripts/validation-schedule.py), with
[regressions](../../scripts/test-validation-schedule.py). CPA bridge process fixtures
run in an isolated lane before other listeners; Gateway runtime consumers wait for
-prepared product roles.
+prepared product roles. [Smoke preparation](../../tools/product-e2e/src/smoke/preparation.rs)
+and [Gateway preparation](../../tools/e2e-harness/src/p0/production/candidate.rs) bind
+the exact receipts consumed by admission; same-byte caches from another recipe do
+not replace them. Failed admission retains independent results and missing coverage.
Ordinary local/SSH totals and module reports share Cargo-target attribution in
[validation-report.py](../../scripts/validation-report.py). The
diff --git a/docs/validation-routing.md b/docs/validation-routing.md
index 8fe11965..dec980d8 100644
--- a/docs/validation-routing.md
+++ b/docs/validation-routing.md
@@ -547,6 +547,15 @@ finish. Missing or changed artifacts and missing listed cases fail rather than
falling back to a runtime Cargo build. Default failure handling stops admitting
new targets; `--no-fail-fast` explicitly collects the whole selected scope.
+The preparation report binds exact product and Gateway receipt paths and their
+file digests. Admission verifies only those selected receipts, preserving their
+attested candidate, executable identity, checkout containment and Cargo runtime
+environment. Different recipes (including different compiler job counts) can
+retain identical executable bytes in the same checkout; cache-wide digest
+uniqueness is not required. Missing or changed selected evidence still fails.
+If preparation fails, completed independent-target results remain in the aggregate,
+with the unexecuted target inventory and doctest admission explicitly recorded.
+
The run result distinguishes correctness from the five-minute performance target.
`queue_seconds`, `execution_seconds` and `end_to_end_seconds` use the workbench's
monotonic clock; execution starts when capacity is acquired and includes checkout,
diff --git a/issue-spec/specs/product-smoke-verification/spec.md b/issue-spec/specs/product-smoke-verification/spec.md
index e1ef6c2f..c4ee0f3c 100644
--- a/issue-spec/specs/product-smoke-verification/spec.md
+++ b/issue-spec/specs/product-smoke-verification/spec.md
@@ -71,6 +71,23 @@ relevant oracle. Evidence-generation integrity is verified before/after launch;
large artifact verification time is runner overhead, not consumed from product-step
budgets. Build and scenario time are recorded separately.
+Build preparation MUST bind each selected product role to its exact build receipt
+and receipt digest, including the recipe/environment identity verified by its
+producer. Different valid recipes MAY retain identical executable bytes in the
+same checkout. Unrelated receipts MUST NOT make the selected preparation ambiguous
+or substitute a different runtime environment. Admission still verifies the source,
+receipt and executable digests, checkout path containment and consistent selected
+environment. Missing, changed or conflicting selected evidence MUST fail admission.
+
+#### Scenario: Continue focused smoke with a full workspace preparation
+
+- **WHEN** focused smoke and full workspace preparation use one candidate checkout
+ with different build environments and identical resulting executable bytes
+- **THEN** each preparation selects its own exact product and Gateway receipts and
+ runtime environment without removing or rewriting the other receipts
+- **AND** a missing, corrupt, escaping or conflicting selected receipt remains a
+ failure even when another valid receipt has the same executable digest.
+
### Requirement: Validation tracks have independent evidence limits
| Track | Can establish | Cannot substitute for |
@@ -129,6 +146,11 @@ worktree's normal target, and artifact cleanup preserves active use and unique
evidence. A leak scan failure blocks acceptance. Concrete measured timing belongs
to the runner/run record, not an unmeasured duration promise here.
+If preparation fails after independent targets execute, the run MUST retain their
+attributable terminal results and counts, the preparation failure, and explicit
+unexecuted target/doctest coverage. Completed independent targets cannot establish
+complete validation when dependent targets were not admitted.
+
## Implementation references
- [Product control journey](../../../tools/product-e2e/tests/control_shell.rs),
@@ -136,3 +158,6 @@ to the runner/run record, not an unmeasured duration promise here.
[controlled Gateway runtime](../../../tools/e2e-harness/tests/p0_gateway_runtime.rs).
- [Evidence/fixture map](../../../docs/code-map/testing.md) and
[validation selection](../../../docs/test-selection.md).
+- [Smoke preparation](../../../tools/product-e2e/src/smoke/preparation.rs),
+ [Gateway preparation](../../../tools/e2e-harness/src/p0/production/candidate.rs), and
+ [DAG admission and regressions](../../../scripts/test-validation-schedule.py).
diff --git a/scripts/test-validation-schedule.py b/scripts/test-validation-schedule.py
index 3122723a..05099029 100644
--- a/scripts/test-validation-schedule.py
+++ b/scripts/test-validation-schedule.py
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
"""Prove phase isolation, exact coverage and shared-slot execution without Rust."""
import contextlib
+import hashlib
import importlib.util
import io
import json
@@ -50,7 +51,7 @@ sys.exit(101 if bad else 0)
class Schedule(unittest.TestCase):
def run_dag_isolation_fixture(self, fail_cpa=False, fail_fast=True, omit_cpa=False,
fail_observation=False, omit_observation=False,
- duplicate_observation=False):
+ duplicate_observation=False, preparation_failure=None):
"""Exercise the real DAG controller with bounded fake build/target processes."""
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
@@ -71,6 +72,7 @@ class Schedule(unittest.TestCase):
cpa_done = threading.Event()
observation_done = threading.Event()
admitted = []
+ ordinary_done = threading.Event()
class FixtureRunner:
@staticmethod
@@ -126,15 +128,23 @@ class Schedule(unittest.TestCase):
or (fail_observation and name == 'hiroute_observation'))
path = run / ('fixture-' + name + '.log')
path.write_text(name + '\n')
- results.append(dict(name=name, log=str(path), complete=not failed,
+ results.append(dict(name=name, index=item['index'], log=str(path), complete=not failed,
process_exit=int(failed),
counts=dict(passed=int(not failed), failed=int(failed), ignored=0)))
stop_event = kwargs.get('stop_event', _args[-1] if _args else None)
if failed and fail_fast and stop_event is not None:
stop_event.set()
break
+ if any(item['name'] == 'hiroute_gateway' for item in selected):
+ ordinary_done.set()
return results
+ def prepare(*_args):
+ if preparation_failure:
+ self.assertTrue(ordinary_done.wait(5), 'Independent target must finish first')
+ raise ValueError(preparation_failure)
+ return {}
+
mocks = dict(dag_catalog=lambda *_args: targets,
dag_list=lambda source, *_args: ['default_two_domain_smoke' if source == 'smoke_cli' else source],
dag_snapshot=lambda *_args: None,
@@ -142,7 +152,7 @@ class Schedule(unittest.TestCase):
dag_bind_daemon_fixture=lambda *_args: None,
dag_bind_metadata=lambda *_args: None,
dag_toolchain_context=lambda *_args: {'path': '/fixture/toolchain'},
- dag_products=lambda *_args: {},
+ dag_products=prepare,
dag_publication_daemon=lambda *_args: root,
dag_run_targets=run_targets)
output = io.StringIO()
@@ -164,6 +174,27 @@ class Schedule(unittest.TestCase):
self.assertEqual(result['executed_targets'], 4)
self.assertEqual(result['counts'], dict(passed=5, failed=0, ignored=0))
+ def test_preparation_failure_preserves_completed_targets_and_unexecuted_coverage(self):
+ for fail_fast in (True, False):
+ with self.subTest(fail_fast=fail_fast):
+ result, admitted, log = self.run_dag_isolation_fixture(
+ preparation_failure='selected receipt changed', fail_fast=fail_fast)
+ self.assertFalse(result['complete'])
+ self.assertEqual(result['process_exit'], 1)
+ self.assertEqual(result['error'], 'selected receipt changed')
+ self.assertEqual(admitted, ['hiroute_cpa_bridge', 'hiroute_observation',
+ 'hiroute_gateway'])
+ self.assertEqual(result['executed_targets'], 3)
+ self.assertEqual(len(result['target_results']), 3)
+ self.assertEqual(result['counts'], dict(passed=3, failed=0, ignored=0))
+ self.assertEqual([row['name'] for row in result['unexecuted_targets']], ['smoke_cli'])
+ self.assertEqual(result['unexecuted_targets'][0]['cases'], ['default_two_domain_smoke'])
+ self.assertFalse(result['doctests_executed'])
+ self.assertIn('hiroute_gateway', log)
+ summary = runner.scheduled_test_summary(log, COMMAND, result)
+ self.assertEqual(summary['counts']['passed'], 3)
+ self.assertNotEqual(summary['state'], 'rust_tests_passed')
+
def test_observation_deadline_isolation_preserves_parallelism_and_all_targets(self):
result, admitted, _ = self.run_dag_isolation_fixture()
self.assertTrue(result['complete'], result)
@@ -499,7 +530,7 @@ class Schedule(unittest.TestCase):
test_binary.chmod(0o700)
(root / 'tools/product-e2e').mkdir(parents=True)
cargo = binary / 'cargo'
- cargo.write_text('#!' + sys.executable + '\n' + '''import os,sys,time,pathlib,json
+ cargo.write_text('#!' + sys.executable + '\n' + '''import os,sys,time,pathlib,json,hashlib
a=sys.argv[1:]; root=pathlib.Path.cwd(); mode=os.environ['FIXTURE_MODE']
if '--no-run' in a:
assert os.environ['CARGO_BUILD_JOBS']=='64'
@@ -516,12 +547,18 @@ if 'prepare_default_smoke_builds' in a:
env={'CARGO_BUILD_JOBS':'8','CARGO_PKG_NAME':'hiroute-product-e2e','CARGO_BIN_EXE_hiroute-smoke':'/fixture/hiroute-smoke','RUST_RECURSION_COUNT':'1'}
artifacts=[]
for package in ('hiroute-e2e','hiroute-cli','hiroute-daemon'):
- sha='sha256:'+package
- artifacts.append({'package':package,'sha256':sha})
+ binary={'hiroute-e2e':'hiroute-e2e','hiroute-cli':'hiroute','hiroute-daemon':'hirouted'}[package]
+ cached=root/'target/smoke/builds'/('f'*64)/package/binary
+ cached.parent.mkdir(parents=True)
+ cached.write_text(package)
+ sha='sha256:'+hashlib.sha256(cached.read_bytes()).hexdigest()
+ artifact={'package':package,'binary':binary,'sha256':sha,'source_revision':'a'*40}
+ receipt=cached.with_name('receipt.json')
+ value={'artifact':artifact,'path':str(cached),'recipe':{'source_revision':'a'*40,'package':package,'binary':binary,'environment':env}}
+ body=json.dumps(value)
+ artifacts.append(dict(artifact,receipt={'path':str(receipt),'sha256':'sha256:'+hashlib.sha256(body.encode()).hexdigest()}))
if mode=='missing-receipt' and package=='hiroute-cli': continue
- receipt=root/'target/smoke/builds/fixture'/package/'receipt.json'
- receipt.parent.mkdir(parents=True)
- receipt.write_text(json.dumps({'artifact':{'sha256':sha},'recipe':{'source_revision':'a'*40,'environment':env}}))
+ receipt.write_text(body)
report=root/'target/smoke/prepare-fixture/preparation.json'
report.parent.mkdir(parents=True)
report.write_text(json.dumps({'schema':'hiroute.smoke.preparation/v1','source_revision':'a'*40,'scenarios_executed':0,'artifacts':artifacts}))
@@ -568,7 +605,7 @@ print(f'test result: ok. {passed} passed; 0 failed; 1 ignored; 0 measured; 0 fil
def test_unattested_preparation_never_starts_runtime(self):
result, _ = self.run_fixture('missing-receipt')
self.assertNotEqual(result['process_exit'], 0)
- self.assertIn('unique matching receipt', result['error'])
+ self.assertIn('receipt.json', result['error'])
def test_red_or_omitted_tests_cannot_produce_aggregate_green(self):
for mode in ('red-smoke', 'missing-remainder'):
@@ -578,5 +615,180 @@ print(f'test result: ok. {passed} passed; 0 failed; 1 ignored; 0 measured; 0 fil
self.assertFalse(result['complete'])
+class Preparation(unittest.TestCase):
+ def setUp(self):
+ temporary = tempfile.TemporaryDirectory()
+ self.addCleanup(temporary.cleanup)
+ self.root = Path(temporary.name).resolve()
+ self.sha = 'a' * 40
+
+ def bind(self, path, value):
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps(value))
+ return dict(path=str(path), sha256='sha256:' + scheduler.file_digest(path))
+
+ def prepare(self, jobs):
+ environment = dict(CARGO_BUILD_JOBS=str(jobs), CARGO_PKG_NAME='hiroute-product-e2e',
+ **{'CARGO_BIN_EXE_hiroute-smoke': '/fixture/hiroute-smoke',
+ 'RUST_RECURSION_COUNT': '1'})
+ artifacts = []
+ for package, binary in (('hiroute-e2e', 'hiroute-e2e'), ('hiroute-cli', 'hiroute'),
+ ('hiroute-daemon', 'hirouted'), ('hiroute-gateway', 'hirouted')):
+ key = hashlib.sha256(f'{package}-{jobs}'.encode()).hexdigest()
+ path = self.root / 'target/smoke/builds' / key / package / binary
+ path.parent.mkdir(parents=True)
+ path.write_bytes(package.encode()) # Deliberately identical bytes across recipes.
+ digest = 'sha256:' + scheduler.file_digest(path)
+ if package == 'hiroute-gateway':
+ attestation = dict(schema_version='hiroute.e2e.sut-build-attestation/v3',
+ source_revision=self.sha, executable_path=str(path),
+ executable_sha256=digest, build_nonce=str(jobs))
+ receipt = dict(recipe='sha256:' + key, attestation=attestation)
+ gateway = dict(schema='hiroute.e2e.current-preparation/v1', prepared=True,
+ source_revision=self.sha, executable_sha256=digest,
+ receipt=self.bind(path.with_name('receipt.json'), receipt))
+ else:
+ artifact = dict(package=package, binary=binary, source_revision=self.sha, sha256=digest)
+ recipe = dict(package=package, binary=binary, source_revision=self.sha,
+ environment=environment)
+ receipt = dict(recipe=recipe, artifact=artifact, path=str(path))
+ artifacts.append(dict(artifact, receipt=self.bind(path.with_name('receipt.json'), receipt)))
+ report = self.root / f'target/smoke/prepare-{jobs}/preparation.json'
+ self.bind(report, dict(schema='hiroute.smoke.preparation/v1', source_revision=self.sha,
+ scenarios_executed=0, artifacts=artifacts, gateway=gateway))
+ return report
+
+ def execute(self, report):
+ return scheduler.dag_products('build preparation=' + str(report) + '\n',
+ self.root, self.sha, 'fixture')
+
+ def test_focused_then_dag_selects_exact_receipts_without_removing_other_recipes(self):
+ focused = self.prepare(8)
+ full = self.prepare(32)
+ before = {path: path.read_bytes() for path in self.root.glob('target/smoke/builds/*/*/receipt.json')}
+ self.assertEqual(len(before), 8)
+ env8 = scheduler.prepared_environment('build preparation=' + str(focused), self.root, self.sha)
+ env32 = self.execute(full)
+ self.assertEqual(env8['CARGO_BUILD_JOBS'], '8')
+ self.assertEqual(env32['CARGO_BUILD_JOBS'], '32')
+ report = json.loads(full.read_text())
+ selected = Path(report['gateway']['receipt']['path']).with_name('hirouted')
+ self.assertEqual(env32['HIROUTE_VALIDATION_GATEWAY_BIN'], str(selected))
+ directory = Path(env32['HIROUTE_VALIDATION_PRODUCT_BIN_DIR'])
+ self.assertEqual((directory / 'hiroute').read_bytes(), b'hiroute-cli')
+ self.assertEqual((directory / 'hirouted').read_bytes(), b'hiroute-daemon')
+ self.assertEqual(before, {path: path.read_bytes() for path in before})
+
+ def test_ambiguous_role_selection_is_rejected_even_with_matching_bytes(self):
+ report = self.prepare(32)
+ data = json.loads(report.read_text())
+ data['artifacts'].append(data['artifacts'][0])
+ report.write_text(json.dumps(data))
+ with self.assertRaisesRegex(ValueError, 'incomplete artifacts'):
+ self.execute(report)
+
+ def test_multiple_selected_receipts_cannot_stand_in_for_one_exact_identity(self):
+ report = self.prepare(32)
+ data = json.loads(report.read_text())
+ for item in (data['artifacts'][0], data['gateway']):
+ binding = item['receipt']
+ item['receipt'] = [binding, binding]
+ report.write_text(json.dumps(data))
+ with self.assertRaisesRegex(ValueError, 'no exact receipt binding'):
+ self.execute(report)
+ item['receipt'] = binding
+
+ def test_selected_receipt_integrity_source_paths_and_environment_remain_required(self):
+ # Every mutation leaves unrelated same-byte receipts available. They must
+ # never rescue invalid or missing selected preparation evidence.
+ mutations = ('missing-binding', 'missing-receipt', 'corrupt-receipt', 'wrong-receipt-hash',
+ 'wrong-source', 'wrong-artifact-source', 'changed-binary', 'wrong-binary-hash',
+ 'escaping-receipt', 'escaping-binary', 'other-role-binary',
+ 'wrong-jobs', 'different-environment')
+ for package in ('hiroute-cli', 'hiroute-gateway'):
+ for mutation in mutations:
+ if package == 'hiroute-gateway' and mutation in (
+ 'wrong-artifact-source', 'wrong-jobs', 'different-environment'):
+ continue
+ with self.subTest(package=package, mutation=mutation), tempfile.TemporaryDirectory() as temp:
+ self.root = Path(temp).resolve()
+ self.prepare(8)
+ report = self.prepare(32)
+ data = json.loads(report.read_text())
+ item = (data['gateway'] if package == 'hiroute-gateway'
+ else next(row for row in data['artifacts'] if row['package'] == package))
+ binding = item['receipt']
+ receipt_path = Path(binding['path'])
+ receipt = json.loads(receipt_path.read_text())
+ facts = receipt['attestation'] if package == 'hiroute-gateway' else receipt['recipe']
+ binary_key = 'executable_path' if package == 'hiroute-gateway' else 'path'
+ binary_facts = receipt['attestation'] if package == 'hiroute-gateway' else receipt
+ binary = Path(binary_facts[binary_key])
+ if mutation == 'missing-binding':
+ del item['receipt']
+ elif mutation == 'missing-receipt':
+ receipt_path.unlink()
+ elif mutation == 'corrupt-receipt':
+ receipt_path.write_text('{')
+ elif mutation == 'wrong-receipt-hash':
+ binding['sha256'] = 'sha256:' + '0' * 64
+ elif mutation == 'wrong-source':
+ facts['source_revision'] = 'b' * 40
+ elif mutation == 'wrong-artifact-source':
+ receipt['artifact']['source_revision'] = 'b' * 40
+ elif mutation == 'changed-binary':
+ binary.write_bytes(b'changed')
+ elif mutation == 'wrong-binary-hash':
+ if package == 'hiroute-gateway':
+ receipt['attestation']['executable_sha256'] = item['executable_sha256'] = 'sha256:wrong'
+ else:
+ receipt['artifact']['sha256'] = item['sha256'] = 'sha256:wrong'
+ elif mutation == 'escaping-receipt':
+ escaped = self.root / 'outside-receipt.json'
+ escaped.write_bytes(receipt_path.read_bytes())
+ receipt_path.unlink()
+ receipt_path.symlink_to(escaped)
+ elif mutation == 'escaping-binary':
+ escaped = self.root / 'outside-binary'
+ escaped.write_bytes(binary.read_bytes())
+ binary.unlink()
+ binary.symlink_to(escaped)
+ elif mutation == 'other-role-binary':
+ other = binary.parent / 'other-binary'
+ other.write_bytes(binary.read_bytes())
+ binary_facts[binary_key] = str(other)
+ elif mutation == 'wrong-jobs':
+ for row in data['artifacts']:
+ path = Path(row['receipt']['path'])
+ value = json.loads(path.read_text())
+ value['recipe']['environment']['CARGO_BUILD_JOBS'] = '8'
+ row['receipt'] = self.bind(path, value)
+ receipt = json.loads(receipt_path.read_text())
+ elif mutation == 'different-environment':
+ facts['environment']['RUST_RECURSION_COUNT'] = '2'
+ if mutation in ('wrong-source', 'wrong-artifact-source', 'wrong-binary-hash',
+ 'other-role-binary', 'different-environment'):
+ item['receipt'] = self.bind(receipt_path, receipt)
+ report.write_text(json.dumps(data))
+ with self.assertRaises((ValueError, OSError)):
+ self.execute(report)
+ self.assertFalse((self.root / 'target/validation-schedule').exists())
+
+ def test_report_source_containment_and_unique_identity_are_required(self):
+ report = self.prepare(32)
+ log = 'build preparation=' + str(report) + '\n'
+ for value in ('b' * 40,):
+ with self.assertRaises(ValueError):
+ scheduler.prepared_environment(log, self.root, value, jobs=32)
+ with self.assertRaisesRegex(ValueError, 'exactly one report'):
+ scheduler.prepared_environment(log + log, self.root, self.sha, jobs=32)
+ escaped = self.root / 'preparation.json'
+ escaped.write_bytes(report.read_bytes())
+ report.unlink()
+ report.symlink_to(escaped)
+ with self.assertRaisesRegex(ValueError, 'outside this checkout'):
+ self.execute(report)
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/scripts/validation-schedule.py b/scripts/validation-schedule.py
index 164ed831..a3083a52 100644
--- a/scripts/validation-schedule.py
+++ b/scripts/validation-schedule.py
@@ -76,12 +76,44 @@ def snapshot_copy(source, destination):
shutil.copy2(source, destination)
-def prepared_environment(log, checkout, sha, jobs=8):
- """Recover Cargo's test runtime variables from the attested preparation."""
+def preparation_receipt(binding, target, package):
+ """Read only the receipt bound by this preparation, never a digest cache search."""
+ if (not isinstance(binding, dict) or set(binding) != {"path", "sha256"}
+ or not isinstance(binding["path"], str)):
+ raise ValueError("Build preparation has no exact receipt binding: " + package)
+ path = Path(binding["path"])
+ resolved = path.resolve(strict=True)
+ build_root = target / "smoke/builds"
+ try:
+ relative = resolved.relative_to(build_root)
+ except ValueError:
+ raise ValueError("Prepared receipt escapes this checkout: " + package) from None
+ if (not path.is_absolute() or path != resolved or len(relative.parts) != 3
+ or not re.fullmatch(r"[0-9a-f]{64}", relative.parts[0])
+ or relative.parts[1:] != (package, "receipt.json")):
+ raise ValueError("Prepared receipt has an invalid cache path: " + package)
+ if "sha256:" + file_digest(resolved) != binding["sha256"]:
+ raise ValueError("Prepared receipt changed: " + package)
+ return json.loads(resolved.read_text()), resolved.parent
+
+
+def prepared_binary(value, directory, binary, digest):
+ path = Path(value).resolve(strict=True)
+ if path != directory / binary:
+ raise ValueError("Product artifact escapes selected receipt directory")
+ if "sha256:" + file_digest(path) != digest:
+ raise ValueError("Prepared product binary changed")
+ return path
+
+
+def prepared_products(log, checkout, sha, jobs):
+ """Verify the exact product receipts and their captured Cargo environment."""
paths = re.findall(r"^build preparation=(\S+)$", log, re.M)
if len(paths) != 1:
raise ValueError("Build preparation did not identify exactly one report")
target = (Path(checkout) / "target").resolve(strict=True)
+ if target != Path(checkout).resolve(strict=True) / "target":
+ raise ValueError("Prepared target is outside this checkout")
report = Path(paths[0]).resolve(strict=True)
if (report.parent.parent != target / "smoke" or report.name != "preparation.json"
or not report.parent.name.startswith("prepare-")):
@@ -97,19 +129,21 @@ def prepared_environment(log, checkout, sha, jobs=8):
!= {"hiroute-e2e", "hiroute-cli", "hiroute-daemon"}):
raise ValueError("Build preparation has incomplete artifacts")
receipts = []
+ products = {}
for artifact in artifacts:
package = artifact["package"]
- if package not in {"hiroute-e2e", "hiroute-cli", "hiroute-daemon"}:
- raise ValueError("Build preparation contains an unexpected package")
- matches = []
- for path in (target / "smoke/builds").glob("*/" + package + "/receipt.json"):
- receipt = json.loads(path.read_text())
- if (receipt.get("artifact", {}).get("sha256") == artifact["sha256"]
- and receipt.get("recipe", {}).get("source_revision") == sha):
- matches.append(receipt)
- if len(matches) != 1:
- raise ValueError("Build preparation artifact has no unique matching receipt: " + package)
- receipts.append(matches[0])
+ receipt, directory = preparation_receipt(artifact.get("receipt"), target, package)
+ binary = {"hiroute-e2e": "hiroute-e2e", "hiroute-cli": "hiroute",
+ "hiroute-daemon": "hirouted"}[package]
+ facts = {key: value for key, value in artifact.items() if key != "receipt"}
+ recipe = receipt["recipe"]
+ if (receipt["artifact"] != facts or facts.get("source_revision") != sha
+ or facts.get("binary") != binary
+ or recipe.get("source_revision") != sha or recipe.get("package") != package
+ or recipe.get("binary") != binary):
+ raise ValueError("Prepared product receipt does not match the candidate: " + package)
+ products[package] = prepared_binary(receipt["path"], directory, binary, artifact["sha256"])
+ receipts.append(receipt)
environments = [item["recipe"]["environment"] for item in receipts]
if not all(env == environments[0] for env in environments):
raise ValueError("Build preparation receipts disagree on runtime environment")
@@ -120,7 +154,11 @@ def prepared_environment(log, checkout, sha, jobs=8):
or environment.get("CARGO_PKG_NAME") != "hiroute-product-e2e"
or not environment.get("CARGO_BIN_EXE_hiroute-smoke")):
raise ValueError("Build preparation has invalid Cargo test environment")
- return environment
+ return data, environment, products
+
+
+def prepared_environment(log, checkout, sha, jobs=8):
+ return prepared_products(log, checkout, sha, jobs)[1]
def lane_summary(parser, log):
@@ -489,48 +527,20 @@ def dag_toolchain_context(runner, checkout, sha, run):
def dag_products(log, checkout, sha, run_id):
- environment = prepared_environment(log, checkout, sha, jobs=32)
- report_path = re.findall(r"^build preparation=(\S+)$", log, re.M)[0]
- report = json.loads(Path(report_path).read_text())
- build_root = Path(checkout) / "target/smoke/builds"
- def within(path):
- try:
- path.relative_to(build_root.resolve(strict=True))
- return True
- except ValueError:
- return False
- artifacts = {}
- for item in report["artifacts"]:
- package = item["package"]
- matches = []
- for receipt_path in build_root.glob("*/" + package + "/receipt.json"):
- receipt = json.loads(receipt_path.read_text())
- artifact = receipt["artifact"]
- if (receipt["recipe"]["source_revision"] == sha
- and artifact["sha256"] == item["sha256"]):
- path = Path(receipt["path"]).resolve(strict=True)
- if not within(path):
- raise ValueError("Product artifact escapes prepared target")
- if "sha256:" + file_digest(path) != artifact["sha256"]:
- raise ValueError("Prepared product binary changed")
- matches.append(path)
- if len(matches) != 1:
- raise ValueError("Prepared product receipt is not unique: " + package)
- artifacts[package] = matches[0]
- gateway = []
- for receipt_path in build_root.glob("*/hiroute-gateway/receipt.json"):
- receipt = json.loads(receipt_path.read_text())
- attestation = receipt["attestation"]
- if (attestation["schema_version"] == "hiroute.e2e.sut-build-attestation/v3"
- and attestation["source_revision"] == sha
- and attestation["executable_sha256"] == report["gateway"]["executable_sha256"]):
- path = Path(attestation["executable_path"]).resolve(strict=True)
- if (not within(path)
- or "sha256:" + file_digest(path) != attestation["executable_sha256"]):
- raise ValueError("Prepared Gateway binary changed")
- gateway.append(path)
- if len(gateway) != 1:
- raise ValueError("Prepared Gateway receipt is not unique")
+ report, environment, artifacts = prepared_products(log, checkout, sha, jobs=32)
+ gateway = report["gateway"]
+ receipt, directory = preparation_receipt(gateway.get("receipt"),
+ Path(checkout).resolve() / "target", "hiroute-gateway")
+ attestation = receipt["attestation"]
+ if (gateway.get("schema") != "hiroute.e2e.current-preparation/v1"
+ or gateway.get("prepared") is not True or gateway.get("source_revision") != sha
+ or attestation["schema_version"] != "hiroute.e2e.sut-build-attestation/v3"
+ or attestation["source_revision"] != sha
+ or attestation["executable_sha256"] != gateway["executable_sha256"]
+ or receipt["recipe"] != "sha256:" + directory.parent.name):
+ raise ValueError("Prepared Gateway receipt does not match the candidate")
+ gateway_path = prepared_binary(attestation["executable_path"], directory, "hirouted",
+ gateway["executable_sha256"])
directory = Path(checkout) / "target/validation-schedule" / run_id / "products"
directory.mkdir(parents=True, mode=0o700, exist_ok=False)
for package, name in (("hiroute-cli", "hiroute"), ("hiroute-daemon", "hirouted")):
@@ -540,7 +550,7 @@ def dag_products(log, checkout, sha, run_id):
raise ValueError("Prepared product copy changed")
environment.update(HIROUTE_VALIDATION_EXECUTION="1",
HIROUTE_VALIDATION_PRODUCT_BIN_DIR=str(directory),
- HIROUTE_VALIDATION_GATEWAY_BIN=str(gateway[0]),
+ HIROUTE_VALIDATION_GATEWAY_BIN=str(gateway_path),
HIROUTE_SMOKE_REQUIRE_PREPARED="1")
return environment
@@ -649,7 +659,7 @@ def dag_run_targets(runner, root, run, targets, checkout, lease, environment, de
or sum(summary["counts"].values()) != len(item["cases"])
or dag_observed_cases(text) != set(item["cases"])):
raise ValueError("Test target did not complete its listed cases: " + name)
- return dict(name=name, log=str(path), process_exit=code,
+ return dict(name=name, index=item["index"], log=str(path), process_exit=code,
complete=summary["complete"] and code == 0,
counts=summary["counts"], exec_audit=audit,
started_at=started_at, finished_at=time.time(),
@@ -676,7 +686,9 @@ def dag_run_targets(runner, root, run, targets, checkout, lease, environment, de
try:
result = future.result()
except Exception as error:
- result = dict(name=item["name"], complete=False, process_exit=1,
+ result = dict(name=item["name"], index=item["index"],
+ log=str(run / ("dag-test-" + str(item["index"]) + ".log")),
+ complete=False, process_exit=1,
error=str(error), counts=dict(passed=0, failed=0, ignored=0))
results.append(result)
if not result["complete"] and fail_fast and stop_event is not None:
@@ -693,7 +705,10 @@ def execute_dag(runner, root, run, checkout, request, lease, output, reserved_ha
base = [arg for arg in request["command"] if arg != "--no-fail-fast"]
fail_fast = "--no-fail-fast" not in request["command"]
phase_records = []
- outcome = dict(process_exit=1, timed_out=False, phases=phase_records, complete=False)
+ outcome = dict(process_exit=1, timed_out=False, phases=phase_records, complete=False,
+ doctests_executed=False)
+ targets, results = [], []
+ expected = None
def cargo_phase(name, argv, slots):
path = run / ("dag-" + name + ".log")
@@ -777,9 +792,6 @@ def execute_dag(runner, root, run, checkout, request, lease, output, reserved_ha
fail_fast=fail_fast, reserved_handles=reserved_handles,
stop_event=stop_event)
if stop_event is not None and stop_event.is_set():
- for result in results:
- if result.get("log"):
- output.write(Path(result["log"]).read_text(errors="replace") + "\n")
outcome.update(expected_tests=expected, targets=len(targets),
executed_targets=len(results), target_results=results,
error="Isolated test target failed; remaining targets and doctests were not admitted")
@@ -789,37 +801,42 @@ def execute_dag(runner, root, run, checkout, request, lease, output, reserved_ha
ordinary = pool.submit(dag_run_targets, runner, root, run, independent, checkout,
lease, execution_env,
deadline, 4, 4, fail_fast, reserved_handles, stop_event)
- preparation_log = prepared.result()
- summary = runner.test_summary(preparation_log, commands(base)["prepare"])
- if summary["counts"] != dict(passed=1, failed=0, ignored=0):
- raise ValueError("Build preparation did not execute exactly once")
- prepared_env = dag_products(preparation_log, checkout, request["sha"], run.name)
- prepared_env["HIROUTE_VALIDATION_TOOLCHAIN_JSON"] = toolchain_artifact["path"]
- hooks_command = ["cargo", "build", "--locked", "-p", "hiroute-daemon",
- "--bin", "hirouted", "--features", "integration-test-hooks",
- "--profile", "dev", "--message-format=json"]
- hooks_log = cargo_phase("publication-daemon", hooks_command, 4)
- hooks_dir = dag_publication_daemon(hooks_log, checkout, metadata, run.name,
- prepared_env, request["sha"], run)
- for item in dependent:
- if item["name"] == "publication_process":
- item["environment"] = {
- "HIROUTE_VALIDATION_PRODUCT_BIN_DIR": str(hooks_dir)}
- remaining = dag_run_targets(runner, root, run, dependent, checkout, lease,
- dict(os.environ, **prepared_env), deadline,
- slots=4, workers=4, fail_fast=fail_fast,
- reserved_handles=reserved_handles,
- stop_event=stop_event)
- first = ordinary.result()
- results += first + remaining
- for result in sorted(results, key=lambda item: item.get("log", "")):
- if result.get("log"):
- output.write(Path(result["log"]).read_text(errors="replace") + "\n")
+ try:
+ preparation_log = prepared.result()
+ summary = runner.test_summary(preparation_log, commands(base)["prepare"])
+ if summary["counts"] != dict(passed=1, failed=0, ignored=0):
+ raise ValueError("Build preparation did not execute exactly once")
+ prepared_env = dag_products(preparation_log, checkout, request["sha"], run.name)
+ prepared_env["HIROUTE_VALIDATION_TOOLCHAIN_JSON"] = toolchain_artifact["path"]
+ hooks_command = ["cargo", "build", "--locked", "-p", "hiroute-daemon",
+ "--bin", "hirouted", "--features", "integration-test-hooks",
+ "--profile", "dev", "--message-format=json"]
+ hooks_log = cargo_phase("publication-daemon", hooks_command, 4)
+ hooks_dir = dag_publication_daemon(hooks_log, checkout, metadata, run.name,
+ prepared_env, request["sha"], run)
+ for item in dependent:
+ if item["name"] == "publication_process":
+ item["environment"] = {
+ "HIROUTE_VALIDATION_PRODUCT_BIN_DIR": str(hooks_dir)}
+ results.extend(dag_run_targets(runner, root, run, dependent, checkout, lease,
+ dict(os.environ, **prepared_env), deadline,
+ slots=4, workers=4, fail_fast=fail_fast,
+ reserved_handles=reserved_handles,
+ stop_event=stop_event))
+ except Exception:
+ if stop_event is not None:
+ stop_event.set()
+ raise
+ finally:
+ # Preparation can fail after independent targets already finished.
+ # Retain those results even when dependent admission never begins.
+ results.extend(ordinary.result())
if stop_event is not None and stop_event.is_set():
outcome.update(expected_tests=expected, targets=len(targets),
executed_targets=len(results), target_results=results,
error="Test target failed; remaining targets and doctests were not admitted")
return outcome
+ outcome["doctests_executed"] = True
doc_log = cargo_phase("doctest", [*base, "--doc"], 8)
output.write(doc_log + "\n")
doc_summary = lane_summary(runner.reporting(), doc_log)
@@ -843,4 +860,17 @@ def execute_dag(runner, root, run, checkout, request, lease, output, reserved_ha
except Exception as error:
outcome["error"] = str(error)
outcome["timed_out"] = time.monotonic() >= deadline
+ finally:
+ for result in sorted(results, key=lambda item: item.get("log", "")):
+ if result.get("log") and Path(result["log"]).is_file():
+ output.write(Path(result["log"]).read_text(errors="replace") + "\n")
+ if expected is not None:
+ executed = {row["index"] for row in results}
+ outcome.update(expected_tests=expected, targets=len(targets),
+ executed_targets=len(results), target_results=results,
+ unexecuted_targets=[dict(index=item["index"], name=item["name"],
+ package=item["package_name"], cases=item["cases"])
+ for item in targets if item["index"] not in executed])
+ outcome.setdefault("counts", {key: sum(row["counts"][key] for row in results)
+ for key in ("passed", "failed", "ignored")})
return outcome
diff --git a/tools/e2e-harness/src/p0/production/candidate.rs b/tools/e2e-harness/src/p0/production/candidate.rs
index 025d70dc..809297e6 100644
--- a/tools/e2e-harness/src/p0/production/candidate.rs
+++ b/tools/e2e-harness/src/p0/production/candidate.rs
@@ -475,9 +475,11 @@ pub fn prepare_current_candidate(root: &Path) -> Result<Value, ProductionError>
.as_ref()
.expect("current bundle has identity");
identity.verify_live_sut(&sut)?;
+ let receipt = sut.canonical_path.with_file_name("receipt.json");
Ok(
json!({"schema":"hiroute.e2e.current-preparation/v1", "prepared":true,
- "source_revision":identity.revision, "executable_sha256":sut.executable_sha256}),
+ "source_revision":identity.revision, "executable_sha256":sut.executable_sha256,
+ "receipt":{"path":receipt, "sha256":sha256_hex(&fs::read(&receipt)?)}}),
)
}
diff --git a/tools/product-e2e/src/smoke/preparation.rs b/tools/product-e2e/src/smoke/preparation.rs
index af5586ba..56348010 100644
--- a/tools/product-e2e/src/smoke/preparation.rs
+++ b/tools/product-e2e/src/smoke/preparation.rs
@@ -46,12 +46,23 @@ pub fn prepare(root: &Path, cancel: Arc<AtomicBool>) -> Result<PathBuf> {
"gateway_preparation_identity_changed",
)?;
tool.verify(&source)?;
+ // Bind the exact receipts selected by get_or_build, including their recipes.
+ // Distinct environments may legitimately produce identical executable bytes.
+ let mut records = Vec::new();
+ for artifact in &artifacts {
+ let receipt = artifact.path.with_file_name("receipt.json");
+ let mut record = serde_json::to_value(artifact)?;
+ record["receipt"] = json!({
+ "path":receipt, "sha256":super::digest(&std::fs::read(&receipt)?),
+ });
+ records.push(record);
+ }
let report = directory.join("preparation.json");
std::fs::write(
&report,
serde_json::to_vec_pretty(&json!({
"schema":"hiroute.smoke.preparation/v1", "source_revision":source.revision,
- "build_ms":started.elapsed().as_millis(), "artifacts":artifacts, "gateway":gateway,
+ "build_ms":started.elapsed().as_millis(), "artifacts":records, "gateway":gateway,
"scenarios_executed":0,
}))?,
)?;