Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.
Rust ai-agentsai-gatewayclaude-codecodexcoding-agentcost-optimizationdeepseekllm-gatewayllm-routerlocal-firstmodel-routingrusttask-routingtauri
fix(validation): exclude nested child counts from focused summaries (#196)
澄潭ece8e991a579
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.mdindex 55d563f7..94d9cc94 100644--- a/docs/code-map/testing.md+++ b/docs/code-map/testing.md@@ -218,6 +218,11 @@ Final backend DAG admission and preparation are owned byrun in an isolated lane before other listeners; Gateway runtime consumers wait forprepared product roles.+Ordinary local/SSH totals and module reports share Cargo-target attribution in+[validation-report.py](../../scripts/validation-report.py). The+[runner regressions](../../scripts/test-remote-rust.py) exercise worker results and+compact waits with nested child summaries, failed and unfinished parents, and multiple targets.+Shared private filesystem behavior has one [platform contract suite](../../crates/platform/src/private_fs/tests.rs): exclusive creation, protected handle reads, permission preservation and identity-bound cleanup. Storage and Worker tests keep domain errors, recovery and lifecycle assertions; the native Windows cases additionally exercise ACLs and held handles.The [Windows iteration route](../validation-routing.md#windows-iteration-and-release-validation)diff --git a/docs/validation-feedback.md b/docs/validation-feedback.mdindex 74bab76b..ee7d9f79 100644--- a/docs/validation-feedback.md+++ b/docs/validation-feedback.md@@ -111,6 +111,12 @@ with build features, so only unique logical target identities are compared acrosUnattributed summaries cannot establish module coverage. A successful Cargo process isnot a Product scenario green verdict, and zero/ignored cases are not coverage evidence.+Ordinary local/SSH run results and compact SSH waits use the same complete Cargo-target+counts as the module report. Nested child summaries stay in the command log for+diagnosis, without adding independent cases. An unfinished parent stays incomplete+even when a child's selected count matches its own; a later successful target cannot+erase an earlier failure. Process exit is retained separately from these test counts.+Outer compile events, nested/later build events, execution times, queue/setup/checkouttimings and total time use separate fields. They must not be blindly added together.Unknown time is not zero. Smoke's existing `build_ms`, `execution_ms` anddiff --git a/issue-spec/specs/product-smoke-verification/spec.md b/issue-spec/specs/product-smoke-verification/spec.mdindex 916b3780..e1ef6c2f 100644--- a/issue-spec/specs/product-smoke-verification/spec.md+++ b/issue-spec/specs/product-smoke-verification/spec.md@@ -36,6 +36,25 @@ is not-executed with reason, not green or executed expected_red. Required red/miexpected-red capability prevents acceptance even if a fixture test legitimately exitszero. Existing runner result contracts cannot be weakened to obtain green.+Ordinary Rust validation results, compact waits, and module reports MUST count+independent cases from complete Cargo-target terminal summaries, excluding nested+child-process summaries. Child assertions and diagnostic output MUST remain available.+Unattributed output, ambiguous target identity, or an unfinished parent MUST NOT+establish successful test execution, even if a child's test count matches its parent.+Every completed target contributes its own passing, failing and ignored counts;+a later successful target MUST NOT erase an earlier failed target. A nonzero command+exit MUST fail the run independently of otherwise successful test summaries.++#### Scenario: Nested subprocesses finish inside a successful parent target++- **WHEN** a Cargo target completes its selected parent tests after printing nested child summaries+- **THEN** run results, compact waits and module reports expose the parent target counts, retaining the child output without adding independent coverage.++#### Scenario: A child finishes but its parent target is interrupted++- **WHEN** the final observed child summary has no completed parent summary, including when both selected the same number of tests+- **THEN** validation remains incomplete and the child summary cannot establish parent success or independent parent counts.+### Requirement: Evidence binds actual revision, artifact, role, and pathReport MUST include checked source revision, build/executable identity, platform,diff --git a/scripts/remote-rust.py b/scripts/remote-rust.pyindex 96e0ad00..88ae7b6d 100644--- a/scripts/remote-rust.py+++ b/scripts/remote-rust.py@@ -483,22 +483,26 @@ def gc_terminal_runs(root, older_than_days, apply, include_retained):def test_summary(log, cargo):- matches = re.findall(r"test result: \w+\. (\d+) passed; (\d+) failed; (\d+) ignored", log)- counts = dict(zip(("passed", "failed", "ignored"),- [sum(int(row[i]) for row in matches) for i in range(3)]))+ parsed = reporting().parse_log(log)+ modules = parsed["modules"]+ # Only complete, uniquely attributed Cargo targets are independent results.+ # A child summary may be the last observed output of an interrupted parent.+ counts = {key: sum(module["counts"][key] for module in modules if module["complete"])+ for key in ("passed", "failed", "ignored")}if cargo[1] != "test" or "--no-run" in cargo:state = "not_assessed"- elif not matches:- state = "unrecognized_test_output"elif counts["failed"]:state = "tests_failed"+ elif not modules or parsed["unassigned_summaries"] or not all(module["complete"] for module in modules):+ state = "unrecognized_test_output"elif not counts["passed"]:state = "zero_tests_passed"else:state = "rust_tests_passed"return {"counts": counts, "state": state,"scenario_state": "not_assessed",- "note": "Rust counts do not establish Product E2E green/expected_red/red; inspect scenario artifacts."}+ "note": "Counts use complete Cargo targets and exclude nested child summaries. "+ "Rust counts do not establish Product E2E green/expected_red/red; inspect scenario artifacts."}def scheduled_test_summary(log, cargo, schedule_result=None):diff --git a/scripts/test-local-rust.py b/scripts/test-local-rust.pyindex 2b69793f..f7528b8d 100644--- a/scripts/test-local-rust.py+++ b/scripts/test-local-rust.py@@ -189,6 +189,8 @@ class Cleanup(unittest.TestCase):' print("cargo fixture")\n''else:\n'' (target/"debug").mkdir(parents=True)\n'+ ' print("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)")\n'+ ' print("running 1 test")\n'' print("test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out")\n')cargo.chmod(0o700)for name in ('rustc', 'sccache'):@@ -536,6 +538,8 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))' (p/"debug").mkdir(parents=True)\n'' (p/"debug/binary").write_text("fixture")\n'' import time; time.sleep(' + str(delay) + ')\n'+ ' print("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)")\n'+ ' print("running 1 test")\n'' print("test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out")\n'' sys.exit(' + str(exit_code) + ')\n')cargo.chmod(0o700)@@ -585,8 +589,8 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))report = json.loads(Path(result['validation_report']['path']).read_text())self.assertEqual(report['sha'], result['sha'])self.assertEqual(report['process_exit'], 0)- # This fake tool prints only an unattributed summary; it is not target evidence.- self.assertFalse(report['complete'])+ self.assertTrue(report['complete'])+ self.assertEqual(report['modules'][0]['counts'], dict(passed=1, failed=0, ignored=0))self.assertIsNotNone(report['timing']['command_seconds'])def test_invalid_jobs_refuses_before_run_record(self):diff --git a/scripts/test-remote-rust.py b/scripts/test-remote-rust.pyindex 7fb59e0c..62344082 100644--- a/scripts/test-remote-rust.py+++ b/scripts/test-remote-rust.py@@ -1,6 +1,7 @@#!/usr/bin/env python3"""Non-compiling tests for the SSH workbench contract."""import importlib.util+import contextlibimport fcntlfrom pathlib import Pathimport signal@@ -623,11 +624,13 @@ sys.exit(101 if bad else 0)runner.validate(ref, sha, ["cargo", "test"])def test_zero_selected_is_not_success(self):- result = runner.test_summary("test result: ok. 0 passed; 0 failed; 80 ignored", ["cargo", "test"])+ result = runner.test_summary("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)\n"+ "running 80 tests\ntest result: ok. 0 passed; 0 failed; 80 ignored", ["cargo", "test"])self.assertEqual(result["state"], "zero_tests_passed")def test_test_exit_is_not_scenario_verdict(self):- result = runner.test_summary("test result: ok. 80 passed; 0 failed; 0 ignored", ["cargo", "test"])+ result = runner.test_summary("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)\n"+ "running 80 tests\ntest result: ok. 80 passed; 0 failed; 0 ignored", ["cargo", "test"])self.assertEqual(result["counts"]["passed"], 80)self.assertEqual(result["scenario_state"], "not_assessed")@@ -644,6 +647,103 @@ sys.exit(101 if bad else 0)def test_compile_is_not_test(self):self.assertEqual(runner.test_summary("", ["cargo", "check"])["state"], "not_assessed")+ def test_focused_worker_result_and_wait_use_parent_target_counts(self):+ # Real worker, Git checkouts, command processes and compact wait; fake Cargo.+ parent = ("Running unittests src/lib.rs (target/debug/deps/parent-abcdef0123456789)\n"+ "running 2 tests\nrunning 1 test\ntest child ... ok\n"+ "test result: ok. 1 passed; 0 failed; 0 ignored; finished in 0.01s\n"+ "test parent ... ok\ntest sibling ... ok\n"+ "test result: ok. 2 passed; 0 failed; 0 ignored; finished in 0.02s\n")+ other = ("Running tests/other.rs (target/debug/deps/other-abcdef0123456789)\n"+ "running 3 tests\ntest one ... ok\ntest two ... ok\ntest three ... ok\n"+ "test result: ok. 3 passed; 0 failed; 0 ignored; finished in 0.01s\n")+ failed = parent.replace("test sibling ... ok", "test sibling ... FAILED").replace(+ "test result: ok. 2 passed; 0 failed", "test result: FAILED. 1 passed; 1 failed")+ interrupted = parent[:parent.index("test parent ... ok")]+ zero = ("Running tests/zero.rs (target/debug/deps/zero-abcdef0123456789)\n"+ "running 0 tests\ntest result: ok. 0 passed; 0 failed; 0 ignored; finished in 0.00s\n")+ cases = [+ ("nested", parent, 0, "completed", "rust_tests_passed", (2, 0, 0)),+ ("expected-child-failure", parent.replace("test child ... ok", "test child ... FAILED").replace(+ "test result: ok. 1 passed; 0 failed", "test result: FAILED. 0 passed; 1 failed"),+ 0, "completed", "rust_tests_passed", (2, 0, 0)),+ ("failed-parent", failed, 101, "failed", "tests_failed", (1, 1, 0)),+ ("incomplete-parent", interrupted, 0, "failed", "unrecognized_test_output", (0, 0, 0)),+ ("same-sized-child", interrupted.replace("running 2 tests", "running 1 test"),+ 0, "failed", "unrecognized_test_output", (0, 0, 0)),+ ("zero-selected", zero, 0, "failed", "zero_tests_passed", (0, 0, 0)),+ ("ignored-only", zero.replace("running 0 tests", "running 2 tests").replace(+ "0 ignored", "2 ignored"), 0, "failed", "zero_tests_passed", (0, 0, 2)),+ ("multiple-targets", parent + other, 0, "completed", "rust_tests_passed", (5, 0, 0)),+ ("early-failure", failed + other, 101, "failed", "tests_failed", (4, 1, 0)),+ ("later-incomplete", parent + other[:other.index("test one")],+ 0, "failed", "unrecognized_test_output", (2, 0, 0)),+ ("unassigned", "test result: ok. 1 passed; 0 failed; 0 ignored\n" + parent,+ 0, "failed", "unrecognized_test_output", (2, 0, 0)),+ ("repeated-identity", parent + parent, 0, "failed", "unrecognized_test_output", (0, 0, 0)),+ ("nonzero-exit", parent, 7, "failed", "rust_tests_passed", (2, 0, 0)),+ ]+ with tempfile.TemporaryDirectory(dir='/tmp') as directory:+ root = Path(directory)+ source, home, workbench = root / 'source', root / 'home', root / 'workbench'+ subprocess.run(['git', 'init', '-q', str(source)], check=True)+ git = ['git', '-C', str(source)]+ subprocess.run([*git, '-c', 'user.name=Fixture', '-c', '[email protected]',+ '-c', 'commit.gpgsign=false', 'commit', '--allow-empty', '-qm', 'fixture'], check=True)+ sha = subprocess.check_output([*git, 'rev-parse', 'HEAD'], text=True).strip()+ ref = subprocess.check_output([*git, 'symbolic-ref', 'HEAD'], text=True).strip()+ binary = home / '.cargo/bin'+ binary.mkdir(parents=True)+ fake = '''import sys,json,pathlib+home=pathlib.Path(__file__).parents[2]+if '--version' in sys.argv: print('fixture tool'); sys.exit(0)+if 'metadata' in sys.argv:+ print(json.dumps({'target_directory':str(pathlib.Path.cwd()/'target')})); sys.exit(0)+print((home/'output.log').read_text(),end='')+sys.exit(int((home/'exit-code').read_text()))+'''+ for name in ('cargo', 'rustc'):+ path = binary / name+ path.write_text('#!' + sys.executable + '\n' + fake)+ path.chmod(0o700)+ for index, (name, log, exit_code, status, state, counts) in enumerate(cases):+ with self.subTest(case=name):+ (home / 'output.log').write_text(log)+ (home / 'exit-code').write_text(str(exit_code))+ key = f'20261021-0000{index:02}-abcdef12'+ run = workbench / 'runs' / key+ run.mkdir(parents=True)+ request = dict(id=key, ref=ref, sha=sha, executor_sha=sha, origin=str(source),+ command=['cargo', 'test', '--no-fail-fast'], retention='always',+ exclusive=False, timeout=10, plan='focused-counts', phase='focused')+ runner.save(run / 'request.json', request)+ with patch.object(runner.Path, 'home', return_value=home), \+ patch.dict(os.environ, {}, clear=False), contextlib.redirect_stdout(io.StringIO()), \+ patch.object(runner.tempfile, 'gettempdir', return_value=str(root.resolve())), \+ patch.object(runner, 'MIN_FREE_BYTES', 0), \+ patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}):+ runner.worker(run / 'request.json')+ result = json.loads((run / 'result.json').read_text())+ if result.get('temporary_directory'):+ runner.cleanup_temp_directory(result['temporary_directory'])+ observed = subprocess.run([sys.executable, '-c', runner.WAIT_BOOTSTRAP,+ key, '0', str(workbench)], capture_output=True,+ text=True, check=True)+ compact = json.loads(observed.stdout)+ self.assertEqual(result['status'], status, result)+ self.assertEqual(result['process_exit'], exit_code)+ self.assertEqual(result['tests']['state'], state)+ self.assertEqual(result['tests']['counts'], dict(zip(('passed', 'failed', 'ignored'), counts)))+ self.assertEqual(result['tests']['scenario_state'], 'not_assessed')+ self.assertEqual(compact['status'], status)+ self.assertEqual(compact['process_exit'], exit_code)+ self.assertEqual(compact['tests'], {k: result['tests'][k] for k in ('state', 'counts')})+ self.assertTrue((run / 'command.log').read_text().endswith(log))+ if status == 'completed':+ report = json.loads((run / 'validation-report.json').read_text())+ self.assertTrue(report['complete'])+ self.assertEqual(sum(m['counts']['passed'] for m in report['modules']), counts[0])+def test_atomic_result(self):with tempfile.TemporaryDirectory() as directory:path = Path(directory) / "result.json"diff --git a/scripts/test-validation-report.py b/scripts/test-validation-report.pyindex abb1960e..bd053307 100644--- a/scripts/test-validation-report.py+++ b/scripts/test-validation-report.py@@ -46,13 +46,15 @@ class Parsing(unittest.TestCase):self.assertEqual(result["build_events"], [dict(seconds=69., line=1, scope="outer")])def test_child_summary_does_not_complete_interrupted_parent(self):- log = ("Running unittests src/lib.rs (target/debug/deps/hiroute_daemon-abcdef0123456789)\n"- "running 2 tests\nrunning 1 test\ntest child ... ok\n"- "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20s\n")- result = report.build_report(row(process_exit=-9), log)- self.assertFalse(result["complete"])- self.assertFalse(result["modules"][0]["complete"])- self.assertEqual(result["top_modules"], [])+ for count in (1, 2):+ with self.subTest(parent_count=count):+ log = ("Running unittests src/lib.rs (target/debug/deps/hiroute_daemon-abcdef0123456789)\n"+ f"running {count} tests\nrunning 1 test\ntest child ... ok\n"+ "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20s\n")+ result = report.build_report(row(process_exit=-9), log)+ self.assertFalse(result["complete"])+ self.assertFalse(result["modules"][0]["complete"])+ self.assertEqual(result["top_modules"], [])def test_failure_and_unfinished_target_remain_distinct(self):log = target(failed=True) + "Running tests/next.rs (target/debug/deps/next-abcdef0123456789)\nrunning 1 test\n"diff --git a/scripts/test-validation-schedule.py b/scripts/test-validation-schedule.pyindex 82b2b615..3122723a 100644--- a/scripts/test-validation-schedule.py+++ b/scripts/test-validation-schedule.py@@ -526,6 +526,8 @@ if 'prepare_default_smoke_builds' in a:report.parent.mkdir(parents=True)report.write_text(json.dumps({'schema':'hiroute.smoke.preparation/v1','source_revision':'a'*40,'scenarios_executed':0,'artifacts':artifacts}))print('build preparation='+str(report))+ print('Running tests/smoke_cli.rs (target/debug/deps/smoke_cli-deadbeef01234567)')+ print(f'running {n} tests')print(f'test result: ok. {n} passed; 0 failed; 0 ignored'); sys.exit(0)assert (root/'prepared').exists()assert '--skip' in adiff --git a/scripts/validation-report.py b/scripts/validation-report.pyindex 9ab42f9f..12578caa 100644--- a/scripts/validation-report.py+++ b/scripts/validation-report.py@@ -75,6 +75,7 @@ def parse_log(log):"""modules, builds, unassigned = [], [], []current = None+ open_harnesses = 0for number, raw in enumerate(log.splitlines(), 1):line = ANSI.sub("", raw)header = HEADER.match(line)@@ -91,9 +92,12 @@ def parse_log(log):current = dict(module=key, identity=identity, label=label, line=number, cases={}, summaries=[],seconds=None, counts=None, result=None, complete=False, expected_count=None)modules.append(current)+ open_harnesses = 0running = re.fullmatch(r"running (\d+) tests?", line)- if running and current is not None and current["expected_count"] is None:- current["expected_count"] = int(running[1])+ if running and current is not None:+ open_harnesses += 1+ if current["expected_count"] is None:+ current["expected_count"] = int(running[1])summary = SUMMARY.match(line)if summary:state, passed, failed, ignored = summary.groups()@@ -101,9 +105,12 @@ def parse_log(log):row = dict(result=state, counts=dict(passed=int(passed), failed=int(failed), ignored=int(ignored)),seconds=float(duration[1]) if duration else None, summary_line=number)if current is not None:+ # A same-sized child cannot close a still-running parent harness.+ open_harnesses = max(0, open_harnesses - 1)current["summaries"].append(row)current.update(row)- current["complete"] = current["expected_count"] is not None and sum(row["counts"].values()) == current["expected_count"]+ current["complete"] = (open_harnesses == 0 and current["expected_count"] is not None+ and sum(row["counts"].values()) == current["expected_count"])else:unassigned.append(row)case = CASE.match(line)