Skip to content

Make Codex and Claude Code subscriptions go further: a local model router that gives routine agent work to economical models and keeps your strongest model for the hard parts.

fix(validation): exclude nested child counts from focused summaries (#196)

澄潭ece8e991a579
diff --git a/docs/code-map/testing.md b/docs/code-map/testing.md
index 55d563f7..94d9cc94 100644
--- a/docs/code-map/testing.md
+++ b/docs/code-map/testing.md
@@ -218,6 +218,11 @@ Final backend DAG admission and preparation are owned by
run in an isolated lane before other listeners; Gateway runtime consumers wait for
prepared product roles.
+Ordinary local/SSH totals and module reports share Cargo-target attribution in
+[validation-report.py](../../scripts/validation-report.py). The
+[runner regressions](../../scripts/test-remote-rust.py) exercise worker results and
+compact waits with nested child summaries, failed and unfinished parents, and multiple targets.
+
Shared private filesystem behavior has one [platform contract suite](../../crates/platform/src/private_fs/tests.rs): exclusive creation, protected handle reads, permission preservation and identity-bound cleanup. Storage and Worker tests keep domain errors, recovery and lifecycle assertions; the native Windows cases additionally exercise ACLs and held handles.
The [Windows iteration route](../validation-routing.md#windows-iteration-and-release-validation)
diff --git a/docs/validation-feedback.md b/docs/validation-feedback.md
index 74bab76b..ee7d9f79 100644
--- a/docs/validation-feedback.md
+++ b/docs/validation-feedback.md
@@ -111,6 +111,12 @@ with build features, so only unique logical target identities are compared acros
Unattributed summaries cannot establish module coverage. A successful Cargo process is
not a Product scenario green verdict, and zero/ignored cases are not coverage evidence.
+Ordinary local/SSH run results and compact SSH waits use the same complete Cargo-target
+counts as the module report. Nested child summaries stay in the command log for
+diagnosis, without adding independent cases. An unfinished parent stays incomplete
+even when a child's selected count matches its own; a later successful target cannot
+erase an earlier failure. Process exit is retained separately from these test counts.
+
Outer compile events, nested/later build events, execution times, queue/setup/checkout
timings and total time use separate fields. They must not be blindly added together.
Unknown time is not zero. Smoke's existing `build_ms`, `execution_ms` and
diff --git a/issue-spec/specs/product-smoke-verification/spec.md b/issue-spec/specs/product-smoke-verification/spec.md
index 916b3780..e1ef6c2f 100644
--- a/issue-spec/specs/product-smoke-verification/spec.md
+++ b/issue-spec/specs/product-smoke-verification/spec.md
@@ -36,6 +36,25 @@ is not-executed with reason, not green or executed expected_red. Required red/mi
expected-red capability prevents acceptance even if a fixture test legitimately exits
zero. Existing runner result contracts cannot be weakened to obtain green.
+Ordinary Rust validation results, compact waits, and module reports MUST count
+independent cases from complete Cargo-target terminal summaries, excluding nested
+child-process summaries. Child assertions and diagnostic output MUST remain available.
+Unattributed output, ambiguous target identity, or an unfinished parent MUST NOT
+establish successful test execution, even if a child's test count matches its parent.
+Every completed target contributes its own passing, failing and ignored counts;
+a later successful target MUST NOT erase an earlier failed target. A nonzero command
+exit MUST fail the run independently of otherwise successful test summaries.
+
+#### Scenario: Nested subprocesses finish inside a successful parent target
+
+- **WHEN** a Cargo target completes its selected parent tests after printing nested child summaries
+- **THEN** run results, compact waits and module reports expose the parent target counts, retaining the child output without adding independent coverage.
+
+#### Scenario: A child finishes but its parent target is interrupted
+
+- **WHEN** the final observed child summary has no completed parent summary, including when both selected the same number of tests
+- **THEN** validation remains incomplete and the child summary cannot establish parent success or independent parent counts.
+
### Requirement: Evidence binds actual revision, artifact, role, and path
Report MUST include checked source revision, build/executable identity, platform,
diff --git a/scripts/remote-rust.py b/scripts/remote-rust.py
index 96e0ad00..88ae7b6d 100644
--- a/scripts/remote-rust.py
+++ b/scripts/remote-rust.py
@@ -483,22 +483,26 @@ def gc_terminal_runs(root, older_than_days, apply, include_retained):
def test_summary(log, cargo):
- matches = re.findall(r"test result: \w+\. (\d+) passed; (\d+) failed; (\d+) ignored", log)
- counts = dict(zip(("passed", "failed", "ignored"),
- [sum(int(row[i]) for row in matches) for i in range(3)]))
+ parsed = reporting().parse_log(log)
+ modules = parsed["modules"]
+ # Only complete, uniquely attributed Cargo targets are independent results.
+ # A child summary may be the last observed output of an interrupted parent.
+ counts = {key: sum(module["counts"][key] for module in modules if module["complete"])
+ for key in ("passed", "failed", "ignored")}
if cargo[1] != "test" or "--no-run" in cargo:
state = "not_assessed"
- elif not matches:
- state = "unrecognized_test_output"
elif counts["failed"]:
state = "tests_failed"
+ elif not modules or parsed["unassigned_summaries"] or not all(module["complete"] for module in modules):
+ state = "unrecognized_test_output"
elif not counts["passed"]:
state = "zero_tests_passed"
else:
state = "rust_tests_passed"
return {"counts": counts, "state": state,
"scenario_state": "not_assessed",
- "note": "Rust counts do not establish Product E2E green/expected_red/red; inspect scenario artifacts."}
+ "note": "Counts use complete Cargo targets and exclude nested child summaries. "
+ "Rust counts do not establish Product E2E green/expected_red/red; inspect scenario artifacts."}
def scheduled_test_summary(log, cargo, schedule_result=None):
diff --git a/scripts/test-local-rust.py b/scripts/test-local-rust.py
index 2b69793f..f7528b8d 100644
--- a/scripts/test-local-rust.py
+++ b/scripts/test-local-rust.py
@@ -189,6 +189,8 @@ class Cleanup(unittest.TestCase):
' print("cargo fixture")\n'
'else:\n'
' (target/"debug").mkdir(parents=True)\n'
+ ' print("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)")\n'
+ ' print("running 1 test")\n'
' print("test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out")\n')
cargo.chmod(0o700)
for name in ('rustc', 'sccache'):
@@ -536,6 +538,8 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
' (p/"debug").mkdir(parents=True)\n'
' (p/"debug/binary").write_text("fixture")\n'
' import time; time.sleep(' + str(delay) + ')\n'
+ ' print("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)")\n'
+ ' print("running 1 test")\n'
' print("test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out")\n'
' sys.exit(' + str(exit_code) + ')\n')
cargo.chmod(0o700)
@@ -585,8 +589,8 @@ print(json.dumps({'preview':preview['state'],'apply_rejected':rejected}))
report = json.loads(Path(result['validation_report']['path']).read_text())
self.assertEqual(report['sha'], result['sha'])
self.assertEqual(report['process_exit'], 0)
- # This fake tool prints only an unattributed summary; it is not target evidence.
- self.assertFalse(report['complete'])
+ self.assertTrue(report['complete'])
+ self.assertEqual(report['modules'][0]['counts'], dict(passed=1, failed=0, ignored=0))
self.assertIsNotNone(report['timing']['command_seconds'])
def test_invalid_jobs_refuses_before_run_record(self):
diff --git a/scripts/test-remote-rust.py b/scripts/test-remote-rust.py
index 7fb59e0c..62344082 100644
--- a/scripts/test-remote-rust.py
+++ b/scripts/test-remote-rust.py
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
"""Non-compiling tests for the SSH workbench contract."""
import importlib.util
+import contextlib
import fcntl
from pathlib import Path
import signal
@@ -623,11 +624,13 @@ sys.exit(101 if bad else 0)
runner.validate(ref, sha, ["cargo", "test"])
def test_zero_selected_is_not_success(self):
- result = runner.test_summary("test result: ok. 0 passed; 0 failed; 80 ignored", ["cargo", "test"])
+ result = runner.test_summary("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)\n"
+ "running 80 tests\ntest result: ok. 0 passed; 0 failed; 80 ignored", ["cargo", "test"])
self.assertEqual(result["state"], "zero_tests_passed")
def test_test_exit_is_not_scenario_verdict(self):
- result = runner.test_summary("test result: ok. 80 passed; 0 failed; 0 ignored", ["cargo", "test"])
+ result = runner.test_summary("Running tests/fixture.rs (target/debug/deps/fixture-abcdef0123456789)\n"
+ "running 80 tests\ntest result: ok. 80 passed; 0 failed; 0 ignored", ["cargo", "test"])
self.assertEqual(result["counts"]["passed"], 80)
self.assertEqual(result["scenario_state"], "not_assessed")
@@ -644,6 +647,103 @@ sys.exit(101 if bad else 0)
def test_compile_is_not_test(self):
self.assertEqual(runner.test_summary("", ["cargo", "check"])["state"], "not_assessed")
+ def test_focused_worker_result_and_wait_use_parent_target_counts(self):
+ # Real worker, Git checkouts, command processes and compact wait; fake Cargo.
+ parent = ("Running unittests src/lib.rs (target/debug/deps/parent-abcdef0123456789)\n"
+ "running 2 tests\nrunning 1 test\ntest child ... ok\n"
+ "test result: ok. 1 passed; 0 failed; 0 ignored; finished in 0.01s\n"
+ "test parent ... ok\ntest sibling ... ok\n"
+ "test result: ok. 2 passed; 0 failed; 0 ignored; finished in 0.02s\n")
+ other = ("Running tests/other.rs (target/debug/deps/other-abcdef0123456789)\n"
+ "running 3 tests\ntest one ... ok\ntest two ... ok\ntest three ... ok\n"
+ "test result: ok. 3 passed; 0 failed; 0 ignored; finished in 0.01s\n")
+ failed = parent.replace("test sibling ... ok", "test sibling ... FAILED").replace(
+ "test result: ok. 2 passed; 0 failed", "test result: FAILED. 1 passed; 1 failed")
+ interrupted = parent[:parent.index("test parent ... ok")]
+ zero = ("Running tests/zero.rs (target/debug/deps/zero-abcdef0123456789)\n"
+ "running 0 tests\ntest result: ok. 0 passed; 0 failed; 0 ignored; finished in 0.00s\n")
+ cases = [
+ ("nested", parent, 0, "completed", "rust_tests_passed", (2, 0, 0)),
+ ("expected-child-failure", parent.replace("test child ... ok", "test child ... FAILED").replace(
+ "test result: ok. 1 passed; 0 failed", "test result: FAILED. 0 passed; 1 failed"),
+ 0, "completed", "rust_tests_passed", (2, 0, 0)),
+ ("failed-parent", failed, 101, "failed", "tests_failed", (1, 1, 0)),
+ ("incomplete-parent", interrupted, 0, "failed", "unrecognized_test_output", (0, 0, 0)),
+ ("same-sized-child", interrupted.replace("running 2 tests", "running 1 test"),
+ 0, "failed", "unrecognized_test_output", (0, 0, 0)),
+ ("zero-selected", zero, 0, "failed", "zero_tests_passed", (0, 0, 0)),
+ ("ignored-only", zero.replace("running 0 tests", "running 2 tests").replace(
+ "0 ignored", "2 ignored"), 0, "failed", "zero_tests_passed", (0, 0, 2)),
+ ("multiple-targets", parent + other, 0, "completed", "rust_tests_passed", (5, 0, 0)),
+ ("early-failure", failed + other, 101, "failed", "tests_failed", (4, 1, 0)),
+ ("later-incomplete", parent + other[:other.index("test one")],
+ 0, "failed", "unrecognized_test_output", (2, 0, 0)),
+ ("unassigned", "test result: ok. 1 passed; 0 failed; 0 ignored\n" + parent,
+ 0, "failed", "unrecognized_test_output", (2, 0, 0)),
+ ("repeated-identity", parent + parent, 0, "failed", "unrecognized_test_output", (0, 0, 0)),
+ ("nonzero-exit", parent, 7, "failed", "rust_tests_passed", (2, 0, 0)),
+ ]
+ with tempfile.TemporaryDirectory(dir='/tmp') as directory:
+ root = Path(directory)
+ source, home, workbench = root / 'source', root / 'home', root / 'workbench'
+ subprocess.run(['git', 'init', '-q', str(source)], check=True)
+ git = ['git', '-C', str(source)]
+ subprocess.run([*git, '-c', 'user.name=Fixture', '-c', '[email protected]',
+ '-c', 'commit.gpgsign=false', 'commit', '--allow-empty', '-qm', 'fixture'], check=True)
+ sha = subprocess.check_output([*git, 'rev-parse', 'HEAD'], text=True).strip()
+ ref = subprocess.check_output([*git, 'symbolic-ref', 'HEAD'], text=True).strip()
+ binary = home / '.cargo/bin'
+ binary.mkdir(parents=True)
+ fake = '''import sys,json,pathlib
+home=pathlib.Path(__file__).parents[2]
+if '--version' in sys.argv: print('fixture tool'); sys.exit(0)
+if 'metadata' in sys.argv:
+ print(json.dumps({'target_directory':str(pathlib.Path.cwd()/'target')})); sys.exit(0)
+print((home/'output.log').read_text(),end='')
+sys.exit(int((home/'exit-code').read_text()))
+'''
+ for name in ('cargo', 'rustc'):
+ path = binary / name
+ path.write_text('#!' + sys.executable + '\n' + fake)
+ path.chmod(0o700)
+ for index, (name, log, exit_code, status, state, counts) in enumerate(cases):
+ with self.subTest(case=name):
+ (home / 'output.log').write_text(log)
+ (home / 'exit-code').write_text(str(exit_code))
+ key = f'20261021-0000{index:02}-abcdef12'
+ run = workbench / 'runs' / key
+ run.mkdir(parents=True)
+ request = dict(id=key, ref=ref, sha=sha, executor_sha=sha, origin=str(source),
+ command=['cargo', 'test', '--no-fail-fast'], retention='always',
+ exclusive=False, timeout=10, plan='focused-counts', phase='focused')
+ runner.save(run / 'request.json', request)
+ with patch.object(runner.Path, 'home', return_value=home), \
+ patch.dict(os.environ, {}, clear=False), contextlib.redirect_stdout(io.StringIO()), \
+ patch.object(runner.tempfile, 'gettempdir', return_value=str(root.resolve())), \
+ patch.object(runner, 'MIN_FREE_BYTES', 0), \
+ patch.object(runner, 'stats', return_value={'max_cache_size': 30 * 1024**3}):
+ runner.worker(run / 'request.json')
+ result = json.loads((run / 'result.json').read_text())
+ if result.get('temporary_directory'):
+ runner.cleanup_temp_directory(result['temporary_directory'])
+ observed = subprocess.run([sys.executable, '-c', runner.WAIT_BOOTSTRAP,
+ key, '0', str(workbench)], capture_output=True,
+ text=True, check=True)
+ compact = json.loads(observed.stdout)
+ self.assertEqual(result['status'], status, result)
+ self.assertEqual(result['process_exit'], exit_code)
+ self.assertEqual(result['tests']['state'], state)
+ self.assertEqual(result['tests']['counts'], dict(zip(('passed', 'failed', 'ignored'), counts)))
+ self.assertEqual(result['tests']['scenario_state'], 'not_assessed')
+ self.assertEqual(compact['status'], status)
+ self.assertEqual(compact['process_exit'], exit_code)
+ self.assertEqual(compact['tests'], {k: result['tests'][k] for k in ('state', 'counts')})
+ self.assertTrue((run / 'command.log').read_text().endswith(log))
+ if status == 'completed':
+ report = json.loads((run / 'validation-report.json').read_text())
+ self.assertTrue(report['complete'])
+ self.assertEqual(sum(m['counts']['passed'] for m in report['modules']), counts[0])
+
def test_atomic_result(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "result.json"
diff --git a/scripts/test-validation-report.py b/scripts/test-validation-report.py
index abb1960e..bd053307 100644
--- a/scripts/test-validation-report.py
+++ b/scripts/test-validation-report.py
@@ -46,13 +46,15 @@ class Parsing(unittest.TestCase):
self.assertEqual(result["build_events"], [dict(seconds=69., line=1, scope="outer")])
def test_child_summary_does_not_complete_interrupted_parent(self):
- log = ("Running unittests src/lib.rs (target/debug/deps/hiroute_daemon-abcdef0123456789)\n"
- "running 2 tests\nrunning 1 test\ntest child ... ok\n"
- "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20s\n")
- result = report.build_report(row(process_exit=-9), log)
- self.assertFalse(result["complete"])
- self.assertFalse(result["modules"][0]["complete"])
- self.assertEqual(result["top_modules"], [])
+ for count in (1, 2):
+ with self.subTest(parent_count=count):
+ log = ("Running unittests src/lib.rs (target/debug/deps/hiroute_daemon-abcdef0123456789)\n"
+ f"running {count} tests\nrunning 1 test\ntest child ... ok\n"
+ "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20s\n")
+ result = report.build_report(row(process_exit=-9), log)
+ self.assertFalse(result["complete"])
+ self.assertFalse(result["modules"][0]["complete"])
+ self.assertEqual(result["top_modules"], [])
def test_failure_and_unfinished_target_remain_distinct(self):
log = target(failed=True) + "Running tests/next.rs (target/debug/deps/next-abcdef0123456789)\nrunning 1 test\n"
diff --git a/scripts/test-validation-schedule.py b/scripts/test-validation-schedule.py
index 82b2b615..3122723a 100644
--- a/scripts/test-validation-schedule.py
+++ b/scripts/test-validation-schedule.py
@@ -526,6 +526,8 @@ if 'prepare_default_smoke_builds' in a:
report.parent.mkdir(parents=True)
report.write_text(json.dumps({'schema':'hiroute.smoke.preparation/v1','source_revision':'a'*40,'scenarios_executed':0,'artifacts':artifacts}))
print('build preparation='+str(report))
+ print('Running tests/smoke_cli.rs (target/debug/deps/smoke_cli-deadbeef01234567)')
+ print(f'running {n} tests')
print(f'test result: ok. {n} passed; 0 failed; 0 ignored'); sys.exit(0)
assert (root/'prepared').exists()
assert '--skip' in a
diff --git a/scripts/validation-report.py b/scripts/validation-report.py
index 9ab42f9f..12578caa 100644
--- a/scripts/validation-report.py
+++ b/scripts/validation-report.py
@@ -75,6 +75,7 @@ def parse_log(log):
"""
modules, builds, unassigned = [], [], []
current = None
+ open_harnesses = 0
for number, raw in enumerate(log.splitlines(), 1):
line = ANSI.sub("", raw)
header = HEADER.match(line)
@@ -91,9 +92,12 @@ def parse_log(log):
current = dict(module=key, identity=identity, label=label, line=number, cases={}, summaries=[],
seconds=None, counts=None, result=None, complete=False, expected_count=None)
modules.append(current)
+ open_harnesses = 0
running = re.fullmatch(r"running (\d+) tests?", line)
- if running and current is not None and current["expected_count"] is None:
- current["expected_count"] = int(running[1])
+ if running and current is not None:
+ open_harnesses += 1
+ if current["expected_count"] is None:
+ current["expected_count"] = int(running[1])
summary = SUMMARY.match(line)
if summary:
state, passed, failed, ignored = summary.groups()
@@ -101,9 +105,12 @@ def parse_log(log):
row = dict(result=state, counts=dict(passed=int(passed), failed=int(failed), ignored=int(ignored)),
seconds=float(duration[1]) if duration else None, summary_line=number)
if current is not None:
+ # A same-sized child cannot close a still-running parent harness.
+ open_harnesses = max(0, open_harnesses - 1)
current["summaries"].append(row)
current.update(row)
- current["complete"] = current["expected_count"] is not None and sum(row["counts"].values()) == current["expected_count"]
+ current["complete"] = (open_harnesses == 0 and current["expected_count"] is not None
+ and sum(row["counts"].values()) == current["expected_count"])
else:
unassigned.append(row)
case = CASE.match(line)