The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
test(scripts): consolidate stable shard selection cases (#16109)
Dotta5c65525219b1
diff --git a/doc/plans/2026-10-13-vitest-stable-shard-test-audit.md b/doc/plans/2026-10-13-vitest-stable-shard-test-audit.mdnew file mode 100644index 0000000000..04d44e50ed--- /dev/null+++ b/doc/plans/2026-10-13-vitest-stable-shard-test-audit.md@@ -0,0 +1,260 @@+# Proven shard test consolidation++Historical proof base: #16099 exact tree 8e955181d8a3776acc86e4662de3c77933231a76, extracted with git archive into an isolated temporary snapshot. Node built-in tests require no dependency installation. All runs serial; no shared checkout edits, commits, pushes or PRs.++Historical result: PROVEN. Retire three repeated executions, preserving all original assertions in named keepers. Test lines 547 -> 521 (26 net removed, including redundant blank normalization); registered cases 32 -> 29; exactly 22 repeated dry-run Node calls removed: 9 serialized partition launches, 12 PR partition launches, 1 general shard-zero launch. The subprocess delta comes from exact unchanged loop bounds/args, not a sampled timing estimate. No CI-time improvement claimed.++| Complete-owner phase | Tests | Pass | Fail | Exit |+|---|---:|---:|---:|---:|+| baseline | 32 | 32 | 0 | 0 |+| candidate | 29 | 29 | 0 | 0 |+| serialized-duplicate-original | 32 | 31 | 1 | 1 |+| serialized-duplicate-keeper | 29 | 28 | 1 | 1 |+| balance-wrong-bin-original | 32 | 28 | 4 | 1 |+| balance-wrong-bin-keeper | 29 | 25 | 4 | 1 |+| route-wrong-lane-original | 32 | 30 | 2 | 1 |+| route-wrong-lane-keeper | 29 | 28 | 1 | 1 |+| restored-candidate | 29 | 29 | 0 | 0 |+| stack-candidate | 29 | 29 | 0 | 0 |++Every run has 0 skipped/cancelled/todo. The 13 failures in mutation runs are deliberate direct assertion failures; no setup, worker or timeout failures. Unmutated baseline/candidate/restored/integrated controls all pass.++## Exact keeper and regression guards++- Serialized completeness moves into serialized duration-balance keeper. Mutate selectSerializedSuites to duplicate one actual selected file on shard zero. Original and keeper fail `every serialized suite must be selected exactly once`: 159 actual vs 158 expected. Each full owner has exactly one failing case.+- PR duration-balance moves into PR exact-coverage keeper. Invert scheduler's lightest-bin `<` comparison to `>`. Original and keeper directly fail `shard weight spread 5003719.5ms exceeds heaviest suite 136113ms`. Each full owner has four failing cases: synthetic skew balance, empty-manifest uniform balance, real serialized balance, real general balance. All are direct meaningful scheduling assertions; complete coverage alone would miss the regression.+- Route/authz exclusion moves into general partition keeper's existing per-file loop, extending the unchanged guard from shard zero to all shards. Actual executed mutation adds a real serialized route to the dry-run general shard-zero selection: `server/src/__tests__/access-routes-hidden-floor.test.ts`. Original route guard and keeper fail the identical route-exclusion assertion naming that actual path. Original owner has two failures (also total count); keeper has one, reaching the relocated route guard before the count assertion. The executable mutation differs from the discovery-map suggestion only in using the selected dry-run membership expression directly; it protects the same wrong-lane behavior.++Production modifications were applied one at a time; each pair ran complete original and candidate test owners under the same mutated bytes. Original test restored between groups. Every production owner restored in finally and hash-verified. Final restored-candidate run passes 29/29, and isolated baseline snapshot subsequently restores all three owners to exact base bytes.++## Hashes and stacked integration++- Original test SHA256: 3b658c7ff55e35619d70ec8381325d07777c3a46a851edc7f5c5d84a7976eacc+- Proven candidate test SHA256: 58ecff8ddceecb8e0c44240fa3dbf7ae6a9756dbac35b5ad52a14e655cfc8327+- Restored run-vitest-stable.mjs SHA256: 71da62d4317591b590f877436ac16c6185b650b497b7fa843b68c801ecb4f353+- Restored general-server-shard.mjs SHA256: ce809911cac307063fc2746118c1d6229c280e3d866b82ee5b776c89f5c016f3++Fetched public branch into an isolated bare Git directory after the default sandbox attempt failed DNS resolution. Fetch succeeds through approved network escalation and resolves exactly #16106 head 397459400c6e7b18b462dcac3d8886b390ca8e8c. All seven relevant test/production owner blobs were identical to the #16099 proof base. The exact proven patch passes git apply --check, applies without conflict to an extracted #16106 snapshot, retains the candidate hash, and passes the complete owner 29/29 on that snapshot. No cherry-pick or rebase commit was created.++## Inspectable reproduction++The results above are historical, recorded on 2026-10-13. This documentation repair+checks the Git source and mutation hashes; it does not claim fresh test results.+Historical raw reports remain local and are not bundled. The relevant failure+excerpts and exact executed production edits are included here; no private proof+file is needed to reconstruct the checks.++Run the block from a repository checkout with the public `base` and `head` Git+revisions below. Fetch those public PR/revisions first if they are absent.+Python 3.12+, Git and Node are required. No dependency installation is needed.++The script uses an isolated snapshot and obtains the candidate test directly+from its Git revision. It runs the full baseline and candidate owners, applies+each mutation independently, checks the intended failing case, restores exact+source bytes in `finally`, and reruns the full candidate. Counts are+`[total, passed, failed, skipped-or-filtered]`. Original-owner mutation counts+above are historical; this recipe repeats candidate sensitivity checks.+Prefix the command with `AUDIT_RECONSTRUCT_ONLY=1` to verify source and mutation+hashes without extracting a snapshot, installing dependencies or running tests.++```sh+python3 - <<'PY'+import hashlib, io, json, os, re, subprocess, tarfile, tempfile+from pathlib import Path++spec = {'base': '8e955181d8a3776acc86e4662de3c77933231a76',+ 'head': 'f750c5af0581ea844997d9ea4ee55053a29c775d',+ 'owner': 'scripts/__tests__/run-vitest-stable-shard.test.mjs',+ 'kind': 'tap',+ 'hashes': {'scripts/__tests__/run-vitest-stable-shard.test.mjs': '3b658c7ff55e35619d70ec8381325d07777c3a46a851edc7f5c5d84a7976eacc',+ 'scripts/run-vitest-stable.mjs': '71da62d4317591b590f877436ac16c6185b650b497b7fa843b68c801ecb4f353',+ 'scripts/general-server-shard.mjs': 'ce809911cac307063fc2746118c1d6229c280e3d866b82ee5b776c89f5c016f3'},+ 'candidate_hash': '58ecff8ddceecb8e0c44240fa3dbf7ae6a9756dbac35b5ad52a14e655cfc8327',+ 'baseline': [32, 32, 0, 0],+ 'candidate': [29, 29, 0, 0],+ 'mutations': [{'name': 'balance-wrong-bin',+ 'path': 'scripts/general-server-shard.mjs',+ 'changes': [[' if (shards[index].totalWeight < shards[target].totalWeight) {\n',+ ' if (shards[index].totalWeight > shards[target].totalWeight) {\n']],+ 'hash': 'a654c5cbfed571ffff107623bc1dec25a5aa3982084d608f6cade19a0d84d75d',+ 'counts': [29, 25, 4, 0],+ 'guard': '12 PR without-chat shards plus the dedicated chat and native-runner lanes '+ 'cover the original server group exactly',+ 'message': 'shard weight spread 5003719.5ms exceeds heaviest suite 136113ms'},+ {'name': 'route-wrong-lane',+ 'path': 'scripts/run-vitest-stable.mjs',+ 'changes': [[' ? selectGeneralServerShard(\n',+ ' ? [...(options.shardIndex === 0 ? '+ '[routeTests.find((routeTest) => '+ 'routeTestPattern.test(routeTest.repoPath)).repoPath] : []), '+ '...selectGeneralServerShard(\n'],+ [' )\n', ' )]\n']],+ 'hash': '230d86c678511ae08bdeaea1c86b6b79c433ffedacffc28eb4ede7d4821a743f',+ 'counts': [29, 28, 1, 0],+ 'guard': 'the general-server shards form a complete, non-overlapping partition',+ 'message': 'route/authz suite must stay in the serialized lane, not general-server: '+ 'server/src/__tests__/access-routes-hidden-floor.test.ts'},+ {'name': 'serialized-duplicate',+ 'path': 'scripts/run-vitest-stable.mjs',+ 'changes': [[' return shardFiles.map((file) => byRepoPath.get(file));\n',+ ' return [...shardFiles, ...(shardIndex === 0 ? shardFiles.slice(0, 1) : '+ '[])].map((file) => byRepoPath.get(file));\n']],+ 'hash': '13c8a46d0037c0594510404c6e6e81c1bf8559d157754082982a2796fbe0d604',+ 'counts': [29, 28, 1, 0],+ 'guard': 'the real serialized shard partition is complete, non-overlapping and '+ 'duration-balanced',+ 'message': 'every serialized suite must be selected exactly once'}]}++def digest(data):+ return hashlib.sha256(data).hexdigest()++def blob(rev, path):+ return subprocess.check_output(["git", "show", rev + ":" + path])++owner = spec["owner"]+original = {p: blob(spec["base"], p) for p in spec["hashes"]}+candidate = blob(spec["head"], owner)+assert all(digest(b) == spec["hashes"][p] for p, b in original.items())+assert digest(candidate) == spec["candidate_hash"]+mutants = []+for m in spec["mutations"]:+ text = original[m["path"]].decode()+ for before, after in m["changes"]:+ assert text.count(before) == 1, (m["name"], "ambiguous mutation")+ text = text.replace(before, after)+ data = text.encode()+ assert digest(data) == m["hash"], m["name"]+ mutants.append((m, data))+if os.environ.get("AUDIT_RECONSTRUCT_ONLY") == "1":+ print("Original, candidate and all mutation hashes match historical proof.")+ raise SystemExit(0)++archive = subprocess.check_output(["git", "archive", spec["base"]])+with tempfile.TemporaryDirectory(prefix="paperclip-audit-") as directory:+ root = Path(directory)+ with tarfile.open(fileobj=io.BytesIO(archive)) as tar:+ tar.extractall(root, filter="data")+ if spec["kind"] != "tap":+ assert subprocess.check_output(["pnpm", "--version"], cwd=root, text=True).strip() == "9.15.4"+ subprocess.run(["pnpm", "install", "--frozen-lockfile"],+ cwd=root, check=True)++ def run(label, expected, mutation=None):+ report = root / "audit-report.json"+ if report.exists():+ report.unlink()+ if spec["kind"] == "tap":+ command = ["node", "--test", "--test-reporter=tap", owner]+ elif spec["kind"] == "ui":+ command = ["pnpm", "--filter", "@paperclipai/ui",+ "exec", "vitest", "run", "src/pages/CompanyExport.test.tsx",+ "--maxWorkers=1", "--reporter=default", "--reporter=json",+ "--outputFile.json=" + str(report)]+ else:+ command = ["node", "node_modules/vitest/vitest.mjs", "run", "--config",+ "packages/adapters/codex-local/vitest.config.ts", owner,+ "--reporter=json", "--outputFile=" + str(report)]+ if mutation and mutation.get("selector"):+ command += ["-t", mutation["selector"]]+ result = subprocess.run(command, cwd=root, capture_output=True, text=True,+ timeout=180)+ output = result.stdout + result.stderr+ print(label, "exit", result.returncode, output)+ if spec["kind"] == "tap":+ counts = {k: int(v) for k, v in re.findall(+ r"^# (tests|pass|fail|skipped) (\d+)$", output, re.M)}+ actual = [counts[k] for k in ["tests", "pass", "fail", "skipped"]]+ if mutation:+ failures = re.findall(+ r"^not ok \d+ - (.*?)\n(.*?)(?=^(?:ok|not ok) \d+ - |^1\.\.|\Z)",+ output, re.M | re.S)+ assert any(name == mutation["guard"] and mutation["message"] in body+ and "ERR_ASSERTION" in body for name, body in failures), label+ else:+ data = json.loads(report.read_text())+ actual = [data[k] for k in ["numTotalTests", "numPassedTests",+ "numFailedTests", "numPendingTests"]]+ if mutation:+ hits = [a for suite in data["testResults"] for a in suite["assertionResults"]+ if a["status"] == "failed" and mutation["guard"] in a["fullName"]]+ assert any(mutation["message"] in msg for a in hits+ for msg in a["failureMessages"]), label+ if mutation.get("line"):+ assert any(f'CompanyExport.test.tsx:{mutation["line"]}:' in msg+ for a in hits for msg in a["failureMessages"]), label+ assert actual == expected, (label, actual, expected)+ assert result.returncode == (1 if mutation else 0), label++ try:+ run("baseline", spec["baseline"])+ (root / owner).write_bytes(candidate)+ run("candidate", spec["candidate"])+ for m, data in mutants:+ try:+ (root / owner).write_bytes(candidate)+ (root / m["path"]).write_bytes(data)+ run(m["name"] + "-keeper", m["counts"], m)+ finally:+ for path, data in original.items():+ (root / path).write_bytes(data)+ assert all((root / p).read_bytes() == b for p, b in original.items())+ (root / owner).write_bytes(candidate)+ run("restored-candidate", spec["candidate"])+ finally:+ for path, data in original.items():+ (root / path).write_bytes(data)+ assert all((root / p).read_bytes() == b for p, b in original.items())+PY+```++Recorded intended failure excerpts (fixture path prefixes omitted):++```text+balance-wrong-bin-original+duration-aware partition balances skewed weights better than round-robin+'expected near-even shard weights, got 90600, 0, 0'+a missing or malformed manifest degrades to uniform weights+|-+ Expected values to be strictly equal:++ 4 !== 0+the real serialized shard partition is duration-balanced+'serialized shard weight spread 2061439ms exceeds heaviest suite 165058ms: 2061439, 0, 0, 0, 0, 0, 0, 0, 0'+the real shard partition is duration-balanced+'shard weight spread 5003719.5ms exceeds heaviest suite 136113ms: 5003719.5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0'+balance-wrong-bin-keeper+duration-aware partition balances skewed weights better than round-robin+'expected near-even shard weights, got 90600, 0, 0'+a missing or malformed manifest degrades to uniform weights+|-+ Expected values to be strictly equal:++ 4 !== 0+the real serialized shard partition is complete, non-overlapping and duration-balanced+'serialized shard weight spread 2061439ms exceeds heaviest suite 165058ms: 2061439, 0, 0, 0, 0, 0, 0, 0, 0'+12 PR without-chat shards plus the dedicated chat and native-runner lanes cover the original server group exactly+'shard weight spread 5003719.5ms exceeds heaviest suite 136113ms: 5003719.5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0'+route-wrong-lane-original+the general-server shards form a complete, non-overlapping partition+|-+ every suite must be selected exactly once++ 890 !== 889+a route/authz suite never leaks into the general-server shards+'route/authz suite must stay in the serialized lane, not general-server: server/src/__tests__/access-routes-hidden-floor.test.ts'+route-wrong-lane-keeper+the general-server shards form a complete, non-overlapping partition+'route/authz suite must stay in the serialized lane, not general-server: server/src/__tests__/access-routes-hidden-floor.test.ts'+serialized-duplicate-original+the serialized shards form a complete, non-overlapping partition+|-+ every serialized suite must be selected exactly once++ 159 !== 158+serialized-duplicate-keeper+the real serialized shard partition is complete, non-overlapping and duration-balanced+|-+ every serialized suite must be selected exactly once++ 159 !== 158+```diff --git a/scripts/__tests__/run-vitest-stable-shard.test.mjs b/scripts/__tests__/run-vitest-stable-shard.test.mjsindex 3dd3e9680e..9b92a9e174 100644--- a/scripts/__tests__/run-vitest-stable-shard.test.mjs+++ b/scripts/__tests__/run-vitest-stable-shard.test.mjs@@ -52,18 +52,6 @@ function dryRunJson(args, envOverrides = {}) {const SHARD_COUNT = 12;const SERIALIZED_SHARD_COUNT = 9;--test("the serialized shards form a complete, non-overlapping partition", () => {- const shards = Array.from({ length: SERIALIZED_SHARD_COUNT }, (_, index) =>- dryRunJson(["--mode", "serialized", "--shard-index", String(index), "--shard-count", String(SERIALIZED_SHARD_COUNT)]),- );-- const total = shards[0].serializedSuiteCount;- const selected = shards.flatMap((shard) => shard.selectedSerializedSuites);- assert.equal(selected.length, total, "every serialized suite must be selected exactly once");- assert.equal(new Set(selected).size, total, "serialized shards must not overlap");-});-test("the general-server shards form a complete, non-overlapping partition", () => {const shards = Array.from({ length: SHARD_COUNT }, (_, index) =>dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)]),@@ -77,6 +65,10 @@ test("the general-server shards form a complete, non-overlapping partition", ()for (const shard of shards) {assert.equal(shard.generalServerSuiteCount, total, "suite count must be stable across shards");for (const file of shard.selectedGeneralServerSuites) {+ assert.ok(+ !(file.startsWith("server/src/__tests__/") && /[^/]*(?:route|routes|authz)[^/]*\.test\.ts$/.test(file)),+ `route/authz suite must stay in the serialized lane, not general-server: ${file}`,+ );assert.ok(!seen.has(file), `suite assigned to more than one shard: ${file}`);seen.add(file);selectedTotal += 1;@@ -88,16 +80,6 @@ test("the general-server shards form a complete, non-overlapping partition", ()assert.equal(seen.size, total, "union of shards must cover the whole suite set");});-test("a route/authz suite never leaks into the general-server shards", () => {- const shard = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", SHARD_COUNT.toString()]);- for (const file of shard.selectedGeneralServerSuites) {- assert.ok(- !(file.startsWith("server/src/__tests__/") && /[^/]*(?:route|routes|authz)[^/]*\.test\.ts$/.test(file)),- `route/authz suite must stay in the serialized lane, not general-server: ${file}`,- );- }-});-test("general shards retain scripts and route-named suites outside the serialized directory", () => {const general = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);for (const file of [@@ -306,13 +288,18 @@ test("the checked-in serialized manifest loads and covers most of the current su);});-test("the real serialized shard partition is duration-balanced", () => {+test("the real serialized shard partition is complete, non-overlapping and duration-balanced", () => {const durations = loadShardDurations(serializedDurationsManifest);const fallback = defaultSuiteWeight(durations);const shards = Array.from({ length: SERIALIZED_SHARD_COUNT }, (_, index) =>dryRunJson(["--mode", "serialized", "--shard-index", String(index), "--shard-count", String(SERIALIZED_SHARD_COUNT)]),);+ const total = shards[0].serializedSuiteCount;+ const selected = shards.flatMap((shard) => shard.selectedSerializedSuites);+ assert.equal(selected.length, total, "every serialized suite must be selected exactly once");+ assert.equal(new Set(selected).size, total, "serialized shards must not overlap");+const totals = shards.map((shard) =>shard.selectedSerializedSuites.reduce((sum, file) => sum + (durations[file] ?? fallback), 0),);@@ -326,19 +313,26 @@ test("the real serialized shard partition is duration-balanced", () => {);});-test("the real shard partition is duration-balanced", () => {- // Mirrors the PR matrix: general-server-without-chat across SHARD_COUNT- // runners, with the chat suite carried by the dedicated general-chat lanes- // and the native-runner suite by the Rust-cached PR vitest lane.+const chatSuitePath = "server/src/__tests__/chat-channels.integration.test.ts";+const nativeRunnerSuitePath =+ "server/src/services/native-runtime/native-codex-runner.integration.test.ts";+const dotRunnerSuitePath = "server/src/__tests__/dot-runner.test.ts";+const restartRecoverySuitePath =+ "server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts";++// Mirrors pr-trusted.yml (12 shards, called by pr.yml so GITHUB_WORKFLOW is+// "PR"): the chat suite runs in its dedicated lanes and the cargo-dependent+// native-runner suite in the Rust-cached final Verify Paperclip Runner vitest+// shard, so together the three cover the full server group exactly.+test("12 PR without-chat shards plus the dedicated chat and native-runner lanes cover the original server group exactly", () => {const durations = loadShardDurations(durationsManifest);const fallback = defaultSuiteWeight(durations);- const shards = Array.from({ length: SHARD_COUNT }, (_, index) =>- dryRunJson(- ["--mode", "general", "--group", "general-server-without-chat", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)],- { GITHUB_WORKFLOW: "PR" },- ),- );-+ const prEnv = { GITHUB_WORKFLOW: "PR" };+ const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], prEnv);+ const shards = Array.from({ length: 12 }, (_, index) => dryRunJson([+ "--mode", "general", "--group", "general-server-without-chat",+ "--shard-index", String(index), "--shard-count", "12",+ ], prEnv));const totals = shards.map((shard) =>shard.selectedGeneralServerSuites.reduce((sum, file) => sum + (durations[file] ?? fallback), 0),);@@ -359,27 +353,7 @@ test("the real shard partition is duration-balanced", () => {maxTotal - minTotal <= heaviest,`shard weight spread ${maxTotal - minTotal}ms exceeds heaviest suite ${heaviest}ms: ${totals.join(", ")}`,);-});---const chatSuitePath = "server/src/__tests__/chat-channels.integration.test.ts";-const nativeRunnerSuitePath =- "server/src/services/native-runtime/native-codex-runner.integration.test.ts";-const dotRunnerSuitePath = "server/src/__tests__/dot-runner.test.ts";-const restartRecoverySuitePath =- "server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts";-// Mirrors pr-trusted.yml (12 shards, called by pr.yml so GITHUB_WORKFLOW is-// "PR"): the chat suite runs in its dedicated lanes and the cargo-dependent-// native-runner suite in the Rust-cached final Verify Paperclip Runner vitest-// shard, so together the three cover the full server group exactly.-test("12 PR without-chat shards plus the dedicated chat and native-runner lanes cover the original server group exactly", () => {- const prEnv = { GITHUB_WORKFLOW: "PR" };- const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], prEnv);- const shards = Array.from({ length: 12 }, (_, index) => dryRunJson([- "--mode", "general", "--group", "general-server-without-chat",- "--shard-index", String(index), "--shard-count", "12",- ], prEnv));const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);assert.ok(!files.includes(chatSuitePath));assert.ok(!files.includes(nativeRunnerSuitePath));