The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
perf(ci): shorten the test execution critical path (#16434)
Dotta68a09dcb5f50
diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.ymlindex d870c77d9e..682fc9762f 100644--- a/.github/workflows/pr-trusted.yml+++ b/.github/workflows/pr-trusted.yml@@ -354,26 +354,9 @@ jobs:fail-fast: falsematrix:include:- # The server suite is pinned to maxWorkers=1 (server/vitest.config.ts),- # so it can only be parallelized across runners. Shard it to keep this- # lane off the PR critical path. The suite has grown to ~3184s of- # serial vitest wall time (mean of runs 35036001734 and 35024948947,- # 2026-09-15): at five plain general-server shards with stale- # recorded durations the worst shard ran 806s of tests while the- # best ran 417s. This now uses the same shape as release-verify.yml:- # the ~429s chat integration suite splits by collected test location- # across three dedicated runners (~143s each), and the remaining- # ~2755s levels across twelve duration-balanced shards at ~230s- # each, in line with the other ~200-290s lanes.- #- # The balance only holds while the manifest is fresh. By 2026-10-10- # the without-chat group had grown to 871 suites and ~5031s of serial- # vitest time (mean of runs 38076844891 and 38076972444), 230 of- # them unrecorded and weighted at the median, so the twelve shards- # ran 313-554s of tests (server 2/12 at 618s was the slowest check- # in the run). Re-recorded durations level them to ~415s each; the- # shard count stays at twelve because Build and workspaces-b sit- # above that already.+ # Keep twelve duration-balanced server shards. Each shard overlaps+ # at most two isolated fork workers; cases within a file stay serial.+ # The chat fixture keeps three independent database-backed shards.- group: general-server-without-chatgroup_label: server (1/12)shard_index: 0diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.mdindex 09e8038a57..6fb6f0f664 100644--- a/doc/DEVELOPING.md+++ b/doc/DEVELOPING.md@@ -75,6 +75,16 @@ When changing the workflow path or branch, authorize the new reference beforeupdating the caller. Retain older authorized SHA references while queued runs orsupported reruns still use them.+General server shards run at most two test files at a time in separate Vitest+forks. Each file retains its own home, config and process state, and its test+cases run serially. An explicit `PAPERCLIP_TEST_DATABASE_URL` keeps those shards+serial because the database is shared. Unsharded local tests, route/authz lanes,+and native Runner lanes retain their existing concurrency. Sharded chat jobs run+two independent partitions, each with its own embedded PostgreSQL, home and+config; an explicit shared test database retains one partition per job.+The Runner transport lease-expiry and cold-admission regressions live in their+own file so those real-clock waits overlap other files in the existing pool.+For a full wall-clock profile of all main PR CI jobs, add `profile-ci` to the PR.See [CI profiling](ci-profiling.md) for the Honeycomb waterfall, coverage,operator setup, and activation requirements.diff --git a/doc/ci-profiling.md b/doc/ci-profiling.mdindex fe8472a757..49aac6a4eb 100644--- a/doc/ci-profiling.md+++ b/doc/ci-profiling.md@@ -162,7 +162,7 @@ readiness fixtures, `profile-ci` also runs **CI work benchmark**. It uses oneGitHub-hosted runner without secrets or AWS access. It compares the unchangedPR base and candidate sequentially. It records complete command times, testinventories, revisions, and failure logs in the `ci-work-benchmark` artifact.-The serialized shard must retain the same cases. The fixture comparison retains+The complete general-server shard must retain the same cases. The fixture comparison retainsthe database contract and splits large Git scanning from bounded staging.These command timings exclude queue time and setup; use the full Honeycombwaterfall to assess the complete CI critical path.diff --git a/packages/adapter-utils/src/workspace-snapshot-thread.test.ts b/packages/adapter-utils/src/workspace-snapshot-thread.test.tsindex 5830b7039a..53aaba4b87 100644--- a/packages/adapter-utils/src/workspace-snapshot-thread.test.ts+++ b/packages/adapter-utils/src/workspace-snapshot-thread.test.ts@@ -81,6 +81,9 @@ describe("workspace snapshot threads", () => {await expect(fs.stat(created[0]!)).rejects.toMatchObject({ code: "ENOENT" });});+ // Multiple real TypeScript worker cold starts can exceed the default 5s on+ // CI (three queued workers use two slots). These cases verify cleanup and+ // error propagation, not a worker startup deadline. Keep completion bounded.it("preserves filesystem errors, cleans failed workers and releases queued jobs", async () => {const root = await fixture();const file = path.join(root, "file");@@ -103,7 +106,7 @@ describe("workspace snapshot threads", () => {if (results[2].status === "fulfilled") await disposeDirectorySnapshot(results[2].value);expect(created).toHaveLength(3);for (const directory of created) await expect(fs.stat(directory)).rejects.toMatchObject({ code: "ENOENT" });- });+ }, 15_000);it.each(["unchanged", "changed", "sparse"] as const)("isolates restore captures and cleans their manifests: %s", async mode => {const target = await fixture(), source = await fixture();@@ -129,7 +132,7 @@ describe("workspace snapshot threads", () => {expect(await fs.readFile(path.join(target, "file"), "utf8")).toBe(mode === "unchanged" ? "before" : "after");expect(await fs.readFile(path.join(target, "concurrent"), "utf8")).toBe("host bytes");for (const directory of created) await expect(fs.stat(directory)).rejects.toMatchObject({ code: "ENOENT" });- });+ }, 15_000);it("retains the in-memory snapshot contract when disk backing is not requested", async () => {const root = await fixture();diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport-deadlines.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport-deadlines.test.tsnew file mode 100644index 0000000000..1f0fbaef19--- /dev/null+++ b/packages/paperclip-runner/src/live/runnerd-codex-transport-deadlines.test.ts@@ -0,0 +1,152 @@+import { chmod, cp, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";+import { readFileSync } from "node:fs";+import { createHash } from "node:crypto";+import { tmpdir } from "node:os";+import { join, resolve } from "node:path";+import { fileURLToPath } from "node:url";+import { expect, it, vi } from "vitest";+import { QUALIFIED_ACPX_PROFILES } from "../drivers/acpx/qualified-profiles.js";+import { codexSemanticToolSpecs } from "../drivers/codex/codex-app-server-driver.js";+import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js";+import * as durableControlPlane from "../control-plane/durable-prp-control-plane.js";+import {+ createCapabilityRunnerdCodexTransport,+ defaultCapabilityRunnerdBinary as qualifiedCapabilityRunnerdBinary,+} from "./runnerd-codex-transport.js";++// These regressions must cross real production timing boundaries. Keep their+// waits intact, but let the existing file worker pool overlap them with the+// rest of the transport suite instead of extending its serial critical path.+const defaultCapabilityRunnerdBinary = () =>+ process.env.PAPERCLIP_ATTACH_TRANSITION_RUNNER ?? qualifiedCapabilityRunnerdBinary();+const fakeCodex = resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server");+function fakeCodexArgs(stateDirectory: string, ...args: string[]): string[] {+ return ["--state-file", join(stateDirectory, "fake-codex-state.json"), ...args];+}++it("keeps a quiet active Codex turn in the same process across connection lease expiry", async () => {+ const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-renew-active-"));+ const callsPath = join(stateDirectory, "calls.log");+ const cores: DurablePrpControlPlane[] = [];+ const OriginalCore = durableControlPlane.DurablePrpControlPlane;+ const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane")+ .mockImplementation(function(options: ConstructorParameters<typeof OriginalCore>[0]) {+ const core = new OriginalCore({ ...options, connectionLeaseTtlMs: 60_000 });+ cores.push(core);+ return core;+ } as unknown as typeof OriginalCore);+ const bundle = createCapabilityRunnerdCodexTransport({+ runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex,+ codexArgs: fakeCodexArgs(stateDirectory, "--hold-turn", "--record-process-start", "--call-log", callsPath),+ stateDirectory, runnerReconnectGraceMs: 5_000,+ });+ try {+ const opened = await bundle.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [] });+ await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Keep working quietly." }] });+ const runnerPid = bundle.evidence().runnerPid;+ const codexPid = bundle.evidence().codexPid;+ expect(runnerPid).toBeGreaterThan(0);+ expect(codexPid).toBeGreaterThan(0);+ const core = cores[0]!;+ const original = structuredClone(Object.values(core.store.state.leases)[0]!);+ await vi.waitFor(() => {+ expect(Date.now()).toBeGreaterThan(original.expiresAtUnixMs + 1_000);+ }, { timeout: 75_000, interval: 1_000 });+ const current = Object.values(core.store.state.leases)[0]!;+ expect(current.expiresAtUnixMs).toBeGreaterThan(original.expiresAtUnixMs);+ expect(current.leaseId).toBe(original.leaseId);+ expect(core.store.state.connectionCount).toBe(1);+ expect(bundle.evidence().runnerPid).toBe(runnerPid);+ expect(bundle.evidence().codexPid).toBe(codexPid);+ process.kill(runnerPid!, 0);+ process.kill(codexPid!, 0);+ const calls = (await readFile(callsPath, "utf8")).trim().split(/\r?\n/);+ expect(calls.filter(call => call === "process-start")).toHaveLength(1);+ expect(calls.filter(call => call === "thread/start")).toHaveLength(1);+ expect(calls.filter(call => call === "turn/start")).toHaveLength(1);+ expect(calls).not.toContain("turn/interrupt");+ expect(calls).not.toContain("thread/resume");+ const state = JSON.parse(await readFile(join(stateDirectory, "fake-codex-state.json"), "utf8"));+ expect(state.threadId).toBe(opened.thread.id);+ expect(state.activeTurnId).toBe("provider-turn-1");+ } finally {+ await bundle.transport.close();+ coreSpy.mockRestore();+ await rm(stateDirectory, { recursive: true, force: true });+ }+}, 90_000);++it("reopens Pi after a completed turn within its cold admission budget during warm attachment", async () => {+ const root = await mkdtemp(join(tmpdir(), "runnerd-pi-warm-admission-"));+ const providerNode = process.platform === "linux" ? join(root, "node") : process.execPath;+ if (process.platform === "linux") {+ await cp(process.execPath, providerNode, { dereference: true });+ await chmod(providerNode, 0o500);+ expect((await stat(providerNode)).mode & 0o777).toBe(0o500);+ }+ const cliRoot = join(root, "dist/cli");+ await mkdir(cliRoot, { recursive: true, mode: 0o700 });+ const sidecarPath = join(cliRoot, "acpx-runtime-sidecar.cjs");+ const journal = join(root, "commands.ndjson");+ const fixture = await readFile(fileURLToPath(new URL("./fixtures/fake-pi-warm-sidecar.cjs", import.meta.url)), "utf8");+ await writeFile(sidecarPath, fixture.replace("/* fixture-config */ null", JSON.stringify({ journal, resumeDelayMs: 32_000, commandDigest: QUALIFIED_ACPX_PROFILES.pi.commandDigest })), { mode: 0o600 });+ const digest = (path: string) => `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`;+ const bundle = createCapabilityRunnerdCodexTransport({+ provider: "acpx", acpxAgent: "pi", piThinkingLevel: "low", acpxPermissionMode: "deny-all",+ runnerBinary: defaultCapabilityRunnerdBinary(), stateDirectory: root, runnerFilesystemRoot: root,+ providerNodeCommand: providerNode, providerNodeCommandSha256: digest(providerNode),+ acpxSidecarPath: sidecarPath, acpxSidecarSha256: digest(sidecarPath),+ providerPackAuthorityDigest: `sha256:${"d".repeat(64)}`,+ lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 },+ environment: { PATH: "/usr/bin:/bin" },+ });+ bundle.transport.setServerRequestHandler(async () => ({+ success: true,+ contentItems: [{ type: "inputText", text: "Completion report accepted." }],+ }));+ let failure: unknown;+ try {+ await bundle.transport.request("initialize", {});+ await bundle.transport.request("thread/start", {+ cwd: root, model: "openrouter/deepseek/deepseek-v4-flash-0731",+ dynamicTools: codexSemanticToolSpecs(),+ completionContract: { revision: "warm-pi-contract", criterionIds: [] },+ });+ const runnerPid = bundle.evidence().runnerPid;+ const notifications = bundle.transport.notifications()[Symbol.asyncIterator]();+ const completeTurn = async () => {+ await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Complete the fixture turn." }] });+ for (let index = 0; index < 64; index += 1) {+ const next = await Promise.race([+ notifications.next(),+ new Promise<never>((_, reject) => setTimeout(() => reject(new Error("Fixture completion timed out")), 5_000)),+ ]);+ if (next.value?.method === "turn/completed") return;+ }+ throw new Error("Fixture turn did not complete");+ };+ await completeTurn();+ const started = Date.now();+ await bundle.transport.attachRun!({ runId: "run-pi-warm-second", turnId: "turn-pi-warm-second", itemId: "item-pi-warm-second" });+ expect(Date.now() - started).toBeGreaterThan(30_000);+ await completeTurn();+ expect(bundle.evidence()).toMatchObject({ runnerPid, runnerExited: false });+ const commands = (await readFile(journal, "utf8")).trim().split("\n").map((line) => JSON.parse(line));+ expect(commands.filter((entry) => entry.event === "spawn")).toHaveLength(2);+ expect(commands.filter((entry) => entry.command === "session.open")).toHaveLength(2);+ expect(commands.filter((entry) => entry.resumed)).toHaveLength(1);+ expect(commands.filter((entry) => entry.command === "turn.start")).toHaveLength(2);+ expect(commands.filter((entry) => entry.command === "session.close")).toHaveLength(1);+ } catch (error) {+ failure = error;+ throw error;+ } finally {+ try {+ await bundle.transport.close();+ } catch (cleanupError) {+ if (failure) throw new AggregateError([failure, cleanupError], "Warm attachment and strict cleanup failed");+ throw cleanupError;+ }+ await rm(root, { recursive: true, force: true });+ }+}, 75_000);diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.tsindex a67cd97afd..4491e8b1fc 100644--- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts+++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts@@ -1,4 +1,3 @@-import { QUALIFIED_ACPX_PROFILES } from "../drivers/acpx/qualified-profiles.js";import { coldAdmissionTimeoutMs, waitForRunnerCommand } from "./runnerd-codex-transport.js";import {chmod,@@ -3341,58 +3340,6 @@ it("expands coalesced canonical items without dropping strict bindings", () => {]);});-it("keeps a quiet active Codex turn in the same process across connection lease expiry", async () => {- const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-renew-active-"));- const callsPath = join(stateDirectory, "calls.log");- const cores: DurablePrpControlPlane[] = [];- const OriginalCore = durableControlPlane.DurablePrpControlPlane;- const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane")- .mockImplementation(function(options: ConstructorParameters<typeof OriginalCore>[0]) {- const core = new OriginalCore({ ...options, connectionLeaseTtlMs: 60_000 });- cores.push(core);- return core;- } as unknown as typeof OriginalCore);- const bundle = createCapabilityRunnerdCodexTransport({- runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex,- codexArgs: fakeCodexArgs(stateDirectory, "--hold-turn", "--record-process-start", "--call-log", callsPath),- stateDirectory, runnerReconnectGraceMs: 5_000,- });- try {- const opened = await bundle.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [] });- await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Keep working quietly." }] });- const runnerPid = bundle.evidence().runnerPid;- const codexPid = bundle.evidence().codexPid;- expect(runnerPid).toBeGreaterThan(0);- expect(codexPid).toBeGreaterThan(0);- const core = cores[0]!;- const original = structuredClone(Object.values(core.store.state.leases)[0]!);- await vi.waitFor(() => {- expect(Date.now()).toBeGreaterThan(original.expiresAtUnixMs + 1_000);- }, { timeout: 75_000, interval: 1_000 });- const current = Object.values(core.store.state.leases)[0]!;- expect(current.expiresAtUnixMs).toBeGreaterThan(original.expiresAtUnixMs);- expect(current.leaseId).toBe(original.leaseId);- expect(core.store.state.connectionCount).toBe(1);- expect(bundle.evidence().runnerPid).toBe(runnerPid);- expect(bundle.evidence().codexPid).toBe(codexPid);- process.kill(runnerPid!, 0);- process.kill(codexPid!, 0);- const calls = (await readFile(callsPath, "utf8")).trim().split(/\r?\n/);- expect(calls.filter(call => call === "process-start")).toHaveLength(1);- expect(calls.filter(call => call === "thread/start")).toHaveLength(1);- expect(calls.filter(call => call === "turn/start")).toHaveLength(1);- expect(calls).not.toContain("turn/interrupt");- expect(calls).not.toContain("thread/resume");- const state = JSON.parse(await readFile(join(stateDirectory, "fake-codex-state.json"), "utf8"));- expect(state.threadId).toBe(opened.thread.id);- expect(state.activeTurnId).toBe("provider-turn-1");- } finally {- await bundle.transport.close();- coreSpy.mockRestore();- await rm(stateDirectory, { recursive: true, force: true });- }-}, 90_000);-it("runs the lab provider boundary through authenticated durable PRP", async () => {const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-lab-provider-"));const bundle = createCapabilityRunnerdCodexTransport({@@ -7005,81 +6952,6 @@ it("rotates PRP authority in place for a warm cross-run attachment", async () =>}}, 30_000);-it("reopens Pi after a completed turn within its cold admission budget during warm attachment", async () => {- const root = await mkdtemp(join(tmpdir(), "runnerd-pi-warm-admission-"));- const providerNode = process.platform === "linux" ? join(root, "node") : process.execPath;- if (process.platform === "linux") {- await cp(process.execPath, providerNode, { dereference: true });- await chmod(providerNode, 0o500);- expect((await stat(providerNode)).mode & 0o777).toBe(0o500);- }- const cliRoot = join(root, "dist/cli");- await mkdir(cliRoot, { recursive: true, mode: 0o700 });- const sidecarPath = join(cliRoot, "acpx-runtime-sidecar.cjs");- const journal = join(root, "commands.ndjson");- const fixture = await readFile(fileURLToPath(new URL("./fixtures/fake-pi-warm-sidecar.cjs", import.meta.url)), "utf8");- await writeFile(sidecarPath, fixture.replace("/* fixture-config */ null", JSON.stringify({ journal, resumeDelayMs: 32_000, commandDigest: QUALIFIED_ACPX_PROFILES.pi.commandDigest })), { mode: 0o600 });- const digest = (path: string) => `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`;- const bundle = createCapabilityRunnerdCodexTransport({- provider: "acpx", acpxAgent: "pi", piThinkingLevel: "low", acpxPermissionMode: "deny-all",- runnerBinary: defaultCapabilityRunnerdBinary(), stateDirectory: root, runnerFilesystemRoot: root,- providerNodeCommand: providerNode, providerNodeCommandSha256: digest(providerNode),- acpxSidecarPath: sidecarPath, acpxSidecarSha256: digest(sidecarPath),- providerPackAuthorityDigest: `sha256:${"d".repeat(64)}`,- lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 },- environment: { PATH: "/usr/bin:/bin" },- });- bundle.transport.setServerRequestHandler(async () => ({- success: true,- contentItems: [{ type: "inputText", text: "Completion report accepted." }],- }));- let failure: unknown;- try {- await bundle.transport.request("initialize", {});- await bundle.transport.request("thread/start", {- cwd: root, model: "openrouter/deepseek/deepseek-v4-flash-0731",- dynamicTools: codexSemanticToolSpecs(),- completionContract: { revision: "warm-pi-contract", criterionIds: [] },- });- const runnerPid = bundle.evidence().runnerPid;- const notifications = bundle.transport.notifications()[Symbol.asyncIterator]();- const completeTurn = async () => {- await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Complete the fixture turn." }] });- for (let index = 0; index < 64; index += 1) {- const next = await Promise.race([- notifications.next(),- new Promise<never>((_, reject) => setTimeout(() => reject(new Error("Fixture completion timed out")), 5_000)),- ]);- if (next.value?.method === "turn/completed") return;- }- throw new Error("Fixture turn did not complete");- };- await completeTurn();- const started = Date.now();- await bundle.transport.attachRun!({ runId: "run-pi-warm-second", turnId: "turn-pi-warm-second", itemId: "item-pi-warm-second" });- expect(Date.now() - started).toBeGreaterThan(30_000);- await completeTurn();- expect(bundle.evidence()).toMatchObject({ runnerPid, runnerExited: false });- const commands = (await readFile(journal, "utf8")).trim().split("\n").map((line) => JSON.parse(line));- expect(commands.filter((entry) => entry.event === "spawn")).toHaveLength(2);- expect(commands.filter((entry) => entry.command === "session.open")).toHaveLength(2);- expect(commands.filter((entry) => entry.resumed)).toHaveLength(1);- expect(commands.filter((entry) => entry.command === "turn.start")).toHaveLength(2);- expect(commands.filter((entry) => entry.command === "session.close")).toHaveLength(1);- } catch (error) {- failure = error;- throw error;- } finally {- try {- await bundle.transport.close();- } catch (cleanupError) {- if (failure) throw new AggregateError([failure, cleanupError], "Warm attachment and strict cleanup failed");- throw cleanupError;- }- await rm(root, { recursive: true, force: true });- }-}, 75_000);-it("waits for a warm runner to re-authenticate before probing attachment readiness", async () => {const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-warm-reattach-before-probe-"),diff --git a/scripts/__tests__/run-vitest-stable-shard.test.mjs b/scripts/__tests__/run-vitest-stable-shard.test.mjsindex 9b92a9e174..8c9ddedf96 100644--- a/scripts/__tests__/run-vitest-stable-shard.test.mjs+++ b/scripts/__tests__/run-vitest-stable-shard.test.mjs@@ -1,6 +1,7 @@import assert from "node:assert/strict";import { spawnSync } from "node:child_process";-import { readFileSync } from "node:fs";+import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import os from "node:os";import path from "node:path";import { fileURLToPath } from "node:url";import test from "node:test";@@ -519,3 +520,69 @@ test("filtered collection must match the exact assigned case identities, includiassert.throws(() => assertSelectedTests(expected, actual, lineShardFile));}});+++test("parallel chat jobs form a complete disjoint partition", () => {+ for (const count of [2, 3, 4]) {+ const shards = Array.from({ length: count }, (_, index) =>+ dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", String(count)],+ { PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards,+ );+ assert.ok(shards.every((workers) => workers.length === 2));+ assert.deepEqual(shards.flat().map(({ index }) => index).sort((a, b) => a - b),+ Array.from({ length: count * 2 }, (_, index) => index));+ assert.ok(shards.flat().every((shard) => shard.count === count * 2));+ }+});++test("shared-database and unsharded chat runs retain one serial partition", () => {+ for (let index = 0; index < 3; index++) {+ const result = dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", "3"],+ { PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test" });+ assert.deepEqual(result.chatVitestShards, [{ index, count: 3 }]);+ }+ assert.deepEqual(dryRunJson(["--mode", "general", "--group", "general-chat"],+ { PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, [{ index: 0, count: 1 }]);+});+++test("a failed chat child fails the job after its sibling finishes", { skip: process.platform === "win32" }, () => {+ const directory = mkdtempSync(path.join(os.tmpdir(), "chat-launch-proof-"));+ const file = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");+ const lines = readFileSync(file, "utf8").split("\n")+ .flatMap((line, index) => /^\s*it\(/.test(line) ? [index + 1] : []).slice(0, 6);+ assert.equal(lines.length, 6);+ const tests = lines.map((line) => ({ file, projectName: "@paperclipai/server",+ name: `launcher fixture ${line}`, location: { line } }));+ const receipt = path.join(directory, "sibling-finished");+ const fakePnpm = path.join(directory, "pnpm");+ try {+ writeFileSync(fakePnpm, `#!${process.execPath}+ const fs = require('node:fs');+ const args = process.argv.slice(2);+ const tests = ${JSON.stringify(tests)};+ const selected = args.flatMap(arg => {+ const match = arg.match(/chat-channels\\.integration\\.test\\.ts:(\\d+)$/);+ return match ? [Number(match[1])] : [];+ });+ if (args.includes('list')) {+ const output = args.find(arg => arg.startsWith('--json=')).slice(7);+ fs.writeFileSync(output, JSON.stringify(selected.length+ ? tests.filter(test => selected.includes(test.location.line)) : tests));+ } else if (selected.includes(${lines[0]})) process.exit(7);+ else setTimeout(() => fs.writeFileSync(${JSON.stringify(receipt)}, 'finished'), 200);+ `);+ chmodSync(fakePnpm, 0o755);+ const result = spawnSync(process.execPath, [script, "--mode", "general", "--group", "general-chat",+ "--shard-index", "0", "--shard-count", "3"], {+ cwd: repoRoot, encoding: "utf8", timeout: 30_000,+ env: { ...process.env, PATH: `${directory}${path.delimiter}${process.env.PATH}`,+ PAPERCLIP_TEST_DATABASE_URL: "", PAPERCLIP_CI_PROFILE: "false" },+ });+ assert.equal(result.status, 1, result.stdout + result.stderr);+ assert.match(result.stderr, /failed \(7\)/);+ assert.equal(readFileSync(receipt, "utf8"), "finished", "a failed worker must not abandon its sibling");+ } finally {+ rmSync(directory, { recursive: true, force: true });+ }+});diff --git a/scripts/__tests__/vitest-worker-isolation.test.mjs b/scripts/__tests__/vitest-worker-isolation.test.mjsindex bd70f733df..2c76272875 100644--- a/scripts/__tests__/vitest-worker-isolation.test.mjs+++ b/scripts/__tests__/vitest-worker-isolation.test.mjs@@ -11,7 +11,7 @@ const vitest = path.join(root, 'node_modules/vitest/vitest.mjs');const api = fileURLToPath(new URL('../../node_modules/vitest/dist/index.js', import.meta.url));const setup = path.join(root, 'server/src/__tests__/setup-supertest.ts');-test('one controller retains fresh fork, globals, environment, timers and home per serialized file', () => {+for (const parallel of [false, true]) test(`isolated forks retain fresh state with file parallelism ${parallel}`, () => {const directory = realpathSync(mkdtempSync(path.join(os.tmpdir(), 'pc-fork-proof-')));try {const record = path.join(directory, 'workers.ndjson');@@ -20,7 +20,7 @@ test('one controller retains fresh fork, globals, environment, timers and home pfor (const file of files) writeFileSync(path.join(directory, file), `import { expect, it, vi } from ${JSON.stringify(api)};import { appendFileSync, existsSync, writeFileSync } from 'node:fs';- it('starts clean and can contaminate only its own worker', () => {+ it('starts clean and can contaminate only its own worker', async () => {expect(globalThis.serializedLeak).toBeUndefined();expect(process.env.PAPERCLIP_SERIALIZED_LEAK).toBeUndefined();expect(vi.isFakeTimers()).toBe(false);@@ -30,6 +30,16 @@ test('one controller retains fresh fork, globals, environment, timers and home pconfig: process.env.PAPERCLIP_CONFIG, instance: process.env.PAPERCLIP_INSTANCE_ID,}) + '\\n');writeFileSync(process.env.PAPERCLIP_CONFIG, '{}');+ // In parallel mode both real workers must arrive before either exits.+ // This fails if the controller silently falls back to serial execution.+ if (${parallel}) {+ const { readFileSync } = await import('node:fs');+ const deadline = Date.now() + 10_000;+ while (readFileSync(${JSON.stringify(record)}, 'utf8').trim().split('\\n').length < 2) {+ if (Date.now() > deadline) throw new Error('second isolated worker did not start');+ await new Promise(resolve => setTimeout(resolve, 10));+ }+ }globalThis.serializedLeak = true;process.env.PAPERCLIP_SERIALIZED_LEAK = 'leaked';vi.useFakeTimers();@@ -40,8 +50,8 @@ test('one controller retains fresh fork, globals, environment, timers and home pwriteFileSync(config, `import Sequencer from './sequencer.mjs'; export default ${JSON.stringify({root: directory,test: {- include: order, pool: 'forks', isolate: true, fileParallelism: false,- maxWorkers: 1, setupFiles: [setup],+ include: order, pool: 'forks', isolate: true, fileParallelism: parallel,+ maxWorkers: parallel ? 2 : 1, setupFiles: [setup],sequence: { sequencer: '__SEQUENCER__' },},}).replace('"__SEQUENCER__"', 'Sequencer')};`);@@ -57,7 +67,8 @@ test('one controller retains fresh fork, globals, environment, timers and home passert.equal(result.status, 0, `${result.error ?? ''}\n${result.stdout}\n${result.stderr}`);const workers = readFileSync(record, 'utf8').trim().split('\n').map((line) => JSON.parse(line));assert.equal(workers.length, 2);- assert.deepEqual(workers.map((worker) => worker.name), order);+ assert.deepEqual(workers.map((worker) => worker.name).sort(), [...order].sort());+ if (!parallel) assert.deepEqual(workers.map((worker) => worker.name), order);for (const key of ['pid', 'home', 'config', 'instance']) assert.equal(new Set(workers.map((worker) => worker[key])).size, 2, key);}} finally {diff --git a/scripts/benchmark-ci-work.mjs b/scripts/benchmark-ci-work.mjsindex 7fa6dec766..6bc2c8b99e 100644--- a/scripts/benchmark-ci-work.mjs+++ b/scripts/benchmark-ci-work.mjs@@ -11,7 +11,7 @@ const output = path.resolve(process.argv[3] ?? path.join(candidate, '.ci-benchmamkdirSync(output, { recursive: true });const results = [];const commands = [- ['serialized', process.execPath, ['scripts/run-vitest-stable.mjs', '--mode', 'serialized', '--shard-index', '0', '--shard-count', '9']],+ ['general-server', process.execPath, ['scripts/run-vitest-stable.mjs', '--mode', 'general', '--group', 'general-server-without-chat-or-native-runner', '--shard-index', '0', '--shard-count', '12']],['fixtures', 'pnpm', ['exec', 'vitest', 'run', '--project', '@paperclipai/server','server/src/services/workspace-git-snapshot-streaming.test.ts', 'server/src/__tests__/workspace-runtime.test.ts','-t', 'streams.*all four real Git|commits fixed-port reservations']],@@ -47,7 +47,7 @@ for (const [name] of commands) {const after = results.find((result) => result.variant === 'candidate' && result.name === name);assert.ok(before.tests.length > 0, `${name}: missing test evidence`);assert.deepEqual(after.files, before.files, `${name}: file inventory changed`);- if (name === 'serialized') assert.deepEqual(after.tests, before.tests, 'serialized case inventory changed');+ if (name === 'general-server') assert.deepEqual(after.tests, before.tests, 'general server case inventory changed');else {const contracts = (tests) => {const found = new Set();diff --git a/scripts/run-vitest-stable.mjs b/scripts/run-vitest-stable.mjsindex 847716b706..7f283c9aca 100644--- a/scripts/run-vitest-stable.mjs+++ b/scripts/run-vitest-stable.mjs@@ -1,5 +1,5 @@#!/usr/bin/env node-import { spawnSync } from "node:child_process";+import { spawn, spawnSync } from "node:child_process";import { mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, statSync, writeFileSync } from "node:fs";import os from "node:os";import path from "node:path";@@ -377,7 +377,7 @@ function selectSerializedSuites(routeTests, shardIndex, shardCount) {return shardFiles.map((file) => byRepoPath.get(file));}-function runVitest(args, label, testShard = null) {+async function runVitest(args, label, testShard = null) {console.log(`\n[test:run] ${label}`);invocationIndex += 1;const tempRootParent = process.platform === "win32" ? os.tmpdir() : "/tmp";@@ -408,7 +408,7 @@ function runVitest(args, label, testShard = null) {...filters, "--allowOnly=false", "--includeTaskLocation", `--json=${output}`], {cwd: repoRoot, env, stdio: "inherit",});- if (result.error || result.status !== 0) fail(`Vitest collection failed: ${result.error?.message ?? result.status}`);+ if (result.error || result.status !== 0) throw new Error(`Vitest collection failed: ${result.error?.message ?? result.status}`);return JSON.parse(readFileSync(output, "utf8"));};const allCollected = collect(args, "all");@@ -434,27 +434,27 @@ function runVitest(args, label, testShard = null) {const runFilters = selected.lines.map((line) => `${chatSuite}:${chatSuiteRunLine(sourceLines, line)}`);args = [...args.filter((arg) => arg !== chatSuite), ...runFilters, "--allowOnly=false"];}- const result = spawnSync("pnpm", ["exec", "vitest", "run", ...sourceOnlyVitestArgs, ...args], {- cwd: repoRoot,- env: invocationEnv("vitest", env),- stdio: "inherit",+ await new Promise((resolve, reject) => {+ const child = spawn("pnpm", ["exec", "vitest", "run", ...sourceOnlyVitestArgs, ...args], {+ cwd: repoRoot,+ env: invocationEnv("vitest", env),+ stdio: "inherit",+ });+ child.once("error", reject);+ child.once("close", (code, signal) => {+ if (code === 0) resolve();+ else reject(new Error(`${label} failed (${signal ?? code})`));+ });});- if (result.error) {- console.error(`[test:run] Failed to start Vitest: ${result.error.message}`);- process.exit(1);- }- if (result.status !== 0) {- process.exit(result.status ?? 1);- }}-function runGeneralSuites(routeTests) {+async function runGeneralSuites(routeTests) {for (const groupName of generalGroupNames) {- runGeneralGroup(routeTests, groupName);+ await runGeneralGroup(routeTests, groupName);}}-function runProjectGroup(projects, groupName, shardIndex = null, shardCount = null) {+async function runProjectGroup(projects, groupName, shardIndex = null, shardCount = null) {// With shard args, lean on Vitest's native --shard: each matrix job runs the// same per-project invocations but only its slice of each project's test// files. Vitest's sharding is deterministic for an identical file list, so@@ -463,18 +463,32 @@ function runProjectGroup(projects, groupName, shardIndex = null, shardCount = nushardCount !== null && shardCount > 1 ? [`--shard=${shardIndex + 1}/${shardCount}`] : [];const shardSuffix = shardArgs.length > 0 ? ` shard ${shardIndex + 1}/${shardCount}` : "";for (const project of projects) {- runVitest(["--project", project, ...shardArgs], `${groupName} project ${project}${shardSuffix}`);+ await runVitest(["--project", project, ...shardArgs], `${groupName} project ${project}${shardSuffix}`);}}-function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = null) {+function chatShardWorkers(index, count) {+ const workers = count > 1 && !process.env.PAPERCLIP_TEST_DATABASE_URL ? 2 : 1;+ return Array.from({ length: workers }, (_, offset) => ({+ index: (index ?? 0) * workers + offset,+ count: (count ?? 1) * workers,+ }));+}++async function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = null) {if (groupName === generalChatGroupName) {- runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite],- "chat integration test shard", { index: shardIndex ?? 0, count: shardCount ?? 1 });+ // Each invocation gets its own temporary home and embedded PostgreSQL.+ // Wait for both children even on failure so no sibling is abandoned.+ const results = await Promise.allSettled(chatShardWorkers(shardIndex, shardCount).map((shard) =>+ runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite],+ "chat integration test shard", shard),+ ));+ const failed = results.find((result) => result.status === "rejected");+ if (failed) throw failed.reason;return;}if (groupName === generalServerNativeRunnerGroupName) {- runVitest(+ await runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, ...nativeRunnerSuites],"native runner vertical-slice suite",);@@ -497,11 +511,16 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount =return;}- runVitest(+ await runVitest(["--project","@paperclipai/server",- ...serializedServerVitestArgs,+ // Files have separate fork processes and isolated homes/configs.+ // Keep each file's cases serial; overlap only two independent files.+ // An explicit shared test database cannot be isolated this way.+ ...(process.env.PAPERCLIP_TEST_DATABASE_URL+ ? serializedServerVitestArgs+ : ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=2"]),...shardFiles,],`${groupName} shard ${shardIndex + 1}/${shardCount}`,@@ -513,7 +532,7 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount =for (const suite of excludedSuites) {excludeRouteArgs.push("--exclude", suite.replace(/^server\//, ""));}- runVitest(+ await runVitest(["--project","@paperclipai/server",@@ -530,31 +549,31 @@ function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount =// 31371439296, 2026-08-10, where workspaces-a was the slowest PR check).// Its 439 test files shard cleanly with Vitest's native --shard, so the// lane splits across runners without a duration manifest.- runProjectGroup(generalWorkspacesAProjects, groupName, shardIndex, shardCount);+ await runProjectGroup(generalWorkspacesAProjects, groupName, shardIndex, shardCount);return;}if (groupName === generalWorkspacesBGroupName) {- runProjectGroup(generalWorkspacesBProjects, groupName);+ await runProjectGroup(generalWorkspacesBProjects, groupName);return;}if (groupName === generalWorkspacesCGroupName) {- runProjectGroup(generalWorkspacesCProjects, groupName);+ await runProjectGroup(generalWorkspacesCProjects, groupName);return;}fail(`Unknown group "${groupName}".`);}-function runSerializedSuites(routeTests, shardIndex, shardCount) {+async function runSerializedSuites(routeTests, shardIndex, shardCount) {const shardTests = selectSerializedSuites(routeTests, shardIndex, shardCount);console.log(`\n[test:run] serialized shard ${shardIndex + 1}/${shardCount} running ${shardTests.length} of ${routeTests.length} suites`,);if (shardTests.length > 0) {- runVitest(+ await runVitest(["--project","@paperclipai/server",@@ -578,9 +597,9 @@ const routeTests = walk(serverTestsDir)// Every server test file that the general-server group is responsible for,// i.e. the whole server project minus the route/authz suites that run in the-// dedicated serialized shards. Sharding this list across runners is what keeps-// the general-server lane from becoming the PR critical path: the server vitest-// config pins maxWorkers to 1, so the only way to parallelize is across jobs.+// dedicated serialized shards. The default server config keeps files serial;+// the sharded entry point overlaps two isolated files per runner unless an+// explicit shared test database requires serial execution.// Suites are partitioned by recorded duration (scripts/general-server-shard.mjs)// rather than round-robin, so one slow suite cluster can't stretch a single shard.const serializedRepoPaths = new Set(routeTests.map(test => test.repoPath));@@ -625,6 +644,8 @@ if (options.dryRun) {generalServerShardDurations,): null,+ chatVitestShards: options.group === generalChatGroupName+ ? chatShardWorkers(options.shardIndex, options.shardCount) : null,workspaceProjects:options.group === generalWorkspacesAGroupName? generalWorkspacesAProjects@@ -647,14 +668,19 @@ if (options.dryRun) {process.exit(0);}-if (options.mode === generalModeName || options.mode === allModeName) {- if (options.group) {- runGeneralGroup(routeTests, options.group, options.shardIndex, options.shardCount);- } else {- runGeneralSuites(routeTests);+try {+ if (options.mode === generalModeName || options.mode === allModeName) {+ if (options.group) {+ await runGeneralGroup(routeTests, options.group, options.shardIndex, options.shardCount);+ } else {+ await runGeneralSuites(routeTests);+ }}-}-if (options.mode === serializedModeName || options.mode === allModeName) {- runSerializedSuites(routeTests, options.shardIndex ?? 0, options.shardCount ?? 1);+ if (options.mode === serializedModeName || options.mode === allModeName) {+ await runSerializedSuites(routeTests, options.shardIndex ?? 0, options.shardCount ?? 1);+ }+} catch (error) {+ console.error(`[test:run] ${error.message}`);+ process.exitCode = 1;}