The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
The end-to-end suite runs in CI on Linux, and three Linux daemon bugs it found are fixed (#843)
Deeabdcbe067e9f
diff --git a/.github/workflows/cli-e2e.yml b/.github/workflows/cli-e2e.ymlnew file mode 100644index 000000000..499a88ae4--- /dev/null+++ b/.github/workflows/cli-e2e.yml@@ -0,0 +1,189 @@+name: CLI end to end++# The end-to-end suite (cli/e2e): real daemons under throwaway homes, private tmux servers and fake+# Claude Code and Codex engines. About 60 files and 70 minutes of test time, run one file at a time;+# until this workflow it ran only on developers' machines, and a full run held a laptop for over an+# hour. Here it is eight shards on standard Linux runners, free for a public repository.+#+# ADVISORY until it has stayed green, and so it does not run on pull requests. A failing check on a+# PR, required or not, makes GitHub report the PR UNSTABLE, and scripts/merge-validated-pr.py (`make+# merge-pr`) refuses to merge anything GitHub does not report clean: a per-PR run that failed would+# block every CLI merge. The suite had only ever run on macOS, and its first Linux runs found real+# failures (zsh's new-user menu took every agent's pane, engineLaunch.ts zshNewUserGuard) beside+# runner-specific ones. So it runs after a merge to main, every night, and on demand from a branch+# (Actions -> CLI end to end -> Run workflow) when a change needs it. Promote it to pull requests,+# and then to a required check collected by record-ci-validation.py, once it has stayed green.+on:+ push:+ branches: [main]+ paths:+ - 'cli/**'+ - '.github/workflows/cli-e2e.yml'+ - 'scripts/verify-cli-test-shards.py'+ schedule:+ - cron: '17 6 * * *'+ workflow_dispatch:++permissions:+ contents: read++# A newer merge to main replaces the run of an older one; a nightly or dispatched run is somebody's+# record and is left to finish.+concurrency:+ group: cli-e2e-${{ github.ref }}-${{ github.event_name }}+ cancel-in-progress: ${{ github.event_name == 'push' }}++# The managed runtime the CLI ships on; keep ci.yml's and release.yml's pins in sync.+env:+ CLI_NODE_VERSION: 22.23.2++jobs:+ e2e:+ name: e2e (${{ matrix.shard }})+ strategy:+ fail-fast: false+ matrix:+ shard: [1, 2, 3, 4, 5, 6, 7, 8]+ runs-on: ubuntu-latest+ # About nine minutes of tests per shard by the timing hints, plus install and the bundle.+ timeout-minutes: 45+ defaults:+ run:+ working-directory: cli+ env:+ # tmux and Linux procps mangle what they print without a UTF-8 locale (hook/notify.mjs).+ LANG: C.UTF-8+ # The login shell every daemon under test launches its engines through, as on a Mac: the+ # dotfiles and engine-homes tests write a .zshrc and expect it read.+ SHELL: /bin/zsh+ steps:+ - uses: actions/checkout@v7+ - uses: actions/setup-node@v7+ with:+ node-version: ${{ env.CLI_NODE_VERSION }}+ cache: npm+ cache-dependency-path: cli/package-lock.json+ - name: Install tmux, the shells the tests launch through, and lsof+ run: |+ sudo apt-get update -q+ sudo apt-get install -y -q --no-install-recommends tmux zsh tcsh dash lsof+ # Every daemon watches its data folder and the engines' transcript folders, and a file+ # starts dozens of daemons one after another; the runner's default instance limit is 128.+ sudo sysctl -w fs.inotify.max_user_instances=1024 fs.inotify.max_user_watches=524288+ tmux -V && zsh --version && tcsh --version && command -v dash+ # Ubuntu's global zshrc runs compinit, which stops at "insecure directories ... continue [y]+ # or abort compinit [n]?" when a completion folder is writable by others or owned by someone+ # else, and the runner image ships /usr/share/zsh and its vendor-completions that way: every+ # zsh agent's pane waited at that question and no engine started. A person's machine has+ # none, or they fix it as zsh says, which is this. Fails here if any is left.+ insecure=$(zsh -fc 'autoload -U compaudit && compaudit' 2>/dev/null | grep '^/' | sort -u || true)+ if [ -n "$insecure" ]; then+ echo "$insecure" | xargs sudo chown root:root+ echo "$insecure" | xargs sudo chmod g-w,o-w+ fi+ zsh -fc 'autoload -U compaudit && compaudit'+ - run: npm ci --no-audit --no-fund+ # The complete file list, before sharding: the summary requires each file exactly once.+ - name: Discover the complete end-to-end inventory+ run: npx vitest list --config vitest.e2e.ci.config.ts --filesOnly --json="$RUNNER_TEMP/inventory-${{ matrix.shard }}.json"+ # One bundle for every daemon in the shard (e2e/harness/bundle.ts): less CPU per start, and the+ # build that ships under test. Files run one at a time (E2E_WORKERS unset), so a test's timing+ # is the daemon's rather than the runner's contention; the eight shards are the parallelism.+ - name: Run end-to-end shard+ env:+ E2E_BUNDLE: '1'+ E2E_ARTIFACTS_DIR: ${{ runner.temp }}/e2e-daemon-logs+ run: >-+ npx vitest run --config vitest.e2e.ci.config.ts --shard=${{ matrix.shard }}/8+ --reporter=default --reporter=json --outputFile.json="$RUNNER_TEMP/result-${{ matrix.shard }}.json"+ - name: Preserve inventory and test results+ if: ${{ !cancelled() }}+ uses: actions/upload-artifact@v4+ with:+ name: cli-e2e-shard-${{ matrix.shard }}+ path: |+ ${{ runner.temp }}/inventory-${{ matrix.shard }}.json+ ${{ runner.temp }}/result-${{ matrix.shard }}.json+ if-no-files-found: error+ overwrite: true+ retention-days: 7+ # Only the failing tests' folders are left there (e2e/harness/artifacts.ts): each daemon's whole+ # log and its fake engines' hook log, by file and test.+ - name: Preserve the failing tests' daemon logs+ if: ${{ failure() }}+ uses: actions/upload-artifact@v4+ with:+ name: cli-e2e-daemon-logs-${{ matrix.shard }}+ path: ${{ runner.temp }}/e2e-daemon-logs/+ if-no-files-found: ignore+ overwrite: true+ retention-days: 7++ # The tests that need macOS: a data folder on a small disk image that fills (hdiutil). Opt-in+ # locally (DISKFULL=1); here on one macOS runner, filtered to those tests alone, a few minutes.+ e2e-macos-disk:+ name: e2e (macOS, full disk)+ runs-on: macos-15+ timeout-minutes: 30+ defaults:+ run:+ working-directory: cli+ steps:+ - uses: actions/checkout@v7+ - uses: actions/setup-node@v7+ with:+ node-version: ${{ env.CLI_NODE_VERSION }}+ cache: npm+ cache-dependency-path: cli/package-lock.json+ - run: command -v tmux || brew install tmux+ - run: npm ci --no-audit --no-fund+ - name: A full disk, and an update on a disk that fills+ env:+ DISKFULL: '1'+ E2E_BUNDLE: '1'+ E2E_ARTIFACTS_DIR: ${{ runner.temp }}/e2e-daemon-logs+ run: >-+ npx vitest run --config vitest.e2e.config.ts e2e/diskfull.e2e.ts e2e/updateHostile.e2e.ts+ -t 'a full disk|an update on a disk that fills'+ - name: Preserve the failing tests' daemon logs+ if: ${{ failure() }}+ uses: actions/upload-artifact@v4+ with:+ name: cli-e2e-daemon-logs-macos-disk+ path: ${{ runner.temp }}/e2e-daemon-logs/+ if-no-files-found: ignore+ overwrite: true+ retention-days: 7++ # The one check to read (and, once promoted, to require): every discovered file ran exactly once+ # and passed, on the same rules as ci.yml's `typecheck-test` for the default suite. Its summary+ # carries each file's duration, which refreshes cli/ci-e2e-durations.json.+ e2e-summary:+ needs: e2e+ if: ${{ !cancelled() }}+ runs-on: ubuntu-latest+ timeout-minutes: 3+ steps:+ - uses: actions/checkout@v7+ with:+ sparse-checkout: scripts/verify-cli-test-shards.py+ sparse-checkout-cone-mode: false+ - uses: actions/download-artifact@v4+ with:+ pattern: cli-e2e-shard-*+ path: ${{ runner.temp }}/cli-e2e-shards+ merge-multiple: true+ - name: Require every discovered file exactly once and every shard passing+ env:+ E2E_TEST_RESULT: ${{ needs.e2e.result }}+ run: >-+ python3 scripts/verify-cli-test-shards.py "$RUNNER_TEMP/cli-e2e-shards" --suite e2e+ --root "$GITHUB_WORKSPACE/cli" --shards 8 --tests-result "$E2E_TEST_RESULT"+ --output "$RUNNER_TEMP/cli-e2e-summary.json"+ - uses: actions/upload-artifact@v4+ with:+ name: cli-e2e-summary+ path: ${{ runner.temp }}/cli-e2e-summary.json+ if-no-files-found: error+ overwrite: true+ retention-days: 7diff --git a/cli/ci-e2e-durations.json b/cli/ci-e2e-durations.jsonnew file mode 100644index 000000000..0d5e17318--- /dev/null+++ b/cli/ci-e2e-durations.json@@ -0,0 +1,22 @@+{+ "schema": 1,+ "source": "Local end-to-end runs on a 12-core Mac, 2026-10-05: each file's own time in a run that did not filter its tests. Replace with the durations_ms of a passing cli-e2e-summary artifact.",+ "description": "Scheduling hints only, for the end-to-end CI shards (vitest.e2e.ci.config.ts). Files without a hint cost the default. Discovery always determines the complete test set.",+ "defaultDurationMs": 60000,+ "durationMs": {+ "e2e/bounded.e2e.ts": 115300,+ "e2e/chaos.e2e.ts": 148400,+ "e2e/compaction.e2e.ts": 50000,+ "e2e/core.e2e.ts": 121200,+ "e2e/enginehomes.e2e.ts": 131900,+ "e2e/history.e2e.ts": 113000,+ "e2e/races.e2e.ts": 264300,+ "e2e/recovery.e2e.ts": 28700,+ "e2e/shells.e2e.ts": 130400,+ "e2e/soak.e2e.ts": 180100,+ "e2e/startup.e2e.ts": 19700,+ "e2e/terminal.e2e.ts": 155900,+ "e2e/tmuxconf.e2e.ts": 74500,+ "e2e/updateHostile.e2e.ts": 392800+ }+}diff --git a/cli/e2e/folders.e2e.ts b/cli/e2e/folders.e2e.tsindex aae3a08b8..ba0a4724f 100644--- a/cli/e2e/folders.e2e.ts+++ b/cli/e2e/folders.e2e.ts@@ -114,9 +114,12 @@ describe('the folders agents work in', () => {rmSync(join(d.projectsDir, 'gone-by-morning'), { recursive: true, force: true })await d.start()client = await LocalClient.connect(d)+ // Back means its engine is up again, not only its row: a restore lists the agent, on its conversation,+ // from the moment it starts relaunching it, and a message sent before the engine is there is refused+ // (on Linux the relaunch was still under way when the row first read as active).await until('the agent whose folder is still there to come back', async () => {const now = await row(client, kept.id)- return now?.status === 'active' && now.sessionId === kept.sessionId ? now : null+ return now?.status === 'active' && now.sessionId === kept.sessionId && now.launch?.state !== 'starting' ? now : null}, 60_000, 500)await turn(client, kept.id, 'restored beside one that could not be')// The one whose folder is gone is not shown as active, and the daemon did not restart over it.diff --git a/cli/e2e/harness/artifacts.ts b/cli/e2e/harness/artifacts.tsnew file mode 100644index 000000000..8c5be5591--- /dev/null+++ b/cli/e2e/harness/artifacts.ts@@ -0,0 +1,76 @@+/**+ * `E2E_ARTIFACTS_DIR=<folder>`: keep the whole log of every daemon a failing test ran, as files, for+ * CI to upload (.github/workflows/cli-e2e.yml). A failing test prints the tail of a daemon's log to the+ * console (`onTestFailed` in each file), which is what a developer at the terminal reads; a CI run has+ * eight runners' worth of console and no machine to look at afterwards, and the line that explains a+ * failure is often hundreds of lines above that tail.+ *+ * Every daemon writes its output here as it goes (`IsolatedDaemon` calls `artifactLog`), into the+ * folder of the test running at that moment: `<dir>/<file>/<test>/<daemon>.log`. Output written while+ * no test runs (a `beforeAll`'s daemon starting) goes to `<dir>/<file>/outside-tests/`. Once a test has+ * finished, after its own `afterEach` has closed its daemons, its folder is removed if it passed; the+ * file's `outside-tests` folder is removed at the end of the file if every test in it passed. What is+ * left is the failures, and only those are uploaded. Written synchronously, so a worker that dies or a+ * test that times out leaves everything up to that moment.+ *+ * A vitest setup file (vitest.e2e.config.ts): its hooks are the first registered, so they run around+ * every test's own.+ */+import { appendFileSync, mkdirSync, readdirSync, rmSync } from 'node:fs'+import { basename, join } from 'node:path'+import { afterAll, beforeEach, expect } from 'vitest'++const ROOT = process.env.E2E_ARTIFACTS_DIR || null++/** A test's name as a folder: what a person can still read, and nothing a path cannot hold. */+export function folderName(name: string): string {+ const safe = name.replace(/[^A-Za-z0-9._-]+/g, '-').replace(/^-+|-+$/g, '')+ return (safe || 'unnamed').slice(0, 120)+}++/** The folder of the test file running in this worker ([path], or the one vitest says is running). */+function fileFolder(path = expect.getState().testPath): string | null {+ if (!ROOT || !path) return null+ return join(ROOT, folderName(basename(path).replace(/\.e2e\.ts$/, '')))+}++let current: string | null = null++/** Whether this run keeps artifacts at all: callers skip work whose only use is an artifact. */+export function artifactsEnabled(): boolean {+ return ROOT !== null+}++/** Append [text] to `<the running test's folder>/<name>`. A no-op unless E2E_ARTIFACTS_DIR is set. */+export function artifactLog(name: string, text: string): void {+ const folder = current ?? (fileFolder() && join(fileFolder()!, 'outside-tests'))+ if (!folder || !text) return+ try {+ mkdirSync(folder, { recursive: true })+ appendFileSync(join(folder, folderName(name)), text)+ } catch { /* a full or missing disk must not fail the test this only records */ }+}++if (ROOT) {+ let failed = false+ beforeEach((context) => {+ // A test skipped from inside (`ctx.skip()`) never calls its onTestFinished: nothing it wrote failed.+ if (current) rmSync(current, { recursive: true, force: true })+ const file = fileFolder(context.task.file.filepath)+ if (!file) return+ const folder = join(file, folderName(context.task.fullTestName ?? context.task.name))+ current = folder+ // After every afterEach, so a daemon closed in one has said all it will.+ context.onTestFinished(() => {+ if (context.task.result?.state === 'fail') failed = true+ else rmSync(folder, { recursive: true, force: true })+ if (current === folder) current = null+ })+ })+ afterAll(() => {+ const file = fileFolder()+ if (!file) return+ if (!failed) rmSync(join(file, 'outside-tests'), { recursive: true, force: true })+ try { if (!readdirSync(file).length) rmSync(file, { recursive: true, force: true }) } catch { /* never made */ }+ })+}diff --git a/cli/e2e/harness/daemon.ts b/cli/e2e/harness/daemon.tsindex 99a7b67d5..ffff1714a 100644--- a/cli/e2e/harness/daemon.ts+++ b/cli/e2e/harness/daemon.ts@@ -18,6 +18,7 @@ import { dirname, isAbsolute, join, relative, resolve } from 'node:path'import { fileURLToPath, pathToFileURL } from 'node:url'import { promisify } from 'node:util'import { isolatedTmux, type IsolatedTmux } from '../../src/testing/isolatedTmux.js'+import { artifactLog, artifactsEnabled } from './artifacts.js'const exec = promisify(execFile)const here = dirname(fileURLToPath(import.meta.url))@@ -193,6 +194,11 @@ export class IsolatedDaemon {projects: join(root, 'projects'),}for (const dir of Object.values(dirs)) await mkdir(dir, { recursive: true })+ // A zsh user as most are, with a .zshrc of their own (empty: nothing of anyone's). With none, the zsh+ // of Debian, Ubuntu and Fedora opens its new-user setup menu in every terminal tile and every shell an+ // engine leaves behind, and a test typing into one types into the menu. The person with no startup+ // files at all has their own case (shells.e2e.ts). Appended, never truncated: a test's own stays.+ await writeFile(join(dirs.home, '.zshrc'), '', { flag: 'a' })const config: EngineConfig = {port, dataDir: dirs.data, claudeProjectsDir: dirs.claudeProjects, codexHome: dirs.codexHome,claudeModel: options.claudeModel, codexModel: options.codexModel,@@ -299,7 +305,14 @@ export class IsolatedDaemon {cwd: CLI_ROOT, env: this.env, stdio: ['ignore', 'pipe', 'pipe'],})this.child = child- const take = (chunk: Buffer) => { this.output += chunk.toString('utf8') }+ // The whole log, as files, for a failing test's CI artifacts (E2E_ARTIFACTS_DIR, artifacts.ts).+ const logName = `daemon-${this.port}.log`+ artifactLog(logName, `---- ${new Date().toISOString()} start (pid ${child.pid}, ${script.at(-1)} ${entry}, data ${this.dataDir})\n`)+ const take = (chunk: Buffer) => {+ const text = chunk.toString('utf8')+ this.output += text+ artifactLog(logName, text)+ }child.stdout!.on('data', take)child.stderr!.on('data', take)const exited = new Promise<never>((_, reject) => child.once('exit', (code, signal) => {@@ -372,14 +385,35 @@ export class IsolatedDaemon {return readFileSync(join(this.dataDir, 'hook-credential'), 'utf8').trim()}+ /**+ * What every pane on this daemon's tmux server shows, for a failing test's artifacts: its command, how+ * it started, and its screen with some history. A pane is the one thing the daemon's log cannot say:+ * the suite's first Linux run failed on a setup menu zsh drew in every agent's pane, and the logs said+ * only that no engine ever appeared.+ */+ private async keepPanes(): Promise<void> {+ try {+ const panes = await this.tmux.run('list-panes', '-a', '-F', '#{pane_id}\t#{session_name}\t#{pane_dead}\t#{pane_current_command}\t#{pane_start_command}')+ let text = ''+ for (const line of panes.split('\n').filter(Boolean)) {+ const screen = await this.tmux.run('capture-pane', '-p', '-J', '-S', '-200', '-t', line.split('\t')[0]).catch((error) => String(error))+ text += `==== ${line}\n${screen}\n`+ }+ artifactLog(`daemon-${this.port}-panes.txt`, text)+ } catch { /* no server any more: nothing on screen to keep */ }+ }+async close(): Promise<void> {const core = this.options.noMaster ? null : this.corePid()+ // While the panes are still there: a daemon that stops can take its agents' panes with it.+ if (artifactsEnabled()) await this.keepPanes()await this.stop().catch(() => {})// Once the daemon has said all it will, and before the root (and the engines' hook log) is removed.let escaped: unknown = nulltry { this.assertHooksStayedInside() } catch (error) { escaped = error }// A failed test prints it after this close has removed the root.this.closedHookLog = this.hookLog()+ artifactLog(`daemon-${this.port}-engine-hooks.log`, this.closedHookLog)// A core whose master died during the test must not outlive it: 20 such cores, left by a run// whose masters crashed, spun on their closed output pipes for the rest of a parallel suite. Only// this test's core is killed — its pid, still running this daemon's entry, orphaned or ours.diff --git a/cli/e2e/shells.e2e.ts b/cli/e2e/shells.e2e.tsindex 6ab753fd2..0a8b63ef0 100644--- a/cli/e2e/shells.e2e.ts+++ b/cli/e2e/shells.e2e.ts@@ -5,7 +5,7 @@* ships with macOS, so it stands in for every shell that is not POSIX: an agent must start, in its* own folder, and work, whichever shell it is.*/-import { mkdirSync, realpathSync } from 'node:fs'+import { existsSync, mkdirSync, realpathSync, rmSync } from 'node:fs'import { join } from 'node:path'import { afterEach, describe, expect, it, onTestFailed } from 'vitest'import { LocalClient, type Frame } from './harness/client.js'@@ -46,7 +46,9 @@ describe('the person\'s login shell', () => {const answer = await client.request('terminal_info', { agentId: created.agent.id }, 10_000)return answer.command && !answer.error && answer.command !== 'sh' ? answer : null}, 30_000, 500)- expect(info.command).toBe(shell.split('/').at(-1))+ // tmux names the command from argv[0] on Linux, and Debian's tcsh rewrites its own to "-bin/tcsh",+ // which tmux reads as "bin/tcsh"; macOS's tmux reads the process name. The shell is the same.+ expect(String(info.command).split('/').at(-1)).toBe(shell.split('/').at(-1))expect(realpathSync(info.path)).toBe(realpathSync(cwd))client.close()}, 120_000)@@ -72,4 +74,29 @@ describe('the person\'s login shell', () => {client.close()}, 240_000)}++ // A zsh user with no startup files of their own. The zsh of Debian, Ubuntu, Fedora and Arch then opens+ // its new-user setup menu on any terminal before anything else, and every agent's pane showed that menu+ // instead of starting its engine: on the suite's first Linux run, no zsh agent bound at all. The launch+ // keeps the menu out of an agent's pane (engineLaunch.ts zshNewUserGuard). macOS's zsh has no such menu.+ it.skipIf(!existsSync('/bin/zsh')).each(['claude', 'codex'] as const)('%s for a zsh user with no startup files: an agent starts and works', async (engine: Engine) => {+ const d = await IsolatedDaemon.create({ env: { SHELL: '/bin/zsh' } })+ daemon = d+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-150).join('\n')}`) })+ rmSync(join(d.env.ZDOTDIR!, '.zshrc'))+ await d.start()+ const client = await LocalClient.connect(d)+ const cwd = join(d.projectsDir, `new-zsh-user-${engine}`)+ mkdirSync(cwd, { recursive: true })+ const created = await client.request('agent_create', { engine, cwd, bypassPermission: true }, 120_000)+ expect(created.error, JSON.stringify(created)).toBeUndefined()+ const agent = await until(`the ${engine} agent to bind`, async () => {+ const now = await row(client, created.agent.id)+ return now?.sessionId && now.status === 'active' ? now : null+ }, 60_000, 500)+ await turn(client, agent.id, 'from a new zsh user')+ // Nothing was written into the person's home to get there.+ expect(['.zshenv', '.zprofile', '.zshrc', '.zlogin'].filter((name) => existsSync(join(d.env.ZDOTDIR!, name)))).toEqual([])+ client.close()+ }, 240_000)})diff --git a/cli/e2e/suspend.e2e.ts b/cli/e2e/suspend.e2e.tsindex 6dba45578..c04710998 100644--- a/cli/e2e/suspend.e2e.ts+++ b/cli/e2e/suspend.e2e.ts@@ -72,6 +72,9 @@ async function lineMode(d: IsolatedDaemon, pane: string): Promise<boolean> {const tty = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_tty}')).trim()return / icanon /.test(` ${execFileSync('/bin/sh', ['-c', 'stty -a < "$0"', tty], { encoding: 'utf8' }).replace(/\s+/g, ' ')} `)}+// The pane's command as tmux names it: on Linux, from the process's argv[0], and Debian's tcsh rewrites+// its own to "-bin/tcsh", which tmux reads as "bin/tcsh" (macOS's tmux reads the process name, "tcsh").+// Compared by basename.const processState = (pid: number): string => {try { return execFileSync('ps', ['-o', 'stat=', '-p', String(pid)], { encoding: 'utf8' }).trim() } catch { return '' }}@@ -140,7 +143,7 @@ describe('an engine that is stopped, or killed', () => {await until('the agent to stop counting as active', async () => (await row(client, agent.id))?.status !== 'active' || null, 45_000, 500)expect(await until('the pane to be a shell', async () => {const [dead, command] = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_dead} #{pane_current_command}')).trim().split(' ')- return dead === '0' && command === basename(shell) ? command : null+ return dead === '0' && basename(command) === basename(shell) ? basename(command) : null}, 30_000, 200)).toBe(basename(shell))expect(d.coresStarted()).toBe(1)client.close()@@ -170,7 +173,7 @@ describe('an engine that is stopped, or killed', () => {await until('the agent to stop counting as active', async () => (await row(client, agent.id))?.status !== 'active' || null, 45_000, 500)expect(await until('the pane to be a shell', async () => {const [dead, command] = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_dead} #{pane_current_command}')).trim().split(' ')- return dead === '0' && command === basename(shell) ? command : null+ return dead === '0' && basename(command) === basename(shell) ? basename(command) : null}, 30_000, 200)).toBe(basename(shell))expect(d.coresStarted()).toBe(1)client.close()diff --git a/cli/src/__fixtures__/ciTestSequencer.ts b/cli/src/__fixtures__/ciTestSequencer.tsindex 128c18799..1b8d4b7eb 100644--- a/cli/src/__fixtures__/ciTestSequencer.ts+++ b/cli/src/__fixtures__/ciTestSequencer.ts@@ -2,9 +2,13 @@ import { readFileSync } from 'node:fs'import { relative } from 'node:path'import { BaseSequencer, type TestSpecification } from 'vitest/node'-type TimingHints = { defaultDurationMs: number; durationMs: Record<string, number> }+export type TimingHints = { defaultDurationMs: number; durationMs: Record<string, number> }-export function readTimingHints(value: unknown): TimingHints {+/** The files each suite's hints may name: the default suite's specs, and the end-to-end files. */+export const UNIT_HINT_PATH = /^src\/.+\.(spec|test)\.ts$/+export const E2E_HINT_PATH = /^e2e\/.+\.e2e\.ts$/++export function readTimingHints(value: unknown, paths: RegExp = UNIT_HINT_PATH): TimingHints {const raw = value as { schema?: unknown; defaultDurationMs?: unknown; durationMs?: unknown } | nullconst positive = (n: unknown): n is number => typeof n === 'number' && Number.isFinite(n) && n > 0if (raw?.schema !== 1 || !positive(raw.defaultDurationMs) || !raw.durationMs@@ -12,14 +16,17 @@ export function readTimingHints(value: unknown): TimingHints {throw new Error('Invalid CLI test timing hints')}for (const [path, duration] of Object.entries(raw.durationMs)) {- if (!/^src\/.+\.(spec|test)\.ts$/.test(path) || path.includes('\\') || path.split('/').includes('..') || !positive(duration)) {+ if (!paths.test(path) || path.includes('\\') || path.split('/').includes('..') || !positive(duration)) {throw new Error(`Invalid CLI test timing hint: ${path}`)}}return { defaultDurationMs: raw.defaultDurationMs, durationMs: raw.durationMs as Record<string, number> }}-const hints = readTimingHints(JSON.parse(readFileSync(new URL('../../ci-test-durations.json', import.meta.url), 'utf8')))+/** A suite's hints file (`ci-test-durations.json`, `ci-e2e-durations.json`), read and checked. */+export function readTimingHintsFile(name: string, paths: RegExp): TimingHints {+ return readTimingHints(JSON.parse(readFileSync(new URL(`../../${name}`, import.meta.url), 'utf8')), paths)+}/** Longest estimated files first, each assigned to the least-loaded shard. Timing hints never* select tests: the caller's complete discovery list is the only source of entries. New files@@ -44,17 +51,24 @@ export function durationShards<T>(entries: Array<{ key: string; durationMs: numbreturn shards.map(shard => shard.values)}-export class DurationSequencer extends BaseSequencer {- override async shard(files: TestSpecification[]): Promise<TestSpecification[]> {- const shard = this.ctx.config.shard- if (!shard) return files- const entries = files.map(spec => {- const path = relative(this.ctx.config.root, spec.moduleId).replaceAll('\\', '/')- return { key: `${spec.project.name}\0${spec.pool}\0${path}`,- durationMs: hints.durationMs[path] ?? hints.defaultDurationMs, value: spec }- })- return durationShards(entries, shard.count)[shard.index - 1]+/** A sequencer that shards by [hints]; one per suite, since each has its own files and timings. */+export function createDurationSequencer(hints: TimingHints): typeof BaseSequencer {+ return class extends BaseSequencer {+ override async shard(files: TestSpecification[]): Promise<TestSpecification[]> {+ const shard = this.ctx.config.shard+ if (!shard) return files+ const entries = files.map(spec => {+ const path = relative(this.ctx.config.root, spec.moduleId).replaceAll('\\', '/')+ return { key: `${spec.project.name}\0${spec.pool}\0${path}`,+ durationMs: hints.durationMs[path] ?? hints.defaultDurationMs, value: spec }+ })+ return durationShards(entries, shard.count)[shard.index - 1]+ }+ // Keep Vitest's normal failed-first/cache-aware ordering within each shard, and its+ // project/isolation/group ordering. Unsharded local runs retain the default behavior.}- // Keep Vitest's normal failed-first/cache-aware ordering within each shard, and its- // project/isolation/group ordering. Unsharded local runs retain the default behavior.}++/** The default suite's CI shards (`vitest.ci.config.ts`). The end-to-end suite builds its own from+ * `ci-e2e-durations.json` (`vitest.e2e.ci.config.ts`). */+export const DurationSequencer = createDurationSequencer(readTimingHintsFile('ci-test-durations.json', UNIT_HINT_PATH))diff --git a/cli/src/architecture.spec.ts b/cli/src/architecture.spec.tsindex fd0a78154..d251bf0df 100644--- a/cli/src/architecture.spec.ts+++ b/cli/src/architecture.spec.ts@@ -150,8 +150,13 @@ const SERVICE_MAY_IMPORT: Record<string, string> = {** Measured at 105,714 lines in 450 files on 6 October, when runForeground moved out of cli.ts (step 1),* against 114,622 in 488 walked from cli.ts: the CLI's own commands left the core's process.+ *+ * Grew by 54 the same day for three Linux bugs in the core's own launch and hook paths, found by the+ * end-to-end suite's first Linux runs (#843): zsh's new-user menu kept out of an agent's pane+ * (lib/engineLaunch.ts), a hook's ancestry read as it arrives (core/engines/hooks.ts), and a relaunch+ * that must stay up to count (core/agents/swap.ts).*/-const CORE_CLOSURE_BUDGET = 105_825+const CORE_CLOSURE_BUDGET = 105_879/** What is not the core's, by path: each goes to a service or its own process, in the plan's order. */const EDGE: RegExp[] = [diff --git a/cli/src/ciTestSequencer.spec.ts b/cli/src/ciTestSequencer.spec.tsindex fda019c61..e390fd2b3 100644--- a/cli/src/ciTestSequencer.spec.ts+++ b/cli/src/ciTestSequencer.spec.ts@@ -1,7 +1,7 @@import { resolve } from 'node:path'import { describe, expect, it } from 'vitest'import type { TestSpecification, Vitest } from 'vitest/node'-import { durationShards, DurationSequencer, readTimingHints } from './__fixtures__/ciTestSequencer.js'+import { createDurationSequencer, durationShards, DurationSequencer, E2E_HINT_PATH, readTimingHints, readTimingHintsFile } from './__fixtures__/ciTestSequencer.js'const entries = (costs: number[]) => costs.map((durationMs, value) => ({ key: `file-${value}`, durationMs, value }))@@ -68,4 +68,27 @@ describe('duration-based CLI shards', () => {expect(new Set(shards.flat())).toEqual(new Set(files))expect(shards.flat()).toHaveLength(files.length)})++ it('reads the end-to-end hints with their own file pattern, and neither suite accepts the other\'s files', () => {+ const e2e = { schema: 1, defaultDurationMs: 60_000, durationMs: { 'e2e/races.e2e.ts': 264_300 } }+ expect(readTimingHints(e2e, E2E_HINT_PATH)).toEqual({ defaultDurationMs: 60_000, durationMs: e2e.durationMs })+ expect(() => readTimingHints(e2e)).toThrow('timing hint: e2e/races.e2e.ts')+ expect(() => readTimingHints({ ...e2e, durationMs: { 'src/slow.spec.ts': 10 } }, E2E_HINT_PATH)).toThrow('timing hint')+ expect(() => readTimingHints({ ...e2e, durationMs: { 'e2e/../races.e2e.ts': 10 } }, E2E_HINT_PATH)).toThrow('timing hint')+ // The checked-in hints parse: a malformed file would fail every CI shard before a test ran.+ expect(readTimingHintsFile('ci-e2e-durations.json', E2E_HINT_PATH).defaultDurationMs).toBeGreaterThan(0)+ })++ it('splits end-to-end files by their hints, so the longest files land on different shards', async () => {+ const root = resolve('worktree')+ const hints = { defaultDurationMs: 60_000, durationMs: { 'e2e/a.e2e.ts': 400_000, 'e2e/b.e2e.ts': 300_000, 'e2e/c.e2e.ts': 250_000 } }+ const paths = ['e2e/a.e2e.ts', 'e2e/b.e2e.ts', 'e2e/c.e2e.ts', 'e2e/d.e2e.ts', 'e2e/e.e2e.ts', 'e2e/f.e2e.ts']+ const files = paths.map(path => ({ moduleId: resolve(root, path), project: { name: '' }, pool: 'forks' }) as TestSpecification)+ const Sequencer = createDurationSequencer(hints)+ const shards = await Promise.all([1, 2, 3].map(index => new Sequencer({+ config: { root, shard: { index, count: 3 } },+ } as Vitest).shard(files).then(shard => shard.map(spec => paths.find(path => resolve(root, path) === spec.moduleId)!))))+ expect(shards.flat().sort()).toEqual(paths)+ expect(shards.map(shard => shard.filter(path => path in hints.durationMs))).toEqual([['e2e/a.e2e.ts'], ['e2e/b.e2e.ts'], ['e2e/c.e2e.ts']])+ })})diff --git a/cli/src/core/agents/swap.spec.ts b/cli/src/core/agents/swap.spec.tsindex cbde981ef..456fbe0fc 100644--- a/cli/src/core/agents/swap.spec.ts+++ b/cli/src/core/agents/swap.spec.ts@@ -5,7 +5,7 @@ import { buildEngineLaunchArgv } from '../../lib/engineLaunch.js'import type { RegisteredSession } from '../../lib/registry.js'import { processRows } from '../../lib/terminalAgentDiscovery.js'import { bypassPermissionActive, resolvePaneEngineProcess } from '../../lib/tmux.js'-import { createPaneSwap, type PaneSwapDeps } from './swap.js'+import { createPaneSwap, SWAP_SETTLE_MS, type PaneSwapDeps } from './swap.js'vi.mock('../../lib/deleteAgentFallback.js', async (real) => ({ ...await real<object>(), terminateDeletedAgent: vi.fn(async () => ({ ok: true })) }))vi.mock('../../lib/engineLaunch.js', async (real) => ({ ...await real<object>(), buildEngineLaunchArgv: vi.fn(() => ['zsh', '-lc', 'claude']) }))@@ -94,9 +94,9 @@ describe('the pane-process swap', () => {vi.useFakeTimers()const { swap } = setup()const found = { pid: 99, startMarker: 'now' }- vi.mocked(resolvePaneEngineProcess).mockResolvedValueOnce(null).mockResolvedValueOnce(found as never)+ vi.mocked(resolvePaneEngineProcess).mockResolvedValueOnce(null).mockResolvedValueOnce(found as never).mockResolvedValueOnce({ ...found } as never)const waiting = swap.paneSwapDeps(session(), runtime).waitForProcess()- await vi.advanceTimersByTimeAsync(150 + 300)+ await vi.advanceTimersByTimeAsync(150 + 300 + SWAP_SETTLE_MS)expect(await waiting).toEqual(found)const never = swap.paneSwapDeps(session(), runtime).waitForProcess()// 150, 300, 600, then 750 ms at a time: the budget is spent on the look that crosses 8 s.@@ -104,6 +104,32 @@ describe('the pane-process swap', () => {expect(await never).toBeNull()})+ // A launch the engine refuses (`codex resume` after an update dropped it) runs for an instant: seen in+ // that instant, a restart called the conversation resumed and never fell back to a fresh start.+ it('counts a process as come up only when it is still the pane\'s engine a moment later', async () => {+ vi.useFakeTimers()+ const { swap } = setup()+ const refused = { pid: 98, startMarker: 'then' }+ const lasting = { pid: 99, startMarker: 'now' }+ vi.mocked(resolvePaneEngineProcess)+ .mockResolvedValueOnce(refused as never).mockResolvedValueOnce(null)+ // Another process in its place by the second look is not the one seen: it is looked at afresh.+ .mockResolvedValueOnce(lasting as never).mockResolvedValueOnce({ pid: 99, startMarker: 'later' } as never)+ .mockResolvedValueOnce(lasting as never).mockResolvedValueOnce({ ...lasting } as never)+ const waiting = swap.paneSwapDeps(session(), runtime).waitForProcess()+ await vi.advanceTimersByTimeAsync(150 + SWAP_SETTLE_MS + 300 + SWAP_SETTLE_MS + 600 + SWAP_SETTLE_MS)+ expect(await waiting).toEqual(lasting)+ expect(resolvePaneEngineProcess).toHaveBeenCalledTimes(6)+ // One that never stays up is no engine: the wait ends as one that found none.+ vi.mocked(resolvePaneEngineProcess).mockReset().mockImplementation(async () => {+ refused.pid+++ return { ...refused } as never+ })+ const never = swap.paneSwapDeps(session(), runtime).waitForProcess()+ await vi.advanceTimersByTimeAsync(12_000)+ expect(await never).toBeNull()+ })+it('builds the relaunch argv with the row\'s mode, folder, the caller\'s argv and the keys a grid must clear', () => {const { swap } = setup()const log = vi.spyOn(console, 'log').mockImplementation(() => {})diff --git a/cli/src/core/agents/swap.ts b/cli/src/core/agents/swap.tsindex aff2d47b6..c38c13f7c 100644--- a/cli/src/core/agents/swap.ts+++ b/cli/src/core/agents/swap.ts@@ -24,6 +24,9 @@ export interface PaneSwapDeps {keepAbandonedConversation: (left: RegisteredSession) => void}+/** How long a relaunched engine must stay running before a restart counts it as come up (`waitForProcess`). */+export const SWAP_SETTLE_MS = 500+export function createPaneSwap({ byAgent, tmuxBackend, prepareSessionResume, keepAbandonedConversation }: PaneSwapDeps) {/*** The dependencies a pane-process swap needs, for both callers that do one.@@ -85,7 +88,15 @@ export function createPaneSwap({ byAgent, tmuxBackend, prepareSessionResume, keeawait new Promise((resolve) => setTimeout(resolve, delayMs))waited += delayMsconst found = await resolvePaneEngineProcess(runtime.paneId, session.engine)- if (found) return found+ if (found) {+ // Up means still up a moment later: a launch the engine refuses (`codex resume` after an update+ // dropped it) runs for an instant, and seen then, the restart said "resumed" and never fell back+ // to a fresh start (e2e/updates.e2e.ts on Linux, whose faster start put it in the first look).+ await new Promise((resolve) => setTimeout(resolve, SWAP_SETTLE_MS))+ waited += SWAP_SETTLE_MS+ const still = await resolvePaneEngineProcess(runtime.paneId, session.engine)+ if (still && still.pid === found.pid && still.startMarker === found.startMarker) return found+ }delayMs = Math.min(delayMs * 2, 750)}return nulldiff --git a/cli/src/core/engines/hooks.spec.ts b/cli/src/core/engines/hooks.spec.tsindex 3504a150b..c1863f847 100644--- a/cli/src/core/engines/hooks.spec.ts+++ b/cli/src/core/engines/hooks.spec.ts@@ -68,6 +68,21 @@ describe('which agent a hook belongs to', () => {expect(console.log).not.toHaveBeenCalled()})+ it('reads the table again when the one in flight began before the hook\'s own process, and matches it', async () => {+ const a1 = agent('a1', 100)+ vi.mocked(processRows)+ .mockResolvedValueOnce(tree([200, 100], [100, 1]) as never)+ .mockResolvedValueOnce(tree([300, 200], [200, 100], [100, 1]) as never)+ expect(await setup({ '%1': a1 }).hooks.resolveHookAgent({ engine: 'claude', runtimeHints: [hint('%1')], callerPid: 300 })).toBe(a1)+ expect(processRows).toHaveBeenCalledTimes(2)+ expect(console.log).not.toHaveBeenCalled()+ })++ it('none when the table read again for a missing caller cannot be read', async () => {+ vi.mocked(processRows).mockResolvedValueOnce(tree([100, 1]) as never).mockResolvedValueOnce(null)+ expect(await setup({ '%1': agent('a1', 100) }).hooks.resolveHookAgent({ engine: 'claude', runtimeHints: [hint('%1')], callerPid: 300 })).toBeNull()+ })+it('none when the process table cannot be read', async () => {vi.mocked(processRows).mockResolvedValueOnce(null)expect(await setup({ '%1': agent('a1', 100) }).hooks.resolveHookAgent({ engine: 'claude', runtimeHints: [hint('%1')], callerPid: 300 })).toBeNull()@@ -130,11 +145,13 @@ describe('which agent a hook belongs to', () => {// The new engine is 200 and the hook's process, 300, is its child. Pid 100 was the engine a restart// killed; nothing in the table is it any more.const newEngine = () => tree([300, 200], [200, 1])- /** Asks, and says when the hook has started waiting: after it read the process table once. */+ /** Asks, and says when the hook has started waiting: once it has been answered with `onWait`. */const ask = (hooks: ReturnType<typeof setup>['hooks'], panes = ['%1'], onWait?: () => void) => {let answer: RegisteredSession | null | undefined- const asked = hooks.resolveHookAgent({ engine: 'claude', runtimeHints: panes.map(hint), callerPid: 300, ...(onWait ? { onWait } : {}) }).then((agent) => { answer = agent })- const waiting = () => vi.waitFor(() => expect(processRows).toHaveBeenCalledTimes(1))+ let waited = false+ const asked = hooks.resolveHookAgent({ engine: 'claude', runtimeHints: panes.map(hint), callerPid: 300, onWait: () => { waited = true; onWait?.() } })+ .then((agent) => { answer = agent })+ const waiting = () => vi.waitFor(() => expect(waited).toBe(true))return { asked, waiting, answer: () => answer }}@@ -187,6 +204,30 @@ describe('which agent a hook belongs to', () => {expect(hook.answer()).toEqual(agent('a1', 200))})+ // On Linux the hook runs under dash, whose `sh -c` stays the hook's parent and exits with it: once+ // `onWait` has answered the hook, the caller is gone from the table. It still belongs to its engine.+ it('belongs to the agent after the wait though its own process has exited by then, by its ancestry as it arrived', async () => {+ const onPane: Record<string, RegisteredSession> = { '%1': agent('a1', 100) }+ vi.mocked(processRows).mockResolvedValueOnce(newEngine() as never).mockResolvedValue(tree([200, 1]) as never)+ const hook = ask(setup(onPane).hooks)+ await hook.waiting()+ onPane['%1'] = agent('a1', 200)+ await hook.asked+ expect(hook.answer()).toEqual(agent('a1', 200))+ })++ it('reads who it came from as it arrives, while the reconcile pass for its pane still runs', async () => {+ const onPane: Record<string, RegisteredSession> = { '%1': agent('a1', 200) }+ vi.mocked(processRows).mockResolvedValueOnce(newEngine() as never)+ let readDuringPass = -1+ const { hooks } = setup(onPane, { agentReconciler: {+ triggerHint: vi.fn(async () => { readDuringPass = vi.mocked(processRows).mock.calls.length; return true }),+ trigger: vi.fn(async () => true),+ } as unknown as EngineHookDeps['agentReconciler'] })+ expect(await hooks.resolveHookAgent({ engine: 'claude', runtimeHints: [hint('%1')], callerPid: 300 })).toEqual(agent('a1', 200))+ expect(readDuringPass).toBe(1)+ })+it('none when what is recorded is a process it does not come from, or the table cannot be read again', async () => {const onPane: Record<string, RegisteredSession> = { '%1': agent('a1') }vi.mocked(processRows).mockResolvedValue(newEngine() as never)diff --git a/cli/src/core/engines/hooks.ts b/cli/src/core/engines/hooks.tsindex bcce161bf..36cc5bf33 100644--- a/cli/src/core/engines/hooks.ts+++ b/cli/src/core/engines/hooks.ts@@ -16,6 +16,7 @@ import {import { sid } from '../../lib/log.js'import type { registry, RegisteredSession } from '../../lib/registry.js'import { processRows } from '../../lib/terminalAgentDiscovery.js'+import type { ProcessRow } from '../../lib/tmux.js'import type { TerminalAgentReconciler } from '../../lib/terminalAgentReconciler.js'import type { TerminalRuntimeRef } from '../../lib/terminalTypes.js'@@ -50,26 +51,19 @@ export function createEngineHooks({ tmuxBackend, agentReconciler, registry }: En/*** The agents on the hinted panes: those whose recorded process the caller descends from, the rest,* and of the rest those with no live process recorded at all (none yet, or one that has exited).+ * `ancestry` is the hook's own process and those it descends from, read as the hook arrived+ * (`resolveHookAgent`); `table`, the process table to judge the recorded processes by, else a new read.*/- const matchCaller = async (resolved: TerminalRuntimeRef[], engine: HookQuery['engine'], callerPid: number) => {- const rows = await processRows()+ const matchCaller = async (resolved: TerminalRuntimeRef[], engine: HookQuery['engine'], ancestry: ReadonlySet<number>, table?: ProcessRow[]) => {+ const rows = table ?? await processRows()if (!rows) return null- const byPid = new Map(rows.map((row) => [row.pid, row.parentPid]))const recordedAlive = (session: RegisteredSession): boolean => {const recorded = session.processIdentityreturn !!recorded && rows.some((row) => row.pid === recorded.pid && row.startMarker === recorded.startMarker)}const callerBelongsTo = (session: RegisteredSession): boolean => {const expectedPid = session.processIdentity?.pid- if (!expectedPid) return false- let pid = callerPid- const visited = new Set<number>()- while (pid > 0 && !visited.has(pid)) {- if (pid === expectedPid) return true- visited.add(pid)- pid = byPid.get(pid) ?? 0- }- return false+ return !!expectedPid && ancestry.has(expectedPid)}const candidates = new Map<string, RegisteredSession>()/**@@ -113,6 +107,11 @@ export function createEngineHooks({ tmuxBackend, agentReconciler, registry }: Enconst resolveHookAgent: ResolveHookAgent = async ({ engine, runtimeHints, callerPid, onWait }) => {if (!callerPid) return null+ // Who the hook came from, read as it arrives. On Linux dash keeps its `sh -c` as the hook's parent, a+ // shell that exits with the hook (its 500ms, or `onWait`'s answer); read after the reconcile pass or+ // the wait, the ancestry stopped at a pid already gone and a restart's new conversation was never+ // bound (e2e/updates.e2e.ts). macOS's /bin/sh is a bash that execs the hook: never seen there.+ const arrival = processRows()const resolved: TerminalRuntimeRef[] = []for (const hint of runtimeHints ?? []) {if (tmuxBackend) resolved.push({ backend: 'tmux', paneId: hint.paneId })@@ -121,8 +120,15 @@ export function createEngineHooks({ tmuxBackend, agentReconciler, registry }: Enlet overdue = falsefor (const runtime of resolved) if (!await agentReconciler.triggerHint(runtime, engine)) overdue = true- let found = await matchCaller(resolved, engine, callerPid)- if (!found) return null+ let table = await arrival+ // A `ps` already running (processRows shares it) may predate the hook's process; the next read has it.+ if (table && !table.some((row) => row.pid === callerPid)) table = await processRows()+ if (!table) return null+ const parents = new Map(table.map((row) => [row.pid, row.parentPid]))+ const ancestry = new Set<number>()+ for (let pid = callerPid; pid > 0 && !ancestry.has(pid); pid = parents.get(pid) ?? 0) ancestry.add(pid)++ let found = (await matchCaller(resolved, engine, ancestry, table))!let choice = chooseHookAgent([...found.candidates.values()], [...found.onHintedRuntime.values()], engine)// Every relaunch starts its engine before it records the new process — create, fork, resume, a// restart and the new conversation it falls back to, a retarget — so a SessionStart in between came@@ -135,8 +141,9 @@ export function createEngineHooks({ tmuxBackend, agentReconciler, registry }: Enif (!choice.agent && choice.reason === 'none' && (found.unrecorded.length || overdue)) {onWait?.()await recordChanged(resolved, engine)- found = await matchCaller(resolved, engine, callerPid)- if (!found) return null+ const again = await matchCaller(resolved, engine, ancestry)+ if (!again) return null+ found = againchoice = chooseHookAgent([...found.candidates.values()], [...found.onHintedRuntime.values()], engine)}const { candidates } = founddiff --git a/cli/src/lib/engineLaunch.spec.ts b/cli/src/lib/engineLaunch.spec.tsindex 550c14980..44baf9c97 100644--- a/cli/src/lib/engineLaunch.spec.ts+++ b/cli/src/lib/engineLaunch.spec.ts@@ -37,6 +37,8 @@ import {supportsFirstPrompt,supportsNamedAgent,unreadableCwdGuard,+ ZSH_GUARD_ZSHENV,+ zshNewUserGuard,} from './engineLaunch.js'import { ENGINES, type AgentEngine } from '../engines/types.js'import { engineBin } from './engineBin.js'@@ -54,6 +56,19 @@ const developersOwnGridBin = process.env.HARNESS_GRID_BINbeforeEach(() => { delete process.env.HARNESS_GRID_BIN })afterAll(() => { if (developersOwnGridBin !== undefined) process.env.HARNESS_GRID_BIN = developersOwnGridBin })+// A zsh user as most are: with a startup file of their own, so the zsh launches below are the plain+// ones. vitest.setup.ts points ZDOTDIR at an empty folder, which is the new-user case `zshNewUserGuard`+// handles; that case has its own tests at the end of this file.+const suiteZdotdir = process.env.ZDOTDIR+const zshUserHome = mkdtempSync(join(tmpdir(), 'launch-zdotdir-'))+writeFileSync(join(zshUserHome, '.zshrc'), '')+beforeEach(() => { process.env.ZDOTDIR = zshUserHome })+afterAll(() => {+ if (suiteZdotdir === undefined) delete process.env.ZDOTDIR+ else process.env.ZDOTDIR = suiteZdotdir+ rmSync(zshUserHome, { recursive: true, force: true })+})+/** The prelude every case below gets by default: no managed grid on this machine, so PATH is left* alone and only grid's update check is turned off. */const GRID_PRELUDE = gridPanePrelude('grid')@@ -1416,3 +1431,95 @@ async function runPaneScript()})}++describe('zsh\'s new-user menu, kept out of an agent\'s pane', () => {+ // Debian, Ubuntu, Fedora and Arch build zsh with its zsh/newuser module: on a terminal, for a person+ // with no .zshenv, .zprofile, .zshrc or .zlogin, it runs a full-screen setup menu before anything else.+ // On the end-to-end suite's first Linux run every zsh agent's pane showed it, and no engine started.+ let root = ''+ let savedDataDir = ''+ beforeEach(() => {+ root = mkdtempSync(join(tmpdir(), 'launch-newuser-'))+ savedDataDir = env.ADAPTER_DATA_DIR+ env.ADAPTER_DATA_DIR = join(root, 'data')+ })+ afterEach(() => {+ env.ADAPTER_DATA_DIR = savedDataDir+ rmSync(root, { recursive: true, force: true })+ })+ const folder = () => join(root, 'data', 'zsh-startup')++ it('points a person with no zsh startup files at a .zshenv of Harness\'s that puts their ZDOTDIR back', () => {+ const home = join(root, 'home')+ mkdirSync(home)+ expect(zshNewUserGuard({ HOME: home })).toEqual([`ZDOTDIR=${folder()}`])+ expect(zshNewUserGuard({ HOME: home, ZDOTDIR: home })).toEqual([`ZDOTDIR=${folder()}`, `HARNESS_ZDOTDIR=${home}`])+ expect(readFileSync(join(folder(), '.zshenv'), 'utf8')).toBe(ZSH_GUARD_ZSHENV)+ expect(statSync(join(folder(), '.zshenv')).mode & 0o777).toBe(0o600)+ // In the launch itself, before the shell and its flags.+ process.env.ZDOTDIR = home+ expect(buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, undefined, NO_TMUX).slice(0, 6)).toEqual([+ '/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', `ZDOTDIR=${folder()}`, `HARNESS_ZDOTDIR=${home}`, '/bin/zsh', '-lic',+ ])+ })++ it.each(['.zshenv', '.zprofile', '.zshrc', '.zlogin'])('leaves anyone with a %s exactly as they were', (name) => {+ const home = join(root, 'home')+ mkdirSync(home)+ writeFileSync(join(home, name), '')+ expect(zshNewUserGuard({ HOME: home })).toEqual([])+ expect(zshNewUserGuard({ HOME: join(root, 'elsewhere'), ZDOTDIR: home })).toEqual([])+ expect(existsSync(folder())).toBe(false)+ })++ it('launches as before when it cannot write its .zshenv, or there is no home to look in', () => {+ writeFileSync(join(root, 'data'), 'a file where the data folder should be')+ expect(zshNewUserGuard({ HOME: root })).toEqual([])+ expect(zshNewUserGuard({})).toEqual([])+ })++ // What the .zshenv does, in a real zsh: a person's ZDOTDIR, set or unset, is theirs again before any+ // of their own files is read, and a startup file of theirs that appears later is read from it.+ it.skipIf(!existsSync('/bin/zsh'))('gives a real zsh back the person\'s ZDOTDIR, set or unset, and reads their files from it', () => {+ const home = join(root, 'home')+ mkdirSync(home)+ const run = (environment: NodeJS.ProcessEnv) => {+ const guard = zshNewUserGuard(environment)+ expect(guard).not.toEqual([])+ const out = execFileSync('/usr/bin/env', [...guard, '/bin/zsh', '-lic', 'print -r -- "${ZDOTDIR-unset}|${+HARNESS_ZDOTDIR}|${HARNESS_PROFILE-}"'], {+ encoding: 'utf8', env: { PATH: process.env.PATH, ...environment }, stdio: ['ignore', 'pipe', 'ignore'],+ })+ return out.trim().split('\n').at(-1)+ }+ expect(run({ HOME: home })).toBe('unset|0|')+ expect(run({ HOME: join(root, 'not-here'), ZDOTDIR: home })).toBe(`${home}|0|`)+ // A .zshenv only at the guard's decision: one written after it is still the person's to run.+ const later = join(root, 'later')+ mkdirSync(later)+ const guard = zshNewUserGuard({ HOME: later })+ writeFileSync(join(later, '.zshenv'), 'export HARNESS_PROFILE=theirs\n')+ const out = execFileSync('/usr/bin/env', [...guard, '/bin/zsh', '-lic', 'print -r -- "${ZDOTDIR-unset}|${HARNESS_PROFILE-}"'], {+ encoding: 'utf8', env: { PATH: process.env.PATH, HOME: later }, stdio: ['ignore', 'pipe', 'ignore'],+ })+ expect(out.trim().split('\n').at(-1)).toBe('unset|theirs')+ })++ // The menu itself needs zsh's newuser script (not on macOS) and a terminal, which a pseudo-terminal+ // from python's pty module provides. Without the guard the menu waits for a key and the marker never+ // comes; with it the shell runs the command at once.+ const newuserScript = ['/usr/share/zsh', '/usr/local/share/zsh'].flatMap((base) => {+ try { return readdirSync(base).map((version) => join(base, version, 'scripts', 'newuser')) } catch { return [] }+ }).find((file) => existsSync(file))+ it.skipIf(!existsSync('/bin/zsh') || !newuserScript || !existsSync('/usr/bin/python3'))('runs a new user\'s pane straight to its command, with no setup menu', () => {+ const home = join(root, 'home')+ mkdirSync(home)+ const argv = ['/usr/bin/env', ...zshNewUserGuard({ HOME: home }), '/bin/zsh', '-lic', 'print HARNESS-MARKER; exit']+ const pty = `import os, pty, sys, select, time\npid, fd = pty.fork()\nif pid == 0:\n os.execvpe(sys.argv[1], sys.argv[1:], os.environ)\nout = b''\nend = time.time() + 8\nwhile time.time() < end and b'HARNESS-MARKER' not in out:\n r, _, _ = select.select([fd], [], [], 0.2)\n if r:\n try: out += os.read(fd, 4096)\n except OSError: break\nos.kill(pid, 9)\nsys.stdout.write(out.decode('utf8', 'replace'))\n`+ const out = execFileSync('/usr/bin/python3', ['-c', pty, ...argv], {+ encoding: 'utf8', env: { PATH: process.env.PATH, HOME: home, TERM: 'xterm', LINES: '40', COLUMNS: '120' }, timeout: 20_000,+ })+ expect(out).not.toContain('zsh-newuser-install')+ expect(out).toContain('HARNESS-MARKER')+ })+})+diff --git a/cli/src/lib/engineLaunch.ts b/cli/src/lib/engineLaunch.tsindex 0a0f5b45e..35a7dafa8 100644--- a/cli/src/lib/engineLaunch.ts+++ b/cli/src/lib/engineLaunch.ts@@ -1,6 +1,6 @@import { execFile } from 'node:child_process'import { randomUUID } from 'node:crypto'-import { existsSync, mkdirSync, readdirSync, realpathSync, rmSync, statSync, writeFileSync } from 'node:fs'+import { existsSync, mkdirSync, readdirSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from 'node:fs'import { homedir, userInfo } from 'node:os'import { isAbsolute, basename, dirname, join } from 'node:path'import { env } from '../config/env.js'@@ -440,11 +440,47 @@ export function interactiveEngineShell(shell: string | undefined = undefined): Ifunction engineShellArgv(shell: InteractiveEngineShell, args: readonly string[]): string[] {if (!isPosixShell(shell.path)) return throughPosixShell(shell, args)const prefix = basename(shell.path).toLowerCase() === 'zsh'- ? ['/usr/bin/env', 'DISABLE_AUTO_UPDATE=true']+ ? ['/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', ...zshNewUserGuard()]: []return [...prefix, shell.path, ...shell.args, ...args]}+/** The startup files zsh's new-user module looks for (zshmodules(1), zsh/newuser). */+const ZSH_STARTUP_FILES = ['.zshenv', '.zprofile', '.zshrc', '.zlogin'] as const++/** The .zshenv in Harness's own ZDOTDIR (`zshNewUserGuard`): the person's ZDOTDIR back as it was, or unset,+ * before anything of theirs is read; zsh then reads .zprofile, .zshrc and .zlogin from theirs. */+export const ZSH_GUARD_ZSHENV = `# Written by Harness (engineLaunch.ts zshNewUserGuard): keeps zsh's new-user menu out of an agent's pane.+if (( \${+HARNESS_ZDOTDIR} )); then ZDOTDIR="\$HARNESS_ZDOTDIR"; unset HARNESS_ZDOTDIR; else unset ZDOTDIR; fi+[[ -r "\${ZDOTDIR:-\$HOME}/.zshenv" ]] && builtin source "\${ZDOTDIR:-\$HOME}/.zshenv"+`++/**+ * Keeps zsh's new-user menu out of an engine's pane. Debian, Ubuntu, Fedora and Arch ship zsh's+ * `zsh/newuser` module: on a terminal, for someone with none of the four startup files in $ZDOTDIR (else+ * $HOME), it runs a full-screen menu that waits for a key, so every agent's pane showed it and no engine+ * started (the end-to-end suite's first Linux runs, 2026-10-06; macOS's zsh has no such module). The+ * module looks only there, right after the global zshenv: for such a person ZDOTDIR points at a folder of+ * Harness's whose .zshenv puts theirs back (`HARNESS_ZDOTDIR`; absent means unset). Anyone with a startup+ * file of their own starts exactly as before.+ */+export function zshNewUserGuard(environment: NodeJS.ProcessEnv = process.env): string[] {+ const dotdir = environment.ZDOTDIR || environment.HOME+ if (!dotdir || ZSH_STARTUP_FILES.some((name) => existsSync(join(dotdir, name)))) return []+ const folder = join(env.ADAPTER_DATA_DIR, 'zsh-startup')+ try {+ mkdirSync(folder, { recursive: true, mode: 0o700 })+ const file = join(folder, '.zshenv')+ let current: string | null = null+ try { current = readFileSync(file, 'utf8') } catch { /* not written yet */ }+ if (current !== ZSH_GUARD_ZSHENV) writeFileSync(file, ZSH_GUARD_ZSHENV, { mode: 0o600 })+ } catch {+ // A ZDOTDIR without its .zshenv would leave the person's own ZDOTDIR unrestored: launch as before.+ return []+ }+ return [`ZDOTDIR=${folder}`, ...(environment.ZDOTDIR !== undefined ? [`HARNESS_ZDOTDIR=${environment.ZDOTDIR}`] : [])]+}+/*** For a shell that is not POSIX: it loads the person's environment (their PATH, their version* managers), then hands the script to a POSIX shell (`posixRunner`). The script and its arguments godiff --git a/cli/vitest.e2e.ci.config.ts b/cli/vitest.e2e.ci.config.tsnew file mode 100644index 000000000..634dfa915--- /dev/null+++ b/cli/vitest.e2e.ci.config.ts@@ -0,0 +1,14 @@+import { defineConfig, mergeConfig } from 'vitest/config'+import config from './vitest.e2e.config.js'+import { createDurationSequencer, E2E_HINT_PATH, readTimingHintsFile } from './src/__fixtures__/ciTestSequencer.js'++/**+ * The end-to-end suite as CI runs it (.github/workflows/cli-e2e.yml): the same files, timeouts and+ * one-file-at-a-time default as `vitest.e2e.config.ts`, split into shards by the timing hints in+ * `ci-e2e-durations.json`. Vitest's own `--shard` splits by file count, and the files here run from+ * twenty seconds to six and a half minutes: by count, one runner could draw the four longest.+ * `--config vitest.e2e.ci.config.ts --shard=N/8` reproduces a CI shard's files locally.+ */+export default mergeConfig(config, defineConfig({+ test: { sequence: { sequencer: createDurationSequencer(readTimingHintsFile('ci-e2e-durations.json', E2E_HINT_PATH)) } },+}))diff --git a/cli/vitest.e2e.config.ts b/cli/vitest.e2e.config.tsindex 919c1de4b..55763d935 100644--- a/cli/vitest.e2e.config.ts+++ b/cli/vitest.e2e.config.ts@@ -12,13 +12,16 @@ import { defineConfig } from 'vitest/config'** `E2E_BUNDLE=1` starts every daemon from one bundle built for the run (e2e/harness/bundle.ts):* less CPU per start, the code frozen as the run began, and the build that ships under test.+ *+ * `E2E_ARTIFACTS_DIR=<folder>` keeps the whole log of every daemon a failing test ran there, as files+ * (e2e/harness/artifacts.ts); CI uploads them.*/const workers = Math.max(1, Number(process.env.E2E_WORKERS ?? 1) || 1)export default defineConfig({test: {environment: 'node',- setupFiles: ['./vitest.setup.ts'],+ setupFiles: ['./vitest.setup.ts', './e2e/harness/artifacts.ts'],globalSetup: ['./e2e/harness/bundle.ts'],include: ['e2e/**/*.e2e.ts'],testTimeout: 180_000,diff --git a/docs/validation-and-release.md b/docs/validation-and-release.mdindex faf2aafb6..5bf3aae01 100644--- a/docs/validation-and-release.md+++ b/docs/validation-and-release.md@@ -69,6 +69,22 @@ shuffling, is unchanged. Use `--config vitest.ci.config.ts --shard=N/4` to reprothe CI assignment locally. The aggregate still requires every discovered fileexactly once, independent of these estimates.+The CLI's end-to-end suite (`cli/e2e`) runs in its own workflow, **CLI end to end**+(`.github/workflows/cli-e2e.yml`): after each merge to `main` that touches `cli/`, nightly,+and on demand from a branch (Actions → CLI end to end → Run workflow).+Eight Linux runners each take a shard, planned from `cli/ci-e2e-durations.json` the same+way (`--config vitest.e2e.ci.config.ts --shard=N/8` reproduces one locally). Each shard+installs tmux, zsh, tcsh and dash, starts every daemon from one bundle (`E2E_BUNDLE=1`) and+runs its files one at a time. The full-disk tests, which need macOS disk images, run on one+macOS runner with `DISKFULL=1`. `e2e-summary` applies the default suite's rule: every+discovered file exactly once, every shard passing. Its summary records each file's duration+for refreshing the hints. A failing test's complete daemon logs and engine hook logs are+uploaded as `cli-e2e-daemon-logs-N`, by file and test (`E2E_ARTIFACTS_DIR`, which works+locally too). It is advisory, and so it does not run on pull requests: a failed check makes+GitHub report a PR unstable, which `make merge-pr` refuses to merge. Once it has stayed+green, run it on pull requests, make `e2e-summary` required and add it to the evidence+collector.+For repository process tooling only, `scope=process` runs its Python regressiontests without installing or building unrelated components. It does not validateapplication changes. Workflow edits also need `actionlint` and a run exercisingdiff --git a/scripts/tests/test_cli_test_shards.py b/scripts/tests/test_cli_test_shards.pyindex 977cf869e..eae5f8b16 100644--- a/scripts/tests/test_cli_test_shards.py+++ b/scripts/tests/test_cli_test_shards.py@@ -107,6 +107,74 @@ class ShardTests(unittest.TestCase):self.assertEqual(result.returncode, 1)self.assertFalse(output.exists())+ def test_durations_are_recorded_rounded_up_for_the_next_runs_hints(self):+ self.edit("result-1.json", lambda r: r["testResults"][0].update(startTime=1_000, endTime=2_201))+ self.edit("result-2.json", lambda r: r["testResults"][0].update(startTime=5, endTime="late"))+ result = self.verify()+ self.assertEqual(result["durations_ms"], {"src/case-0.spec.ts": 1300})+ self.assertEqual(result["suite"], "unit")++ def test_the_unit_suite_still_requires_its_contract_job(self):+ with self.assertRaisesRegex(ValueError, "upstream jobs"):+ self.verify(contracts=None)+++class EndToEndShardTests(ShardTests):+ """The same contract for the end-to-end suite: its own folder and names, and no contract job."""++ def setUp(self):+ super().setUp()+ self.names = [str(self.root / "e2e" / f"case-{i}.e2e.ts") for i in range(4)]+ for number in range(1, 5):+ self.edit(f"inventory-{number}.json", lambda paths: [item.update(file=self.names[i]) for i, item in enumerate(paths)])+ self.edit(f"result-{number}.json", lambda r, n=number: r["testResults"][0].update(name=self.names[n - 1]))++ def verify(self, tests="success", contracts=None):+ return verifier.verify(self.directory, self.root, 4, tests, contracts, suite="e2e")++ def test_complete_partition_preserves_pass_skip_counts_and_hashes(self):+ result = self.verify()+ self.assertEqual((result["suite"], result["files"], result["passed"], result["skipped"]), ("e2e", 4, 4, 4))+ self.assertEqual(result["verified_files"], [f"e2e/case-{i}.e2e.ts" for i in range(4)])++ def test_json_success_cannot_override_nonpassing_jobs(self):+ for state in ["failure", "cancelled", "skipped", ""]:+ with self.subTest(tests=state), self.assertRaisesRegex(ValueError, "upstream jobs"):+ self.verify(state)+ # A contract verdict, when one is given, still has to be a pass.+ with self.assertRaisesRegex(ValueError, "upstream jobs"):+ self.verify("success", "failure")++ def test_the_unit_suite_still_requires_its_contract_job(self):+ self.assertEqual(self.verify(contracts="success")["files"], 4)++ def test_durations_are_recorded_rounded_up_for_the_next_runs_hints(self):+ self.edit("result-3.json", lambda r: r["testResults"][0].update(startTime=0, endTime=264_249))+ self.assertEqual(self.verify()["durations_ms"], {"e2e/case-2.e2e.ts": 264_300})++ def test_a_unit_spec_cannot_count_as_an_end_to_end_file_or_the_reverse(self):+ unit = str(self.root / "src" / "case-0.spec.ts")+ self.edit("result-1.json", lambda r: r["testResults"][0].update(name=unit))+ with self.assertRaisesRegex(ValueError, "unexpected test path"):+ self.verify()+ with self.assertRaisesRegex(ValueError, "unexpected test path"):+ verifier.verify(self.directory, self.root, 4, "success", "success")++ def test_command_writes_a_verified_summary_and_fails_on_incomplete_artifacts(self):+ output = self.directory / "summary.json"+ command = [sys.executable, str(SCRIPT), str(self.directory), "--root", str(self.root),+ "--shards", "4", "--tests-result", "success", "--suite", "e2e", "--output", str(output)]+ result = subprocess.run(command, capture_output=True, text=True, timeout=5)+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertEqual(json.loads(output.read_text())["files"], 4)+ output.unlink()+ # Without --suite it is the default suite, which needs its contract job's verdict.+ unit = [arg for arg in command if arg not in ("--suite", "e2e")]+ result = subprocess.run(unit, capture_output=True, text=True, timeout=5)+ self.assertEqual(result.returncode, 2)+ self.assertIn("--contracts-result", result.stderr)+ self.assertFalse(output.exists())+if __name__ == "__main__":unittest.main()diff --git a/scripts/verify-cli-test-shards.py b/scripts/verify-cli-test-shards.pyindex 97e1ae790..ff2330492 100644--- a/scripts/verify-cli-test-shards.py+++ b/scripts/verify-cli-test-shards.py@@ -1,5 +1,11 @@#!/usr/bin/env python3-"""Verify that passing Vitest CI shards cover the discovered file set exactly once."""+"""Verify that passing Vitest CI shards cover the discovered file set exactly once.++Two suites use it: the default suite (`--suite unit`, ci.yml, four shards beside the contract job) and+the end-to-end suite (`--suite e2e`, cli-e2e.yml, which has no contract job). The summary also records+each file's duration, rounded up to 100ms: the timing hints the next run's shards are planned with+(cli/ci-test-durations.json, cli/ci-e2e-durations.json) are refreshed from it.+"""import argparsefrom collections import Counterimport hashlib@@ -8,14 +14,21 @@ from pathlib import Pathimport sys-def verify(directory, root, shards, tests_result, contracts_result):+# Where each suite's files live, relative to cli/, and how they are named.+SUITES = {"unit": ("src/", (".spec.ts", ".test.ts")), "e2e": ("e2e/", (".e2e.ts",))}+++def verify(directory, root, shards, tests_result, contracts_result, suite="unit"):+ folder, suffixes = SUITES[suite]# Vitest JSON can say success even when an unhandled error makes its process- # fail. Require the actual matrix/contract job verdicts, not JSON alone.- if tests_result != "success" or contracts_result != "success":+ # fail. Require the actual matrix/contract job verdicts, not JSON alone. Only the+ # end-to-end suite runs without a contract job; the default suite always has one.+ if tests_result != "success" or (contracts_result != "success" and not (suite == "e2e" and contracts_result is None)):raise ValueError(f"upstream jobs must pass: tests={tests_result}, contracts={contracts_result}")expected, seen = None, Counter()totals = Counter(passed=0, skipped=0, todo=0)receipts = []+ durations = {}def relative(value):if not isinstance(value, str):@@ -24,7 +37,7 @@ def verify(directory, root, shards, tests_result, contracts_result):if not path.is_absolute():raise ValueError(f"expected an absolute Vitest file path: {value}")result = path.resolve().relative_to(root.resolve()).as_posix()- if not result.startswith("src/") or not result.endswith((".spec.ts", ".test.ts")):+ if not result.startswith(folder) or not result.endswith(suffixes):raise ValueError(f"unexpected test path: {result}")return result@@ -53,6 +66,11 @@ def verify(directory, root, shards, tests_result, contracts_result):seen[name] += 1if file.get("status") != "passed" or file.get("message"):raise ValueError(f"shard {shard}: file failed: {name}")+ # Scheduling hints only: a missing or odd time is left out, never a reason to fail.+ elapsed = file.get("endTime", 0) - file.get("startTime", 0) if all(+ type(file.get(key)) in (int, float) for key in ("startTime", "endTime")) else 0+ if elapsed > 0:+ durations[name] = int(-(-elapsed // 100) * 100)for case in file["assertionResults"]:status = case.get("status")if status not in {"passed", "skipped", "todo"} or case.get("failureMessages"):@@ -69,8 +87,8 @@ def verify(directory, root, shards, tests_result, contracts_result):if set(seen) != expected or any(count != 1 for count in seen.values()):raise ValueError(f"file partition mismatch: missing={sorted(expected - set(seen))}, "f"unexpected={sorted(set(seen) - expected)}, duplicates={sorted(p for p, n in seen.items() if n != 1)}")- return {"schema": 1, "status": "passed", "files": len(seen), **dict(totals),- "shards": receipts, "verified_files": sorted(seen)}+ return {"schema": 1, "suite": suite, "status": "passed", "files": len(seen), **dict(totals),+ "shards": receipts, "verified_files": sorted(seen), "durations_ms": dict(sorted(durations.items()))}def main():@@ -79,14 +97,17 @@ def main():parser.add_argument("--root", type=Path, required=True)parser.add_argument("--shards", type=int, required=True)parser.add_argument("--tests-result", required=True)- parser.add_argument("--contracts-result", required=True)+ parser.add_argument("--contracts-result", help="the contract job's verdict; required for the unit suite")+ parser.add_argument("--suite", choices=sorted(SUITES), default="unit")parser.add_argument("--output", type=Path, required=True)args = parser.parse_args()if args.shards < 1:parser.error("shards must be positive")- summary = verify(args.directory, args.root, args.shards, args.tests_result, args.contracts_result)+ if args.suite == "unit" and args.contracts_result is None:+ parser.error("the unit suite requires --contracts-result")+ summary = verify(args.directory, args.root, args.shards, args.tests_result, args.contracts_result, args.suite)args.output.write_text(json.dumps(summary, indent=2) + "\n")- print(f"Verified {summary['files']} files exactly once: {summary['passed']} passed, "+ print(f"Verified {summary['files']} {args.suite} files exactly once: {summary['passed']} passed, "f"{summary['skipped']} skipped, {summary['todo']} todo across {args.shards} shards")return 0