Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

The end-to-end suite runs in CI on Linux, and three Linux daemon bugs it found are fixed (#843)

Deeabdcbe067e9f
diff --git a/.github/workflows/cli-e2e.yml b/.github/workflows/cli-e2e.yml
new file mode 100644
index 000000000..499a88ae4
--- /dev/null
+++ b/.github/workflows/cli-e2e.yml
@@ -0,0 +1,189 @@
+name: CLI end to end
+
+# The end-to-end suite (cli/e2e): real daemons under throwaway homes, private tmux servers and fake
+# Claude Code and Codex engines. About 60 files and 70 minutes of test time, run one file at a time;
+# until this workflow it ran only on developers' machines, and a full run held a laptop for over an
+# hour. Here it is eight shards on standard Linux runners, free for a public repository.
+#
+# ADVISORY until it has stayed green, and so it does not run on pull requests. A failing check on a
+# PR, required or not, makes GitHub report the PR UNSTABLE, and scripts/merge-validated-pr.py (`make
+# merge-pr`) refuses to merge anything GitHub does not report clean: a per-PR run that failed would
+# block every CLI merge. The suite had only ever run on macOS, and its first Linux runs found real
+# failures (zsh's new-user menu took every agent's pane, engineLaunch.ts zshNewUserGuard) beside
+# runner-specific ones. So it runs after a merge to main, every night, and on demand from a branch
+# (Actions -> CLI end to end -> Run workflow) when a change needs it. Promote it to pull requests,
+# and then to a required check collected by record-ci-validation.py, once it has stayed green.
+on:
+ push:
+ branches: [main]
+ paths:
+ - 'cli/**'
+ - '.github/workflows/cli-e2e.yml'
+ - 'scripts/verify-cli-test-shards.py'
+ schedule:
+ - cron: '17 6 * * *'
+ workflow_dispatch:
+
+permissions:
+ contents: read
+
+# A newer merge to main replaces the run of an older one; a nightly or dispatched run is somebody's
+# record and is left to finish.
+concurrency:
+ group: cli-e2e-${{ github.ref }}-${{ github.event_name }}
+ cancel-in-progress: ${{ github.event_name == 'push' }}
+
+# The managed runtime the CLI ships on; keep ci.yml's and release.yml's pins in sync.
+env:
+ CLI_NODE_VERSION: 22.23.2
+
+jobs:
+ e2e:
+ name: e2e (${{ matrix.shard }})
+ strategy:
+ fail-fast: false
+ matrix:
+ shard: [1, 2, 3, 4, 5, 6, 7, 8]
+ runs-on: ubuntu-latest
+ # About nine minutes of tests per shard by the timing hints, plus install and the bundle.
+ timeout-minutes: 45
+ defaults:
+ run:
+ working-directory: cli
+ env:
+ # tmux and Linux procps mangle what they print without a UTF-8 locale (hook/notify.mjs).
+ LANG: C.UTF-8
+ # The login shell every daemon under test launches its engines through, as on a Mac: the
+ # dotfiles and engine-homes tests write a .zshrc and expect it read.
+ SHELL: /bin/zsh
+ steps:
+ - uses: actions/checkout@v7
+ - uses: actions/setup-node@v7
+ with:
+ node-version: ${{ env.CLI_NODE_VERSION }}
+ cache: npm
+ cache-dependency-path: cli/package-lock.json
+ - name: Install tmux, the shells the tests launch through, and lsof
+ run: |
+ sudo apt-get update -q
+ sudo apt-get install -y -q --no-install-recommends tmux zsh tcsh dash lsof
+ # Every daemon watches its data folder and the engines' transcript folders, and a file
+ # starts dozens of daemons one after another; the runner's default instance limit is 128.
+ sudo sysctl -w fs.inotify.max_user_instances=1024 fs.inotify.max_user_watches=524288
+ tmux -V && zsh --version && tcsh --version && command -v dash
+ # Ubuntu's global zshrc runs compinit, which stops at "insecure directories ... continue [y]
+ # or abort compinit [n]?" when a completion folder is writable by others or owned by someone
+ # else, and the runner image ships /usr/share/zsh and its vendor-completions that way: every
+ # zsh agent's pane waited at that question and no engine started. A person's machine has
+ # none, or they fix it as zsh says, which is this. Fails here if any is left.
+ insecure=$(zsh -fc 'autoload -U compaudit && compaudit' 2>/dev/null | grep '^/' | sort -u || true)
+ if [ -n "$insecure" ]; then
+ echo "$insecure" | xargs sudo chown root:root
+ echo "$insecure" | xargs sudo chmod g-w,o-w
+ fi
+ zsh -fc 'autoload -U compaudit && compaudit'
+ - run: npm ci --no-audit --no-fund
+ # The complete file list, before sharding: the summary requires each file exactly once.
+ - name: Discover the complete end-to-end inventory
+ run: npx vitest list --config vitest.e2e.ci.config.ts --filesOnly --json="$RUNNER_TEMP/inventory-${{ matrix.shard }}.json"
+ # One bundle for every daemon in the shard (e2e/harness/bundle.ts): less CPU per start, and the
+ # build that ships under test. Files run one at a time (E2E_WORKERS unset), so a test's timing
+ # is the daemon's rather than the runner's contention; the eight shards are the parallelism.
+ - name: Run end-to-end shard
+ env:
+ E2E_BUNDLE: '1'
+ E2E_ARTIFACTS_DIR: ${{ runner.temp }}/e2e-daemon-logs
+ run: >-
+ npx vitest run --config vitest.e2e.ci.config.ts --shard=${{ matrix.shard }}/8
+ --reporter=default --reporter=json --outputFile.json="$RUNNER_TEMP/result-${{ matrix.shard }}.json"
+ - name: Preserve inventory and test results
+ if: ${{ !cancelled() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: cli-e2e-shard-${{ matrix.shard }}
+ path: |
+ ${{ runner.temp }}/inventory-${{ matrix.shard }}.json
+ ${{ runner.temp }}/result-${{ matrix.shard }}.json
+ if-no-files-found: error
+ overwrite: true
+ retention-days: 7
+ # Only the failing tests' folders are left there (e2e/harness/artifacts.ts): each daemon's whole
+ # log and its fake engines' hook log, by file and test.
+ - name: Preserve the failing tests' daemon logs
+ if: ${{ failure() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: cli-e2e-daemon-logs-${{ matrix.shard }}
+ path: ${{ runner.temp }}/e2e-daemon-logs/
+ if-no-files-found: ignore
+ overwrite: true
+ retention-days: 7
+
+ # The tests that need macOS: a data folder on a small disk image that fills (hdiutil). Opt-in
+ # locally (DISKFULL=1); here on one macOS runner, filtered to those tests alone, a few minutes.
+ e2e-macos-disk:
+ name: e2e (macOS, full disk)
+ runs-on: macos-15
+ timeout-minutes: 30
+ defaults:
+ run:
+ working-directory: cli
+ steps:
+ - uses: actions/checkout@v7
+ - uses: actions/setup-node@v7
+ with:
+ node-version: ${{ env.CLI_NODE_VERSION }}
+ cache: npm
+ cache-dependency-path: cli/package-lock.json
+ - run: command -v tmux || brew install tmux
+ - run: npm ci --no-audit --no-fund
+ - name: A full disk, and an update on a disk that fills
+ env:
+ DISKFULL: '1'
+ E2E_BUNDLE: '1'
+ E2E_ARTIFACTS_DIR: ${{ runner.temp }}/e2e-daemon-logs
+ run: >-
+ npx vitest run --config vitest.e2e.config.ts e2e/diskfull.e2e.ts e2e/updateHostile.e2e.ts
+ -t 'a full disk|an update on a disk that fills'
+ - name: Preserve the failing tests' daemon logs
+ if: ${{ failure() }}
+ uses: actions/upload-artifact@v4
+ with:
+ name: cli-e2e-daemon-logs-macos-disk
+ path: ${{ runner.temp }}/e2e-daemon-logs/
+ if-no-files-found: ignore
+ overwrite: true
+ retention-days: 7
+
+ # The one check to read (and, once promoted, to require): every discovered file ran exactly once
+ # and passed, on the same rules as ci.yml's `typecheck-test` for the default suite. Its summary
+ # carries each file's duration, which refreshes cli/ci-e2e-durations.json.
+ e2e-summary:
+ needs: e2e
+ if: ${{ !cancelled() }}
+ runs-on: ubuntu-latest
+ timeout-minutes: 3
+ steps:
+ - uses: actions/checkout@v7
+ with:
+ sparse-checkout: scripts/verify-cli-test-shards.py
+ sparse-checkout-cone-mode: false
+ - uses: actions/download-artifact@v4
+ with:
+ pattern: cli-e2e-shard-*
+ path: ${{ runner.temp }}/cli-e2e-shards
+ merge-multiple: true
+ - name: Require every discovered file exactly once and every shard passing
+ env:
+ E2E_TEST_RESULT: ${{ needs.e2e.result }}
+ run: >-
+ python3 scripts/verify-cli-test-shards.py "$RUNNER_TEMP/cli-e2e-shards" --suite e2e
+ --root "$GITHUB_WORKSPACE/cli" --shards 8 --tests-result "$E2E_TEST_RESULT"
+ --output "$RUNNER_TEMP/cli-e2e-summary.json"
+ - uses: actions/upload-artifact@v4
+ with:
+ name: cli-e2e-summary
+ path: ${{ runner.temp }}/cli-e2e-summary.json
+ if-no-files-found: error
+ overwrite: true
+ retention-days: 7
diff --git a/cli/ci-e2e-durations.json b/cli/ci-e2e-durations.json
new file mode 100644
index 000000000..0d5e17318
--- /dev/null
+++ b/cli/ci-e2e-durations.json
@@ -0,0 +1,22 @@
+{
+ "schema": 1,
+ "source": "Local end-to-end runs on a 12-core Mac, 2026-10-05: each file's own time in a run that did not filter its tests. Replace with the durations_ms of a passing cli-e2e-summary artifact.",
+ "description": "Scheduling hints only, for the end-to-end CI shards (vitest.e2e.ci.config.ts). Files without a hint cost the default. Discovery always determines the complete test set.",
+ "defaultDurationMs": 60000,
+ "durationMs": {
+ "e2e/bounded.e2e.ts": 115300,
+ "e2e/chaos.e2e.ts": 148400,
+ "e2e/compaction.e2e.ts": 50000,
+ "e2e/core.e2e.ts": 121200,
+ "e2e/enginehomes.e2e.ts": 131900,
+ "e2e/history.e2e.ts": 113000,
+ "e2e/races.e2e.ts": 264300,
+ "e2e/recovery.e2e.ts": 28700,
+ "e2e/shells.e2e.ts": 130400,
+ "e2e/soak.e2e.ts": 180100,
+ "e2e/startup.e2e.ts": 19700,
+ "e2e/terminal.e2e.ts": 155900,
+ "e2e/tmuxconf.e2e.ts": 74500,
+ "e2e/updateHostile.e2e.ts": 392800
+ }
+}
diff --git a/cli/e2e/folders.e2e.ts b/cli/e2e/folders.e2e.ts
index aae3a08b8..ba0a4724f 100644
--- a/cli/e2e/folders.e2e.ts
+++ b/cli/e2e/folders.e2e.ts
@@ -114,9 +114,12 @@ describe('the folders agents work in', () => {
rmSync(join(d.projectsDir, 'gone-by-morning'), { recursive: true, force: true })
await d.start()
client = await LocalClient.connect(d)
+ // Back means its engine is up again, not only its row: a restore lists the agent, on its conversation,
+ // from the moment it starts relaunching it, and a message sent before the engine is there is refused
+ // (on Linux the relaunch was still under way when the row first read as active).
await until('the agent whose folder is still there to come back', async () => {
const now = await row(client, kept.id)
- return now?.status === 'active' && now.sessionId === kept.sessionId ? now : null
+ return now?.status === 'active' && now.sessionId === kept.sessionId && now.launch?.state !== 'starting' ? now : null
}, 60_000, 500)
await turn(client, kept.id, 'restored beside one that could not be')
// The one whose folder is gone is not shown as active, and the daemon did not restart over it.
diff --git a/cli/e2e/harness/artifacts.ts b/cli/e2e/harness/artifacts.ts
new file mode 100644
index 000000000..8c5be5591
--- /dev/null
+++ b/cli/e2e/harness/artifacts.ts
@@ -0,0 +1,76 @@
+/**
+ * `E2E_ARTIFACTS_DIR=<folder>`: keep the whole log of every daemon a failing test ran, as files, for
+ * CI to upload (.github/workflows/cli-e2e.yml). A failing test prints the tail of a daemon's log to the
+ * console (`onTestFailed` in each file), which is what a developer at the terminal reads; a CI run has
+ * eight runners' worth of console and no machine to look at afterwards, and the line that explains a
+ * failure is often hundreds of lines above that tail.
+ *
+ * Every daemon writes its output here as it goes (`IsolatedDaemon` calls `artifactLog`), into the
+ * folder of the test running at that moment: `<dir>/<file>/<test>/<daemon>.log`. Output written while
+ * no test runs (a `beforeAll`'s daemon starting) goes to `<dir>/<file>/outside-tests/`. Once a test has
+ * finished, after its own `afterEach` has closed its daemons, its folder is removed if it passed; the
+ * file's `outside-tests` folder is removed at the end of the file if every test in it passed. What is
+ * left is the failures, and only those are uploaded. Written synchronously, so a worker that dies or a
+ * test that times out leaves everything up to that moment.
+ *
+ * A vitest setup file (vitest.e2e.config.ts): its hooks are the first registered, so they run around
+ * every test's own.
+ */
+import { appendFileSync, mkdirSync, readdirSync, rmSync } from 'node:fs'
+import { basename, join } from 'node:path'
+import { afterAll, beforeEach, expect } from 'vitest'
+
+const ROOT = process.env.E2E_ARTIFACTS_DIR || null
+
+/** A test's name as a folder: what a person can still read, and nothing a path cannot hold. */
+export function folderName(name: string): string {
+ const safe = name.replace(/[^A-Za-z0-9._-]+/g, '-').replace(/^-+|-+$/g, '')
+ return (safe || 'unnamed').slice(0, 120)
+}
+
+/** The folder of the test file running in this worker ([path], or the one vitest says is running). */
+function fileFolder(path = expect.getState().testPath): string | null {
+ if (!ROOT || !path) return null
+ return join(ROOT, folderName(basename(path).replace(/\.e2e\.ts$/, '')))
+}
+
+let current: string | null = null
+
+/** Whether this run keeps artifacts at all: callers skip work whose only use is an artifact. */
+export function artifactsEnabled(): boolean {
+ return ROOT !== null
+}
+
+/** Append [text] to `<the running test's folder>/<name>`. A no-op unless E2E_ARTIFACTS_DIR is set. */
+export function artifactLog(name: string, text: string): void {
+ const folder = current ?? (fileFolder() && join(fileFolder()!, 'outside-tests'))
+ if (!folder || !text) return
+ try {
+ mkdirSync(folder, { recursive: true })
+ appendFileSync(join(folder, folderName(name)), text)
+ } catch { /* a full or missing disk must not fail the test this only records */ }
+}
+
+if (ROOT) {
+ let failed = false
+ beforeEach((context) => {
+ // A test skipped from inside (`ctx.skip()`) never calls its onTestFinished: nothing it wrote failed.
+ if (current) rmSync(current, { recursive: true, force: true })
+ const file = fileFolder(context.task.file.filepath)
+ if (!file) return
+ const folder = join(file, folderName(context.task.fullTestName ?? context.task.name))
+ current = folder
+ // After every afterEach, so a daemon closed in one has said all it will.
+ context.onTestFinished(() => {
+ if (context.task.result?.state === 'fail') failed = true
+ else rmSync(folder, { recursive: true, force: true })
+ if (current === folder) current = null
+ })
+ })
+ afterAll(() => {
+ const file = fileFolder()
+ if (!file) return
+ if (!failed) rmSync(join(file, 'outside-tests'), { recursive: true, force: true })
+ try { if (!readdirSync(file).length) rmSync(file, { recursive: true, force: true }) } catch { /* never made */ }
+ })
+}
diff --git a/cli/e2e/harness/daemon.ts b/cli/e2e/harness/daemon.ts
index 99a7b67d5..ffff1714a 100644
--- a/cli/e2e/harness/daemon.ts
+++ b/cli/e2e/harness/daemon.ts
@@ -18,6 +18,7 @@ import { dirname, isAbsolute, join, relative, resolve } from 'node:path'
import { fileURLToPath, pathToFileURL } from 'node:url'
import { promisify } from 'node:util'
import { isolatedTmux, type IsolatedTmux } from '../../src/testing/isolatedTmux.js'
+import { artifactLog, artifactsEnabled } from './artifacts.js'
const exec = promisify(execFile)
const here = dirname(fileURLToPath(import.meta.url))
@@ -193,6 +194,11 @@ export class IsolatedDaemon {
projects: join(root, 'projects'),
}
for (const dir of Object.values(dirs)) await mkdir(dir, { recursive: true })
+ // A zsh user as most are, with a .zshrc of their own (empty: nothing of anyone's). With none, the zsh
+ // of Debian, Ubuntu and Fedora opens its new-user setup menu in every terminal tile and every shell an
+ // engine leaves behind, and a test typing into one types into the menu. The person with no startup
+ // files at all has their own case (shells.e2e.ts). Appended, never truncated: a test's own stays.
+ await writeFile(join(dirs.home, '.zshrc'), '', { flag: 'a' })
const config: EngineConfig = {
port, dataDir: dirs.data, claudeProjectsDir: dirs.claudeProjects, codexHome: dirs.codexHome,
claudeModel: options.claudeModel, codexModel: options.codexModel,
@@ -299,7 +305,14 @@ export class IsolatedDaemon {
cwd: CLI_ROOT, env: this.env, stdio: ['ignore', 'pipe', 'pipe'],
})
this.child = child
- const take = (chunk: Buffer) => { this.output += chunk.toString('utf8') }
+ // The whole log, as files, for a failing test's CI artifacts (E2E_ARTIFACTS_DIR, artifacts.ts).
+ const logName = `daemon-${this.port}.log`
+ artifactLog(logName, `---- ${new Date().toISOString()} start (pid ${child.pid}, ${script.at(-1)} ${entry}, data ${this.dataDir})\n`)
+ const take = (chunk: Buffer) => {
+ const text = chunk.toString('utf8')
+ this.output += text
+ artifactLog(logName, text)
+ }
child.stdout!.on('data', take)
child.stderr!.on('data', take)
const exited = new Promise<never>((_, reject) => child.once('exit', (code, signal) => {
@@ -372,14 +385,35 @@ export class IsolatedDaemon {
return readFileSync(join(this.dataDir, 'hook-credential'), 'utf8').trim()
}
+ /**
+ * What every pane on this daemon's tmux server shows, for a failing test's artifacts: its command, how
+ * it started, and its screen with some history. A pane is the one thing the daemon's log cannot say:
+ * the suite's first Linux run failed on a setup menu zsh drew in every agent's pane, and the logs said
+ * only that no engine ever appeared.
+ */
+ private async keepPanes(): Promise<void> {
+ try {
+ const panes = await this.tmux.run('list-panes', '-a', '-F', '#{pane_id}\t#{session_name}\t#{pane_dead}\t#{pane_current_command}\t#{pane_start_command}')
+ let text = ''
+ for (const line of panes.split('\n').filter(Boolean)) {
+ const screen = await this.tmux.run('capture-pane', '-p', '-J', '-S', '-200', '-t', line.split('\t')[0]).catch((error) => String(error))
+ text += `==== ${line}\n${screen}\n`
+ }
+ artifactLog(`daemon-${this.port}-panes.txt`, text)
+ } catch { /* no server any more: nothing on screen to keep */ }
+ }
+
async close(): Promise<void> {
const core = this.options.noMaster ? null : this.corePid()
+ // While the panes are still there: a daemon that stops can take its agents' panes with it.
+ if (artifactsEnabled()) await this.keepPanes()
await this.stop().catch(() => {})
// Once the daemon has said all it will, and before the root (and the engines' hook log) is removed.
let escaped: unknown = null
try { this.assertHooksStayedInside() } catch (error) { escaped = error }
// A failed test prints it after this close has removed the root.
this.closedHookLog = this.hookLog()
+ artifactLog(`daemon-${this.port}-engine-hooks.log`, this.closedHookLog)
// A core whose master died during the test must not outlive it: 20 such cores, left by a run
// whose masters crashed, spun on their closed output pipes for the rest of a parallel suite. Only
// this test's core is killed — its pid, still running this daemon's entry, orphaned or ours.
diff --git a/cli/e2e/shells.e2e.ts b/cli/e2e/shells.e2e.ts
index 6ab753fd2..0a8b63ef0 100644
--- a/cli/e2e/shells.e2e.ts
+++ b/cli/e2e/shells.e2e.ts
@@ -5,7 +5,7 @@
* ships with macOS, so it stands in for every shell that is not POSIX: an agent must start, in its
* own folder, and work, whichever shell it is.
*/
-import { mkdirSync, realpathSync } from 'node:fs'
+import { existsSync, mkdirSync, realpathSync, rmSync } from 'node:fs'
import { join } from 'node:path'
import { afterEach, describe, expect, it, onTestFailed } from 'vitest'
import { LocalClient, type Frame } from './harness/client.js'
@@ -46,7 +46,9 @@ describe('the person\'s login shell', () => {
const answer = await client.request('terminal_info', { agentId: created.agent.id }, 10_000)
return answer.command && !answer.error && answer.command !== 'sh' ? answer : null
}, 30_000, 500)
- expect(info.command).toBe(shell.split('/').at(-1))
+ // tmux names the command from argv[0] on Linux, and Debian's tcsh rewrites its own to "-bin/tcsh",
+ // which tmux reads as "bin/tcsh"; macOS's tmux reads the process name. The shell is the same.
+ expect(String(info.command).split('/').at(-1)).toBe(shell.split('/').at(-1))
expect(realpathSync(info.path)).toBe(realpathSync(cwd))
client.close()
}, 120_000)
@@ -72,4 +74,29 @@ describe('the person\'s login shell', () => {
client.close()
}, 240_000)
}
+
+ // A zsh user with no startup files of their own. The zsh of Debian, Ubuntu, Fedora and Arch then opens
+ // its new-user setup menu on any terminal before anything else, and every agent's pane showed that menu
+ // instead of starting its engine: on the suite's first Linux run, no zsh agent bound at all. The launch
+ // keeps the menu out of an agent's pane (engineLaunch.ts zshNewUserGuard). macOS's zsh has no such menu.
+ it.skipIf(!existsSync('/bin/zsh')).each(['claude', 'codex'] as const)('%s for a zsh user with no startup files: an agent starts and works', async (engine: Engine) => {
+ const d = await IsolatedDaemon.create({ env: { SHELL: '/bin/zsh' } })
+ daemon = d
+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-150).join('\n')}`) })
+ rmSync(join(d.env.ZDOTDIR!, '.zshrc'))
+ await d.start()
+ const client = await LocalClient.connect(d)
+ const cwd = join(d.projectsDir, `new-zsh-user-${engine}`)
+ mkdirSync(cwd, { recursive: true })
+ const created = await client.request('agent_create', { engine, cwd, bypassPermission: true }, 120_000)
+ expect(created.error, JSON.stringify(created)).toBeUndefined()
+ const agent = await until(`the ${engine} agent to bind`, async () => {
+ const now = await row(client, created.agent.id)
+ return now?.sessionId && now.status === 'active' ? now : null
+ }, 60_000, 500)
+ await turn(client, agent.id, 'from a new zsh user')
+ // Nothing was written into the person's home to get there.
+ expect(['.zshenv', '.zprofile', '.zshrc', '.zlogin'].filter((name) => existsSync(join(d.env.ZDOTDIR!, name)))).toEqual([])
+ client.close()
+ }, 240_000)
})
diff --git a/cli/e2e/suspend.e2e.ts b/cli/e2e/suspend.e2e.ts
index 6dba45578..c04710998 100644
--- a/cli/e2e/suspend.e2e.ts
+++ b/cli/e2e/suspend.e2e.ts
@@ -72,6 +72,9 @@ async function lineMode(d: IsolatedDaemon, pane: string): Promise<boolean> {
const tty = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_tty}')).trim()
return / icanon /.test(` ${execFileSync('/bin/sh', ['-c', 'stty -a < "$0"', tty], { encoding: 'utf8' }).replace(/\s+/g, ' ')} `)
}
+// The pane's command as tmux names it: on Linux, from the process's argv[0], and Debian's tcsh rewrites
+// its own to "-bin/tcsh", which tmux reads as "bin/tcsh" (macOS's tmux reads the process name, "tcsh").
+// Compared by basename.
const processState = (pid: number): string => {
try { return execFileSync('ps', ['-o', 'stat=', '-p', String(pid)], { encoding: 'utf8' }).trim() } catch { return '' }
}
@@ -140,7 +143,7 @@ describe('an engine that is stopped, or killed', () => {
await until('the agent to stop counting as active', async () => (await row(client, agent.id))?.status !== 'active' || null, 45_000, 500)
expect(await until('the pane to be a shell', async () => {
const [dead, command] = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_dead} #{pane_current_command}')).trim().split(' ')
- return dead === '0' && command === basename(shell) ? command : null
+ return dead === '0' && basename(command) === basename(shell) ? basename(command) : null
}, 30_000, 200)).toBe(basename(shell))
expect(d.coresStarted()).toBe(1)
client.close()
@@ -170,7 +173,7 @@ describe('an engine that is stopped, or killed', () => {
await until('the agent to stop counting as active', async () => (await row(client, agent.id))?.status !== 'active' || null, 45_000, 500)
expect(await until('the pane to be a shell', async () => {
const [dead, command] = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_dead} #{pane_current_command}')).trim().split(' ')
- return dead === '0' && command === basename(shell) ? command : null
+ return dead === '0' && basename(command) === basename(shell) ? basename(command) : null
}, 30_000, 200)).toBe(basename(shell))
expect(d.coresStarted()).toBe(1)
client.close()
diff --git a/cli/src/__fixtures__/ciTestSequencer.ts b/cli/src/__fixtures__/ciTestSequencer.ts
index 128c18799..1b8d4b7eb 100644
--- a/cli/src/__fixtures__/ciTestSequencer.ts
+++ b/cli/src/__fixtures__/ciTestSequencer.ts
@@ -2,9 +2,13 @@ import { readFileSync } from 'node:fs'
import { relative } from 'node:path'
import { BaseSequencer, type TestSpecification } from 'vitest/node'
-type TimingHints = { defaultDurationMs: number; durationMs: Record<string, number> }
+export type TimingHints = { defaultDurationMs: number; durationMs: Record<string, number> }
-export function readTimingHints(value: unknown): TimingHints {
+/** The files each suite's hints may name: the default suite's specs, and the end-to-end files. */
+export const UNIT_HINT_PATH = /^src\/.+\.(spec|test)\.ts$/
+export const E2E_HINT_PATH = /^e2e\/.+\.e2e\.ts$/
+
+export function readTimingHints(value: unknown, paths: RegExp = UNIT_HINT_PATH): TimingHints {
const raw = value as { schema?: unknown; defaultDurationMs?: unknown; durationMs?: unknown } | null
const positive = (n: unknown): n is number => typeof n === 'number' && Number.isFinite(n) && n > 0
if (raw?.schema !== 1 || !positive(raw.defaultDurationMs) || !raw.durationMs
@@ -12,14 +16,17 @@ export function readTimingHints(value: unknown): TimingHints {
throw new Error('Invalid CLI test timing hints')
}
for (const [path, duration] of Object.entries(raw.durationMs)) {
- if (!/^src\/.+\.(spec|test)\.ts$/.test(path) || path.includes('\\') || path.split('/').includes('..') || !positive(duration)) {
+ if (!paths.test(path) || path.includes('\\') || path.split('/').includes('..') || !positive(duration)) {
throw new Error(`Invalid CLI test timing hint: ${path}`)
}
}
return { defaultDurationMs: raw.defaultDurationMs, durationMs: raw.durationMs as Record<string, number> }
}
-const hints = readTimingHints(JSON.parse(readFileSync(new URL('../../ci-test-durations.json', import.meta.url), 'utf8')))
+/** A suite's hints file (`ci-test-durations.json`, `ci-e2e-durations.json`), read and checked. */
+export function readTimingHintsFile(name: string, paths: RegExp): TimingHints {
+ return readTimingHints(JSON.parse(readFileSync(new URL(`../../${name}`, import.meta.url), 'utf8')), paths)
+}
/** Longest estimated files first, each assigned to the least-loaded shard. Timing hints never
* select tests: the caller's complete discovery list is the only source of entries. New files
@@ -44,17 +51,24 @@ export function durationShards<T>(entries: Array<{ key: string; durationMs: numb
return shards.map(shard => shard.values)
}
-export class DurationSequencer extends BaseSequencer {
- override async shard(files: TestSpecification[]): Promise<TestSpecification[]> {
- const shard = this.ctx.config.shard
- if (!shard) return files
- const entries = files.map(spec => {
- const path = relative(this.ctx.config.root, spec.moduleId).replaceAll('\\', '/')
- return { key: `${spec.project.name}\0${spec.pool}\0${path}`,
- durationMs: hints.durationMs[path] ?? hints.defaultDurationMs, value: spec }
- })
- return durationShards(entries, shard.count)[shard.index - 1]
+/** A sequencer that shards by [hints]; one per suite, since each has its own files and timings. */
+export function createDurationSequencer(hints: TimingHints): typeof BaseSequencer {
+ return class extends BaseSequencer {
+ override async shard(files: TestSpecification[]): Promise<TestSpecification[]> {
+ const shard = this.ctx.config.shard
+ if (!shard) return files
+ const entries = files.map(spec => {
+ const path = relative(this.ctx.config.root, spec.moduleId).replaceAll('\\', '/')
+ return { key: `${spec.project.name}\0${spec.pool}\0${path}`,
+ durationMs: hints.durationMs[path] ?? hints.defaultDurationMs, value: spec }
+ })
+ return durationShards(entries, shard.count)[shard.index - 1]
+ }
+ // Keep Vitest's normal failed-first/cache-aware ordering within each shard, and its
+ // project/isolation/group ordering. Unsharded local runs retain the default behavior.
}
- // Keep Vitest's normal failed-first/cache-aware ordering within each shard, and its
- // project/isolation/group ordering. Unsharded local runs retain the default behavior.
}
+
+/** The default suite's CI shards (`vitest.ci.config.ts`). The end-to-end suite builds its own from
+ * `ci-e2e-durations.json` (`vitest.e2e.ci.config.ts`). */
+export const DurationSequencer = createDurationSequencer(readTimingHintsFile('ci-test-durations.json', UNIT_HINT_PATH))
diff --git a/cli/src/architecture.spec.ts b/cli/src/architecture.spec.ts
index fd0a78154..d251bf0df 100644
--- a/cli/src/architecture.spec.ts
+++ b/cli/src/architecture.spec.ts
@@ -150,8 +150,13 @@ const SERVICE_MAY_IMPORT: Record<string, string> = {
*
* Measured at 105,714 lines in 450 files on 6 October, when runForeground moved out of cli.ts (step 1),
* against 114,622 in 488 walked from cli.ts: the CLI's own commands left the core's process.
+ *
+ * Grew by 54 the same day for three Linux bugs in the core's own launch and hook paths, found by the
+ * end-to-end suite's first Linux runs (#843): zsh's new-user menu kept out of an agent's pane
+ * (lib/engineLaunch.ts), a hook's ancestry read as it arrives (core/engines/hooks.ts), and a relaunch
+ * that must stay up to count (core/agents/swap.ts).
*/
-const CORE_CLOSURE_BUDGET = 105_825
+const CORE_CLOSURE_BUDGET = 105_879
/** What is not the core's, by path: each goes to a service or its own process, in the plan's order. */
const EDGE: RegExp[] = [
diff --git a/cli/src/ciTestSequencer.spec.ts b/cli/src/ciTestSequencer.spec.ts
index fda019c61..e390fd2b3 100644
--- a/cli/src/ciTestSequencer.spec.ts
+++ b/cli/src/ciTestSequencer.spec.ts
@@ -1,7 +1,7 @@
import { resolve } from 'node:path'
import { describe, expect, it } from 'vitest'
import type { TestSpecification, Vitest } from 'vitest/node'
-import { durationShards, DurationSequencer, readTimingHints } from './__fixtures__/ciTestSequencer.js'
+import { createDurationSequencer, durationShards, DurationSequencer, E2E_HINT_PATH, readTimingHints, readTimingHintsFile } from './__fixtures__/ciTestSequencer.js'
const entries = (costs: number[]) => costs.map((durationMs, value) => ({ key: `file-${value}`, durationMs, value }))
@@ -68,4 +68,27 @@ describe('duration-based CLI shards', () => {
expect(new Set(shards.flat())).toEqual(new Set(files))
expect(shards.flat()).toHaveLength(files.length)
})
+
+ it('reads the end-to-end hints with their own file pattern, and neither suite accepts the other\'s files', () => {
+ const e2e = { schema: 1, defaultDurationMs: 60_000, durationMs: { 'e2e/races.e2e.ts': 264_300 } }
+ expect(readTimingHints(e2e, E2E_HINT_PATH)).toEqual({ defaultDurationMs: 60_000, durationMs: e2e.durationMs })
+ expect(() => readTimingHints(e2e)).toThrow('timing hint: e2e/races.e2e.ts')
+ expect(() => readTimingHints({ ...e2e, durationMs: { 'src/slow.spec.ts': 10 } }, E2E_HINT_PATH)).toThrow('timing hint')
+ expect(() => readTimingHints({ ...e2e, durationMs: { 'e2e/../races.e2e.ts': 10 } }, E2E_HINT_PATH)).toThrow('timing hint')
+ // The checked-in hints parse: a malformed file would fail every CI shard before a test ran.
+ expect(readTimingHintsFile('ci-e2e-durations.json', E2E_HINT_PATH).defaultDurationMs).toBeGreaterThan(0)
+ })
+
+ it('splits end-to-end files by their hints, so the longest files land on different shards', async () => {
+ const root = resolve('worktree')
+ const hints = { defaultDurationMs: 60_000, durationMs: { 'e2e/a.e2e.ts': 400_000, 'e2e/b.e2e.ts': 300_000, 'e2e/c.e2e.ts': 250_000 } }
+ const paths = ['e2e/a.e2e.ts', 'e2e/b.e2e.ts', 'e2e/c.e2e.ts', 'e2e/d.e2e.ts', 'e2e/e.e2e.ts', 'e2e/f.e2e.ts']
+ const files = paths.map(path => ({ moduleId: resolve(root, path), project: { name: '' }, pool: 'forks' }) as TestSpecification)
+ const Sequencer = createDurationSequencer(hints)
+ const shards = await Promise.all([1, 2, 3].map(index => new Sequencer({
+ config: { root, shard: { index, count: 3 } },
+ } as Vitest).shard(files).then(shard => shard.map(spec => paths.find(path => resolve(root, path) === spec.moduleId)!))))
+ expect(shards.flat().sort()).toEqual(paths)
+ expect(shards.map(shard => shard.filter(path => path in hints.durationMs))).toEqual([['e2e/a.e2e.ts'], ['e2e/b.e2e.ts'], ['e2e/c.e2e.ts']])
+ })
})
diff --git a/cli/src/core/agents/swap.spec.ts b/cli/src/core/agents/swap.spec.ts
index cbde981ef..456fbe0fc 100644
--- a/cli/src/core/agents/swap.spec.ts
+++ b/cli/src/core/agents/swap.spec.ts
@@ -5,7 +5,7 @@ import { buildEngineLaunchArgv } from '../../lib/engineLaunch.js'
import type { RegisteredSession } from '../../lib/registry.js'
import { processRows } from '../../lib/terminalAgentDiscovery.js'
import { bypassPermissionActive, resolvePaneEngineProcess } from '../../lib/tmux.js'
-import { createPaneSwap, type PaneSwapDeps } from './swap.js'
+import { createPaneSwap, SWAP_SETTLE_MS, type PaneSwapDeps } from './swap.js'
vi.mock('../../lib/deleteAgentFallback.js', async (real) => ({ ...await real<object>(), terminateDeletedAgent: vi.fn(async () => ({ ok: true })) }))
vi.mock('../../lib/engineLaunch.js', async (real) => ({ ...await real<object>(), buildEngineLaunchArgv: vi.fn(() => ['zsh', '-lc', 'claude']) }))
@@ -94,9 +94,9 @@ describe('the pane-process swap', () => {
vi.useFakeTimers()
const { swap } = setup()
const found = { pid: 99, startMarker: 'now' }
- vi.mocked(resolvePaneEngineProcess).mockResolvedValueOnce(null).mockResolvedValueOnce(found as never)
+ vi.mocked(resolvePaneEngineProcess).mockResolvedValueOnce(null).mockResolvedValueOnce(found as never).mockResolvedValueOnce({ ...found } as never)
const waiting = swap.paneSwapDeps(session(), runtime).waitForProcess()
- await vi.advanceTimersByTimeAsync(150 + 300)
+ await vi.advanceTimersByTimeAsync(150 + 300 + SWAP_SETTLE_MS)
expect(await waiting).toEqual(found)
const never = swap.paneSwapDeps(session(), runtime).waitForProcess()
// 150, 300, 600, then 750 ms at a time: the budget is spent on the look that crosses 8 s.
@@ -104,6 +104,32 @@ describe('the pane-process swap', () => {
expect(await never).toBeNull()
})
+ // A launch the engine refuses (`codex resume` after an update dropped it) runs for an instant: seen in
+ // that instant, a restart called the conversation resumed and never fell back to a fresh start.
+ it('counts a process as come up only when it is still the pane\'s engine a moment later', async () => {
+ vi.useFakeTimers()
+ const { swap } = setup()
+ const refused = { pid: 98, startMarker: 'then' }
+ const lasting = { pid: 99, startMarker: 'now' }
+ vi.mocked(resolvePaneEngineProcess)
+ .mockResolvedValueOnce(refused as never).mockResolvedValueOnce(null)
+ // Another process in its place by the second look is not the one seen: it is looked at afresh.
+ .mockResolvedValueOnce(lasting as never).mockResolvedValueOnce({ pid: 99, startMarker: 'later' } as never)
+ .mockResolvedValueOnce(lasting as never).mockResolvedValueOnce({ ...lasting } as never)
+ const waiting = swap.paneSwapDeps(session(), runtime).waitForProcess()
+ await vi.advanceTimersByTimeAsync(150 + SWAP_SETTLE_MS + 300 + SWAP_SETTLE_MS + 600 + SWAP_SETTLE_MS)
+ expect(await waiting).toEqual(lasting)
+ expect(resolvePaneEngineProcess).toHaveBeenCalledTimes(6)
+ // One that never stays up is no engine: the wait ends as one that found none.
+ vi.mocked(resolvePaneEngineProcess).mockReset().mockImplementation(async () => {
+ refused.pid++
+ return { ...refused } as never
+ })
+ const never = swap.paneSwapDeps(session(), runtime).waitForProcess()
+ await vi.advanceTimersByTimeAsync(12_000)
+ expect(await never).toBeNull()
+ })
+
it('builds the relaunch argv with the row\'s mode, folder, the caller\'s argv and the keys a grid must clear', () => {
const { swap } = setup()
const log = vi.spyOn(console, 'log').mockImplementation(() => {})
diff --git a/cli/src/core/agents/swap.ts b/cli/src/core/agents/swap.ts
index aff2d47b6..c38c13f7c 100644
--- a/cli/src/core/agents/swap.ts
+++ b/cli/src/core/agents/swap.ts
@@ -24,6 +24,9 @@ export interface PaneSwapDeps {
keepAbandonedConversation: (left: RegisteredSession) => void
}
+/** How long a relaunched engine must stay running before a restart counts it as come up (`waitForProcess`). */
+export const SWAP_SETTLE_MS = 500
+
export function createPaneSwap({ byAgent, tmuxBackend, prepareSessionResume, keepAbandonedConversation }: PaneSwapDeps) {
/**
* The dependencies a pane-process swap needs, for both callers that do one.
@@ -85,7 +88,15 @@ export function createPaneSwap({ byAgent, tmuxBackend, prepareSessionResume, kee
await new Promise((resolve) => setTimeout(resolve, delayMs))
waited += delayMs
const found = await resolvePaneEngineProcess(runtime.paneId, session.engine)
- if (found) return found
+ if (found) {
+ // Up means still up a moment later: a launch the engine refuses (`codex resume` after an update
+ // dropped it) runs for an instant, and seen then, the restart said "resumed" and never fell back
+ // to a fresh start (e2e/updates.e2e.ts on Linux, whose faster start put it in the first look).
+ await new Promise((resolve) => setTimeout(resolve, SWAP_SETTLE_MS))
+ waited += SWAP_SETTLE_MS
+ const still = await resolvePaneEngineProcess(runtime.paneId, session.engine)
+ if (still && still.pid === found.pid && still.startMarker === found.startMarker) return found
+ }
delayMs = Math.min(delayMs * 2, 750)
}
return null
diff --git a/cli/src/core/engines/hooks.spec.ts b/cli/src/core/engines/hooks.spec.ts
index 3504a150b..c1863f847 100644
--- a/cli/src/core/engines/hooks.spec.ts
+++ b/cli/src/core/engines/hooks.spec.ts
@@ -68,6 +68,21 @@ describe('which agent a hook belongs to', () => {
expect(console.log).not.toHaveBeenCalled()
})
+ it('reads the table again when the one in flight began before the hook\'s own process, and matches it', async () => {
+ const a1 = agent('a1', 100)
+ vi.mocked(processRows)
+ .mockResolvedValueOnce(tree([200, 100], [100, 1]) as never)
+ .mockResolvedValueOnce(tree([300, 200], [200, 100], [100, 1]) as never)
+ expect(await setup({ '%1': a1 }).hooks.resolveHookAgent({ engine: 'claude', runtimeHints: [hint('%1')], callerPid: 300 })).toBe(a1)
+ expect(processRows).toHaveBeenCalledTimes(2)
+ expect(console.log).not.toHaveBeenCalled()
+ })
+
+ it('none when the table read again for a missing caller cannot be read', async () => {
+ vi.mocked(processRows).mockResolvedValueOnce(tree([100, 1]) as never).mockResolvedValueOnce(null)
+ expect(await setup({ '%1': agent('a1', 100) }).hooks.resolveHookAgent({ engine: 'claude', runtimeHints: [hint('%1')], callerPid: 300 })).toBeNull()
+ })
+
it('none when the process table cannot be read', async () => {
vi.mocked(processRows).mockResolvedValueOnce(null)
expect(await setup({ '%1': agent('a1', 100) }).hooks.resolveHookAgent({ engine: 'claude', runtimeHints: [hint('%1')], callerPid: 300 })).toBeNull()
@@ -130,11 +145,13 @@ describe('which agent a hook belongs to', () => {
// The new engine is 200 and the hook's process, 300, is its child. Pid 100 was the engine a restart
// killed; nothing in the table is it any more.
const newEngine = () => tree([300, 200], [200, 1])
- /** Asks, and says when the hook has started waiting: after it read the process table once. */
+ /** Asks, and says when the hook has started waiting: once it has been answered with `onWait`. */
const ask = (hooks: ReturnType<typeof setup>['hooks'], panes = ['%1'], onWait?: () => void) => {
let answer: RegisteredSession | null | undefined
- const asked = hooks.resolveHookAgent({ engine: 'claude', runtimeHints: panes.map(hint), callerPid: 300, ...(onWait ? { onWait } : {}) }).then((agent) => { answer = agent })
- const waiting = () => vi.waitFor(() => expect(processRows).toHaveBeenCalledTimes(1))
+ let waited = false
+ const asked = hooks.resolveHookAgent({ engine: 'claude', runtimeHints: panes.map(hint), callerPid: 300, onWait: () => { waited = true; onWait?.() } })
+ .then((agent) => { answer = agent })
+ const waiting = () => vi.waitFor(() => expect(waited).toBe(true))
return { asked, waiting, answer: () => answer }
}
@@ -187,6 +204,30 @@ describe('which agent a hook belongs to', () => {
expect(hook.answer()).toEqual(agent('a1', 200))
})
+ // On Linux the hook runs under dash, whose `sh -c` stays the hook's parent and exits with it: once
+ // `onWait` has answered the hook, the caller is gone from the table. It still belongs to its engine.
+ it('belongs to the agent after the wait though its own process has exited by then, by its ancestry as it arrived', async () => {
+ const onPane: Record<string, RegisteredSession> = { '%1': agent('a1', 100) }
+ vi.mocked(processRows).mockResolvedValueOnce(newEngine() as never).mockResolvedValue(tree([200, 1]) as never)
+ const hook = ask(setup(onPane).hooks)
+ await hook.waiting()
+ onPane['%1'] = agent('a1', 200)
+ await hook.asked
+ expect(hook.answer()).toEqual(agent('a1', 200))
+ })
+
+ it('reads who it came from as it arrives, while the reconcile pass for its pane still runs', async () => {
+ const onPane: Record<string, RegisteredSession> = { '%1': agent('a1', 200) }
+ vi.mocked(processRows).mockResolvedValueOnce(newEngine() as never)
+ let readDuringPass = -1
+ const { hooks } = setup(onPane, { agentReconciler: {
+ triggerHint: vi.fn(async () => { readDuringPass = vi.mocked(processRows).mock.calls.length; return true }),
+ trigger: vi.fn(async () => true),
+ } as unknown as EngineHookDeps['agentReconciler'] })
+ expect(await hooks.resolveHookAgent({ engine: 'claude', runtimeHints: [hint('%1')], callerPid: 300 })).toEqual(agent('a1', 200))
+ expect(readDuringPass).toBe(1)
+ })
+
it('none when what is recorded is a process it does not come from, or the table cannot be read again', async () => {
const onPane: Record<string, RegisteredSession> = { '%1': agent('a1') }
vi.mocked(processRows).mockResolvedValue(newEngine() as never)
diff --git a/cli/src/core/engines/hooks.ts b/cli/src/core/engines/hooks.ts
index bcce161bf..36cc5bf33 100644
--- a/cli/src/core/engines/hooks.ts
+++ b/cli/src/core/engines/hooks.ts
@@ -16,6 +16,7 @@ import {
import { sid } from '../../lib/log.js'
import type { registry, RegisteredSession } from '../../lib/registry.js'
import { processRows } from '../../lib/terminalAgentDiscovery.js'
+import type { ProcessRow } from '../../lib/tmux.js'
import type { TerminalAgentReconciler } from '../../lib/terminalAgentReconciler.js'
import type { TerminalRuntimeRef } from '../../lib/terminalTypes.js'
@@ -50,26 +51,19 @@ export function createEngineHooks({ tmuxBackend, agentReconciler, registry }: En
/**
* The agents on the hinted panes: those whose recorded process the caller descends from, the rest,
* and of the rest those with no live process recorded at all (none yet, or one that has exited).
+ * `ancestry` is the hook's own process and those it descends from, read as the hook arrived
+ * (`resolveHookAgent`); `table`, the process table to judge the recorded processes by, else a new read.
*/
- const matchCaller = async (resolved: TerminalRuntimeRef[], engine: HookQuery['engine'], callerPid: number) => {
- const rows = await processRows()
+ const matchCaller = async (resolved: TerminalRuntimeRef[], engine: HookQuery['engine'], ancestry: ReadonlySet<number>, table?: ProcessRow[]) => {
+ const rows = table ?? await processRows()
if (!rows) return null
- const byPid = new Map(rows.map((row) => [row.pid, row.parentPid]))
const recordedAlive = (session: RegisteredSession): boolean => {
const recorded = session.processIdentity
return !!recorded && rows.some((row) => row.pid === recorded.pid && row.startMarker === recorded.startMarker)
}
const callerBelongsTo = (session: RegisteredSession): boolean => {
const expectedPid = session.processIdentity?.pid
- if (!expectedPid) return false
- let pid = callerPid
- const visited = new Set<number>()
- while (pid > 0 && !visited.has(pid)) {
- if (pid === expectedPid) return true
- visited.add(pid)
- pid = byPid.get(pid) ?? 0
- }
- return false
+ return !!expectedPid && ancestry.has(expectedPid)
}
const candidates = new Map<string, RegisteredSession>()
/**
@@ -113,6 +107,11 @@ export function createEngineHooks({ tmuxBackend, agentReconciler, registry }: En
const resolveHookAgent: ResolveHookAgent = async ({ engine, runtimeHints, callerPid, onWait }) => {
if (!callerPid) return null
+ // Who the hook came from, read as it arrives. On Linux dash keeps its `sh -c` as the hook's parent, a
+ // shell that exits with the hook (its 500ms, or `onWait`'s answer); read after the reconcile pass or
+ // the wait, the ancestry stopped at a pid already gone and a restart's new conversation was never
+ // bound (e2e/updates.e2e.ts). macOS's /bin/sh is a bash that execs the hook: never seen there.
+ const arrival = processRows()
const resolved: TerminalRuntimeRef[] = []
for (const hint of runtimeHints ?? []) {
if (tmuxBackend) resolved.push({ backend: 'tmux', paneId: hint.paneId })
@@ -121,8 +120,15 @@ export function createEngineHooks({ tmuxBackend, agentReconciler, registry }: En
let overdue = false
for (const runtime of resolved) if (!await agentReconciler.triggerHint(runtime, engine)) overdue = true
- let found = await matchCaller(resolved, engine, callerPid)
- if (!found) return null
+ let table = await arrival
+ // A `ps` already running (processRows shares it) may predate the hook's process; the next read has it.
+ if (table && !table.some((row) => row.pid === callerPid)) table = await processRows()
+ if (!table) return null
+ const parents = new Map(table.map((row) => [row.pid, row.parentPid]))
+ const ancestry = new Set<number>()
+ for (let pid = callerPid; pid > 0 && !ancestry.has(pid); pid = parents.get(pid) ?? 0) ancestry.add(pid)
+
+ let found = (await matchCaller(resolved, engine, ancestry, table))!
let choice = chooseHookAgent([...found.candidates.values()], [...found.onHintedRuntime.values()], engine)
// Every relaunch starts its engine before it records the new process — create, fork, resume, a
// restart and the new conversation it falls back to, a retarget — so a SessionStart in between came
@@ -135,8 +141,9 @@ export function createEngineHooks({ tmuxBackend, agentReconciler, registry }: En
if (!choice.agent && choice.reason === 'none' && (found.unrecorded.length || overdue)) {
onWait?.()
await recordChanged(resolved, engine)
- found = await matchCaller(resolved, engine, callerPid)
- if (!found) return null
+ const again = await matchCaller(resolved, engine, ancestry)
+ if (!again) return null
+ found = again
choice = chooseHookAgent([...found.candidates.values()], [...found.onHintedRuntime.values()], engine)
}
const { candidates } = found
diff --git a/cli/src/lib/engineLaunch.spec.ts b/cli/src/lib/engineLaunch.spec.ts
index 550c14980..44baf9c97 100644
--- a/cli/src/lib/engineLaunch.spec.ts
+++ b/cli/src/lib/engineLaunch.spec.ts
@@ -37,6 +37,8 @@ import {
supportsFirstPrompt,
supportsNamedAgent,
unreadableCwdGuard,
+ ZSH_GUARD_ZSHENV,
+ zshNewUserGuard,
} from './engineLaunch.js'
import { ENGINES, type AgentEngine } from '../engines/types.js'
import { engineBin } from './engineBin.js'
@@ -54,6 +56,19 @@ const developersOwnGridBin = process.env.HARNESS_GRID_BIN
beforeEach(() => { delete process.env.HARNESS_GRID_BIN })
afterAll(() => { if (developersOwnGridBin !== undefined) process.env.HARNESS_GRID_BIN = developersOwnGridBin })
+// A zsh user as most are: with a startup file of their own, so the zsh launches below are the plain
+// ones. vitest.setup.ts points ZDOTDIR at an empty folder, which is the new-user case `zshNewUserGuard`
+// handles; that case has its own tests at the end of this file.
+const suiteZdotdir = process.env.ZDOTDIR
+const zshUserHome = mkdtempSync(join(tmpdir(), 'launch-zdotdir-'))
+writeFileSync(join(zshUserHome, '.zshrc'), '')
+beforeEach(() => { process.env.ZDOTDIR = zshUserHome })
+afterAll(() => {
+ if (suiteZdotdir === undefined) delete process.env.ZDOTDIR
+ else process.env.ZDOTDIR = suiteZdotdir
+ rmSync(zshUserHome, { recursive: true, force: true })
+})
+
/** The prelude every case below gets by default: no managed grid on this machine, so PATH is left
* alone and only grid's update check is turned off. */
const GRID_PRELUDE = gridPanePrelude('grid')
@@ -1416,3 +1431,95 @@ async function runPaneScript(
)
})
}
+
+describe('zsh\'s new-user menu, kept out of an agent\'s pane', () => {
+ // Debian, Ubuntu, Fedora and Arch build zsh with its zsh/newuser module: on a terminal, for a person
+ // with no .zshenv, .zprofile, .zshrc or .zlogin, it runs a full-screen setup menu before anything else.
+ // On the end-to-end suite's first Linux run every zsh agent's pane showed it, and no engine started.
+ let root = ''
+ let savedDataDir = ''
+ beforeEach(() => {
+ root = mkdtempSync(join(tmpdir(), 'launch-newuser-'))
+ savedDataDir = env.ADAPTER_DATA_DIR
+ env.ADAPTER_DATA_DIR = join(root, 'data')
+ })
+ afterEach(() => {
+ env.ADAPTER_DATA_DIR = savedDataDir
+ rmSync(root, { recursive: true, force: true })
+ })
+ const folder = () => join(root, 'data', 'zsh-startup')
+
+ it('points a person with no zsh startup files at a .zshenv of Harness\'s that puts their ZDOTDIR back', () => {
+ const home = join(root, 'home')
+ mkdirSync(home)
+ expect(zshNewUserGuard({ HOME: home })).toEqual([`ZDOTDIR=${folder()}`])
+ expect(zshNewUserGuard({ HOME: home, ZDOTDIR: home })).toEqual([`ZDOTDIR=${folder()}`, `HARNESS_ZDOTDIR=${home}`])
+ expect(readFileSync(join(folder(), '.zshenv'), 'utf8')).toBe(ZSH_GUARD_ZSHENV)
+ expect(statSync(join(folder(), '.zshenv')).mode & 0o777).toBe(0o600)
+ // In the launch itself, before the shell and its flags.
+ process.env.ZDOTDIR = home
+ expect(buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, undefined, NO_TMUX).slice(0, 6)).toEqual([
+ '/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', `ZDOTDIR=${folder()}`, `HARNESS_ZDOTDIR=${home}`, '/bin/zsh', '-lic',
+ ])
+ })
+
+ it.each(['.zshenv', '.zprofile', '.zshrc', '.zlogin'])('leaves anyone with a %s exactly as they were', (name) => {
+ const home = join(root, 'home')
+ mkdirSync(home)
+ writeFileSync(join(home, name), '')
+ expect(zshNewUserGuard({ HOME: home })).toEqual([])
+ expect(zshNewUserGuard({ HOME: join(root, 'elsewhere'), ZDOTDIR: home })).toEqual([])
+ expect(existsSync(folder())).toBe(false)
+ })
+
+ it('launches as before when it cannot write its .zshenv, or there is no home to look in', () => {
+ writeFileSync(join(root, 'data'), 'a file where the data folder should be')
+ expect(zshNewUserGuard({ HOME: root })).toEqual([])
+ expect(zshNewUserGuard({})).toEqual([])
+ })
+
+ // What the .zshenv does, in a real zsh: a person's ZDOTDIR, set or unset, is theirs again before any
+ // of their own files is read, and a startup file of theirs that appears later is read from it.
+ it.skipIf(!existsSync('/bin/zsh'))('gives a real zsh back the person\'s ZDOTDIR, set or unset, and reads their files from it', () => {
+ const home = join(root, 'home')
+ mkdirSync(home)
+ const run = (environment: NodeJS.ProcessEnv) => {
+ const guard = zshNewUserGuard(environment)
+ expect(guard).not.toEqual([])
+ const out = execFileSync('/usr/bin/env', [...guard, '/bin/zsh', '-lic', 'print -r -- "${ZDOTDIR-unset}|${+HARNESS_ZDOTDIR}|${HARNESS_PROFILE-}"'], {
+ encoding: 'utf8', env: { PATH: process.env.PATH, ...environment }, stdio: ['ignore', 'pipe', 'ignore'],
+ })
+ return out.trim().split('\n').at(-1)
+ }
+ expect(run({ HOME: home })).toBe('unset|0|')
+ expect(run({ HOME: join(root, 'not-here'), ZDOTDIR: home })).toBe(`${home}|0|`)
+ // A .zshenv only at the guard's decision: one written after it is still the person's to run.
+ const later = join(root, 'later')
+ mkdirSync(later)
+ const guard = zshNewUserGuard({ HOME: later })
+ writeFileSync(join(later, '.zshenv'), 'export HARNESS_PROFILE=theirs\n')
+ const out = execFileSync('/usr/bin/env', [...guard, '/bin/zsh', '-lic', 'print -r -- "${ZDOTDIR-unset}|${HARNESS_PROFILE-}"'], {
+ encoding: 'utf8', env: { PATH: process.env.PATH, HOME: later }, stdio: ['ignore', 'pipe', 'ignore'],
+ })
+ expect(out.trim().split('\n').at(-1)).toBe('unset|theirs')
+ })
+
+ // The menu itself needs zsh's newuser script (not on macOS) and a terminal, which a pseudo-terminal
+ // from python's pty module provides. Without the guard the menu waits for a key and the marker never
+ // comes; with it the shell runs the command at once.
+ const newuserScript = ['/usr/share/zsh', '/usr/local/share/zsh'].flatMap((base) => {
+ try { return readdirSync(base).map((version) => join(base, version, 'scripts', 'newuser')) } catch { return [] }
+ }).find((file) => existsSync(file))
+ it.skipIf(!existsSync('/bin/zsh') || !newuserScript || !existsSync('/usr/bin/python3'))('runs a new user\'s pane straight to its command, with no setup menu', () => {
+ const home = join(root, 'home')
+ mkdirSync(home)
+ const argv = ['/usr/bin/env', ...zshNewUserGuard({ HOME: home }), '/bin/zsh', '-lic', 'print HARNESS-MARKER; exit']
+ const pty = `import os, pty, sys, select, time\npid, fd = pty.fork()\nif pid == 0:\n os.execvpe(sys.argv[1], sys.argv[1:], os.environ)\nout = b''\nend = time.time() + 8\nwhile time.time() < end and b'HARNESS-MARKER' not in out:\n r, _, _ = select.select([fd], [], [], 0.2)\n if r:\n try: out += os.read(fd, 4096)\n except OSError: break\nos.kill(pid, 9)\nsys.stdout.write(out.decode('utf8', 'replace'))\n`
+ const out = execFileSync('/usr/bin/python3', ['-c', pty, ...argv], {
+ encoding: 'utf8', env: { PATH: process.env.PATH, HOME: home, TERM: 'xterm', LINES: '40', COLUMNS: '120' }, timeout: 20_000,
+ })
+ expect(out).not.toContain('zsh-newuser-install')
+ expect(out).toContain('HARNESS-MARKER')
+ })
+})
+
diff --git a/cli/src/lib/engineLaunch.ts b/cli/src/lib/engineLaunch.ts
index 0a0f5b45e..35a7dafa8 100644
--- a/cli/src/lib/engineLaunch.ts
+++ b/cli/src/lib/engineLaunch.ts
@@ -1,6 +1,6 @@
import { execFile } from 'node:child_process'
import { randomUUID } from 'node:crypto'
-import { existsSync, mkdirSync, readdirSync, realpathSync, rmSync, statSync, writeFileSync } from 'node:fs'
+import { existsSync, mkdirSync, readdirSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from 'node:fs'
import { homedir, userInfo } from 'node:os'
import { isAbsolute, basename, dirname, join } from 'node:path'
import { env } from '../config/env.js'
@@ -440,11 +440,47 @@ export function interactiveEngineShell(shell: string | undefined = undefined): I
function engineShellArgv(shell: InteractiveEngineShell, args: readonly string[]): string[] {
if (!isPosixShell(shell.path)) return throughPosixShell(shell, args)
const prefix = basename(shell.path).toLowerCase() === 'zsh'
- ? ['/usr/bin/env', 'DISABLE_AUTO_UPDATE=true']
+ ? ['/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', ...zshNewUserGuard()]
: []
return [...prefix, shell.path, ...shell.args, ...args]
}
+/** The startup files zsh's new-user module looks for (zshmodules(1), zsh/newuser). */
+const ZSH_STARTUP_FILES = ['.zshenv', '.zprofile', '.zshrc', '.zlogin'] as const
+
+/** The .zshenv in Harness's own ZDOTDIR (`zshNewUserGuard`): the person's ZDOTDIR back as it was, or unset,
+ * before anything of theirs is read; zsh then reads .zprofile, .zshrc and .zlogin from theirs. */
+export const ZSH_GUARD_ZSHENV = `# Written by Harness (engineLaunch.ts zshNewUserGuard): keeps zsh's new-user menu out of an agent's pane.
+if (( \${+HARNESS_ZDOTDIR} )); then ZDOTDIR="\$HARNESS_ZDOTDIR"; unset HARNESS_ZDOTDIR; else unset ZDOTDIR; fi
+[[ -r "\${ZDOTDIR:-\$HOME}/.zshenv" ]] && builtin source "\${ZDOTDIR:-\$HOME}/.zshenv"
+`
+
+/**
+ * Keeps zsh's new-user menu out of an engine's pane. Debian, Ubuntu, Fedora and Arch ship zsh's
+ * `zsh/newuser` module: on a terminal, for someone with none of the four startup files in $ZDOTDIR (else
+ * $HOME), it runs a full-screen menu that waits for a key, so every agent's pane showed it and no engine
+ * started (the end-to-end suite's first Linux runs, 2026-10-06; macOS's zsh has no such module). The
+ * module looks only there, right after the global zshenv: for such a person ZDOTDIR points at a folder of
+ * Harness's whose .zshenv puts theirs back (`HARNESS_ZDOTDIR`; absent means unset). Anyone with a startup
+ * file of their own starts exactly as before.
+ */
+export function zshNewUserGuard(environment: NodeJS.ProcessEnv = process.env): string[] {
+ const dotdir = environment.ZDOTDIR || environment.HOME
+ if (!dotdir || ZSH_STARTUP_FILES.some((name) => existsSync(join(dotdir, name)))) return []
+ const folder = join(env.ADAPTER_DATA_DIR, 'zsh-startup')
+ try {
+ mkdirSync(folder, { recursive: true, mode: 0o700 })
+ const file = join(folder, '.zshenv')
+ let current: string | null = null
+ try { current = readFileSync(file, 'utf8') } catch { /* not written yet */ }
+ if (current !== ZSH_GUARD_ZSHENV) writeFileSync(file, ZSH_GUARD_ZSHENV, { mode: 0o600 })
+ } catch {
+ // A ZDOTDIR without its .zshenv would leave the person's own ZDOTDIR unrestored: launch as before.
+ return []
+ }
+ return [`ZDOTDIR=${folder}`, ...(environment.ZDOTDIR !== undefined ? [`HARNESS_ZDOTDIR=${environment.ZDOTDIR}`] : [])]
+}
+
/**
* For a shell that is not POSIX: it loads the person's environment (their PATH, their version
* managers), then hands the script to a POSIX shell (`posixRunner`). The script and its arguments go
diff --git a/cli/vitest.e2e.ci.config.ts b/cli/vitest.e2e.ci.config.ts
new file mode 100644
index 000000000..634dfa915
--- /dev/null
+++ b/cli/vitest.e2e.ci.config.ts
@@ -0,0 +1,14 @@
+import { defineConfig, mergeConfig } from 'vitest/config'
+import config from './vitest.e2e.config.js'
+import { createDurationSequencer, E2E_HINT_PATH, readTimingHintsFile } from './src/__fixtures__/ciTestSequencer.js'
+
+/**
+ * The end-to-end suite as CI runs it (.github/workflows/cli-e2e.yml): the same files, timeouts and
+ * one-file-at-a-time default as `vitest.e2e.config.ts`, split into shards by the timing hints in
+ * `ci-e2e-durations.json`. Vitest's own `--shard` splits by file count, and the files here run from
+ * twenty seconds to six and a half minutes: by count, one runner could draw the four longest.
+ * `--config vitest.e2e.ci.config.ts --shard=N/8` reproduces a CI shard's files locally.
+ */
+export default mergeConfig(config, defineConfig({
+ test: { sequence: { sequencer: createDurationSequencer(readTimingHintsFile('ci-e2e-durations.json', E2E_HINT_PATH)) } },
+}))
diff --git a/cli/vitest.e2e.config.ts b/cli/vitest.e2e.config.ts
index 919c1de4b..55763d935 100644
--- a/cli/vitest.e2e.config.ts
+++ b/cli/vitest.e2e.config.ts
@@ -12,13 +12,16 @@ import { defineConfig } from 'vitest/config'
*
* `E2E_BUNDLE=1` starts every daemon from one bundle built for the run (e2e/harness/bundle.ts):
* less CPU per start, the code frozen as the run began, and the build that ships under test.
+ *
+ * `E2E_ARTIFACTS_DIR=<folder>` keeps the whole log of every daemon a failing test ran there, as files
+ * (e2e/harness/artifacts.ts); CI uploads them.
*/
const workers = Math.max(1, Number(process.env.E2E_WORKERS ?? 1) || 1)
export default defineConfig({
test: {
environment: 'node',
- setupFiles: ['./vitest.setup.ts'],
+ setupFiles: ['./vitest.setup.ts', './e2e/harness/artifacts.ts'],
globalSetup: ['./e2e/harness/bundle.ts'],
include: ['e2e/**/*.e2e.ts'],
testTimeout: 180_000,
diff --git a/docs/validation-and-release.md b/docs/validation-and-release.md
index faf2aafb6..5bf3aae01 100644
--- a/docs/validation-and-release.md
+++ b/docs/validation-and-release.md
@@ -69,6 +69,22 @@ shuffling, is unchanged. Use `--config vitest.ci.config.ts --shard=N/4` to repro
the CI assignment locally. The aggregate still requires every discovered file
exactly once, independent of these estimates.
+The CLI's end-to-end suite (`cli/e2e`) runs in its own workflow, **CLI end to end**
+(`.github/workflows/cli-e2e.yml`): after each merge to `main` that touches `cli/`, nightly,
+and on demand from a branch (Actions → CLI end to end → Run workflow).
+Eight Linux runners each take a shard, planned from `cli/ci-e2e-durations.json` the same
+way (`--config vitest.e2e.ci.config.ts --shard=N/8` reproduces one locally). Each shard
+installs tmux, zsh, tcsh and dash, starts every daemon from one bundle (`E2E_BUNDLE=1`) and
+runs its files one at a time. The full-disk tests, which need macOS disk images, run on one
+macOS runner with `DISKFULL=1`. `e2e-summary` applies the default suite's rule: every
+discovered file exactly once, every shard passing. Its summary records each file's duration
+for refreshing the hints. A failing test's complete daemon logs and engine hook logs are
+uploaded as `cli-e2e-daemon-logs-N`, by file and test (`E2E_ARTIFACTS_DIR`, which works
+locally too). It is advisory, and so it does not run on pull requests: a failed check makes
+GitHub report a PR unstable, which `make merge-pr` refuses to merge. Once it has stayed
+green, run it on pull requests, make `e2e-summary` required and add it to the evidence
+collector.
+
For repository process tooling only, `scope=process` runs its Python regression
tests without installing or building unrelated components. It does not validate
application changes. Workflow edits also need `actionlint` and a run exercising
diff --git a/scripts/tests/test_cli_test_shards.py b/scripts/tests/test_cli_test_shards.py
index 977cf869e..eae5f8b16 100644
--- a/scripts/tests/test_cli_test_shards.py
+++ b/scripts/tests/test_cli_test_shards.py
@@ -107,6 +107,74 @@ class ShardTests(unittest.TestCase):
self.assertEqual(result.returncode, 1)
self.assertFalse(output.exists())
+ def test_durations_are_recorded_rounded_up_for_the_next_runs_hints(self):
+ self.edit("result-1.json", lambda r: r["testResults"][0].update(startTime=1_000, endTime=2_201))
+ self.edit("result-2.json", lambda r: r["testResults"][0].update(startTime=5, endTime="late"))
+ result = self.verify()
+ self.assertEqual(result["durations_ms"], {"src/case-0.spec.ts": 1300})
+ self.assertEqual(result["suite"], "unit")
+
+ def test_the_unit_suite_still_requires_its_contract_job(self):
+ with self.assertRaisesRegex(ValueError, "upstream jobs"):
+ self.verify(contracts=None)
+
+
+class EndToEndShardTests(ShardTests):
+ """The same contract for the end-to-end suite: its own folder and names, and no contract job."""
+
+ def setUp(self):
+ super().setUp()
+ self.names = [str(self.root / "e2e" / f"case-{i}.e2e.ts") for i in range(4)]
+ for number in range(1, 5):
+ self.edit(f"inventory-{number}.json", lambda paths: [item.update(file=self.names[i]) for i, item in enumerate(paths)])
+ self.edit(f"result-{number}.json", lambda r, n=number: r["testResults"][0].update(name=self.names[n - 1]))
+
+ def verify(self, tests="success", contracts=None):
+ return verifier.verify(self.directory, self.root, 4, tests, contracts, suite="e2e")
+
+ def test_complete_partition_preserves_pass_skip_counts_and_hashes(self):
+ result = self.verify()
+ self.assertEqual((result["suite"], result["files"], result["passed"], result["skipped"]), ("e2e", 4, 4, 4))
+ self.assertEqual(result["verified_files"], [f"e2e/case-{i}.e2e.ts" for i in range(4)])
+
+ def test_json_success_cannot_override_nonpassing_jobs(self):
+ for state in ["failure", "cancelled", "skipped", ""]:
+ with self.subTest(tests=state), self.assertRaisesRegex(ValueError, "upstream jobs"):
+ self.verify(state)
+ # A contract verdict, when one is given, still has to be a pass.
+ with self.assertRaisesRegex(ValueError, "upstream jobs"):
+ self.verify("success", "failure")
+
+ def test_the_unit_suite_still_requires_its_contract_job(self):
+ self.assertEqual(self.verify(contracts="success")["files"], 4)
+
+ def test_durations_are_recorded_rounded_up_for_the_next_runs_hints(self):
+ self.edit("result-3.json", lambda r: r["testResults"][0].update(startTime=0, endTime=264_249))
+ self.assertEqual(self.verify()["durations_ms"], {"e2e/case-2.e2e.ts": 264_300})
+
+ def test_a_unit_spec_cannot_count_as_an_end_to_end_file_or_the_reverse(self):
+ unit = str(self.root / "src" / "case-0.spec.ts")
+ self.edit("result-1.json", lambda r: r["testResults"][0].update(name=unit))
+ with self.assertRaisesRegex(ValueError, "unexpected test path"):
+ self.verify()
+ with self.assertRaisesRegex(ValueError, "unexpected test path"):
+ verifier.verify(self.directory, self.root, 4, "success", "success")
+
+ def test_command_writes_a_verified_summary_and_fails_on_incomplete_artifacts(self):
+ output = self.directory / "summary.json"
+ command = [sys.executable, str(SCRIPT), str(self.directory), "--root", str(self.root),
+ "--shards", "4", "--tests-result", "success", "--suite", "e2e", "--output", str(output)]
+ result = subprocess.run(command, capture_output=True, text=True, timeout=5)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(json.loads(output.read_text())["files"], 4)
+ output.unlink()
+ # Without --suite it is the default suite, which needs its contract job's verdict.
+ unit = [arg for arg in command if arg not in ("--suite", "e2e")]
+ result = subprocess.run(unit, capture_output=True, text=True, timeout=5)
+ self.assertEqual(result.returncode, 2)
+ self.assertIn("--contracts-result", result.stderr)
+ self.assertFalse(output.exists())
+
if __name__ == "__main__":
unittest.main()
diff --git a/scripts/verify-cli-test-shards.py b/scripts/verify-cli-test-shards.py
index 97e1ae790..ff2330492 100644
--- a/scripts/verify-cli-test-shards.py
+++ b/scripts/verify-cli-test-shards.py
@@ -1,5 +1,11 @@
#!/usr/bin/env python3
-"""Verify that passing Vitest CI shards cover the discovered file set exactly once."""
+"""Verify that passing Vitest CI shards cover the discovered file set exactly once.
+
+Two suites use it: the default suite (`--suite unit`, ci.yml, four shards beside the contract job) and
+the end-to-end suite (`--suite e2e`, cli-e2e.yml, which has no contract job). The summary also records
+each file's duration, rounded up to 100ms: the timing hints the next run's shards are planned with
+(cli/ci-test-durations.json, cli/ci-e2e-durations.json) are refreshed from it.
+"""
import argparse
from collections import Counter
import hashlib
@@ -8,14 +14,21 @@ from pathlib import Path
import sys
-def verify(directory, root, shards, tests_result, contracts_result):
+# Where each suite's files live, relative to cli/, and how they are named.
+SUITES = {"unit": ("src/", (".spec.ts", ".test.ts")), "e2e": ("e2e/", (".e2e.ts",))}
+
+
+def verify(directory, root, shards, tests_result, contracts_result, suite="unit"):
+ folder, suffixes = SUITES[suite]
# Vitest JSON can say success even when an unhandled error makes its process
- # fail. Require the actual matrix/contract job verdicts, not JSON alone.
- if tests_result != "success" or contracts_result != "success":
+ # fail. Require the actual matrix/contract job verdicts, not JSON alone. Only the
+ # end-to-end suite runs without a contract job; the default suite always has one.
+ if tests_result != "success" or (contracts_result != "success" and not (suite == "e2e" and contracts_result is None)):
raise ValueError(f"upstream jobs must pass: tests={tests_result}, contracts={contracts_result}")
expected, seen = None, Counter()
totals = Counter(passed=0, skipped=0, todo=0)
receipts = []
+ durations = {}
def relative(value):
if not isinstance(value, str):
@@ -24,7 +37,7 @@ def verify(directory, root, shards, tests_result, contracts_result):
if not path.is_absolute():
raise ValueError(f"expected an absolute Vitest file path: {value}")
result = path.resolve().relative_to(root.resolve()).as_posix()
- if not result.startswith("src/") or not result.endswith((".spec.ts", ".test.ts")):
+ if not result.startswith(folder) or not result.endswith(suffixes):
raise ValueError(f"unexpected test path: {result}")
return result
@@ -53,6 +66,11 @@ def verify(directory, root, shards, tests_result, contracts_result):
seen[name] += 1
if file.get("status") != "passed" or file.get("message"):
raise ValueError(f"shard {shard}: file failed: {name}")
+ # Scheduling hints only: a missing or odd time is left out, never a reason to fail.
+ elapsed = file.get("endTime", 0) - file.get("startTime", 0) if all(
+ type(file.get(key)) in (int, float) for key in ("startTime", "endTime")) else 0
+ if elapsed > 0:
+ durations[name] = int(-(-elapsed // 100) * 100)
for case in file["assertionResults"]:
status = case.get("status")
if status not in {"passed", "skipped", "todo"} or case.get("failureMessages"):
@@ -69,8 +87,8 @@ def verify(directory, root, shards, tests_result, contracts_result):
if set(seen) != expected or any(count != 1 for count in seen.values()):
raise ValueError(f"file partition mismatch: missing={sorted(expected - set(seen))}, "
f"unexpected={sorted(set(seen) - expected)}, duplicates={sorted(p for p, n in seen.items() if n != 1)}")
- return {"schema": 1, "status": "passed", "files": len(seen), **dict(totals),
- "shards": receipts, "verified_files": sorted(seen)}
+ return {"schema": 1, "suite": suite, "status": "passed", "files": len(seen), **dict(totals),
+ "shards": receipts, "verified_files": sorted(seen), "durations_ms": dict(sorted(durations.items()))}
def main():
@@ -79,14 +97,17 @@ def main():
parser.add_argument("--root", type=Path, required=True)
parser.add_argument("--shards", type=int, required=True)
parser.add_argument("--tests-result", required=True)
- parser.add_argument("--contracts-result", required=True)
+ parser.add_argument("--contracts-result", help="the contract job's verdict; required for the unit suite")
+ parser.add_argument("--suite", choices=sorted(SUITES), default="unit")
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
if args.shards < 1:
parser.error("shards must be positive")
- summary = verify(args.directory, args.root, args.shards, args.tests_result, args.contracts_result)
+ if args.suite == "unit" and args.contracts_result is None:
+ parser.error("the unit suite requires --contracts-result")
+ summary = verify(args.directory, args.root, args.shards, args.tests_result, args.contracts_result, args.suite)
args.output.write_text(json.dumps(summary, indent=2) + "\n")
- print(f"Verified {summary['files']} files exactly once: {summary['passed']} passed, "
+ print(f"Verified {summary['files']} {args.suite} files exactly once: {summary['passed']} passed, "
f"{summary['skipped']} skipped, {summary['todo']} todo across {args.shards} shards")
return 0