Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

Ctrl+Z, or a SIGSTOP, no longer loses an agent: the pane's shell continues a stopped engine (round 30) (#847)

Deee0816993b0b2
diff --git a/cli/e2e/harness/fakeEngine.mjs b/cli/e2e/harness/fakeEngine.mjs
index c6981bcaf..14c09c062 100644
--- a/cli/e2e/harness/fakeEngine.mjs
+++ b/cli/e2e/harness/fakeEngine.mjs
@@ -40,11 +40,20 @@
// loaded machine, while the daemon has already found the engine and its conversation. `root` is the
// test's throwaway root, the only place whose hooks it will run, and `hookLog` where it notes every hook
// it ran.
+//
+// Ctrl+Z suspends it, as both CLIs do on Unix (Claude Code 2.1.289: "Claude Code has been suspended.
+// Run `fg` to bring Claude Code back."; Codex 0.160: "`ctrl-z` is reserved for suspending the terminal
+// on Unix"): it gives the terminal back and stops its whole process group with SIGTSTP. Continued from
+// that stop it takes the terminal again; continued from any stop it repaints, and nothing more.
+//
+// Codex runs as npm installs it: a Node wrapper, the command the pane starts, with the engine as its
+// child in the same process group and terminal (`codexWrapper`).
import { execFileSync, spawn } from 'node:child_process'
import { randomUUID } from 'node:crypto'
import { appendFileSync, closeSync, existsSync, mkdirSync, openSync, readFileSync, writeFileSync } from 'node:fs'
import { homedir } from 'node:os'
import { dirname, isAbsolute, join, relative, resolve } from 'node:path'
+import { fileURLToPath, pathToFileURL } from 'node:url'
/**
* The events each CLI fires that the daemon installs hooks for (lib/hooks.ts): Claude Code's session,
@@ -73,7 +82,28 @@ const matcherTakes = (matcher, value) => {
try { return new RegExp(`^(?:${matcher})$`).test(String(value ?? '')) } catch { return false }
}
-export async function run(engine, config) {
+/** What the npm wrapper hands its child, the Codex engine itself: this module, run as a script. */
+const CODEX_NATIVE = 'HARNESS_FAKE_CODEX_NATIVE'
+
+/**
+ * `@openai/codex`'s bin/codex.js (0.160.0): it starts the native binary as its child with the same
+ * terminal, process group and arguments, forwards SIGINT, SIGTERM and SIGHUP to it, and ends as the
+ * child ended — re-raising a signal the child died of, which ends the wrapper with that signal unless
+ * it is one the wrapper listens for (then the wrapper exits 0).
+ */
+async function codexWrapper(config) {
+ const child = spawn(process.execPath, [fileURLToPath(import.meta.url), ...process.argv.slice(2)], {
+ stdio: 'inherit',
+ env: { ...process.env, [CODEX_NATIVE]: JSON.stringify(config) },
+ })
+ for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP']) process.on(signal, () => { try { child.kill(signal) } catch { /* gone */ } })
+ const ended = await new Promise((resolve) => child.on('exit', (code, signal) => resolve({ code, signal })))
+ if (ended.signal) process.kill(process.pid, ended.signal)
+ else process.exit(ended.code ?? 1)
+}
+
+export async function run(engine, config = {}, { native = false } = {}) {
+ if (engine === 'codex' && !native) return codexWrapper(config)
const args = process.argv.slice(2)
const version = config.version ?? (engine === 'claude' ? '2.1.270' : '0.159.0')
const versionLine = engine === 'claude' ? `${version} (Claude Code)` : `codex-cli ${version}`
@@ -792,7 +822,7 @@ export async function run(engine, config) {
await finish(`answer ${turn}: ${prompt}`)
}
- // Raw input: bracketed paste brackets the text, Enter (\r) submits, Ctrl-C interrupts.
+ // Raw input: bracketed paste brackets the text, Enter (\r) submits, Ctrl-C interrupts, Ctrl+Z suspends.
process.stdin.setEncoding('utf8')
let queue = Promise.resolve()
// Inside a bracketed paste a carriage return or newline is a newline in the prompt, as Ink and
@@ -816,7 +846,27 @@ export async function run(engine, config) {
pastes.clear()
return text
}
+ // Ctrl+Z, as the real CLIs do it: give the terminal back, and stop the whole process group (the
+ // npm Codex's wrapper with it) once the terminal is restored; in raw mode the terminal sends no
+ // SIGTSTP of its own. Raw mode comes back only from this stop: Claude Code arms a one-time SIGCONT
+ // handler here, Codex takes the terminal again when its stop returns.
+ const suspend = () => {
+ process.stdout.write('\x1b[?2004l')
+ if (engine === 'claude') process.stdout.write('\r\nClaude Code has been suspended. Run `fg` to bring Claude Code back.\r\n')
+ process.stdin.setRawMode?.(false)
+ process.once('SIGCONT', () => {
+ process.stdin.setRawMode?.(true)
+ process.stdout.write('\x1b[?2004h')
+ })
+ process.kill(0, 'SIGTSTP')
+ }
+ // Continued, whoever stopped it, the renderer repaints, and that is all: after a stop from outside
+ // the terminal is in whatever modes the shell that resumed it left (an interactive bash puts back its
+ // own, line mode), as it is for the real CLIs.
+ process.on('SIGCONT', () => draw())
process.stdin.on('data', (input) => {
+ // The key, not the byte: a Ctrl+Z inside a paste is pasted text.
+ if (input.includes('\x1a') && !pasting && !input.includes('\x1b[200~')) { suspend(); return }
if (dialog) { dialogKeys(input); return }
let chunk = input
if (bottom?.transcript) {
@@ -953,3 +1003,11 @@ export async function run(engine, config) {
process.on('SIGTERM', () => process.exit(0))
setInterval(() => {}, 60_000)
}
+
+// The native Codex: this module run as a script by `codexWrapper`. Last, so everything `run` reads at
+// the top level of the module is there when it starts.
+if (process.env[CODEX_NATIVE] && process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+ const config = JSON.parse(process.env[CODEX_NATIVE])
+ delete process.env[CODEX_NATIVE]
+ void run('codex', config, { native: true })
+}
diff --git a/cli/e2e/races.e2e.ts b/cli/e2e/races.e2e.ts
index ffd30ec75..5f950c919 100644
--- a/cli/e2e/races.e2e.ts
+++ b/cli/e2e/races.e2e.ts
@@ -50,7 +50,9 @@ function install(d: IsolatedDaemon, engine: Engine, extra: Record<string, unknow
writeFileSync(`${wrapper}.new`, `#!${process.execPath}\nimport(${JSON.stringify(module)}).then((m) => m.run(${JSON.stringify(engine)}, ${JSON.stringify(config)}))\n`, { mode: 0o755 })
renameSync(`${wrapper}.new`, wrapper)
}
-/** Engine processes in a pane: a pane must never end with two of them, or none while the agent is active. */
+/** Engines in a pane: a pane must never end with two of them, or none while the agent is active. npm's Codex
+ * is a Node wrapper with the engine as its child (fakeEngine.mjs), one engine in two processes, so a match
+ * whose parent matched too is the same engine. */
async function engineProcesses(daemon: IsolatedDaemon, pane: string): Promise<number> {
const panePid = Number(await daemon.tmux.run('display-message', '-p', '-t', pane, '#{pane_pid}').catch(() => '0'))
if (!panePid) return 0
@@ -61,7 +63,9 @@ async function engineProcesses(daemon: IsolatedDaemon, pane: string): Promise<nu
for (const [, pid, ppid] of table) children.set(Number(ppid), [...(children.get(Number(ppid)) ?? []), Number(pid)])
const tree = new Set<number>([panePid])
for (const pid of tree) for (const child of children.get(pid) ?? []) tree.add(child)
- return table.filter(([, pid, , command]) => tree.has(Number(pid)) && /\b(claude|codex)\b/.test(command) && !/\bzsh\b|\bbash\b|\bsh -c\b/.test(command)).length
+ const engines = table.filter(([, pid, , command]) => tree.has(Number(pid)) && /\b(claude|codex)\b/.test(command) && !/\bzsh\b|\bbash\b|\bsh -c\b/.test(command))
+ const matched = new Set(engines.map(([, pid]) => Number(pid)))
+ return engines.filter(([, , ppid]) => !matched.has(Number(ppid))).length
}
describe('lifecycle requests that race', () => {
diff --git a/cli/e2e/suspend.e2e.ts b/cli/e2e/suspend.e2e.ts
new file mode 100644
index 000000000..6ac5fabcf
--- /dev/null
+++ b/cli/e2e/suspend.e2e.ts
@@ -0,0 +1,177 @@
+/**
+ * An engine that is stopped, for Claude Code and Codex: Ctrl+Z typed into its pane, which both CLIs
+ * answer by suspending themselves ("Claude Code has been suspended. Run `fg` to bring Claude Code
+ * back."), and a SIGSTOP of the engine alone. The engine runs as a job of the pane's interactive shell,
+ * and that shell took the stop for the engine's end and exited, hanging up the stopped engine: the
+ * pane closed and the agent was gone. tmux continues a stopped pane process at once, but nothing below
+ * it, and an agent's pane has no prompt to type `fg` into; the launch script now continues the engine
+ * itself (engineLaunch.ts, STOP_PROOF_FUNCTIONS).
+ *
+ * So the agent stays: the same conversation in the same engine process, working once it is back, with
+ * nothing restarted. And its real exit still ends it the usual way: the pane, marked with the engine's
+ * status, is a shell. So does a death by SIGINT or SIGQUIT, which zsh took for an interrupt of the whole
+ * launch script. Under zsh and bash, and tcsh, which hands the launch to a POSIX shell: dash where
+ * there is one, else /bin/sh.
+ */
+import { execFileSync } from 'node:child_process'
+import { existsSync, mkdirSync, readFileSync } from 'node:fs'
+import { basename, join } from 'node:path'
+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'
+import { LocalClient, type Frame } from './harness/client.js'
+import { IsolatedDaemon, until } from './harness/daemon.js'
+
+type Engine = 'claude' | 'codex'
+type Row = Record<string, any>
+const engines: Engine[] = ['claude', 'codex']
+const SHELLS = ['/bin/zsh', '/bin/bash', '/bin/tcsh'].filter(existsSync)
+/** Whether bash runs the engine as its job: under bash itself, or under tcsh handing off to a /bin/sh
+ * that is bash because there is no dash (Fedora). */
+const bashRuns = (shell: string): boolean => basename(shell) === 'bash'
+ || (basename(shell) === 'tcsh' && !existsSync('/bin/dash') && execFileSync('/bin/sh', ['-c', 'printf %s "${BASH_VERSION:-}"'], { encoding: 'utf8' }) !== '')
+
+const rows = async (client: LocalClient): Promise<Row[]> =>
+ (await client.request<{ agents: Row[] }>('agents_list', { includeStopped: true }, 30_000)).agents
+const row = async (client: LocalClient, agentId: string) => (await rows(client)).find((agent) => agent.id === agentId)
+async function create(d: IsolatedDaemon, client: LocalClient, engine: Engine, folder: string): Promise<Row> {
+ const cwd = join(d.projectsDir, folder)
+ mkdirSync(cwd, { recursive: true })
+ const created = await client.request('agent_create', { engine, cwd, bypassPermission: true }, 120_000)
+ expect(created.error, `${folder}: ${JSON.stringify(created)}`).toBeUndefined()
+ return until(`${folder} to bind its conversation`, async () => {
+ const agent = await row(client, created.agent.id)
+ return agent?.sessionId && agent.status === 'active' ? agent : null
+ }, 60_000, 500)
+}
+const isTurn = (type: string, agentId: string) => (frame: Frame) => frame.type === type && frame.agentId === agentId
+async function turn(client: LocalClient, agentId: string, content: string): Promise<void> {
+ const started = client.next(isTurn('turn_started', agentId), 45_000, `turn_started (${content})`)
+ const ended = client.next(isTurn('turn_ended', agentId), 45_000, `turn_ended (${content})`)
+ client.send('message', { agentId, content })
+ expect((await started).payload?.userMessage).toBe(content)
+ await ended
+}
+
+const processTable = () => execFileSync('ps', ['-A', '-o', 'pid=,ppid=,command='], { encoding: 'utf8' })
+ .split('\n').map((line) => /^\s*(\d+)\s+(\d+)\s+(.*)$/.exec(line)).filter(Boolean)
+ .map((m) => ({ pid: Number(m![1]), ppid: Number(m![2]), command: m![3].trim() }))
+
+/** The engine's own process, the pane shell's job: the one below the shell started by the engine's
+ * command, never the shell itself, whose command line names it too. For Codex that is npm's Node
+ * wrapper (the fake runs as npm installs Codex); `native` asks for the engine below it instead, which
+ * names its process `codex` as the real binary's reads in ps. */
+function enginePid(d: IsolatedDaemon, root: number, engine: Engine, native = false): number | null {
+ const table = processTable()
+ const job = table.find((p) => p.ppid === root && (p.command === engine || p.command.includes(join(d.root, 'bin', engine))))
+ if (!job || !native) return job?.pid ?? null
+ return table.find((p) => p.ppid === job.pid && p.command === engine)?.pid ?? null
+}
+/** Whether the pane's terminal is in line mode rather than raw, as an engine keeps it. */
+async function lineMode(d: IsolatedDaemon, pane: string): Promise<boolean> {
+ const tty = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_tty}')).trim()
+ return / icanon /.test(` ${execFileSync('/bin/sh', ['-c', 'stty -a < "$0"', tty], { encoding: 'utf8' }).replace(/\s+/g, ' ')} `)
+}
+const processState = (pid: number): string => {
+ try { return execFileSync('ps', ['-o', 'stat=', '-p', String(pid)], { encoding: 'utf8' }).trim() } catch { return '' }
+}
+
+describe('an engine that is stopped, or killed', () => {
+ let daemon: IsolatedDaemon | undefined
+ const stopped: number[] = []
+ afterEach(async () => {
+ // A stopped process left behind by a failed test would outlive it.
+ for (const pid of stopped.splice(0)) { try { process.kill(pid, 'SIGCONT') } catch { /* gone */ } }
+ await daemon?.close(); daemon = undefined
+ })
+
+ for (const shell of SHELLS) {
+ it.each(engines)(`%s under ${shell}: Ctrl+Z and a SIGSTOP leave the agent there and working, and its own exit still ends it`, async (engine) => {
+ const d = await IsolatedDaemon.create({ env: { SHELL: shell } })
+ daemon = d
+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-150).join('\n')}`) })
+ await d.start()
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, engine, `suspend-${engine}`)
+ await turn(client, agent.id, 'a first turn')
+ const pane = String(agent.terminal?.runtimes?.[0]?.paneId ?? agent.tmuxPane)
+ const root = Number((await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_pid}')).trim())
+ const pid = await until(`the ${engine} process under the pane`, () => enginePid(d, root, engine), 10_000, 200)
+ const exitMark = async () => (await d.tmux.run('show-option', '-p', '-v', '-t', pane, '@harness_engine_exit').catch(() => '')).trim()
+ onTestFailed(async () => { console.log(`---- pane ${pane}\n${await d.capture(pane).catch((error) => String(error))}`) })
+
+ /** Back, as it was: the same process, running again, in the same conversation, nothing restarted
+ * and no exit marked; and it works. */
+ const back = async (how: string) => {
+ await until(`the engine to run again after ${how}`, () => !processState(pid).startsWith('T') || null, 15_000, 100)
+ const now = await row(client, agent.id)
+ expect(now?.status, how).toBe('active')
+ expect(now?.sessionId, how).toBe(agent.sessionId)
+ expect(enginePid(d, root, engine), how).toBe(pid)
+ expect(await exitMark(), how).toBe('')
+ await turn(client, agent.id, `after ${how}`)
+ expect(d.coresStarted(), how).toBe(1)
+ }
+
+ // Ctrl+Z, typed into the pane. The engine gives the terminal back and stops itself; Claude Code
+ // says so, and the shell reports the stopped job. Read from everything the pane writes from here on,
+ // not its screen: the engine repaints its composer over those lines the moment it is back.
+ const written = join(d.root, `pane-${engine}.out`)
+ await d.tmux.run('pipe-pane', '-t', pane, '-o', `cat >> '${written}'`)
+ const output = () => { try { return readFileSync(written, 'utf8') } catch { return '' } }
+ await d.tmux.run('send-keys', '-t', pane, 'C-z')
+ await until('the engine to suspend itself', () => /suspended|stopped/i.test(output()) || null, 15_000, 100)
+ if (engine === 'claude') await until('Claude Code to say so', () => output().includes('Claude Code has been suspended') || null, 5_000, 100)
+ await back('Ctrl+Z')
+
+ // A SIGSTOP of the engine alone, as a debugger or a person with `kill` sends it. Neither real CLI
+ // takes raw mode again on a SIGCONT it did not ask for, so under bash, which puts back its own
+ // modes when a job stops, the engine comes back to a line-mode terminal (accepted, and seen
+ // here); zsh restores the job's, and dash, which tcsh hands the launch to, leaves them be.
+ process.kill(pid, 'SIGSTOP')
+ stopped.push(pid)
+ await back('a SIGSTOP')
+ stopped.splice(0)
+ expect(await lineMode(d, pane)).toBe(bashRuns(shell))
+
+ // Its own exit is still the end of it: marked on the pane, which is a shell now.
+ client.send('message', { agentId: agent.id, content: '!exit' })
+ expect(await until('the exit to be marked on the pane', () => exitMark(), 30_000, 200)).toBe('0')
+ await until('the agent to stop counting as active', async () => (await row(client, agent.id))?.status !== 'active' || null, 45_000, 500)
+ expect(await until('the pane to be a shell', async () => {
+ const [dead, command] = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_dead} #{pane_current_command}')).trim().split(' ')
+ return dead === '0' && command === basename(shell) ? command : null
+ }, 30_000, 200)).toBe(basename(shell))
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ }, 240_000)
+ }
+
+ for (const shell of SHELLS) {
+ it.each([['claude', 'SIGINT', '130'], ['codex', 'SIGQUIT', '131']] as const)(`%s under ${shell}: an engine that dies of %s still leaves the pane a shell, marked with its status`, async (engine, signal, status) => {
+ // zsh gave up the rest of the launch script when its job was killed by INT or QUIT, so the pane
+ // closed instead of turning into a shell. npm's Codex wrapper ends with the signal its engine died
+ // of, unless it listens for it (INT, TERM, HUP), so a QUIT ends the wrapper too.
+ const d = await IsolatedDaemon.create({ env: { SHELL: shell } })
+ daemon = d
+ onTestFailed(() => { console.log(`---- daemon log\n${d.log().split('\n').slice(-150).join('\n')}`) })
+ await d.start()
+ const client = await LocalClient.connect(d)
+ const agent = await create(d, client, engine, `killed-${engine}`)
+ await turn(client, agent.id, 'a first turn')
+ const pane = String(agent.terminal?.runtimes?.[0]?.paneId ?? agent.tmuxPane)
+ const root = Number((await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_pid}')).trim())
+ const target = await until(`the ${engine} engine under the pane`, () => enginePid(d, root, engine, engine === 'codex'), 10_000, 200)
+ process.kill(target, signal)
+ // bash run as the hand-off's /bin/sh reports a job killed by INT as a success.
+ expect(await until('the death to be marked on the pane', async () =>
+ (await d.tmux.run('show-option', '-p', '-v', '-t', pane, '@harness_engine_exit').catch(() => '')).trim(), 30_000, 200))
+ .toBe(signal === 'SIGINT' && basename(shell) === 'tcsh' && bashRuns(shell) ? '0' : status)
+ await until('the agent to stop counting as active', async () => (await row(client, agent.id))?.status !== 'active' || null, 45_000, 500)
+ expect(await until('the pane to be a shell', async () => {
+ const [dead, command] = (await d.tmux.run('display-message', '-p', '-t', pane, '#{pane_dead} #{pane_current_command}')).trim().split(' ')
+ return dead === '0' && command === basename(shell) ? command : null
+ }, 30_000, 200)).toBe(basename(shell))
+ expect(d.coresStarted()).toBe(1)
+ client.close()
+ }, 240_000)
+ }
+})
diff --git a/cli/package.json b/cli/package.json
index efa20a28c..a97a32c3f 100644
--- a/cli/package.json
+++ b/cli/package.json
@@ -23,7 +23,7 @@
"test:sharing-e2e": "tsx scripts/share-harness-e2e.ts",
"test:remote-viewers": "vitest run --config vitest.viewer.config.ts",
"test:cursor-e2e": "RUN_CURSOR_E2E=1 vitest run src/engines/cursor/localE2e.spec.ts",
- "test:tmux-real": "RUN_REAL_TMUX_DISCOVERY=1 vitest run src/lib/tmuxAgentDiscovery.real.spec.ts src/lib/engineShellStartup.spec.ts src/lib/codexUpdateRestart.real.spec.ts src/lib/restartDiscovery.real.spec.ts src/lib/terminalClose.real.spec.ts --reporter=verbose",
+ "test:tmux-real": "RUN_REAL_TMUX_DISCOVERY=1 vitest run src/lib/tmuxAgentDiscovery.real.spec.ts src/lib/engineShellStartup.spec.ts src/lib/codexUpdateRestart.real.spec.ts src/lib/restartDiscovery.real.spec.ts src/lib/terminalClose.real.spec.ts src/lib/engineStop.real.spec.ts --reporter=verbose",
"test:resume": "vitest run src/lib/sessionRepair.spec.ts src/lib/captureResumeIdentity.spec.ts src/lib/resumeStoppedAgent.spec.ts src/lib/resumeAgentService.spec.ts src/lib/stoppedAgents.spec.ts src/lib/stopAgentService.spec.ts --coverage --coverage.include=src/lib/captureResumeIdentity.ts --coverage.include=src/lib/resumeStoppedAgent.ts --coverage.include=src/lib/resumeAgentService.ts --coverage.include=src/lib/stoppedAgents.ts --coverage.include=src/lib/stopAgentService.ts --coverage.thresholds.lines=100 --coverage.thresholds.statements=100 --coverage.thresholds.branches=100 --coverage.thresholds.functions=100 --coverage.reporter=text --coverage.reporter=json --coverage.reportsDirectory=coverage/resume",
"test:resume-native": "tsx scripts/resume-native-e2e.ts",
"test:portability": "vitest run --config vitest.portability.config.ts"
diff --git a/cli/src/lib/codexStartupRetry.real.spec.ts b/cli/src/lib/codexStartupRetry.real.spec.ts
index bee6493ed..8c1eec33a 100644
--- a/cli/src/lib/codexStartupRetry.real.spec.ts
+++ b/cli/src/lib/codexStartupRetry.real.spec.ts
@@ -74,7 +74,8 @@ real('Codex startup recovery in an isolated real tmux server', () => {
const f = await launch(100)
await eventually(async () => await f.exit() === '1')
expect(await f.attempts()).toHaveLength(3)
- expect(await f.screen()).toContain('This pane is a shell now')
+ // The pane is marked before the input left for the engine is drained, and says so only after.
+ await eventually(async () => (await f.screen()).includes('This pane is a shell now'))
}, 20_000)
it('does not retry other errors after a transient first failure', async () => {
@@ -95,6 +96,7 @@ real('Codex startup recovery in an isolated real tmux server', () => {
await tmux('send-keys', '-t', f.pane, 'C-c')
await eventually(async () => await f.exit() === '130')
expect(await f.attempts()).toHaveLength(1)
- expect(await f.screen()).toContain('This pane is a shell now')
+ // The pane is marked before the input left for the engine is drained, and says so only after.
+ await eventually(async () => (await f.screen()).includes('This pane is a shell now'))
}, 15_000)
})
diff --git a/cli/src/lib/codexUpdateRestart.real.spec.ts b/cli/src/lib/codexUpdateRestart.real.spec.ts
index ccea35614..6907ebfc9 100644
--- a/cli/src/lib/codexUpdateRestart.real.spec.ts
+++ b/cli/src/lib/codexUpdateRestart.real.spec.ts
@@ -131,7 +131,8 @@ if (output !== undefined) {
await f.accept()
await f.waitFor(async () => await f.exited() === String(status))
expect(await f.attempts()).toHaveLength(1)
- expect(await f.screen()).toContain('This pane is a shell now')
+ // The pane is marked before the input left for the engine is drained, and says so only after.
+ await f.waitFor(async () => (await f.screen()).includes('This pane is a shell now'))
})
it('bounds update restarts even when the replacement requests another restart', async () => {
diff --git a/cli/src/lib/engineInstall.spec.ts b/cli/src/lib/engineInstall.spec.ts
index 5635db012..f499462da 100644
--- a/cli/src/lib/engineInstall.spec.ts
+++ b/cli/src/lib/engineInstall.spec.ts
@@ -3,6 +3,7 @@ import { execFileSync } from 'node:child_process'
import { PROCESS_ENGINES } from '../engines/types.js'
import { ENGINE_INSTALL } from './engineInstall.js'
import { buildEngineLaunchArgv } from './engineLaunch.js'
+import { launchScriptOf } from '../testing/launchScript.js'
const OFFICIAL_COMMANDS = {
claude: 'npm install -g @anthropic-ai/claude-code',
@@ -41,11 +42,12 @@ describe('ENGINE_INSTALL', () => {
it('generates valid POSIX pane scripts for every recipe', () => {
for (const engine of PROCESS_ENGINES) {
- const script = buildEngineLaunchArgv(
+ const script = launchScriptOf(buildEngineLaunchArgv(
engine,
{ installIfMissing: ENGINE_INSTALL[engine] },
'/bin/sh',
- )[2]
+ ))
+ expect(script).toContain('harness_find_engine')
expect(() => execFileSync('/bin/sh', ['-n', '-c', script])).not.toThrow()
}
})
diff --git a/cli/src/lib/engineLaunch.spec.ts b/cli/src/lib/engineLaunch.spec.ts
index cd53eb3a0..550c14980 100644
--- a/cli/src/lib/engineLaunch.spec.ts
+++ b/cli/src/lib/engineLaunch.spec.ts
@@ -1,7 +1,7 @@
import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
-import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'
+import { chmodSync, existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, symlinkSync, utimesSync, writeFileSync } from 'node:fs'
import { execFileSync } from 'node:child_process'
-import { tmpdir } from 'node:os'
+import { constants, tmpdir } from 'node:os'
import { join } from 'node:path'
import { env } from '../config/env.js'
import {
@@ -13,6 +13,7 @@ import {
NAMED_AGENT_ARGS,
NamedAgentUnsupportedError,
HARNESS_WORDMARK_LINES,
+ MAX_FIRST_PROMPT_CHARS,
PERMISSION_MODES,
terminalHintLines,
buildEngineCommandArgv,
@@ -39,9 +40,12 @@ import {
} from './engineLaunch.js'
import { ENGINES, type AgentEngine } from '../engines/types.js'
import { engineBin } from './engineBin.js'
-import type { EngineInstallRecipe } from './engineInstall.js'
+import { engineInstallRecipe, type EngineInstallRecipe } from './engineInstall.js'
import { MIN_OPEN_FILES, RAISE_OPEN_FILES_SH } from './openFiles.js'
import { DSH_SESSION_ENV, harnessEnvToClear } from '../dsh/launch.js'
+import { launchScriptOf } from '../testing/launchScript.js'
+import { buildGridEngineLaunch, gridConflictingEnvToClear } from './gridLaunch.js'
+import { TmuxBackend } from './tmuxBackend.js'
// The launch script names the `grid` the daemon resolved, and a developer's own HARNESS_GRID_BIN
// would resolve to THEIR grid. The suite's runtime dir is already a throwaway (vitest.setup.ts), so
@@ -57,6 +61,15 @@ const GRID_PRELUDE = gridPanePrelude('grid')
* `engineFallbackPrelude`. `null` tmux: the suite must not depend on what this machine has. */
const FALLBACK = (engine: AgentEngine, shell: string) => engineFallbackPrelude(engine, shell, null)
const NO_TMUX = null
+/** How a plain launch ends: job control on, the binary its command names run at the script's top
+ * level, where a stop is resumed, then the exit handling (`engineRunScript`). */
+const RUN = 'if [ -n "${ZSH_VERSION:-}" ]; then setopt monitor 2>/dev/null || :; else set -m 2>/dev/null || :; fi\n'
+ + 'trap : INT QUIT\n'
+ + 'harness_engine_bin=$1\nshift\nharness_status=0\n"$harness_engine_bin" "$@" || harness_status=$?\nharness_resume\nharness_after'
+/** The engine's own run, in every launch. */
+const RUN_LINE = '"$harness_engine_bin" "$@" || harness_status=$?'
+/** What a POSIX login shell's `-c` is: its launch script, sourced from a one-time file. */
+const SOURCED = /^\. '.+\.sh'$/
describe('buildEngineLaunchArgv', () => {
it.each(['claude', 'terminal'] as const)('clears inherited harness context before a plain %s session runs', (engine) => {
@@ -72,27 +85,108 @@ describe('buildEngineLaunchArgv', () => {
})
it('wraps zsh in its interactive login form and execs the resolved binary', () => {
- expect(buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, undefined, NO_TMUX)).toEqual([
- '/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', '/bin/zsh', '-lic', `${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}harness_engine "$@"`, 'harness-engine', engineBin('claude'),
+ const argv = buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, undefined, NO_TMUX)
+ expect(argv).toEqual([
+ '/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', '/bin/zsh', '-lic', expect.stringMatching(SOURCED), 'harness-engine', engineBin('claude'),
])
+ expect(launchScriptOf(argv)).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}${RUN}`)
})
it('uses Ubuntu bash interactive startup files without making it a login shell', () => {
- expect(buildEngineLaunchArgv('claude', {}, '/bin/bash', undefined, undefined, NO_TMUX)).toEqual([
- '/bin/bash', '-ic', `${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/bash')}${GRID_PRELUDE}harness_engine "$@"`, 'harness-engine', engineBin('claude'),
- ])
+ const argv = buildEngineLaunchArgv('claude', {}, '/bin/bash', undefined, undefined, NO_TMUX)
+ expect(argv).toEqual(['/bin/bash', '-ic', expect.stringMatching(SOURCED), 'harness-engine', engineBin('claude')])
+ expect(launchScriptOf(argv)).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/bash')}${GRID_PRELUDE}${RUN}`)
+ })
+
+ it('hands the shell its script in a one-time file that removes itself, private to this user', () => {
+ // On the command line the script was most of a launch, and tmux takes the launch as one command of
+ // at most 16KiB: see `launchFile`.
+ const argv = buildEngineLaunchArgv('claude', { cwd: '/work' }, '/bin/zsh', undefined, undefined, NO_TMUX)
+ const file = SOURCED.test(argv[4]) ? argv[4].slice(3, -1) : ''
+ expect(file.startsWith(join(env.ADAPTER_DATA_DIR, 'launch'))).toBe(true)
+ expect(statSync(file).mode & 0o777).toBe(0o600)
+ expect(readFileSync(file, 'utf8').startsWith(`rm -f -- '${file}'\n`)).toBe(true)
+ const run = execFileSync('/bin/sh', ['-c', argv[4], 'harness-engine', '/', '/bin/sh', '-c', 'printf "%s|" "$PWD" "$@"', 'engine', 'a b'], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] })
+ expect(run).toBe('/|a b|')
+ expect(existsSync(file)).toBe(false)
+ })
+
+ it('runs Codex under dash, where there is one, when no login shell resolves', () => {
+ // macOS's /bin/sh is bash 3.2, which never sees the engine stop: the engine would wait, stopped, for
+ // good (STOP_PROOF_FUNCTIONS). dash resumes it like the rest.
+ const argv = buildEngineLaunchArgv('codex', {}, 'relative-shell', undefined, undefined, NO_TMUX)
+ expect(argv.slice(0, 2)).toEqual([existsSync('/bin/dash') ? '/bin/dash' : '/bin/sh', '-c'])
+ expect(argv[2]).toMatch(SOURCED)
+ expect(launchScriptOf(argv)).toContain('harness_resume')
+ })
+
+ it('sweeps a launch file that never ran once it is an hour old, and only then', () => {
+ const directory = join(env.ADAPTER_DATA_DIR, 'launch')
+ mkdirSync(directory, { recursive: true })
+ const stale = join(directory, 'never-ran.sh')
+ const fresh = join(directory, 'about-to-run.sh')
+ writeFileSync(stale, 'exit 0\n')
+ writeFileSync(fresh, 'exit 0\n')
+ utimesSync(stale, new Date(Date.now() - 2 * 3600_000), new Date(Date.now() - 2 * 3600_000))
+ buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, undefined, NO_TMUX)
+ expect(existsSync(stale)).toBe(false)
+ expect(existsSync(fresh)).toBe(true)
+ })
+
+ it('puts the script on the command line when there is no data folder to write it in', () => {
+ const saved = env.ADAPTER_DATA_DIR
+ env.ADAPTER_DATA_DIR = '/dev/null/no-data-folder'
+ try {
+ const argv = buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, undefined, NO_TMUX)
+ expect(argv[4]).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}${RUN}`)
+ } finally { env.ADAPTER_DATA_DIR = saved }
+ })
+
+ it('keeps the longest launch, as the whole tmux command, well inside what tmux takes', async () => {
+ // `tmux new-session` takes a launch as one command, its environment and options included, and
+ // refuses one over 16KiB with "command too long" (measured, tmux 3.7c). The longest a launch gets:
+ // Codex or Claude Code on a grid with its web tools, looked for first, taking over a terminal's
+ // conversation, resuming it, with Harness's Node, and the longest first prompt in three-byte
+ // characters. With the script on the command line Codex's came to 17.9KB; it goes in a file now.
+ const dir = mkdtempSync(join(tmpdir(), 'harness-launch-size-'))
+ const savedPath = process.env.PATH
+ try {
+ writeFileSync(join(dir, 'tmux'), `#!/bin/sh\nfor arg in "$@"; do printf '%s\\000' "$arg"; done > "$(mktemp ${dir}/call.XXXXXX)"\nprintf '%%42\\n'\n`, { mode: 0o755 })
+ process.env.PATH = `${dir}:${savedPath}`
+ const grid = { networkId: 'grid-3378218621364f16', networkName: 'autonomous.ai', baseUrl: 'https://grid.autonomous.ai/grid-3378218621364f16/relay/v1', apiKey: 'gridkey-abc123', model: 'GLM-4.7-Flash', mcpUrl: 'https://api-grid.autonomous.ai/v1/grid/web-mcp/' }
+ for (const engine of ['claude', 'codex'] as const) {
+ for (const call of readdirSync(dir).filter((name) => name.startsWith('call.'))) rmSync(join(dir, call))
+ const built = buildGridEngineLaunch(engine, grid, { hermesSystemManaged: false })
+ if (!built.ok) throw new Error(built.detail)
+ const cwd = '/Users/demo/projects/a project with a fairly long name'
+ const command = buildEngineLaunchArgv(engine, {
+ cwd, firstPrompt: '漢'.repeat(MAX_FIRST_PROMPT_CHARS), resumeSessionId: '019a2b3c-4d5e-7f60-8a9b-0c1d2e3f4a5b',
+ installIfMissing: engineInstallRecipe(engine), waitForPid: { pid: 4242, name: 'Codex' }, bypassPermission: true,
+ extraArgs: built.launch.args, clearEnv: gridConflictingEnvToClear(built.launch), harnessNode: true,
+ }, '/bin/zsh', '/opt/harness/runtime/node/bin/node', '/opt/harness/runtime/grid/grid', '/opt/homebrew/bin/tmux')
+ const created = await new TmuxBackend(undefined, () => 'daemon-0000').create({ cwd, label: `harness-${engine}-1791200000000`, env: built.launch.env, command })
+ expect(created.state, engine).toBe('succeeded')
+ const sizes = readdirSync(dir).filter((name) => name.startsWith('call.')).map((name) => statSync(join(dir, name)).size)
+ expect(Math.max(...sizes), engine).toBeLessThan(15 * 1024)
+ }
+ } finally {
+ process.env.PATH = savedPath
+ rmSync(dir, { recursive: true, force: true })
+ }
})
it('enters the workspace only after interactive startup has completed', () => {
const argv = buildEngineLaunchArgv('claude', { cwd: '/work/project' }, '/bin/zsh', undefined, undefined, NO_TMUX)
expect(argv).toEqual([
- '/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', '/bin/zsh', '-lic',
- `${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}if ! cd -- "$1"; then printf '%s\\n' 'harness: the selected working directory is unavailable.' >&2; exit 1; fi\n${unreadableCwdGuard(process.platform)}shift\nharness_engine "$@"`,
+ '/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', '/bin/zsh', '-lic', expect.stringMatching(SOURCED),
'harness-engine', '/work/project', engineBin('claude'),
])
+ expect(launchScriptOf(argv)).toBe(
+ `${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}if ! cd -- "$1"; then printf '%s\\n' 'harness: the selected working directory is unavailable.' >&2; exit 1; fi\n${unreadableCwdGuard(process.platform)}shift\n${RUN}`,
+ )
})
- describe('the engine runs INSIDE the pane shell (`harness_engine`), never exec\'d over it', () => {
+ describe('the engine runs INSIDE the pane shell (`engineRunScript`), never exec\'d over it', () => {
const run = (argv: string[]) => {
try {
return { status: 0, out: execFileSync(argv[0], argv.slice(1), { stdio: ['ignore', 'pipe', 'pipe'] }).toString() }
@@ -152,8 +246,8 @@ describe('buildEngineLaunchArgv', () => {
const tmux = join(folder, 'tmux')
const calls = join(folder, 'calls')
writeFileSync(tmux, `#!/bin/sh\n${refuse}printf '%s\\n' "$*" >> '${calls}'\n`, { mode: 0o755 })
- const prelude = engineFallbackPrelude('claude', '/bin/sh', tmux)
- execFileSync('/bin/sh', ['-c', `${prelude}harness_engine "$@"`, 'harness-engine', '/bin/sh', '-c', 'exit 3'], {
+ const argv = buildEngineLaunchArgv('claude', {}, '/bin/sh', undefined, undefined, tmux)
+ execFileSync(argv[0], [argv[1], argv[2], 'harness-engine', '/bin/sh', '-c', 'exit 3'], {
stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, TMUX_PANE: '%5' },
}).toString()
} catch (error) {
@@ -178,6 +272,88 @@ describe('buildEngineLaunchArgv', () => {
})
})
+ describe('a stop is not an exit: a stopped engine is continued, never left behind', () => {
+ const SHELLS = ['/bin/sh', '/bin/bash', '/bin/zsh', '/bin/dash'].filter(existsSync)
+ const npmRecipe: EngineInstallRecipe = { command: 'npm install -g fixture', source: 'test fixture', executable: { names: ['fixture'], npmGlobal: true } }
+ const launches: Array<[string, () => string[]]> = [
+ ['claude', () => buildEngineLaunchArgv('claude', { cwd: '/work' }, '/bin/zsh', undefined, undefined, '/opt/homebrew/bin/tmux')],
+ ['codex, run again after a failed startup', () => buildEngineLaunchArgv('codex', { cwd: '/work' }, '/bin/bash', undefined, undefined, '/opt/homebrew/bin/tmux')],
+ ['codex without tmux', () => buildEngineLaunchArgv('codex', {}, '/bin/zsh', undefined, undefined, NO_TMUX)],
+ ['an engine installed when missing', () => buildEngineLaunchArgv('claude', { installIfMissing: npmRecipe }, '/bin/zsh', undefined, undefined, '/opt/homebrew/bin/tmux')],
+ ['codex installed when missing, after a take-over', () => buildEngineLaunchArgv('codex', { installIfMissing: { ...npmRecipe, executable: { names: ['codex'] } }, waitForPid: { pid: 2 ** 22 + 7, name: 'Codex' } }, '/bin/zsh', undefined, undefined, '/opt/homebrew/bin/tmux')],
+ ['an engine installed first', () => buildEngineLaunchArgv('opencode', { installFirst: "curl -fsSL 'https://example.invalid/install' | bash" }, '/bin/bash', undefined, undefined, NO_TMUX)],
+ ]
+ const scriptOf = launchScriptOf
+
+ it.each(launches)('%s: parses in every POSIX shell a pane may run it in', (_name, launch) => {
+ const script = scriptOf(launch())
+ for (const shell of SHELLS) expect(() => execFileSync(shell, ['-n', '-c', script], { stdio: ['ignore', 'pipe', 'pipe'] }), shell).not.toThrow()
+ })
+
+ it.each(launches)('%s: runs the engine, and an install, only at the top level of the script', (name, launch) => {
+ // zsh gives up the whole script when a job stops inside a function, an `if`, a loop or an
+ // `eval`, and bash leaves every loop around one: only a top-level run can be resumed. Every
+ // block in these scripts indents its body, so a run in one would not start the line.
+ const lines = scriptOf(launch()).split('\n')
+ const runs = lines.filter((line) => /"\$harness_engine_bin"(?: \$\{harness_codex_no_daemon:\+--no-daemon\})? "\$@" \|\| harness_status=\$\?$/.test(line.trim()))
+ expect(runs.length).toBe(name.startsWith('codex, run again') || name.startsWith('codex installed') ? 4 : 1)
+ for (const run of runs) expect(run).toMatch(/^(?:\[ "\$harness_codex_go" != 1 \] \|\| )?"\$harness_engine_bin"/)
+ for (const install of lines.filter((line) => line.includes('eval '))) expect(install).toMatch(/^(?:\[ -n "\$harness_engine_bin" \] \|\| )?\((?:export npm_config_prefix=.*; )?eval /)
+ // What follows each run is the resume, at the top level too.
+ for (const run of runs) expect(lines[lines.indexOf(run) + 1]).toBe('harness_resume')
+ })
+
+ // The stop signals' numbers differ between macOS and Linux; dash's `kill -l` takes no names, so
+ // these come from Node's table for this platform.
+ const tstp = constants.signals.SIGTSTP
+ const stop = constants.signals.SIGSTOP
+ // `fg` and `jobs` need a terminal; these stand-ins count the stops and report each like the
+ // shell's own would: 128 + TSTP while the engine keeps stopping, then its exit status.
+ const ttou = constants.signals.SIGTTOU
+ const resume = (shell: string, status: number, stops: number, job = true, { again = tstp, before = '' } = {}) => execFileSync(shell, ['-c', `${engineFallbackPrelude('claude', shell, null)}
+stops=${stops}; resumed=0
+jobs() { ${job ? 'return 0' : 'return 1'}; }
+fg() { resumed=$((resumed + 1)); stops=$((stops - 1)); if [ "$stops" -gt 0 ]; then return ${128 + again}; fi; return 7; }
+${before}
+harness_status=${status}
+harness_resume
+printf '%s %s' "$harness_status" "$resumed"`], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] })
+
+ it.each(SHELLS)('%s: continues a stopped engine as often as it stops, and keeps its real exit status', (shell) => {
+ expect(resume(shell, 128 + tstp, 1)).toBe('7 1')
+ expect(resume(shell, 128 + tstp, 3)).toBe('7 3')
+ expect(resume(shell, 128 + stop, 2)).toBe('7 2')
+ })
+
+ it.each(SHELLS)('%s: leaves an engine that exited alone, and a job it did not start', (shell) => {
+ expect(resume(shell, 0, 1)).toBe('0 0')
+ expect(resume(shell, 3, 1)).toBe('3 0')
+ // An engine killed by a signal, or one that exits with a stop's number: not a stop.
+ expect(resume(shell, 128 + 15, 1)).toBe(`${128 + 15} 0`)
+ expect(resume(shell, tstp, 1)).toBe(`${tstp} 0`)
+ // A stop's status with no job to continue (a rc file's job is not the engine): nothing to resume.
+ expect(resume(shell, 128 + tstp, 1, false)).toBe(`${128 + tstp} 0`)
+ })
+
+ it.each(SHELLS)('%s: lets an engine that stops again at once wait a second between resumes, so nothing spins', (shell) => {
+ // Six stops, each within a second of its resume: three resumed at once, the rest after a pause.
+ const started = Date.now()
+ expect(resume(shell, 128 + tstp, 6)).toBe('7 6')
+ expect(Date.now() - started).toBeGreaterThanOrEqual(2_000)
+ }, 20_000)
+
+ it.each(SHELLS)('%s: ends an engine that cannot keep the terminal after five quick tries', (shell) => {
+ // TTOU over and over: the job is never let have the terminal (as when `fg` could not hand it over).
+ expect(resume(shell, 128 + ttou, 50, true, { again: ttou })).toBe('137 5')
+ }, 20_000)
+
+ it.each(SHELLS)('%s: ends an engine at its thousandth stop where the resume calls itself, and only there', (shell) => {
+ // bash and dash recurse once a stop, and overflow some ten thousand deep; zsh loops.
+ const atTheEnd = { before: 'harness_resumes=998' }
+ expect(resume(shell, 128 + tstp, 5, true, atTheEnd)).toBe(shell === '/bin/zsh' ? '7 5' : '137 1')
+ })
+ })
+
describe('Codex writer ownership', () => {
it.each(['supported', 'older', 'failed'] as const)('probes %s startup options without changing prompt arguments', mode => {
const folder = mkdtempSync(join(tmpdir(), 'harness-codex-launch-'))
@@ -186,8 +362,8 @@ describe('buildEngineLaunchArgv', () => {
writeFileSync(binary, '#!/bin/sh\nif [ "$1" = "--help" ]; then\n'
+ (mode === 'failed' ? 'exit 2\n' : `printf '%s\\n' '${mode === 'supported' ? ' --no-daemon Run locally' : ' --model Choose a model'}'\nexit 0\n`)
+ 'fi\nprintf "<%s>\\n" "$@"\n', { mode: 0o755 })
- const prelude = engineFallbackPrelude('codex', '/bin/sh', null)
- const args = ['-c', `${prelude}harness_engine "$@"`, 'harness-engine', binary, 'resume', 'same-conversation', 'literal $(nothing) `nothing` and spaces']
+ const script = buildEngineLaunchArgv('codex', {}, '/bin/sh', undefined, undefined, NO_TMUX)[2]
+ const args = ['-c', script, 'harness-engine', binary, 'resume', 'same-conversation', 'literal $(nothing) `nothing` and spaces']
if (mode === 'failed') {
try { execFileSync('/bin/sh', args, { stdio: ['ignore', 'pipe', 'pipe'] }); expect.fail('launch must refuse an unknown startup mode') }
catch (error) {
@@ -408,10 +584,10 @@ describe('buildEngineLaunchArgv', () => {
// On this branch every pane script opens with the open-files raise, the engine-in-a-shell
// wrapper and the grid prelude; the Node line lands after them, and a launch without
// `harnessNode` is exactly the baseline above.
- expect(argv[4]).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}${harnessNodePrelude('/opt/harness runtime/bin/node')}harness_engine "$@"`)
+ expect(launchScriptOf(argv)).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}${harnessNodePrelude('/opt/harness runtime/bin/node')}${RUN}`)
expect(harnessNodePrelude('/opt/harness runtime/bin/node')).toBe(
'if ! command -v node >/dev/null 2>&1; then PATH="${PATH:+$PATH:}"\'/opt/harness runtime/bin\'; export PATH; fi\n')
- expect(buildEngineLaunchArgv('claude', { harnessNode: false }, '/bin/zsh', undefined, undefined, NO_TMUX)[4]).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}harness_engine "$@"`)
+ expect(launchScriptOf(buildEngineLaunchArgv('claude', { harnessNode: false }, '/bin/zsh', undefined, undefined, NO_TMUX))).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('claude', '/bin/zsh')}${GRID_PRELUDE}${RUN}`)
})
it('the DSH prelude, run by a real shell, reaches the engine\'s PATH only when node is missing', () => {
@@ -543,26 +719,29 @@ describe('a login shell that is not POSIX', () => {
expect(interactiveEngineShell('/opt/homebrew/bin/nu')?.args).toEqual(['-c'])
})
- it('launches an engine through the shell, which hands a one-time POSIX script to /bin/sh', () => {
+ it('launches an engine through the shell, which hands a one-time POSIX script to dash, or else /bin/sh', () => {
const argv = buildEngineLaunchArgv('claude', { cwd: "/work/it's here" }, '/usr/local/bin/fish', '/opt/node/bin/node', '/opt/grid/bin/grid', '/usr/bin/tmux')
expect(argv.slice(0, 3)).toEqual(['/usr/local/bin/fish', '-i', '-c'])
expect(argv).toHaveLength(4)
- const file = /^exec \/bin\/sh '(.+)'$/.exec(argv[3])?.[1]
+ // dash, where there is one: run like this it still resumes an engine that stopped (bash as sh does not).
+ expect(argv[3].startsWith(`exec ${existsSync('/bin/dash') ? '/bin/dash' : '/bin/sh'} '`)).toBe(true)
+ const file = /^exec \/bin\/(?:da)?sh '(.+)'$/.exec(argv[3])?.[1]
expect(file && file.startsWith(join(dataDir, 'launch'))).toBe(true)
const script = readFileSync(file!, 'utf8')
// It removes itself first, then takes the arguments a POSIX shell would have been given.
- expect(script.startsWith(`rm -f -- "$0"\nset -- '/work/it'"'"'s here' '${engineBin('claude')}'`)).toBe(true)
- expect(script).toContain('harness_engine "$@"')
- // The same launch for zsh is unchanged: no file, the script straight to zsh.
+ expect(script.startsWith(`rm -f -- '${file}'\nset -- '/work/it'"'"'s here' '${engineBin('claude')}'`)).toBe(true)
+ expect(script).toContain(RUN_LINE)
+ // zsh runs its own: it sources the file itself, with the arguments on its command line.
const zsh = buildEngineLaunchArgv('claude', { cwd: '/work' }, '/bin/zsh', '/opt/node/bin/node', '/opt/grid/bin/grid', '/usr/bin/tmux')
expect(zsh.slice(0, 4)).toEqual(['/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', '/bin/zsh', '-lic'])
+ expect(zsh[4]).toMatch(SOURCED)
})
it.skipIf(!existsSync('/bin/tcsh'))('runs that script for real under tcsh, with its arguments intact and the file gone', () => {
const argv = buildEngineLaunchArgv('claude', { cwd: dataDir, installFirst: undefined }, '/bin/tcsh', '/opt/node/bin/node', '/opt/grid/bin/grid', null)
- const file = /^exec \/bin\/sh '(.+)'$/.exec(argv[argv.length - 1])![1]
+ const file = /^exec \/bin\/(?:da)?sh '(.+)'$/.exec(argv[argv.length - 1])![1]
// Run the file the way tcsh would hand it over, with the engine replaced by printf.
- writeFileSync(file, readFileSync(file, 'utf8').replace(/harness_engine "\$@"\s*$/, 'printf "%s|" "$PWD" "$@"'))
+ writeFileSync(file, readFileSync(file, 'utf8').replace(RUN_LINE, 'printf "%s|" "$PWD" "$harness_engine_bin" "$@"'))
const out = execFileSync('/bin/tcsh', ['-i', '-c', argv[argv.length - 1]], { encoding: 'utf8', env: { ...process.env, HOME: dataDir }, stdio: ['ignore', 'pipe', 'ignore'] })
expect(out).toContain(`${dataDir}|${engineBin('claude')}|`)
expect(existsSync(file)).toBe(false)
@@ -1031,15 +1210,15 @@ describe('buildEngineLaunchArgv — the grid the pane finds', () => {
it('puts the resolved grid first on PATH and turns its update check off, before the engine', () => {
const managed = '/opt/harness/runtime/grid-0.3.47-darwin-arm64/grid'
- const script = buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, managed)[4]
+ const script = launchScriptOf(buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, managed))
expect(script).toContain(`PATH='/opt/harness/runtime/grid-0.3.47-darwin-arm64'"\${PATH:+:$PATH}"\nexport PATH\n`)
expect(script).toContain('GRID_NO_UPDATE_CHECK=1\nexport GRID_NO_UPDATE_CHECK\n')
- expect(script.indexOf('export GRID_NO_UPDATE_CHECK')).toBeLessThan(script.indexOf('harness_engine "$@"'))
+ expect(script.indexOf('export GRID_NO_UPDATE_CHECK')).toBeLessThan(script.indexOf(RUN_LINE))
})
it('leaves PATH alone when grid is only a name on it, but still turns the update check off', () => {
- const script = buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, 'grid')[4]
+ const script = launchScriptOf(buildEngineLaunchArgv('claude', {}, '/bin/zsh', undefined, 'grid'))
expect(script).not.toContain('export PATH')
expect(script).toContain('GRID_NO_UPDATE_CHECK=1')
@@ -1094,7 +1273,7 @@ describe('buildEngineLaunchArgv with installFirst', () => {
buildEngineLaunchArgv('opencode', { installFirst: install }, '/bin/zsh')[4]
it('leaves the plain launch alone when nothing has to be installed', () => {
- expect(buildEngineLaunchArgv('opencode', {}, '/bin/zsh', undefined, undefined, NO_TMUX)[4]).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('opencode', '/bin/zsh')}${GRID_PRELUDE}harness_engine "$@"`)
+ expect(launchScriptOf(buildEngineLaunchArgv('opencode', {}, '/bin/zsh', undefined, undefined, NO_TMUX))).toBe(`${RAISE_OPEN_FILES_SH}${FALLBACK('opencode', '/bin/zsh')}${GRID_PRELUDE}${RUN}`)
})
it('keeps the engine argv positional, so the shell never re-parses a path or a flag', () => {
@@ -1104,7 +1283,7 @@ describe('buildEngineLaunchArgv with installFirst', () => {
}, '/bin/zsh')
expect(argv.slice(0, 4)).toEqual(['/usr/bin/env', 'DISABLE_AUTO_UPDATE=true', '/bin/zsh', '-lic'])
expect(argv.slice(5)).toEqual(['harness-engine', ...buildEngineCommandArgv('opencode', { bypassPermission: true })])
- expect(argv[4]).toContain('harness_engine "$@"')
+ expect(launchScriptOf(argv)).toContain(RUN_LINE)
})
it('runs the engine only when the install succeeded', async () => {
@@ -1125,7 +1304,7 @@ describe('buildEngineLaunchArgv with installFirst', () => {
})
it('prints the command before running it, so a long install is not a hung pane', () => {
- expect(script('npm install -g opencode-ai')).toContain('npm install -g opencode-ai')
+ expect(launchScriptOf(['-c', script('npm install -g opencode-ai')])).toContain('npm install -g opencode-ai')
})
it("survives an install line carrying a quote, rather than ending the shell's string", async () => {
@@ -1210,7 +1389,8 @@ if [ "$1" = prefix ]; then exit 0; fi
})
it('does not add the npm bootstrap to non-npm installers', () => {
- expect(script(recipe('true'))).not.toContain('managed Node.js/npm')
+ expect(launchScriptOf(['-c', script(recipe('true'))])).toContain('engine is missing')
+ expect(launchScriptOf(['-c', script(recipe('true'))])).not.toContain('managed Node.js/npm')
})
})
diff --git a/cli/src/lib/engineLaunch.ts b/cli/src/lib/engineLaunch.ts
index d2b3ad206..0a0f5b45e 100644
--- a/cli/src/lib/engineLaunch.ts
+++ b/cli/src/lib/engineLaunch.ts
@@ -1,6 +1,6 @@
import { execFile } from 'node:child_process'
import { randomUUID } from 'node:crypto'
-import { mkdirSync, realpathSync, statSync, writeFileSync } from 'node:fs'
+import { existsSync, mkdirSync, readdirSync, realpathSync, rmSync, statSync, writeFileSync } from 'node:fs'
import { homedir, userInfo } from 'node:os'
import { isAbsolute, basename, dirname, join } from 'node:path'
import { env } from '../config/env.js'
@@ -447,19 +447,71 @@ function engineShellArgv(shell: InteractiveEngineShell, args: readonly string[])
/**
* For a shell that is not POSIX: it loads the person's environment (their PATH, their version
- * managers), then hands the script to /bin/sh. The script and its arguments go in a one-time file, so
- * all the person's shell parses is `exec /bin/sh '<file>'`, which fish, tcsh, nushell and xonsh read
- * alike; the file removes itself as it starts. It sits in the daemon's own data folder, private to
- * this user, and holds no secret: those reach the pane through the session's environment, never its
- * command. `args` is what a POSIX shell would take after `-c`: the script, `$0`, then the arguments.
+ * managers), then hands the script to a POSIX shell (`posixRunner`). The script and its arguments go
+ * in a one-time file (`launchFile`), so all the person's shell parses is `exec /bin/dash '<file>'`,
+ * which fish, tcsh, nushell and xonsh read alike. `args` is what a POSIX shell would take after
+ * `-c`: the script, `$0`, then the arguments.
*/
function throughPosixShell(shell: InteractiveEngineShell, args: readonly string[]): string[] {
const [script = '', , ...positional] = args
+ const file = launchFile((path) => `rm -f -- ${shellSingleQuote(path)}\nset -- ${positional.map(shellSingleQuote).join(' ')}\n${script}`)
+ return [shell.path, ...shell.args, `exec ${posixRunner()} ${shellSingleQuote(file)}`]
+}
+
+/**
+ * The POSIX shell that runs a launch no login shell takes: dash where there is one (macOS ships it,
+ * and it is Debian's and Ubuntu's /bin/sh), else /bin/sh. Run non-interactive, dash resumes an engine
+ * that stopped like the rest (`STOP_PROOF_FUNCTIONS`), and so does Linux's bash 5; the bash 3.2 that
+ * is macOS's /bin/sh never sees the stop, and the engine would wait, stopped, for good.
+ */
+function posixRunner(): string {
+ return existsSync('/bin/dash') ? '/bin/dash' : '/bin/sh'
+}
+
+/**
+ * A one-time file in the daemon's own data folder holding a launch script, which removes itself as it
+ * starts. Private to this user, and no secret: those reach the pane through the session's environment,
+ * never its command. One that never ran (a pane tmux would not make) is swept an hour on.
+ *
+ * Why a file: the launch goes to tmux as one `new-session` command, which tmux refuses past 16KiB
+ * ("command too long", measured with tmux 3.7c), and the script is most of it. With a first prompt of
+ * 2,000 three-byte characters, a grid's arguments and environment and a resume id, a Codex launch
+ * written out on the command line came to 17.9KB. Named by a file, no launch comes near the limit.
+ */
+function launchFile(content: (path: string) => string): string {
const directory = join(env.ADAPTER_DATA_DIR, 'launch')
mkdirSync(directory, { recursive: true, mode: 0o700 })
+ sweepLaunchFiles(directory)
const file = join(directory, `${randomUUID()}.sh`)
- writeFileSync(file, `rm -f -- "$0"\nset -- ${positional.map(shellSingleQuote).join(' ')}\n${script}`, { mode: 0o600 })
- return [shell.path, ...shell.args, `exec /bin/sh ${shellSingleQuote(file)}`]
+ writeFileSync(file, content(file), { mode: 0o600 })
+ return file
+}
+
+const LAUNCH_FILE_TTL_MS = 60 * 60_000
+
+function sweepLaunchFiles(directory: string): void {
+ try {
+ for (const name of readdirSync(directory)) {
+ if (!name.endsWith('.sh')) continue
+ const path = join(directory, name)
+ try { if (Date.now() - statSync(path).mtimeMs > LAUNCH_FILE_TTL_MS) rmSync(path, { force: true }) } catch { /* gone already */ }
+ }
+ } catch { /* nothing to sweep */ }
+}
+
+/**
+ * The `-c` a POSIX login shell is given: `. '<file>'`, the launch script read from a one-time file
+ * (`launchFile`) by the shell itself. Sourced, its commands are the shell's own top level, where a
+ * stop is safe (`STOP_PROOF_FUNCTIONS`; measured the same as an inline script in zsh, bash, sh and
+ * dash), and they keep the shell's positional parameters. Without a data folder to write in, the
+ * script goes on the command line as it used to.
+ */
+function sourcedOnce(script: string): string {
+ try {
+ return `. ${shellSingleQuote(launchFile((path) => `rm -f -- ${shellSingleQuote(path)}\n${script}`))}`
+ } catch {
+ return script
+ }
}
/**
@@ -479,7 +531,7 @@ export function buildEngineLaunchArgv(
if (isTerminalEngine(engine)) return buildTerminalLaunchArgv(opts, shell)
const command = buildEngineCommandArgv(engine, opts)
const interactive = interactiveEngineShell(shell)
- ?? (engine === 'codex' ? { path: '/bin/sh', args: ['-c'], label: 'shell' } : null)
+ ?? (engine === 'codex' ? { path: posixRunner(), args: ['-c'], label: 'shell' } : null)
if (!interactive) return command
const enginePrelude = engineFallbackPrelude(engine, interactive.path, tmuxBinary)
// The clear comes FIRST, before the install as well as before the engine. An installer is a child
@@ -497,26 +549,55 @@ export function buildEngineLaunchArgv(
+ unreadableCwdGuard()
+ 'shift\n'
: ''
- const body = opts.installIfMissing
- ? installIfMissingThenExecScript(opts.installIfMissing, runtimeNode)
+ // The engine is found (or installed first), then run at the script's top level (`engineRunScript`).
+ const engineFound = opts.installIfMissing
+ ? installIfMissingScript(opts.installIfMissing, runtimeNode)
: opts.installFirst
- ? installThenExecScript(opts.installFirst)
- : 'harness_engine "$@"'
+ ? installFirstScript(opts.installFirst)
+ : 'harness_engine_bin=$1\n'
+ const body = `${JOB_CONTROL}${SIGNAL_GUARD}${engineFound}shift\n${engineRunScript(engine, tmuxBinary)}`
// The open-files raise goes ahead of everything, the installer included: a pane inherits the tmux
// SERVER's soft limit, which is launchd's 256 whenever the desktop app started the daemon that
// started the server, and an engine (Claude Code refuses outright) or an npm install under 256 is
// the failure the person then reads in the pane. See openFiles.ts.
const wait = opts.waitForPid ? waitForPidScript(opts.waitForPid) : ''
- return engineShellArgv(interactive, [RAISE_OPEN_FILES_SH + prelude + cwdPrelude + wait + body, 'harness-engine', ...(opts.cwd ? [opts.cwd] : []), ...command])
+ const script = RAISE_OPEN_FILES_SH + prelude + cwdPrelude + wait + body
+ return engineShellArgv(interactive, [isPosixShell(interactive.path) ? sourcedOnce(script) : script, 'harness-engine', ...(opts.cwd ? [opts.cwd] : []), ...command])
}
-/** Waits in the pane for [wait]'s process to end before the engine starts: see `waitForPid`. */
+/**
+ * Job control, before the install and the engine: a pane's interactive shell has it already, the
+ * hand-off's non-interactive one (`throughPosixShell`) gets it here (`STOP_PROOF_FUNCTIONS`), and
+ * without a terminal (the specs) a shell goes on without it. zsh is asked by its own option: a
+ * `set -m` that fails ends a zsh script, `|| :` or not.
+ */
+const JOB_CONTROL = 'if [ -n "${ZSH_VERSION:-}" ]; then setopt monitor 2>/dev/null || :; else set -m 2>/dev/null || :; fi\n'
+
+/**
+ * An engine that dies of SIGINT or SIGQUIT must not take the script with it. zsh, seeing its
+ * foreground job killed by one of those, gives up the rest of the script as if it had been interrupted
+ * itself; dash run interactive drops to its prompt; bash run without a terminal of its own (as the
+ * hand-off's /bin/sh) exits: the exit handling never ran and the pane closed, or sat in a bare shell,
+ * instead of turning into the person's shell (measured on main with zsh 5.9, dash and bash 5.3). npm's
+ * Codex is a Node wrapper that re-raises the signal its native engine died of, and one it does not
+ * listen for itself (QUIT) ends the job that way. With a trap they all go on; interactive bash and sh
+ * went on anyway. The non-interactive bash then reports a job killed by INT as a success (status 0),
+ * which still turns the pane into a shell. The trap does nothing else: the engine still gets the
+ * signals itself, since a caught signal is reset for a child. Set after a take-over's wait, which has
+ * a trap of its own.
+ */
+const SIGNAL_GUARD = 'trap : INT QUIT\n'
+
+/** Waits in the pane for [wait]'s process to end before the engine starts: see `waitForPid`. Its
+ * second's sleep runs in a command substitution, out of a Ctrl+Z's reach (`STOP_PROOF_FUNCTIONS`):
+ * zsh would take a stopped `sleep` for the end of the script, and bash would leave the loop and
+ * start the engine while the terminal's still has the conversation. */
function waitForPidScript(wait: { pid: number; name: string }): string {
const pid = Math.trunc(wait.pid)
const name = wait.name.replace(/[^A-Za-z0-9 ._-]/g, '')
return `printf '%s\\n' 'Waiting for the ${name} in your terminal to finish its turn.' 'It moves here when the turn ends. Ctrl-C leaves it there.'\n`
+ `trap 'printf "\\n%s\\n" "It stays in your terminal."; exit 130' INT\n`
- + `while kill -0 ${pid} 2>/dev/null; do sleep 1; done\n`
+ + `while kill -0 ${pid} 2>/dev/null; do harness_waited=$(sleep 1) || :; done\n`
+ `trap - INT\n`
+ `printf '\\033[H\\033[2J'\n`
}
@@ -538,10 +619,10 @@ export const ENGINE_INPUT_DRAIN_SH = ' if harness_tty=$(stty -g 2>/dev/null) &&
+ ' fi\n'
/**
- * The shell function every engine launch runs its engine through, instead of a bare `exec`.
+ * The shell functions every engine launch runs its engine with, instead of a bare `exec`.
*
* The engine is a CHILD of the pane's shell, and when it exits — `/exit`, Ctrl-C, a crash — the
- * pane does not die with it: the wrapper records the exit status on the pane (`ENGINE_EXIT_PANE_OPTION`,
+ * pane does not die with it: `harness_after` records the exit status on the pane (`ENGINE_EXIT_PANE_OPTION`,
* which is how the daemon tells "the engine left" from "the install is still running") and `exec`s
* the user's interactive shell in its place, exactly as a terminal opened with ⌘⇧T is. The daemon
* then turns the row back into a terminal (`registry.releaseEngine`) rather than deleting it, and
@@ -553,11 +634,12 @@ export const ENGINE_INPUT_DRAIN_SH = ' if harness_tty=$(stty -g 2>/dev/null) &&
* have it on PATH (the managed runtime never is). Without one the marker is skipped and the fallback
* still happens — only a failure at launch then takes the daemon's full wait to notice.
*
- * Without a terminal on stdin there is nobody to hand a shell to, so the function exits with the
+ * Without a terminal on stdin there is nobody to hand a shell to, so `harness_after` exits with the
* engine's status instead — which is also what keeps the specs that run these scripts honest.
*
- * `"$@" || status=$?` rather than `"$@"; status=$?`: a rc file that turned on `set -e` would end
- * the script on the engine's non-zero exit before the fallback ran (see RAISE_OPEN_FILES_SH).
+ * A stop is not an exit: `harness_resume` continues a stopped engine, so only its real exit reaches
+ * `harness_after` (`STOP_PROOF_FUNCTIONS`). The engine's run itself is `engineRunScript`'s, at the
+ * script's top level; for Codex these also hold its startup probe and retry.
*/
export function engineFallbackPrelude(engine: AgentEngine, shellPath: string, tmuxBinary: string | null): string {
const loginArgs = basename(shellPath).toLowerCase() === 'zsh' ? ' -l' : ''
@@ -571,15 +653,8 @@ export function engineFallbackPrelude(engine: AgentEngine, shellPath: string, tm
? ` [ -n "\${TMUX_PANE:-}" ] && { ${tmux} set-option -p -t "$TMUX_PANE" ${ENGINE_EXIT_PANE_OPTION} "$harness_status" >/dev/null 2>&1`
+ ` || ${tmux} set-option -w -t "$TMUX_PANE" ${ENGINE_EXIT_PANE_OPTION} "$harness_status" >/dev/null 2>&1; } || true\n`
: ''
- // A function has its own positional parameters. Reprobe each replacement
- // binary without adding --no-daemon to the saved launch arguments repeatedly.
- return (engine === 'codex'
- ? 'harness_codex_launch() {\n' + codexOwnedLaunchPrelude() + ' "$@"\n}\n'
- : '')
- + 'harness_engine() {\n'
- + (engine === 'codex' && tmuxBinary && isAbsolute(tmuxBinary)
- ? codexStartupRetryScript(tmuxBinary)
- : ` harness_status=0\n ${engine === 'codex' ? 'harness_codex_launch ' : ''}"$@" || harness_status=$?\n`)
+ return STOP_PROOF_FUNCTIONS
+ + 'harness_after() {\n'
+ ' if [ "$harness_status" -eq 127 ]; then exit 127; fi\n'
+ mark
// Only a pane — something with a terminal on stdin — gets a shell to type into. Run without one
@@ -589,6 +664,142 @@ export function engineFallbackPrelude(engine: AgentEngine, shellPath: string, tm
+ ` printf '\\n%s\\n' ${shellSingleQuote(`harness: ${command} exited ($harness_status). This pane is a shell now — run ${command} again, or stop the pane.`).replace('($harness_status)', `('"$harness_status"')`)}\n`
+ ` exec ${shellSingleQuote(shellPath)}${loginArgs}\n`
+ '}\n'
+ + (engine === 'codex' ? codexOwnedLaunchPrelude() : '')
+ + (codexRetries(engine, tmuxBinary) ? codexStartupRetryScript(tmuxBinary) : '')
+}
+
+/**
+ * A stopped engine is continued, never taken for one that exited: Ctrl+Z, which Claude Code and Codex
+ * both answer by suspending themselves (they give the terminal back and stop their whole process
+ * group, "Run `fg` to bring Claude Code back"), or a SIGSTOP from outside. tmux does this for a pane's
+ * own process (server_child_stopped continues it at once), but the engine is the pane shell's child,
+ * and an agent's pane has no prompt to type `fg` into.
+ *
+ * The pane shell has job control: the engine runs as its foreground job, in a process group of its own
+ * that holds the terminal. Seeing that job stop, the shell used to carry on to the end of the launch
+ * script (bash, sh) or give the script up at once (zsh), and exit, hanging up the stopped engine: the
+ * pane closed and the agent was gone. Now `harness_resume` puts a stopped engine back in the
+ * foreground with `fg`, as often as it stops, until it really exits, and keeps that exit status.
+ * Measured in tmux with zsh 5.9, bash 3.2 and 5.3, sh and dash, each its own way:
+ *
+ * - zsh gives up the whole script when a foreground job stops inside a function, an `if`, a loop or
+ * an `eval`, and carries on only after one at the top level; a stop seen by `fg` is fine anywhere.
+ * So the engine, and an install, run at the script's top level (`engineRunScript`). zsh prints its
+ * "suspended" and "continued" lines as they happen, as it would in a terminal.
+ * - bash breaks out of every loop around a job that stops, wherever the loop is, so there (and in
+ * sh and dash) the resume calls itself instead of looping; zsh's loops, which keeps it clear of
+ * zsh's limit on nested calls.
+ * - `fg` keeps its standard error. Run without a terminal of its own, bash takes standard error for
+ * the terminal, so a `fg 2>/dev/null` there never handed the terminal over: the engine stopped
+ * again on its first write to it, at once and for ever, until the recursion overflowed (bash 5.3
+ * as Fedora's /bin/sh, the hand-off of a fish or tcsh login).
+ * - A stop that comes back within a second of its resume, three times running, is resumed only after
+ * a second's pause, so nothing can spin; one that cannot take the terminal (TTIN, TTOU) five times
+ * running, or the thousandth stop of one run where the resume recurses, ends the engine instead
+ * (SIGKILL, status 137), and the pane turns into a shell as after any exit.
+ * - bash puts the terminal back in its own modes when a job stops and does not restore the job's on
+ * `fg` (zsh does; dash leaves them alone). Neither Claude Code nor Codex takes raw mode again on a
+ * plain SIGCONT, only after a stop of its own, so one stopped from outside under bash comes back
+ * to a line-mode terminal until it sets its modes again: accepted, and the e2e shows it. Ctrl+Z is
+ * not affected: the engine gave the terminal back and takes it again itself.
+ * - A stop of a process below the engine's own is seen by no shell, as one below a pane's process
+ * is seen by no tmux: npm's Codex is a Node wrapper with the native engine as its child, and a
+ * SIGSTOP of that child alone leaves it stopped. Its Ctrl+Z stops the whole group, wrapper too.
+ * - ksh93 stops the whole `... || harness_status=$?` list with the job and goes on with a status of
+ * 0, so there a stop still reads as the engine's exit, as it did everywhere before.
+ *
+ * Only a job that stopped is resumed, told by its status — 128 plus STOP, TSTP, TTIN or TTOU, by name,
+ * since the numbers differ between macOS and Linux — and not by `%%` alone, which could be a job a rc
+ * file left running.
+ *
+ * Every other command the script waits on — a probe, a backoff, the wait for a take-over — runs in a
+ * command substitution. That keeps it in the pane shell's own process group, which has no parent in
+ * its session (tmux is outside it), and the kernel discards a terminal stop sent to such a group: a
+ * Ctrl+Z at that moment is ignored rather than stopping a helper in a function, which zsh would again
+ * take for the end of the script.
+ */
+const STOP_PROOF_FUNCTIONS = [
+ 'harness_stopped() {',
+ ' [ "$harness_status" -gt 128 ] 2>/dev/null || return 1',
+ ' harness_signal=$(kill -l "$harness_status" 2>/dev/null) || return 1',
+ ' case $harness_signal in',
+ ' STOP|TSTP|TTIN|TTOU|SIGSTOP|SIGTSTP|SIGTTIN|SIGTTOU) jobs %% >/dev/null 2>&1 ;;',
+ ' *) return 1 ;;',
+ ' esac',
+ '}',
+ 'harness_fg() {',
+ ' harness_now=$(date +%s 2>/dev/null) || harness_now=0',
+ ' if [ $((harness_now - harness_resumed)) -le 1 ]; then harness_quick=$((harness_quick + 1)); else harness_quick=0; fi',
+ ' harness_resumes=$((harness_resumes + 1))',
+ ' case $harness_signal in *TTIN|*TTOU) harness_stuck=$harness_quick ;; *) harness_stuck=0 ;; esac',
+ ' if [ "$harness_stuck" -ge 5 ] || { [ -z "${ZSH_VERSION:-}" ] && [ "$harness_resumes" -ge 1000 ]; }; then',
+ ' kill -9 %% 2>/dev/null',
+ ' harness_status=137',
+ ' return 0',
+ ' fi',
+ ' if [ "$harness_quick" -ge 3 ]; then harness_waited=$(sleep 1) || :; fi',
+ ' harness_resumed=$(date +%s 2>/dev/null) || harness_resumed=0',
+ ' harness_status=0',
+ ' fg >/dev/null || harness_status=$?',
+ '}',
+ 'harness_resume() {',
+ ' if [ -n "${ZSH_VERSION:-}" ]; then',
+ ' while harness_stopped; do harness_fg; done',
+ ' return 0',
+ ' fi',
+ ' harness_stopped || return 0',
+ ' harness_fg',
+ ' harness_resume',
+ '}',
+ 'harness_resumed=0',
+ 'harness_quick=0',
+ 'harness_resumes=0',
+ '',
+].join('\n')
+
+/**
+ * The engine's run, then `harness_after`, at the script's top level: the one place a stop is safe in
+ * every shell (`STOP_PROOF_FUNCTIONS`). `$harness_engine_bin` is the engine and `"$@"` its arguments.
+ *
+ * `... || harness_status=$?` rather than `...; harness_status=$?`: a rc file that turned on `set -e`
+ * would end the script on the engine's non-zero exit before the fallback ran (see RAISE_OPEN_FILES_SH).
+ *
+ * A Codex launch can end before its conversation opens and be run again (`codexStartupRetryScript`):
+ * its runs are written out one after another, since no loop or function may hold the engine.
+ */
+function engineRunScript(engine: AgentEngine, tmuxBinary: string | null): string {
+ const run = `"$harness_engine_bin"${engine === 'codex' ? ' ${harness_codex_no_daemon:+--no-daemon}' : ''} "$@"`
+ if (!codexRetries(engine, tmuxBinary)) {
+ return [
+ ...(engine === 'codex' ? ['harness_codex_probe "$harness_engine_bin"'] : []),
+ 'harness_status=0',
+ `${run} || harness_status=$?`,
+ 'harness_resume',
+ 'harness_after',
+ ].join('\n')
+ }
+ const attempt = [
+ 'harness_codex_start "$harness_engine_bin"',
+ `[ "$harness_codex_go" != 1 ] || ${run} || harness_status=$?`,
+ 'harness_resume',
+ 'harness_codex_next',
+ ]
+ return [
+ 'harness_codex_attempt=1',
+ 'harness_codex_updated=0',
+ 'harness_codex_go=1',
+ ...Array.from({ length: CODEX_STARTUP_RUNS }, () => attempt).flat(),
+ 'harness_after',
+ ].join('\n')
+}
+
+/** Codex's runs at most: the first, one more after a startup update, two more after a timed-out
+ * account lookup (`codexStartupRetryScript`). */
+const CODEX_STARTUP_RUNS = 4
+
+/** Codex is run again after a failed startup only where the pane can be read: through the daemon's tmux. */
+function codexRetries(engine: AgentEngine, tmuxBinary: string | null): tmuxBinary is string {
+ return engine === 'codex' && !!tmuxBinary && isAbsolute(tmuxBinary)
}
/** Keep a successful startup update or transient account lookup failure in the original launch.
@@ -596,49 +807,72 @@ export function engineFallbackPrelude(engine: AgentEngine, shellPath: string, tm
* keeps discovery from archiving the row between attempts. Never reparse "$@": it
* includes the original prompt, images, model, permissions and resume/fork arguments.
* Codex's updater runs before the conversation opens, so replay that exact launch
- * once, without choosing an unrelated conversation via `resume --last`. */
+ * once, without choosing an unrelated conversation via `resume --last`.
+ *
+ * `harness_codex_start` readies a run and `harness_codex_next` decides whether another
+ * follows; the runs are `engineRunScript`'s, at the top level. The probes and the backoff
+ * run in command substitutions, out of a Ctrl+Z's reach (`STOP_PROOF_FUNCTIONS`).
+ *
+ * The probe is written once, as `harness_codex_check`: tmux refuses a command longer than
+ * 16KiB, and the launch, first prompt and all, goes to it as one (`tmux new-session`). */
function codexStartupRetryScript(tmuxBinary: string): string {
- const probe = `${shellSingleQuote(process.execPath)} -e ${shellSingleQuote(CODEX_STARTUP_RETRY_PROBE)}`
- return ' harness_codex_attempt=1\n'
- + ' harness_codex_updated=0\n'
- + ' while :; do\n'
- + ' harness_codex_before=\n'
- + ` if [ -n "\${TMUX_PANE:-}" ]; then harness_codex_before=$(${probe} before ${shellSingleQuote(tmuxBinary)} "$TMUX_PANE") || harness_codex_before=; fi\n`
- + ' harness_status=0\n'
- + ' harness_codex_launch "$@" || harness_status=$?\n'
- + ' if [ "$harness_status" -eq 0 ] && [ "$harness_codex_updated" -eq 0 ] && [ -n "$harness_codex_before" ] &&\n'
- + ` ${probe} after-update ${shellSingleQuote(tmuxBinary)} "$TMUX_PANE" "$harness_codex_before"; then\n`
- + ' harness_codex_updated=1\n'
- + ` printf '\\n%s\\n' 'harness: Codex updated. Continuing startup…'\n`
- + ' continue\n'
- + ' fi\n'
- + ' [ "$harness_status" -eq 1 ] && [ "$harness_codex_attempt" -lt 3 ] && [ -n "$harness_codex_before" ] || break\n'
- + ` ${probe} after ${shellSingleQuote(tmuxBinary)} "$TMUX_PANE" "$harness_codex_before" || break\n`
- + ' harness_codex_delay=$((harness_codex_attempt * 2))\n'
- + ' harness_codex_attempt=$((harness_codex_attempt + 1))\n'
- + ' harness_codex_cancelled=0\n'
- + " trap 'harness_codex_cancelled=1' INT\n"
- + ` printf '\\n%s\\n' "harness: Codex account lookup timed out. Retrying startup ($harness_codex_attempt/3) in \${harness_codex_delay}s; Ctrl-C cancels."\n`
- + ' sleep "$harness_codex_delay" || harness_codex_cancelled=1\n'
- + ' trap - INT\n'
- + ' if [ "$harness_codex_cancelled" -eq 1 ]; then harness_status=130; break; fi\n'
- + ' done\n'
+ const probe = 'harness_codex_check'
+ const tmux = shellSingleQuote(tmuxBinary)
+ return 'harness_codex_check() {\n'
+ + ` ${shellSingleQuote(process.execPath)} -e ${shellSingleQuote(CODEX_STARTUP_RETRY_PROBE)} "$@"\n`
+ + '}\n'
+ + 'harness_codex_start() {\n'
+ + ' [ "$harness_codex_go" = 1 ] || return 0\n'
+ + ' harness_codex_before=\n'
+ + ` if [ -n "\${TMUX_PANE:-}" ]; then harness_codex_before=$(${probe} before ${tmux} "$TMUX_PANE") || harness_codex_before=; fi\n`
+ + ' harness_codex_probe "$1"\n'
+ + ' harness_status=0\n'
+ + '}\n'
+ + 'harness_codex_next() {\n'
+ + ' [ "$harness_codex_go" = 1 ] || return 0\n'
+ + ' harness_codex_go=0\n'
+ + ' if [ "$harness_status" -eq 0 ] && [ "$harness_codex_updated" -eq 0 ] && [ -n "$harness_codex_before" ] &&\n'
+ + ` harness_codex_seen=$(${probe} after-update ${tmux} "$TMUX_PANE" "$harness_codex_before"); then\n`
+ + ' harness_codex_updated=1\n'
+ + ` printf '\\n%s\\n' 'harness: Codex updated. Continuing startup…'\n`
+ + ' harness_codex_go=1\n'
+ + ' return 0\n'
+ + ' fi\n'
+ + ' [ "$harness_status" -eq 1 ] && [ "$harness_codex_attempt" -lt 3 ] && [ -n "$harness_codex_before" ] || return 0\n'
+ + ` harness_codex_seen=$(${probe} after ${tmux} "$TMUX_PANE" "$harness_codex_before") || return 0\n`
+ + ' harness_codex_delay=$((harness_codex_attempt * 2))\n'
+ + ' harness_codex_attempt=$((harness_codex_attempt + 1))\n'
+ + ' harness_codex_cancelled=0\n'
+ + " trap 'harness_codex_cancelled=1' INT\n"
+ + ` printf '\\n%s\\n' "harness: Codex account lookup timed out. Retrying startup ($harness_codex_attempt/3) in \${harness_codex_delay}s; Ctrl-C cancels."\n`
+ + ' harness_codex_seen=$(sleep "$harness_codex_delay") || harness_codex_cancelled=1\n'
+ + ' trap : INT\n'
+ + ' if [ "$harness_codex_cancelled" -eq 1 ]; then harness_status=130; return 0; fi\n'
+ + ' harness_codex_go=1\n'
+ + '}\n'
}
/** Codex 0.157+ otherwise puts the writer outside tmux in a shared server. Keep
* Harness-owned launches process-owned so Close, hook attribution, provider env
* and RAM accounting describe the same lifetime. Probe the binary AFTER any
* install, in the exact pane shell; older versions simply omit the flag. The
- * probe is bounded and never changes the user's Codex configuration. */
+ * probe is bounded and never changes the user's Codex configuration.
+ *
+ * It sets `harness_codex_no_daemon` for the run rather than rewriting "$@", so each
+ * run of the retry probes its binary afresh (an update may have replaced it) without
+ * adding the flag to the saved arguments again; and it runs in a command substitution,
+ * out of a Ctrl+Z's reach (`STOP_PROOF_FUNCTIONS`). */
function codexOwnedLaunchPrelude(): string {
const probe = `const {execFileSync}=require('node:child_process');try { const h=execFileSync(process.argv[1],['--help'],{timeout:5000,maxBuffer:1048576,encoding:'utf8',stdio:['ignore','pipe','pipe']});process.exit(/--no-daemon(?:[^A-Za-z0-9-]|$)/.test(h)?0:64); } catch { process.exit(2); }`
- return ` harness_codex_mode=0\n`
- + ` ${shellSingleQuote(process.execPath)} -e ${shellSingleQuote(probe)} "$1" || harness_codex_mode=$?\n`
- + ` case "$harness_codex_mode" in\n`
- + ` 0) harness_codex_bin="$1"; shift; set -- "$harness_codex_bin" --no-daemon "$@" ;;\n`
- + ` 64) ;;\n`
+ return 'harness_codex_probe() {\n'
+ + ' harness_codex_mode=0\n'
+ + ` harness_codex_seen=$(${shellSingleQuote(process.execPath)} -e ${shellSingleQuote(probe)} "$1") || harness_codex_mode=$?\n`
+ + ' case "$harness_codex_mode" in\n'
+ + ' 0) harness_codex_no_daemon=1 ;;\n'
+ + ' 64) harness_codex_no_daemon= ;;\n'
+ ` *) printf '%s\\n' 'harness: could not verify Codex startup options. Please try opening this session again.' >&2; exit 1 ;;\n`
- + ` esac\n`
+ + ' esac\n'
+ + '}\n'
}
/**
@@ -760,11 +994,11 @@ function clearEnvPrelude(names: readonly string[] | undefined): string {
}
/**
- * Install, then become the engine — or say why not, and stop.
+ * Install, then start the engine — or say why not, and stop.
*
* Three things this script gets right, each of which was a way to lose:
*
- * * **`exec` only on success.** Running the engine after a failed install reproduces the exact
+ * * **The engine only on success.** Running the engine after a failed install reproduces the exact
* `command not found` this feature exists to replace, with a screenful of npm output above it to
* bury the cause.
* * **The install line is not interpolated into a command.** It is the vendor's own published line
@@ -774,15 +1008,21 @@ function clearEnvPrelude(names: readonly string[] | undefined): string {
* * **`"$@"` still carries the engine argv positionally**, so engine paths and flags are never
* re-parsed by the shell. That property is what the plain `exec "$@"` had and it is preserved.
*
+ * The install runs in a subshell at the script's top level, like the engine, so a Ctrl+Z in the
+ * middle of it is resumed rather than ending the pane (`STOP_PROOF_FUNCTIONS`).
+ *
* The banner matters more than it looks. A pane that sits silent for forty seconds of `npm install`
* reads as a hung agent, and the person's next move is to kill it.
*/
-function installThenExecScript(install: string): string {
+function installFirstScript(install: string): string {
return [
`printf '%s\\n' 'harness: installing the engine — this pane becomes the agent when it finishes' 'harness: $ ${install.replace(/'/g, "'\\''")}' ''`,
- `if eval ${JSON.stringify(install)}; then harness_engine "$@"; fi`,
- `printf '\\n%s\\n' 'harness: the install failed, so the agent was not started. The command is above; fix it and create the agent again.'`,
- 'exit 1',
+ 'harness_status=0',
+ `(eval ${JSON.stringify(install)}) || harness_status=$?`,
+ 'harness_resume',
+ `if [ "$harness_status" -ne 0 ]; then printf '\\n%s\\n' 'harness: the install failed, so the agent was not started. The command is above; fix it and create the agent again.'; exit 1; fi`,
+ 'harness_engine_bin=$1',
+ '',
].join('\n')
}
@@ -852,25 +1092,17 @@ export function gridPanePrelude(binary: string): string {
* source-owned candidate paths below bridge that one-shell gap without sourcing arbitrary profile
* files a second time. npm installs also get their active global prefix as a fallback.
*/
-function installIfMissingThenExecScript(recipe: EngineInstallRecipe, runtimeNode: string): string {
+function installIfMissingScript(recipe: EngineInstallRecipe, runtimeNode: string): string {
const install = recipe.command
const names = recipe.executable.names.map(shellSingleQuote).join(' ')
const paths = engineInstallPaths(recipe).map(shellSingleQuote).join(' ')
// Scope both npm env spellings to the installer subprocess. Do not rewrite .npmrc or install
// into a different OS user's shared prefix, and do not tie the engine to a versioned Node folder.
+ // Either way a subshell, run at the top level where a stop is resumed (STOP_PROOF_FUNCTIONS).
const installCommand = recipe.executable.npmGlobal
? `(export npm_config_prefix=${shellSingleQuote(npmEnginePrefix())} NPM_CONFIG_PREFIX=${shellSingleQuote(npmEnginePrefix())}; eval ${shellSingleQuote(install)})`
- : `eval ${shellSingleQuote(install)}`
+ : `(eval ${shellSingleQuote(install)})`
const candidates = [names, paths].filter(Boolean).join(' ')
- const tryCandidates = candidates
- ? `for candidate in ${candidates}; do try_engine "$candidate" "$@" || true; done`
- : ''
- const tryNpmGlobal = recipe.executable.npmGlobal
- ? [
- 'npm_prefix="$(npm prefix -g 2>/dev/null)" || true',
- `if [ -n "$npm_prefix" ]; then for bin in ${names}; do try_engine "$npm_prefix/bin/$bin" "$@" || true; done; fi`,
- ].join('\n')
- : ''
return [
'resolve_engine() {',
' candidate="$1"',
@@ -881,38 +1113,46 @@ function installIfMissingThenExecScript(recipe: EngineInstallRecipe, runtimeNode
' esac',
' [ -n "$resolved" ] && [ -f "$resolved" ] && [ -x "$resolved" ]',
'}',
- 'try_engine() {',
- ' candidate="$1"',
- ' shift',
- ' if ! resolve_engine "$candidate"; then return 1; fi',
- ' shift',
- ' harness_engine "$resolved" "$@"',
+ // The first of these that is there becomes `harness_engine_bin`. Loops are fine here: nothing in
+ // them can stop the way the engine can.
+ 'harness_find_engine() {',
+ ' harness_engine_bin=',
+ ` for candidate in "$1"${candidates ? ` ${candidates}` : ''}; do`,
+ ' if resolve_engine "$candidate"; then harness_engine_bin="$resolved"; return 0; fi',
+ ' done',
+ ...(recipe.executable.npmGlobal ? [
+ ' npm_prefix="$(npm prefix -g 2>/dev/null)" || true',
+ ' if [ -n "$npm_prefix" ]; then',
+ ` for bin in ${names}; do`,
+ ' if resolve_engine "$npm_prefix/bin/$bin"; then harness_engine_bin="$resolved"; return 0; fi',
+ ' done',
+ ' fi',
+ ] : []),
+ ' return 1',
'}',
// A previously installed npm launcher also needs Node. Resolve the runtime before executing
// it, not just before installing it; fresh users often have no system node on PATH.
npmRuntimePrelude(recipe, runtimeNode),
- 'try_engine "$1" "$@" || true',
- tryCandidates,
- tryNpmGlobal,
+ 'if ! harness_find_engine "$1"; then',
...(recipe.executable.npmGlobal ? [
- 'if ! command -v npm >/dev/null 2>&1; then',
- ` printf '%s\\n' 'harness: npm is unavailable and the managed Node.js/npm runtime could not be used.'`,
- ' exit 1',
- 'fi',
+ ' if ! command -v npm >/dev/null 2>&1; then',
+ ` printf '%s\\n' 'harness: npm is unavailable and the managed Node.js/npm runtime could not be used.'`,
+ ' exit 1',
+ ' fi',
] : []),
- `printf '%s\\n' 'harness: engine is missing — installing it in this terminal' 'harness: $ ${install.replace(/'/g, "'\\''")}' ''`,
- ...(recipe.executable.npmGlobal ? [`printf '%s\\n' ${shellSingleQuote(`harness: installing for this user in ${npmEnginePrefix()}`)}`] : []),
- `if ${installCommand}; then`,
+ ` printf '%s\\n' 'harness: engine is missing — installing it in this terminal' 'harness: $ ${install.replace(/'/g, "'\\''")}' ''`,
+ ...(recipe.executable.npmGlobal ? [` printf '%s\\n' ${shellSingleQuote(`harness: installing for this user in ${npmEnginePrefix()}`)}`] : []),
+ 'fi',
+ 'harness_status=0',
+ `[ -n "$harness_engine_bin" ] || ${installCommand} || harness_status=$?`,
+ 'harness_resume',
+ 'if [ -z "$harness_engine_bin" ]; then',
+ ` if [ "$harness_status" -ne 0 ]; then printf '\\n%s\\n' 'harness: the install failed, so the agent was not started. The command is above; fix it and create the agent again.'; exit 1; fi`,
' hash -r 2>/dev/null || true',
- ' try_engine "$1" "$@" || true',
- ` ${tryCandidates}`,
- tryNpmGlobal.split('\n').map((line) => ` ${line}`).join('\n'),
- ` printf '\\n%s\\n' 'harness: the install completed, but its executable could not be found. Check the installer output and PATH above.'`,
- ' exit 1',
+ ` if ! harness_find_engine "$1"; then printf '\\n%s\\n' 'harness: the install completed, but its executable could not be found. Check the installer output and PATH above.'; exit 1; fi`,
'fi',
- `printf '\\n%s\\n' 'harness: the install failed, so the agent was not started. The command is above; fix it and create the agent again.'`,
- 'exit 1',
- ].filter(Boolean).join('\n')
+ '',
+ ].filter(Boolean).join('\n') + '\n'
}
function availabilityScript(recipe: EngineInstallRecipe | undefined): string {
diff --git a/cli/src/lib/engineStop.real.spec.ts b/cli/src/lib/engineStop.real.spec.ts
new file mode 100644
index 000000000..c8c9c1226
--- /dev/null
+++ b/cli/src/lib/engineStop.real.spec.ts
@@ -0,0 +1,290 @@
+/**
+ * A stopped engine, for real: the launch script in an isolated tmux server, under each shell a pane
+ * may run it in, with a stand-in engine that stops the way Claude Code and Codex do. Ctrl+Z: it gives
+ * the terminal back, stops its whole process group, and takes the terminal again only when continued
+ * from that stop; a stop from outside it does not notice, and only repaints. Codex comes as npm
+ * installs it, a Node wrapper with the engine as its child. Each time it must come back on its own and
+ * take keys again, and in the end exit with its own status through the usual exit handling, the pane
+ * turning into a shell; one that dies of a signal must too. A take-over that waits for the terminal's
+ * engine keeps waiting through a Ctrl+Z, and an engine that cannot stay running is ended, not spun.
+ *
+ * Before, the pane's shell took the stop for the end of the engine and exited, hanging up the stopped
+ * engine with the pane. Linux has no tcsh here: a stand-in named fish hands the launch to dash or, with
+ * no dash (Fedora), to bash run as /bin/sh.
+ */
+import { execFileSync, spawn } from 'node:child_process'
+import { existsSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'
+import { env } from '../config/env.js'
+import { isolatedTmux, type IsolatedTmux } from '../testing/isolatedTmux.js'
+import { resolveBinaryOnPath } from './binaryOnPath.js'
+import { buildEngineLaunchArgv } from './engineLaunch.js'
+import type { EngineInstallRecipe } from './engineInstall.js'
+import { ENGINE_EXIT_PANE_OPTION } from './tmux.js'
+
+const tmuxBin = resolveBinaryOnPath('tmux')
+const real = process.env.RUN_REAL_TMUX_DISCOVERY === '1' && tmuxBin ? describe : describe.skip
+
+/** A login shell that is not POSIX, as far as the launch can tell (it goes by the name): `-i -c <it>`
+ * runs what it is given, and on its own it is an interactive shell, as the one exec'd after an exit. */
+const standIn = realpathSync(mkdtempSync(join(tmpdir(), 'harness-not-posix-')))
+const FISH = join(standIn, 'fish')
+writeFileSync(FISH, '#!/bin/sh\n[ $# -eq 0 ] && exec /bin/sh -i\nwhile [ $# -gt 1 ]; do shift; done\nexec /bin/sh -c "$1"\n', { mode: 0o755 })
+
+/** The login shells, as the launch is given them. tcsh where there is one, else the stand-in. */
+const SHELLS = [...['/bin/zsh', '/bin/bash', '/bin/sh', '/bin/dash'].filter(existsSync), existsSync('/bin/tcsh') ? '/bin/tcsh' : FISH]
+/** A login shell that hands the launch to a POSIX shell rather than running it. */
+const handsOff = (shell: string): boolean => shell === FISH || shell.endsWith('/tcsh')
+const isBash = (path: string): boolean => execFileSync(path, ['-c', 'printf %s "${BASH_VERSION:-}"'], { encoding: 'utf8' }) !== ''
+/** Whether bash is the shell with the engine as its job: bash puts the terminal back in its own modes
+ * when the job stops and leaves them when it continues it (`STOP_PROOF_FUNCTIONS`), and run without
+ * a terminal of its own it reports a job killed by SIGINT as a success. The hand-off goes to dash
+ * where there is one, else to /bin/sh, which is bash on Fedora. */
+const bashRuns = (shell: string): boolean => handsOff(shell)
+ ? !existsSync('/bin/dash') && isBash('/bin/sh')
+ : (shell === '/bin/bash' || shell === '/bin/sh') && isBash(shell)
+
+const roots: string[] = [standIn]
+const waitedFor: number[] = []
+const stopped: number[] = []
+let server: IsolatedTmux | undefined
+const saved = { claude: env.CLAUDE_PATH, codex: process.env.CODEX_PATH, data: env.ADAPTER_DATA_DIR }
+
+afterEach(async () => {
+ // A stopped process left behind would outlive the test.
+ for (const pid of stopped.splice(0)) { try { process.kill(pid, 'SIGCONT') } catch { /* gone */ } }
+ await server?.close()
+ server = undefined
+ for (const pid of waitedFor.splice(0)) { try { process.kill(pid) } catch { /* gone */ } }
+ env.CLAUDE_PATH = saved.claude
+ if (saved.codex === undefined) delete process.env.CODEX_PATH
+ else process.env.CODEX_PATH = saved.codex
+ env.ADAPTER_DATA_DIR = saved.data
+ for (const dir of roots.splice(1)) rmSync(dir, { recursive: true, force: true })
+})
+
+/**
+ * A raw-mode engine that logs what happens to it. Ctrl+Z: raw mode off, a one-time SIGCONT handler
+ * that takes it again, and SIGTSTP to its whole process group. Any SIGCONT is logged, and that is all.
+ * `q` exits 7; `t` starts it stopping again the moment it is continued, with SIGTTOU (`runaway=ttou`)
+ * or SIGTSTP (`runaway=tstp`). With `wrapped`, it runs as the child of a Node wrapper, as npm's Codex
+ * does: same group and terminal, INT, TERM and HUP forwarded, the child's end mirrored.
+ */
+function fixture(runaway: '' | 'ttou' | 'tstp' = '', wrapped = false) {
+ const dir = realpathSync(mkdtempSync(join(tmpdir(), 'harness-engine-stop-')))
+ roots.push(dir)
+ const log = join(dir, 'engine.log')
+ writeFileSync(log, '')
+ const engine = join(dir, 'engine')
+ writeFileSync(engine, `#!${process.execPath}
+const fs = require('node:fs');
+const { spawn } = require('node:child_process');
+const log = (line) => fs.appendFileSync(${JSON.stringify(log)}, line + '\\n');
+if (process.argv.includes('--help')) { console.log(' --no-daemon Run in this process'); process.exit(0); }
+if (${wrapped} && !process.env.FIXTURE_CHILD) {
+ const child = spawn(process.execPath, [__filename, ...process.argv.slice(2)], { stdio: 'inherit', env: { ...process.env, FIXTURE_CHILD: '1' } });
+ log('wrapper ' + process.pid);
+ for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP']) process.on(signal, () => { try { child.kill(signal); } catch {} });
+ child.on('exit', (code, signal) => { if (signal) process.kill(process.pid, signal); else process.exit(code ?? 1); });
+ return;
+}
+log('up ' + process.pid + ' ' + JSON.stringify(process.argv.slice(2)));
+process.stdin.setRawMode(true);
+process.stdout.write('ENGINE READY\\r\\n');
+let runaway = false;
+process.on('SIGCONT', () => {
+ log('continued');
+ process.stdout.write('ENGINE BACK\\r\\n');
+ if (runaway) process.kill(0, ${JSON.stringify(runaway === 'ttou' ? 'SIGTTOU' : 'SIGTSTP')});
+});
+process.stdin.on('data', (data) => {
+ const key = data.toString();
+ if (key.includes('\\x1a')) {
+ log('suspending');
+ process.stdin.setRawMode(false);
+ process.once('SIGCONT', () => { process.stdin.setRawMode(true); log('raw again'); });
+ process.kill(0, 'SIGTSTP');
+ return;
+ }
+ if (key.includes('t') && ${JSON.stringify(runaway)}) { runaway = true; process.kill(0, ${JSON.stringify(runaway === 'ttou' ? 'SIGTTOU' : 'SIGTSTP')}); return; }
+ if (key.includes('q')) { log('quit'); process.exit(7); }
+ log('key ' + JSON.stringify(key));
+});
+setInterval(() => {}, 1000);
+`, { mode: 0o755 })
+ const lines = () => readFileSync(log, 'utf8').split('\n').filter(Boolean)
+ return { dir, engine, lines, count: (prefix: string) => lines().filter((line) => line.startsWith(prefix)).length }
+}
+
+async function until<T>(what: string, check: () => T | Promise<T>, ms = 8_000): Promise<NonNullable<T>> {
+ const deadline = Date.now() + ms
+ for (;;) {
+ const value = await check()
+ if (value) return value as NonNullable<T>
+ if (Date.now() > deadline) throw new Error(`timed out waiting for ${what}`)
+ await new Promise((resolve) => setTimeout(resolve, 50))
+ }
+}
+
+const state = (pid: number): string => {
+ try { return execFileSync('ps', ['-o', 'stat=', '-p', String(pid)], { encoding: 'utf8' }).trim() } catch { return '' }
+}
+
+/** A pane running the launch of `engine` under `shell`, from a throwaway home: no rc file of the
+ * developer's runs in it. */
+async function open(shell: string, engine: 'claude' | 'codex', f: ReturnType<typeof fixture>, extra: Record<string, unknown> = {}) {
+ if (engine === 'claude') env.CLAUDE_PATH = f.engine
+ else process.env.CODEX_PATH = f.engine
+ env.ADAPTER_DATA_DIR = f.dir
+ const argv = buildEngineLaunchArgv(engine, { cwd: f.dir, ...extra }, shell, undefined, 'grid', tmuxBin)
+ server = await isolatedTmux()
+ const pane = await server.run('new-session', '-d', '-P', '-F', '#{pane_id}', '-x', '100', '-y', '30', '-s', 'stop',
+ '-e', `HOME=${f.dir}`, '-e', `ZDOTDIR=${f.dir}`, '-e', 'ENV=', ...argv)
+ await server.run('set-option', '-w', '-t', 'stop', 'remain-on-exit', 'on')
+ const tmux = server
+ const screen = () => tmux.run('capture-pane', '-p', '-t', pane)
+ onTestFailed(async () => {
+ let log = ''
+ try { log = f.lines().join('\n') } catch { /* swept already */ }
+ console.log(`---- ${shell} ${engine}\n${log}\n---- pane\n${await screen().catch(() => '')}`)
+ })
+ return {
+ pane,
+ screen,
+ keys: (...keys: string[]) => tmux.run('send-keys', '-t', pane, ...keys),
+ /** The terminal's line discipline: raw (`-icanon`) as an engine keeps it, or line mode. */
+ lineMode: async () => / icanon /.test(` ${execFileSync('/bin/sh', ['-c', 'stty -a < "$0"', (await tmux.run('display-message', '-p', '-t', pane, '#{pane_tty}')).trim()], { encoding: 'utf8' }).replace(/\s+/g, ' ')} `),
+ exitMark: async () => (await tmux.run('show-option', '-p', '-v', '-t', pane, ENGINE_EXIT_PANE_OPTION).catch(() => '')).trim(),
+ dead: async () => (await tmux.run('display-message', '-p', '-t', pane, '#{pane_dead}')).trim(),
+ }
+}
+
+/** Most launches look for the engine first and install it when it is missing; here it is there. */
+const installIfMissing: EngineInstallRecipe = { command: 'false', source: 'test fixture', executable: { names: ['harness-no-such-engine'] } }
+const LAUNCHES = [
+ { engine: 'claude', label: 'claude' },
+ { engine: 'claude', label: 'claude, looked for first', extra: { installIfMissing } },
+ { engine: 'codex', label: 'codex, npm wrapper and all', wrapped: true },
+] as const
+
+real('a stopped engine in a real pane', () => {
+ for (const shell of SHELLS) for (const launch of LAUNCHES) {
+ const engine = launch.engine
+ it(`${shell}, ${launch.label}: comes back after Ctrl+Z and after a SIGSTOP, and its own exit still turns the pane into a shell`, async () => {
+ const f = fixture('', 'wrapped' in launch)
+ const p = await open(shell, engine, f, 'extra' in launch ? launch.extra : {})
+ const up = await until('the engine to start', () => f.lines().find((line) => line.startsWith('up ')))
+ const pid = Number(up.split(' ')[1])
+ // Codex is given its writer flag, then the arguments as they were.
+ if (engine === 'codex') expect(JSON.parse(up.slice(up.indexOf('[')))).toEqual(['--no-daemon'])
+
+ // Ctrl+Z: the whole group stops, the shell continues it, and the engine takes the terminal again.
+ await p.keys('C-z')
+ await until('the engine to take the terminal again after Ctrl+Z', () => f.count('raw again') === 1)
+ await until('the engine to run again', () => !state(pid).startsWith('T'))
+ await p.keys('-l', 'k')
+ await until('a key to reach it', () => f.count('key "k"') === 1)
+ expect(await p.lineMode()).toBe(false)
+
+ // A SIGSTOP of the engine alone, from outside: continued too. It does not take the terminal
+ // again (the real CLIs do not on a plain SIGCONT), so an interactive bash, which put back its own
+ // modes when the job stopped, leaves it in line mode; zsh restores the job's, dash leaves them be.
+ // npm's Codex is the shell's job through its wrapper, and that is the process stopped here: a
+ // stop of the native child alone reaches no shell, as one below a pane's process reaches no tmux.
+ const job = 'wrapped' in launch ? Number(f.lines().find((line) => line.startsWith('wrapper '))!.split(' ')[1]) : pid
+ process.kill(job, 'SIGSTOP')
+ stopped.push(job)
+ await until('the engine to be continued after a SIGSTOP', () => f.count('continued') === 2)
+ await until('the engine to run again', () => !state(pid).startsWith('T'))
+ expect(await p.lineMode()).toBe(bashRuns(shell))
+ await p.keys('-l', 'j')
+ await p.keys('Enter')
+ await until('a key to reach it again', () => f.lines().some((line) => line.startsWith('key "j')))
+ // One engine all along: no second start.
+ expect(f.count('up ')).toBe(1)
+
+ await p.keys('-l', 'q')
+ await p.keys('Enter')
+ expect(await until('the exit to be marked on the pane', () => p.exitMark())).toBe('7')
+ await until('the pane to say it is a shell now', async () => (await p.screen()).includes('This pane is a shell now') || null)
+ expect(await p.dead()).toBe('0')
+ }, 30_000)
+ }
+
+ for (const shell of SHELLS) {
+ it(`${shell}: an engine that dies of SIGINT, or a Codex whose engine dies of SIGQUIT, still turns the pane into a shell`, async () => {
+ // zsh gave up the script when its job was killed by INT or QUIT, and dash dropped to a prompt.
+ // bash run as the hand-off's /bin/sh (Fedora) reports a job killed by INT as a success.
+ for (const [engine, signal, status] of [['claude', 'SIGINT', handsOff(shell) && bashRuns(shell) ? '0' : '130'], ['codex', 'SIGQUIT', '131']] as const) {
+ const f = fixture('', engine === 'codex')
+ const p = await open(shell, engine, f)
+ const up = await until('the engine to start', () => f.lines().find((line) => line.startsWith('up ')))
+ // npm's Codex wrapper ends with the signal its child died of, unless it listens for it (INT).
+ process.kill(Number(up.split(' ')[1]), signal)
+ expect(await until(`the ${signal} death to be marked on the pane`, () => p.exitMark())).toBe(status)
+ await until('the pane to say it is a shell now', async () => (await p.screen()).includes('This pane is a shell now') || null)
+ expect(await p.dead()).toBe('0')
+ await server?.close()
+ server = undefined
+ }
+ }, 30_000)
+ }
+
+ for (const shell of SHELLS.filter((path) => !path.endsWith('tcsh') && path !== FISH)) {
+ it(`${shell}: an engine that cannot keep the terminal is ended after a few tries, not resumed for ever`, async () => {
+ const f = fixture('ttou')
+ const p = await open(shell, 'claude', f)
+ await until('the engine to start', () => f.count('up ') === 1)
+ await p.keys('-l', 't')
+ expect(await until('the engine to be ended', () => p.exitMark(), 20_000)).toBe('137')
+ // Five continues at most, the last ones a second apart: no spinning.
+ expect(f.count('continued')).toBeLessThanOrEqual(6)
+ expect(await p.dead()).toBe('0')
+ }, 30_000)
+
+ it(`${shell}: an engine that stops again at once is resumed a second apart, not spun`, async () => {
+ const f = fixture('tstp')
+ const p = await open(shell, 'claude', f)
+ await until('the engine to start', () => f.count('up ') === 1)
+ await p.keys('-l', 't')
+ await new Promise((resolve) => setTimeout(resolve, 6_000))
+ // Three quick resumes, then about one a second; without the pause it would be hundreds.
+ expect(f.count('continued')).toBeLessThanOrEqual(10)
+ expect(f.count('continued')).toBeGreaterThanOrEqual(4)
+ expect(await p.exitMark()).toBe('')
+ expect(await p.dead()).toBe('0')
+ }, 30_000)
+ }
+})
+
+real('a take-over that waits in a real pane', () => {
+ // The wait loop's `sleep` used to be the shell's foreground job: Ctrl+Z stopped it, and zsh gave the
+ // script up while bash left the loop and started the engine with the terminal's still running.
+ for (const shell of SHELLS.filter((path) => !path.endsWith('tcsh') && path !== FISH)) {
+ it(`${shell}: Ctrl+Z does not end the wait, and Ctrl-C still leaves the conversation where it is`, async () => {
+ const f = fixture()
+ const other = spawn('sleep', ['60'], { detached: true, stdio: 'ignore' })
+ other.unref()
+ waitedFor.push(other.pid!)
+ const p = await open(shell, 'claude', f, { waitForPid: { pid: other.pid!, name: 'Claude Code' } })
+ const deadStatus = async () => (await server!.run('display-message', '-p', '-t', p.pane, '#{pane_dead} #{pane_dead_status}')).trim()
+
+ await until('the pane to say it is waiting', async () => (await p.screen()).includes('Waiting for the Claude Code in your terminal') || null)
+ await p.keys('C-z')
+ // Longer than a turn of the wait: still waiting, and nothing started.
+ await new Promise((resolve) => setTimeout(resolve, 2_500))
+ expect(await deadStatus()).toBe('0')
+ expect(f.lines()).toEqual([])
+
+ await p.keys('C-c')
+ await until('the pane to say the conversation stays', async () => (await p.screen()).includes('It stays in your terminal.') || null)
+ expect(await until('the script to end', async () => {
+ const now = await deadStatus()
+ return now.startsWith('1 ') ? now : null
+ })).toBe('1 130')
+ expect(f.lines()).toEqual([])
+ }, 30_000)
+ }
+})
diff --git a/cli/src/lib/registry.ts b/cli/src/lib/registry.ts
index dd20abfc6..37263e5ba 100644
--- a/cli/src/lib/registry.ts
+++ b/cli/src/lib/registry.ts
@@ -1748,7 +1748,7 @@ class Registry {
* The engine in this pane has exited and the pane is a shell now: the row is a terminal.
*
* For every agent, not only one that began as a terminal — an engine the app launched runs
- * inside its pane's shell too (engineLaunch.ts `harness_engine`), and `/exit` or Ctrl-C leaves
+ * inside its pane's shell too (engineLaunch.ts `engineRunScript`), and `/exit` or Ctrl-C leaves
* that shell at its prompt with the engine's last screen above it.
*
* What goes is what described the PROCESS and its session — session id, transcript, pid,
diff --git a/cli/src/lib/tmux.ts b/cli/src/lib/tmux.ts
index 137b967ff..5520295f6 100644
--- a/cli/src/lib/tmux.ts
+++ b/cli/src/lib/tmux.ts
@@ -1186,7 +1186,7 @@ export async function setPaneStyle(pane: string, style: string): Promise<boolean
/** What tmux knows about a pane right now. See `agentCreateDiagnosis.ts` for why this is read. */
/**
* The pane option an engine's launch wrapper sets when the engine exits and the pane falls back to
- * a shell (engineLaunch.ts, `harness_engine`): the engine's exit status. Empty/absent while the
+ * a shell (engineLaunch.ts, `harness_after`): the engine's exit status. Empty/absent while the
* wrapper is still running the engine — and for the whole life of the fallback shell after that,
* once something reads it, so `respawn` clears it before every new launch in the same pane.
*/
diff --git a/cli/src/testing/launchScript.ts b/cli/src/testing/launchScript.ts
new file mode 100644
index 000000000..716d1be80
--- /dev/null
+++ b/cli/src/testing/launchScript.ts
@@ -0,0 +1,15 @@
+import { readFileSync } from 'node:fs'
+
+/**
+ * The script an engine launch hands its pane's shell, as the shell will read it: from the one-time file
+ * a POSIX login shell sources (`. '<file>'`, engineLaunch.ts `sourcedOnce`), without the line that
+ * removes the file, or the `-c` script itself when it went on the command line.
+ */
+export function launchScriptOf(argv: readonly string[]): string {
+ for (const arg of argv) {
+ const quoted = /^\. '(.+)'$/s.exec(arg)?.[1]
+ if (quoted) return readFileSync(quoted.replace(/'"'"'/g, "'"), 'utf8').replace(/^rm -f -- '[^\n]*'\n/, '')
+ }
+ const flag = argv.findIndex((arg) => arg === '-lic' || arg === '-ic' || arg === '-c')
+ return argv[flag + 1] ?? ''
+}