Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

harness e2e on main, with the daemon's local credential: grid-switch stops failing on 'no machineId' (#1441)

lingrd6bb09d3f9e0a
diff --git a/cli/src/cli.ts b/cli/src/cli.ts
index b51b70d6a..78979da13 100644
--- a/cli/src/cli.ts
+++ b/cli/src/cli.ts
@@ -161,6 +161,8 @@ Machine:
harness help agents the agent commands, states, exit codes and rules on one screen
harness settings [<name> on|off] what agents may do (agents-start-agents, pull-request-line, agent-messages): your account's experiments in Settings ▸ Experimental, each off by default
harness machines delete <id> remove ANOTHER machine (refuses this one; use \`harness logout\`)
+ harness e2e plan which coding tools have a new version out (the grid-switch e2e trigger)
+ harness e2e grid-switch move a fresh agent subscription -> grid -> back home with its tool + MCP; leaves a review bundle
harness remote from a Harness terminal tile: open a terminal on another of your machines and move this tile to it
harness version print the installed version (v${VERSION})
harness update [--force] update to the latest build now (it also self-updates in the background;
@@ -2750,6 +2752,20 @@ switch (cmd) {
color: process.stdout.isTTY === true,
})
break
+ case 'e2e':
+ // The grid-switch e2e: a fresh agent moved subscription -> grid -> back home with the person's
+ // tool + MCP, leaving a review bundle. Env-driven (see src/e2e/runGridSwitchTrace.ts); the
+ // relay-side reviewer (opencode `e2e-watchdog`) lives with the relay, in autonomous-grid-cli.
+ // Shipped with the CLI, so it is always the daemon's own version that drives the daemon.
+ if (args[0] === 'grid-switch') {
+ import('./e2e/runGridSwitchTrace.js')
+ .then(({ runGridSwitch }) => runGridSwitch())
+ .then((r) => { process.exitCode = r.error || r.stuck ? 1 : 0 })
+ .catch(onError)
+ } else if (args[0] === 'plan') {
+ import('./e2e/plan.js').then(({ planCommand }) => planCommand(flags.includes('--json'))).catch(onError)
+ } else { console.error(`Unknown command: e2e ${args[0] ?? ''} (grid-switch | plan)`); usage(1) }
+ break
case 'machines':
if (!args[0]) machinesListCommand(flags.includes('--json')).catch(onError)
else if (args[0] === 'list') machinesListCommand(flags.includes('--json')).catch(onError)
diff --git a/cli/src/e2e/artifacts.ts b/cli/src/e2e/artifacts.ts
new file mode 100644
index 000000000..f9570872c
--- /dev/null
+++ b/cli/src/e2e/artifacts.ts
@@ -0,0 +1,35 @@
+/**
+ * artifacts — where a run leaves what people and the watchdog read AFTERWARDS.
+ *
+ * Two places, on purpose:
+ * - `outDir()`: scratch under $TMPDIR (`grid-matrix-out/`) — the agent's workspace, dry-run reports.
+ * - `runDir(session)`: the durable review bundle, under the harness's own data dir
+ * (`<ADAPTER_DATA_DIR>/e2e/runs/<session>/`), one folder per run:
+ * trace.json the report (legs, trace, pane tails, watchdog verdict)
+ * manifest.json ids + how to reopen everything (opencode session, daemon log, workspace)
+ * daemon.log the daemon's lines for this run's agent/session, cut from harness.log
+ * watchdog.jsonl the opencode agent's raw event stream (every tool call it made)
+ * watchdog.md its review, as text
+ * workspace/ tools/, .codex/ or .mcp.json, .e2e/*.log — the tool/MCP call logs
+ * evidence/ what the watchdog saved while reproducing (proxy req/res, bisect table)
+ */
+import { mkdirSync } from 'node:fs'
+import { join } from 'node:path'
+import { tmpdir } from 'node:os'
+import { env } from '../config/env.js'
+
+export function outDir(): string {
+ const dir = join(tmpdir(), 'grid-matrix-out')
+ mkdirSync(dir, { recursive: true })
+ return dir
+}
+
+export function runsRoot(): string {
+ return join(env.ADAPTER_DATA_DIR, 'e2e', 'runs')
+}
+
+export function runDir(session: string): string {
+ const dir = join(runsRoot(), session.replace(/[^A-Za-z0-9._@>-]/g, '-'))
+ mkdirSync(join(dir, 'evidence'), { recursive: true })
+ return dir
+}
diff --git a/cli/src/e2e/currentVersion.ts b/cli/src/e2e/currentVersion.ts
new file mode 100644
index 000000000..0b5b5e49e
--- /dev/null
+++ b/cli/src/e2e/currentVersion.ts
@@ -0,0 +1,46 @@
+/**
+ * current version provider — reads the INSTALLED version of each coding-agent tool on the server.
+ * The harness already captures this as `cli_version` on a registered agent; here we read it from
+ * each binary's `--version` for the trigger to compare against latest.
+ */
+import { execFile } from 'node:child_process'
+import { promisify } from 'node:util'
+import type { AgentEngine } from '../engines/types.js'
+import type { VersionProvider } from './trigger.js'
+
+const execFileP = promisify(execFile)
+
+/** How to ask each engine binary its version. */
+const VERSION_COMMAND: Partial<Record<AgentEngine, string[]>> = {
+ codex: ['--version'],
+ claude: ['--version'],
+}
+
+/** Nice display names for logs/notifies. */
+export const ENGINE_BIN: Partial<Record<AgentEngine, string>> = {
+ codex: 'codex',
+ claude: 'claude',
+}
+
+/** Extract the dotted version from a `--version` string like `codex-cli 0.155.0` / `2.0.34`. */
+export function parseVersionLine(line: string): string | null {
+ const m = line.match(/\b\d+\.\d+(?:\.\d+)*\b/)
+ return m ? m[0] : null
+}
+
+/** Reads installed versions from the real binaries on PATH. */
+export class InstalledCurrentVersion implements VersionProvider {
+ constructor(private readonly bin: Partial<Record<AgentEngine, string>> = ENGINE_BIN) {}
+
+ async currentVersion(engine: AgentEngine): Promise<string | null> {
+ const argv = VERSION_COMMAND[engine]
+ const binary = this.bin[engine]
+ if (!argv || !binary) return null
+ try {
+ const { stdout } = await execFileP(binary, argv, { timeout: 10_000 })
+ return parseVersionLine(stdout) ?? null
+ } catch {
+ return null
+ }
+ }
+}
diff --git a/cli/src/e2e/e2e.spec.ts b/cli/src/e2e/e2e.spec.ts
new file mode 100644
index 000000000..c670046db
--- /dev/null
+++ b/cli/src/e2e/e2e.spec.ts
@@ -0,0 +1,664 @@
+import { describe, expect, it } from 'vitest'
+import { gridSpawn, backHomeSpawn } from './gridArgv.js'
+import { MockVersionSource, NpmVersionSource } from './versionSource.js'
+import { compareVersions, planMatrixRuns } from './trigger.js'
+import { sessionName, workspaceDirName } from './sessionName.js'
+import { buildMatrixEntry } from './matrix.js'
+import type { AgentEngine } from '../engines/types.js'
+import { SMOKE_CHECKS, SCENARIO, LEGS, firstStuck, plannedLegs, quotaHit, scenarioFor, type CheckId } from './smokeChecks.js'
+import { pickGridModel } from './gridSwitchDriver.js'
+import { prepareWorkspace, readLog, logPath, readWorkspaceFile, preAcceptClaudeBypassMode, CALC_MCP_MJS } from './workspace.js'
+import { toolResult, useByRef } from './sessionTools.js'
+import { execFileSync } from 'node:child_process'
+import { probeLeg, runCheck, dismissStartupDialogs, STARTUP_DIALOGS, type Tmux } from './paneProbe.js'
+import { mkdtempSync, mkdirSync, readFileSync, writeFileSync, chmodSync, existsSync, rmSync } from 'node:fs'
+import { join } from 'node:path'
+import { tmpdir } from 'node:os'
+
+const GRID = {
+ networkId: 'grid-1',
+ networkName: 'someone-grid',
+ baseUrl: 'https://grid.autonomous.ai/grid-1/relay/v1',
+ apiKey: 'sk-probe',
+ model: 'grid:gpt-5-mini',
+}
+
+describe('argv-package (spawn exactly like the harness)', () => {
+ it('codex grid: provider argv + wire_api + clearEnv', () => {
+ const spec = gridSpawn('codex', GRID, 'S1')
+ expect('ok' in spec ? false : true).toBe(true) // not an error spec
+ const s = spec as { command: string[]; env: Record<string, string>; clearEnv: string[] }
+ expect(s.command[0]).toBe('codex')
+ expect(s.command).toContain('resume') // codex resume <id>
+ expect(s.command).toContain('S1')
+ expect(s.command.join(' ')).toContain('model_provider="grid"')
+ expect(s.command.join(' ')).toContain('model_providers.grid.wire_api="responses"')
+ expect(s.command.join(' ')).toContain('https://grid.autonomous.ai/grid-1/relay/v1')
+ expect(Object.keys(s.env).length).toBeGreaterThan(0) // the grid key goes in env, not argv
+ expect(s.clearEnv.length).toBeGreaterThan(0)
+ })
+
+ it('claude grid: --resume + ANTHROPIC_BASE_URL env', () => {
+ const spec = gridSpawn('claude', GRID, 'S1') as { command: string[]; env: Record<string, string> }
+ expect(spec.command[0]).toBe('claude')
+ expect(spec.command).toContain('--resume')
+ expect(spec.env.ANTHROPIC_BASE_URL).toContain('grid.autonomous.ai/grid-1')
+ })
+
+ it('back home clears the grid env vars', () => {
+ const spec = backHomeSpawn('claude', 'S1') as { clearEnv: string[] }
+ expect(spec.clearEnv).toContain('ANTHROPIC_BASE_URL')
+ })
+})
+
+describe('version source (mock first, npm later)', () => {
+ it('mock returns fixtures offline', async () => {
+ const src = new MockVersionSource()
+ expect(await src.latestVersion('codex')).toBe('0.156.0')
+ expect(await src.latestVersion('claude')).toBe('2.1.279')
+ })
+
+ it('npm impl reads the registry via fetch (stubbed)', async () => {
+ let called = ''
+ let sentInit: RequestInit | undefined
+ const fetchImpl = (async (url: string, init?: RequestInit) => {
+ called = url
+ sentInit = init
+ return { ok: true, json: async () => ({ version: '9.9.9' }) } as unknown as Response
+ }) as typeof fetch
+ const src = new NpmVersionSource(fetchImpl)
+ expect(await src.latestVersion('codex')).toBe('9.9.9')
+ expect(called).toContain('@openai/codex')
+ // No Accept header: `/latest` answers 406 to the abbreviated-metadata type, and the failure is
+ // silent (latest=null reads as "nothing new"), so the request is pinned bare. See versionSource.
+ expect(sentInit).toBeUndefined()
+ })
+
+ it('reads each engine\'s STABLE channel, and never a pre-release', async () => {
+ const asked: string[] = []
+ const answer = (version: string) => (async (url: string) => {
+ asked.push(url)
+ return { ok: true, json: async () => ({ version }) } as unknown as Response
+ }) as typeof fetch
+ expect(await new NpmVersionSource(answer('2.1.273')).latestVersion('claude')).toBe('2.1.273')
+ expect(await new NpmVersionSource(answer('0.156.1')).latestVersion('codex')).toBe('0.156.1')
+ // claude's stable is its own `stable` tag, not `latest` (its fast channel); codex's is `latest`.
+ expect(asked).toEqual(['https://registry.npmjs.org/@anthropic-ai/claude-code/stable', 'https://registry.npmjs.org/@openai/codex/latest'])
+ // Whatever tag it came from, a pre-release is never "the new version".
+ for (const v of ['0.158.0-alpha.8', '2.1.300-beta.1', '1.0.0-rc.2', '0.1.2505172116-nightly']) {
+ expect(await new NpmVersionSource(answer(v)).latestVersion('codex'), v).toBeNull()
+ }
+ })
+
+ // A stub cannot catch a registry that refuses our headers, and that exact bug once kept the whole
+ // pipeline quiet. This talks to npm for real, so it is opt-in: `RUN_REAL_NPM=1 vitest run src/e2e`.
+ it.runIf(process.env.RUN_REAL_NPM === '1')('npm impl really answers for both engines', async () => {
+ const src = new NpmVersionSource()
+ for (const engine of ['codex', 'claude'] as const) {
+ expect(await src.latestVersion(engine)).toMatch(/^\d+\.\d+\.\d+/)
+ }
+ }, 30_000)
+})
+
+describe('claude bypass-permissions warning (what killed a run once)', () => {
+ it('accepts it in a config copy, is idempotent, and leaves an unreadable one alone', () => {
+ const home = mkdtempSync(join(tmpdir(), 'wd-home-'))
+ try {
+ expect(preAcceptClaudeBypassMode(home)).toBe('skipped') // no ~/.claude.json at all
+ writeFileSync(join(home, '.claude.json'), JSON.stringify({ projects: { '/x': { allowedTools: [] } } }))
+ expect(preAcceptClaudeBypassMode(home)).toBe('accepted')
+ const after = JSON.parse(readFileSync(join(home, '.claude.json'), 'utf8'))
+ expect(after.bypassPermissionsModeAccepted).toBe(true)
+ expect(after.projects['/x']).toEqual({ allowedTools: [] }) // nothing else touched
+ expect(preAcceptClaudeBypassMode(home)).toBe('already')
+ writeFileSync(join(home, '.claude.json'), 'not json')
+ expect(preAcceptClaudeBypassMode(home)).toBe('skipped')
+ } finally {
+ rmSync(home, { recursive: true, force: true })
+ }
+ })
+
+ it('the probe answers the warning by moving off the default "No, exit"', () => {
+ const dialog = STARTUP_DIALOGS.find((d) => d.id === 'claude-bypass-accept')!
+ expect(dialog.match.test(' WARNING: Claude Code running in Bypass Permissions mode')).toBe(true)
+ expect(dialog.keys).toEqual(['Down', 'Enter'])
+ })
+})
+
+describe('trigger (current vs latest)', () => {
+ it('runs on upgrade, not on unchanged', async () => {
+ const src = new MockVersionSource({ codex: '0.156.0', claude: '2.1.279' })
+ const cur = {
+ currentVersion: async (e: AgentEngine) => (e === 'codex' ? '0.155.1' : '2.1.279'),
+ }
+ const plans = await planMatrixRuns(src, cur, ['codex', 'claude'])
+ const codex = plans.find((p) => p.engine === 'codex')!
+ const claude = plans.find((p) => p.engine === 'claude')!
+ expect(codex.plan).toBe(true)
+ expect(codex.reason).toBe('upgrade')
+ expect(claude.plan).toBe(false)
+ expect(claude.reason).toBe('unchanged')
+ })
+
+ it('compares dotted numeric versions', () => {
+ expect(compareVersions('0.155.0', '0.155.1')).toBe(-1)
+ expect(compareVersions('0.156.0', '0.155.1')).toBe(1)
+ expect(compareVersions('2.1.279', '2.1.279')).toBe(0)
+ })
+})
+
+describe('workspace folder name (what claude 2.1.274 tripped on)', () => {
+ it('keeps the session readable but drops every character a tool might rewrite', () => {
+ const session = sessionName({ engine: 'claude', version: '2.1.274', testcase: 'grid-switch', at: new Date('2026-09-24T20:22:39Z') })
+ expect(session).toContain('->') // the bundle keeps its readable name
+ const dir = workspaceDirName(session)
+ expect(dir).toBe('claude-2.1.274_to_grid-switch-none--20260924T202239Z')
+ expect(dir).toMatch(/^[A-Za-z0-9._-]+$/)
+ })
+})
+
+describe('session naming', () => {
+ it('is namespaced and grep-able', () => {
+ const name = sessionName({ engine: 'codex', version: '0.156.0', testcase: 'grid-switch', gridModel: 'grid:gpt-5-mini', at: new Date('2026-09-21T18:05:00Z') })
+ expect(name).toMatch(/^codex@0\.156\.0->grid-switch@grid-gpt-5-mini--/)
+ })
+})
+
+describe('matrix entry (dry run)', () => {
+ it('emits argv + notify when not logged in', async () => {
+ const plan = { engine: 'claude' as AgentEngine, current: '2.1.278', latest: '2.1.279', reason: 'upgrade' as const, plan: true }
+ const entry = await buildMatrixEntry(plan, {
+ status: async () => ({ binary: true, login: false, grid: true }),
+ grid: async () => null,
+ at: new Date('2026-09-21T18:05:00Z'),
+ })
+ expect(entry.session).toContain('[email protected]')
+ expect(entry.notify.join(' ')).toMatch(/NOT_LOGGED_IN/)
+ expect(entry.status.login).toBe(false)
+ expect(entry.subscriptionModel).toBe('claude-sonnet-5')
+ // Dry run: the journey is planned, nothing typed yet.
+ expect(entry.legs.map((l) => l.leg)).toEqual(['subscription', 'grid'])
+ expect(entry.legs.flatMap((l) => l.checks).every((c) => c.status === 'not-run')).toBe(true)
+ })
+
+ it('uses the fixed models for the subscription leg', async () => {
+ const src = new MockVersionSource({ codex: '0.156.0' })
+ const plans = await planMatrixRuns(src, { currentVersion: async () => '0.155.0' }, ['codex'])
+ const entry = await buildMatrixEntry(plans[0], {
+ status: async () => ({ binary: true, login: true, grid: true }),
+ grid: async () => null,
+ })
+ expect(entry.subscriptionModel).toBe('gpt-5.5')
+ })
+})
+
+describe('scenario (what a person does, on each side of the switch)', () => {
+ it('four plain requests on each side — read, write, edit, then MCP — with nothing shared', () => {
+ expect(LEGS).toEqual(['subscription', 'grid'])
+ // The own account calls the workspace's MCP server; the grid, the harness's web search.
+ expect(SCENARIO.subscription.map((c) => c.id)).toEqual(['read', 'write', 'edit', 'mcp'])
+ expect(SCENARIO.grid.map((c) => c.id)).toEqual(['read', 'write', 'edit', 'web'])
+ expect(SCENARIO.grid[3].sessionTool).toBe('mcp__harness__web_search')
+ // Nothing the first side leaves behind can pass the second: its own numbers and its own files.
+ const patterns = SMOKE_CHECKS.filter((c) => c.logPattern).map((c) => c.logPattern)
+ expect(new Set(patterns).size).toBe(patterns.length)
+ const files = SMOKE_CHECKS.map((c) => c.answerFromFile ?? c.file?.path).filter(Boolean)
+ expect(new Set(files).size).toBe(files.length)
+ expect(plannedLegs().flatMap((l) => l.checks).every((c) => c.status === 'not-run')).toBe(true)
+ })
+
+ it('every prompt is a plain request, and never contains its own proof', () => {
+ for (const c of SMOKE_CHECKS) {
+ expect(c.prompt).not.toMatch(/reply with exactly|<result>|RECALL_|TOOL_|MCP_/i)
+ if (c.logPattern) expect(c.prompt).not.toContain(c.logPattern) // "add 40 2" is asked; "= 42" is only the log's
+ expect(!!c.log || !!c.answerFromFile || !!c.file || !!c.sessionTool).toBe(true) // proven by the machine, never by the reply
+ // The result is what is tested, not the route: no request tells the engine which tool to use.
+ expect(c.prompt).not.toMatch(/\b(Bash|Read|Write|Edit) tool\b|apply_patch|in the shell/)
+ }
+ // And every engine gets the same words.
+ expect(scenarioFor('codex')).toEqual(scenarioFor('claude'))
+ })
+
+ it('firstStuck names the first leg/step that did not pass', () => {
+ const legs = plannedLegs()
+ legs[0].checks.forEach((c) => (c.status = 'ok'))
+ legs[1].checks[0].status = 'ok'
+ legs[1].checks[1].status = 'stuck'
+ expect(firstStuck(legs)).toEqual({ leg: 'grid', check: 'write', status: 'stuck' })
+ legs.forEach((l) => l.checks.forEach((c) => (c.status = 'ok')))
+ expect(firstStuck(legs)).toBeNull()
+ })
+})
+
+/** A pane that answers each typed prompt with `reply(prompt)` on the next capture. */
+function fakePane(reply: (prompt: string) => string | null): Tmux & { typed: string[] } {
+ let screen = '> '
+ let clock = 0
+ const typed: string[] = []
+ return {
+ typed,
+ async capture() {
+ return screen
+ },
+ async type(_pane, text) {
+ typed.push(text)
+ screen += `\n> ${text}`
+ const r = reply(text)
+ if (r !== null) screen += `\n${r}`
+ },
+ async key(_pane, key) {
+ typed.push(`<${key}>`)
+ // Any key answers a dialog: the dialog text leaves the screen.
+ screen = screen.replace(/Update available![^\n]*\n?/, '').replace(/Do you trust[^\n]*\n?/, '')
+ },
+ async sleep(ms) {
+ clock += ms
+ },
+ now: () => clock,
+ }
+}
+
+/** An MCP log that gains a line whenever the fake tool "uses" the server. */
+function fakeLogs() {
+ const lines: Record<'mcp', string[]> = { mcp: [] }
+ return { lines, readLog: (k: 'mcp') => [...lines[k]] }
+}
+
+/**
+ * A fake coding tool on a fake machine: it does what each request asks — reads the file, writes
+ * it, edits it, calls the MCP server (a log line), searches the web (a result in its
+ * session, as `toolResult` would read it). With `lie`, it only SAYS it did those steps, which is the
+ * case the proofs exist for; with `empty`, its web search comes back with nothing.
+ */
+function fakeAgent(opts: { lie?: CheckId[]; empty?: boolean } = {}) {
+ const files: Record<string, string> = {
+ 'notes/info-1.txt': 'kiwi-4821-tulip\n', 'notes/info-2.txt': 'otter-1234-ember\n',
+ 'notes/todo-1.txt': '# todo 1\nstatus: pending\n', 'notes/todo-2.txt': '# todo 2\nstatus: pending\n',
+ }
+ const logs = fakeLogs()
+ const searched: Record<string, string> = {}
+ const lie = new Set(opts.lie ?? [])
+ const reply = (p: string): string | null => {
+ let m: RegExpMatchArray | null
+ if ((m = p.match(/Read (notes\/info-\d\.txt)/))) return lie.has('read') ? 'It says hello.' : `It says: ${files[m[1]].trim()}`
+ if ((m = p.match(/Search the web .*\((ref-[0-9a-z]{6})\)/))) {
+ if (!lie.has('web')) searched[m[1]] = opts.empty ? 'No results.' : 'Node.js 24.11.0 — https://nodejs.org/en/download'
+ return 'The latest stable Node.js is 24.11.0.'
+ }
+ if ((m = p.match(/Create the file (\S+) with this text: (.*) \(ref-/))) {
+ if (!lie.has('write')) files[m[1]] = `${m[2]}\n`
+ return 'Done.'
+ }
+ if ((m = p.match(/In (\S+), change pending to done/))) {
+ if (!lie.has('edit')) files[m[1]] = files[m[1]].replace('pending', 'done')
+ return 'Updated.'
+ }
+ if ((m = p.match(/e2e_calc to add (\d+) and (\d+)/))) {
+ if (!lie.has('mcp')) logs.lines.mcp.push(`t add ${m[1]} ${m[2]} = ${+m[1] + +m[2]}`)
+ return `${+m[1] + +m[2]}`
+ }
+ return null
+ }
+ const pane = fakePane(reply)
+ const toolResult = (ref: string, tool: string) => {
+ const out = tool === 'mcp__harness__web_search' ? searched[ref] : undefined
+ return { called: out !== undefined, answered: !!out && out.includes('https://'), ...(out ? { output: out } : {}) }
+ }
+ const probe = { settleMs: 1, pollMs: 1, checkTimeoutMs: 5_000, readLog: logs.readLog, readFile: (f: string) => files[f] ?? null, toolResult }
+ return { pane, files, logs, probe }
+}
+
+const READ_STEP = SCENARIO.subscription[0]
+const MCP_STEP = SCENARIO.subscription[3]
+const WEB_STEP = SCENARIO.grid[3]
+
+describe('pane probe (live steps on a tmux pane)', () => {
+ it('a tool that really does the four requests passes both sides, and each proof is recorded', async () => {
+ const { pane, probe, files } = fakeAgent()
+ const own = await probeLeg(pane, '%1', 'subscription', probe)
+ const grid = await probeLeg(pane, '%1', 'grid', probe)
+ expect(own.checks.map((c) => `${c.id}=${c.status}`)).toEqual(['read=ok', 'write=ok', 'edit=ok', 'mcp=ok'])
+ expect(grid.checks.map((c) => `${c.id}=${c.status}`)).toEqual(['read=ok', 'write=ok', 'edit=ok', 'web=ok'])
+ expect(own.checks[3].logLines).toEqual(['t add 30 12 = 42'])
+ expect(files['out/hello-2.txt']).toBe('hello from step 2\n')
+ expect(files['notes/todo-1.txt']).toContain('status: done')
+ // Every step carries the tag its prompt was typed with — how its tools are found afterwards.
+ expect([...own.checks, ...grid.checks].every((c) => /^ref-[0-9a-z]{6}$/.test(c.ref ?? ''))).toBe(true)
+ })
+
+ it('saying is not doing: each step fails when the tool only claims it, with what was missing', async () => {
+ const notes: Record<CheckId, RegExp> = {
+ read: /notes\/info-1\.txt's token never appeared/,
+ write: /out\/hello-1\.txt does not exist/,
+ edit: /notes\/todo-1\.txt is "# todo 1\\nstatus: pending"/,
+ mcp: /no mcp log line for "add 30 12 = 42"/,
+ web: /mcp__harness__web_search was never called/,
+ }
+ for (const step of [...SCENARIO.subscription, WEB_STEP]) {
+ const { pane, probe } = fakeAgent({ lie: [step.id] })
+ const out = await runCheck(pane, '%1', step, probe)
+ expect(out.status, step.id).toBe('stuck')
+ expect(out.note, step.id).toMatch(notes[step.id])
+ }
+ })
+
+ it('web: a search that came back with no results is not a search that worked', async () => {
+ const { pane, probe } = fakeAgent({ empty: true })
+ const out = await runCheck(pane, '%1', WEB_STEP, probe)
+ expect(out.status).toBe('stuck')
+ expect(out.note).toBe('mcp__harness__web_search came back without results: "No results."')
+ })
+
+ it('read: the token must come after THIS question — one already on screen is no proof', async () => {
+ const { pane, probe } = fakeAgent({ lie: ['read'] })
+ await pane.type('%1', 'kiwi-4821-tulip') // on screen before the question was asked
+ expect((await runCheck(pane, '%1', READ_STEP, probe)).status).toBe('stuck')
+ })
+
+ it('write: the exact text, not merely a file', async () => {
+ const { pane, probe, files } = fakeAgent({ lie: ['write'] })
+ files['out/hello-1.txt'] = 'hello from step one\n'
+ const out = await runCheck(pane, '%1', SCENARIO.subscription[1], probe)
+ expect(out.status).toBe('stuck')
+ expect(out.note).toBe('out/hello-1.txt is "hello from step one"')
+ })
+
+ it('read with no info file in the workspace is the run\'s setup failing, said as such', async () => {
+ const { pane, probe } = fakeAgent()
+ const out = await runCheck(pane, '%1', READ_STEP, { ...probe, readFile: () => null })
+ expect(out.status).toBe('stuck')
+ expect(out.note).toMatch(/the run's setup, not the tool/)
+ expect(pane.typed).toHaveLength(0) // nothing typed for a step that cannot be proven
+ })
+
+ it('out of usage is recognised at once, with its reset time, instead of waiting out the step', async () => {
+ // What each tool really prints — codex and claude, from their own binaries.
+ const cases: Array<[string, string | null]> = [
+ ["■ You've hit your usage limit. Upgrade to Pro, or try again at Sep 24th, 2026 6:02 PM.", 'try again at Sep 24th, 2026 6:02 PM'],
+ ["You've hit your session limit · resets 6pm", 'resets 6pm'],
+ ["You’ve hit your weekly limit · resets Mon 9am", 'resets Mon 9am'],
+ ["You're out of usage credits. /model to switch models.", null],
+ ]
+ for (const [said, resets] of cases) {
+ const pane = fakePane(() => said)
+ const out = await runCheck(pane, '%1', MCP_STEP, { settleMs: 1, pollMs: 1, checkTimeoutMs: 120_000 })
+ expect(out.status, said).toBe('no-quota')
+ expect(out.resets ?? null, said).toBe(resets)
+ expect(out.elapsedMs!, said).toBeLessThan(10) // one poll, not the whole budget
+ }
+ // Claude's early warning is not the limit: the step carries on and passes.
+ const logs = fakeLogs()
+ const warned = fakePane(() => { logs.lines.mcp.push('t add 30 12 = 42'); return 'Approaching usage limit · resets 6pm\n42' })
+ expect((await runCheck(warned, '%1', MCP_STEP, { settleMs: 1, pollMs: 1, readLog: logs.readLog })).status).toBe('ok')
+ })
+
+ it('an old limit message already on screen does not count against a new request', async () => {
+ const { pane, probe } = fakeAgent()
+ await pane.type('%1', "You've hit your session limit · resets 6pm") // yesterday's, still in the scrollback
+ pane.typed.length = 0
+ expect((await runCheck(pane, '%1', MCP_STEP, probe)).status).toBe('ok')
+ })
+
+ it('out of usage ends the leg: nothing more is typed into an account that cannot answer', async () => {
+ const pane = fakePane(() => "You've hit your usage limit. try again at 6:02 PM.")
+ const leg = await probeLeg(pane, '%1', 'subscription', { settleMs: 1, pollMs: 1, readFile: () => 'kiwi-4821-tulip\n' })
+ expect(leg.checks.map((c) => `${c.id}=${c.status}`)).toEqual(['read=no-quota', 'write=not-run', 'edit=not-run', 'mcp=not-run'])
+ expect(pane.typed).toHaveLength(1)
+ expect(quotaHit([leg])).toEqual({ leg: 'subscription', check: 'read', resets: 'try again at 6:02 PM' })
+ })
+
+ it('answers codex 0.156\'s stacked startup screens — the one on top first, each once', async () => {
+ const { pane, probe } = fakeAgent()
+ // What the pane held on grid-dev: the hooks review, and the model notice drawn under it.
+ await pane.type('%1', 'Hooks need review\n 2 hooks are new or changed.\n› Review hooks\n Trust all and continue')
+ await pane.type('%1', 'GPT-5.5 retires on October 14, 2026. Switch to GPT-5.6 Sol to continue working in Codex.\n› 1. Try new model\n 2. Use existing model')
+ pane.typed.length = 0
+ const answered = await dismissStartupDialogs(pane, '%1', { settleMs: 1, pollMs: 1 })
+ expect(answered).toEqual(['codex-model-retire', 'codex-hooks-trust'])
+ expect(pane.typed).toEqual(['<2>', '<Enter>', '<Down>', '<Enter>'])
+ // And the step typed after them is not disturbed by their text still being in the scrollback.
+ pane.typed.length = 0
+ expect((await runCheck(pane, '%1', MCP_STEP, probe)).status).toBe('ok')
+ expect(pane.typed).toHaveLength(1)
+ })
+
+ it('types the request again when a tool that is still loading dropped it', async () => {
+ const { pane, probe } = fakeAgent()
+ // The first paste vanishes, the way claude's first question did on grid-dev; the second lands.
+ const realType = pane.type.bind(pane)
+ let dropped = false
+ pane.type = async (id, text) => { if (!dropped) { dropped = true; pane.typed.push(text); return } await realType(id, text) }
+ // The real per-step budget: the 5s spent seeing whether the first paste landed comes out of it.
+ const out = await runCheck(pane, '%1', MCP_STEP, { ...probe, pollMs: 100, checkTimeoutMs: 120_000 })
+ expect(out.status).toBe('ok')
+ expect(pane.typed).toHaveLength(2) // typed, not seen, typed again
+ expect(pane.typed[0]).toBe(pane.typed[1]) // the same request with the same tag
+ })
+
+ it('answers claude 2.1.281\'s reworded folder trust by moving off its "No, exit" default', async () => {
+ const pane = fakePane(() => null)
+ await pane.type('%1', 'Quick safety check: Is this a project you created or one you trust?\n ❯ No, exit\n Yes, I trust this folder')
+ pane.typed.length = 0
+ expect(await dismissStartupDialogs(pane, '%1', { settleMs: 1, pollMs: 1 })).toEqual(['claude-trust-v2'])
+ expect(pane.typed).toEqual(['<Down>', '<Enter>'])
+ })
+
+ it('a step that is not proven stops the leg there: nothing more is typed after it', async () => {
+ const { pane, probe } = fakeAgent({ lie: ['write'] })
+ const leg = await probeLeg(pane, '%1', 'subscription', { ...probe, checkTimeoutMs: 5 })
+ expect(leg.checks.map((c) => `${c.id}=${c.status}`)).toEqual(['read=ok', 'write=stuck', 'edit=not-run', 'mcp=not-run'])
+ expect(pane.typed).toHaveLength(2)
+ })
+
+ it('answers a startup dialog (codex update → Skip) before typing the first request, and records it', async () => {
+ const { pane, probe } = fakeAgent()
+ // What a fresh codex shows before its prompt; a request typed here would pick "1. Update now".
+ await pane.type('%1', 'Update available! 0.155.0 -> 0.155.1 › 1. Update now 2. Skip Press enter to continue')
+ pane.typed.length = 0
+ const leg = await probeLeg(pane, '%1', 'subscription', probe)
+ expect(pane.typed.slice(0, 2)).toEqual(['<2>', '<Enter>'])
+ expect(pane.typed[2]).toContain(READ_STEP.prompt)
+ expect(leg.dialogs).toEqual(['codex-update'])
+ expect(leg.checks.every((c) => c.status === 'ok')).toBe(true)
+ expect(await dismissStartupDialogs(pane, '%1', { settleMs: 1 })).toEqual([]) // nothing left on screen
+ })
+})
+
+describe('session tools and models (read from the engine\'s own session file)', () => {
+ // Records shaped exactly like the real ones — a codex 0.156.1 rollout and a claude 2.1.273
+ // session from grid-dev, 2026-09-24 — trimmed to the fields that matter.
+ it('codex: code-mode exec calls, classic calls, a namespaced MCP call, and each turn\'s model', () => {
+ const home = mkdtempSync(join(tmpdir(), 'wd-sess-'))
+ try {
+ const dir = join(home, '.codex', 'sessions', '2026', '09', '24')
+ mkdirSync(dir, { recursive: true })
+ const rec = (o: object) => JSON.stringify(o)
+ const user = (t: string) => rec({ type: 'response_item', payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: t }] } })
+ writeFileSync(join(dir, 'rollout-x.jsonl'), [
+ rec({ type: 'turn_context', payload: { model: 'gpt-6-luna' } }), // written BEFORE the turn's message
+ user('Run tools/calc.sh add 40 2 and tell me the result. (ref-aaaaaa)'),
+ rec({ type: 'response_item', payload: { type: 'custom_tool_call', name: 'exec', input: 'const r = await tools.exec_command({cmd:"tools/calc.sh add 40 2"})' } }),
+ rec({ type: 'turn_context', payload: { model: 'DeepSeek-V4-Flash-0731' } }),
+ user('Use the MCP server e2e_calc to add 60 and 18. (ref-bbbbbb)'),
+ rec({ type: 'response_item', payload: { type: 'function_call', name: 'sub', namespace: 'mcp__e2e_calc', arguments: '{}' } }),
+ rec({ type: 'response_item', payload: { type: 'function_call', name: 'apply_patch', arguments: '{}' } }),
+ ].join('\n'))
+ expect(useByRef('codex', '/unused', ['ref-aaaaaa', 'ref-bbbbbb'], home)).toEqual({
+ 'ref-aaaaaa': { tools: ['exec→exec_command'], models: ['gpt-6-luna'] },
+ 'ref-bbbbbb': { tools: ['mcp__e2e_calc__sub', 'apply_patch'], models: ['DeepSeek-V4-Flash-0731'] },
+ })
+ } finally {
+ rmSync(home, { recursive: true, force: true })
+ }
+ })
+
+ it('claude: tool_use names per step, ToolSearch included, and the model of every reply', () => {
+ const home = mkdtempSync(join(tmpdir(), 'wd-sess-'))
+ try {
+ const cwd = '/tmp/grid-matrix-out/agents/[email protected]>grid-switch@none--20260924T081321Z'
+ const dir = join(home, '.claude', 'projects', '-tmp-grid-matrix-out-agents-claude-2-1-273--grid-switch-none--20260924T081321Z')
+ mkdirSync(dir, { recursive: true })
+ const rec = (o: object) => JSON.stringify(o)
+ const reply = (model: string, ...tools: string[]) => rec({ type: 'assistant', message: { model, content: tools.map((name) => ({ type: 'tool_use', name, input: {} })) } })
+ writeFileSync(join(dir, 's.jsonl'), [
+ rec({ type: 'user', message: { role: 'user', content: 'Read notes/info-1.txt and tell me what it says. (ref-cccccc)' } }),
+ reply('claude-haiku-4-5-20251001', 'Read'),
+ rec({ type: 'user', message: { role: 'user', content: [{ type: 'tool_result', content: 'kiwi-4821-tulip' }] } }),
+ reply('claude-haiku-4-5-20251001'),
+ rec({ type: 'user', message: { role: 'user', content: 'Use the MCP server e2e_calc to add 30 and 12. (ref-dddddd)' } }),
+ reply('DeepSeek-V4-Flash-0731', 'ToolSearch'),
+ reply('DeepSeek-V4-Flash-0731', 'mcp__e2e_calc__add'),
+ reply('<synthetic>'),
+ ].join('\n'))
+ expect(useByRef('claude', cwd, ['ref-cccccc', 'ref-dddddd'], home)).toEqual({
+ 'ref-cccccc': { tools: ['Read'], models: ['claude-haiku-4-5-20251001'] },
+ 'ref-dddddd': { tools: ['ToolSearch', 'mcp__e2e_calc__add'], models: ['DeepSeek-V4-Flash-0731'] },
+ })
+ } finally {
+ rmSync(home, { recursive: true, force: true })
+ }
+ })
+
+ it('codex: the web search of THIS step, and whether it came back with results', () => {
+ const home = mkdtempSync(join(tmpdir(), 'wd-sess-'))
+ try {
+ const dir = join(home, '.codex', 'sessions', '2026', '10', '06')
+ mkdirSync(dir, { recursive: true })
+ const rec = (o: object) => JSON.stringify(o)
+ const item = (payload: object) => rec({ type: 'response_item', payload })
+ const user = (t: string) => item({ type: 'message', role: 'user', content: [{ type: 'input_text', text: t }] })
+ // The output shape codex 0.160.0 recorded for an MCP call on the grid (grid-dev, 2026-10-01).
+ const out = (callId: string, text: string) => item({ type: 'function_call_output', call_id: callId, output: [{ type: 'input_text', text: 'Wall time: 0.4 seconds\nOutput:' }, { type: 'input_text', text }] })
+ writeFileSync(join(dir, 'rollout-x.jsonl'), [
+ user('Delete the file notes/old-2.txt. (ref-aaaaaa)'),
+ user('<environment_context>a context line in the user role, mid-step</environment_context>'),
+ user('Search the web for the latest stable version of Node.js and tell me the version number. (ref-bbbbbb)'),
+ user('<environment_context>a context line in the user role, mid-step</environment_context>'),
+ item({ type: 'function_call', name: 'web_search', namespace: 'mcp__harness', arguments: '{"query":"node"}', call_id: 'c1' }),
+ out('c1', 'Node.js 24.11.0 (LTS) https://nodejs.org/en/download'),
+ user('Search again. (ref-cccccc)'),
+ item({ type: 'function_call', name: 'web_search', namespace: 'mcp__harness', arguments: '{}', call_id: 'c2' }),
+ out('c2', 'MCP tool call requires approval, which the policy denies'),
+ ].join('\n'))
+ expect(toolResult('codex', '/unused', 'ref-aaaaaa', 'mcp__harness__web_search', home)).toEqual({ called: false, answered: false })
+ expect(toolResult('codex', '/unused', 'ref-bbbbbb', 'mcp__harness__web_search', home)).toMatchObject({ called: true, answered: true })
+ // Called, refused: no results, and the refusal is what the probe quotes.
+ expect(toolResult('codex', '/unused', 'ref-cccccc', 'mcp__harness__web_search', home)).toEqual({
+ called: true, answered: false, output: 'Wall time: 0.4 seconds Output: MCP tool call requires approval, which the policy denies',
+ })
+ } finally {
+ rmSync(home, { recursive: true, force: true })
+ }
+ })
+
+ it('claude: the web search tool_use, its tool_result, and an error result', () => {
+ const home = mkdtempSync(join(tmpdir(), 'wd-sess-'))
+ try {
+ const cwd = '/tmp/ws'
+ const dir = join(home, '.claude', 'projects', '-tmp-ws')
+ mkdirSync(dir, { recursive: true })
+ const rec = (o: object) => JSON.stringify(o)
+ const prompt = (t: string) => rec({ type: 'user', message: { role: 'user', content: t } })
+ const call = (id: string) => rec({ type: 'assistant', message: { model: 'DeepSeek-V4-Flash-0731', content: [{ type: 'tool_use', id, name: 'mcp__harness__web_search', input: { query: 'node' } }] } })
+ const result = (id: string, text: string, isError = false) => rec({ type: 'user', message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: id, is_error: isError, content: [{ type: 'text', text }] }] } })
+ writeFileSync(join(dir, 's.jsonl'), [
+ prompt('Search the web for the latest stable version of Node.js and tell me the version number. (ref-dddddd)'),
+ call('t1'), result('t1', 'Node.js 24.11.0 https://nodejs.org'),
+ prompt('Search again. (ref-eeeeee)'),
+ call('t2'), result('t2', 'see https://example.com — upstream timed out', true),
+ ].join('\n'))
+ expect(toolResult('claude', cwd, 'ref-dddddd', 'mcp__harness__web_search', home)).toMatchObject({ called: true, answered: true })
+ expect(toolResult('claude', cwd, 'ref-eeeeee', 'mcp__harness__web_search', home)).toMatchObject({ called: true, answered: false })
+ } finally {
+ rmSync(home, { recursive: true, force: true })
+ }
+ })
+})
+
+describe('grid model pick (live)', () => {
+ it('takes the smallest model on any listed grid, own grid first, even when the top-level list is empty', () => {
+ const pick = pickGridModel({
+ gridName: 'mine',
+ models: [],
+ localModelEngines: null,
+ grids: [
+ { name: 'mine', own: true, models: [] },
+ { name: 'autonomous.ai', own: false, models: [
+ { id: 'DeepSeek-V4-Flash-0731', node: 'a' },
+ { id: 'Qwen/Qwen3.8-27B', node: 'b' },
+ { id: 'Qwen3.6-35B-A3B', node: 'c' },
+ { id: 'gemma-4-31B-it', node: 'd' },
+ ] },
+ ],
+ })
+ expect(pick).toEqual({ model: 'Qwen/Qwen3.8-27B', gridName: 'autonomous.ai' })
+ expect(pickGridModel({ gridName: null, models: [], grids: [], localModelEngines: null })).toBeNull()
+ })
+})
+
+describe('workspace (the person\'s project the agent is created in)', () => {
+ it('lays out the notes, the MCP config for the engine, and the log the MCP step is proven by', () => {
+ const cwd = mkdtempSync(join(tmpdir(), 'wd-ws-'))
+ try {
+ // A stand-in for the codex binary: it records the argv it was called with and answers
+ // `mcp list` the way a codex that accepted the registration would.
+ const codexBin = join(cwd, 'fake-codex')
+ const argvLog = join(cwd, 'fake-codex.log')
+ writeFileSync(codexBin, `#!/bin/sh\necho "$@" >> ${JSON.stringify(argvLog)}\n[ "$2" = list ] && echo "e2e_calc node /x/calc-mcp.mjs"\nexit 0\n`)
+ chmodSync(codexBin, 0o755)
+ // The tool approvals are appended to codex's config: a copy here, never the developer's own.
+ const codexConfig = join(cwd, 'codex-config.toml')
+ writeFileSync(codexConfig, 'model = "gpt-6-luna"\n')
+ const laid = prepareWorkspace(cwd, 'codex', { codexBin, codexConfig })
+ // The file steps' files: a fresh unguessable secret per side, a todo to edit.
+ const secrets = [1, 2].map((n) => readWorkspaceFile(cwd, `notes/info-${n}.txt`)?.trim() ?? '')
+ for (const t of secrets) expect(t).toMatch(/^[a-z]+-\d{4}-[a-z]+$/)
+ expect(secrets[0]).not.toBe(secrets[1])
+ expect(readWorkspaceFile(cwd, 'notes/todo-1.txt')).toContain('status: pending')
+ expect(existsSync(join(cwd, 'out'))).toBe(true)
+ expect(readWorkspaceFile(cwd, 'out/hello-1.txt')).toBeNull() // written by the step, never by us
+ // Fresh per run: two workspaces never share a secret.
+ const other = mkdtempSync(join(tmpdir(), 'wd-ws-'))
+ prepareWorkspace(other, 'claude')
+ expect(readWorkspaceFile(other, 'notes/info-1.txt')).not.toBe(readWorkspaceFile(cwd, 'notes/info-1.txt'))
+ rmSync(other, { recursive: true, force: true })
+ // The registration is codex's own command, with the server and its log as argv — and a stale
+ // entry is removed before the add, so an interrupted run cannot leave one behind.
+ const calls = readFileSync(argvLog, 'utf8').trim().split('\n')
+ expect(calls[0]).toBe('mcp remove e2e_calc')
+ expect(calls[1]).toBe(`mcp add e2e_calc -- node ${laid.mcpServer} ${logPath(cwd, 'mcp')}`)
+ expect(calls[2]).toBe('mcp list')
+ // The string shipped in the bundle is the file in workspace/, byte for byte.
+ expect(CALC_MCP_MJS).toBe(readFileSync(join(__dirname, 'workspace', 'calc-mcp.mjs'), 'utf8'))
+ expect(existsSync(join(cwd, 'tools', 'calc-mcp.mjs'))).toBe(true)
+ // codex has no project-level MCP config — a `<cwd>/.codex/config.toml` is never read (measured:
+ // `codex mcp list` inside such a workspace does not list the server), so the workspace must not
+ // pretend otherwise. Registration goes through `codex mcp add`, which this test does not run:
+ // it would write the developer's own ~/.codex/config.toml. What is pinned is that the run says
+ // out loud which of the two happened.
+ expect(existsSync(join(cwd, '.codex', 'config.toml'))).toBe(false)
+ expect(laid.mcpConfig).toBe(codexConfig)
+ expect(laid.mcpNote).toMatch(/codex mcp add/)
+ // Its tools always allowed, as on a developer's machine: codex then never sends them to its
+ // reviewer, whose `codex-auto-review` model the grid does not serve (openai/codex#24879).
+ const toml = readFileSync(codexConfig, 'utf8')
+ expect(toml.startsWith('model = "gpt-6-luna"\n')).toBe(true) // appended, nothing rewritten
+ for (const tool of ['add', 'sub']) expect(toml).toContain(`[mcp_servers.e2e_calc.tools.${tool}]\napproval_mode = "approve"\n`)
+ // The MCP server answers tools/call and leaves its line in the MCP log.
+ const rpc = ['{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}', '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"sub","arguments":{"a":60,"b":18}}}'].join('\n') + '\n'
+ const out = execFileSync('node', [laid.mcpServer, logPath(cwd, 'mcp')], { input: rpc }).toString()
+ expect(out).toContain('"text":"42"')
+ expect(readLog(cwd, 'mcp').at(-1)).toContain('sub 60 18 = 42')
+ // claude gets .mcp.json instead.
+ const claude = prepareWorkspace(cwd, 'claude')
+ expect(JSON.parse(readFileSync(join(cwd, '.mcp.json'), 'utf8')).mcpServers.e2e_calc.command).toBe('node')
+ expect(claude.mcpConfig).toBe(join(cwd, '.mcp.json'))
+ // …and the project settings a person who said "don't ask again" has: the steps' tools allowed.
+ const settings = JSON.parse(readFileSync(join(cwd, '.claude', 'settings.json'), 'utf8'))
+ expect(settings.permissions.allow).toEqual(expect.arrayContaining(['Write', 'Edit', 'mcp__e2e_calc', 'mcp__harness']))
+ expect(settings.enabledMcpjsonServers).toEqual(['e2e_calc'])
+ } finally {
+ rmSync(cwd, { recursive: true, force: true })
+ }
+ })
+})
diff --git a/cli/src/e2e/fixedModels.ts b/cli/src/e2e/fixedModels.ts
new file mode 100644
index 000000000..c2a714510
--- /dev/null
+++ b/cli/src/e2e/fixedModels.ts
@@ -0,0 +1,13 @@
+/**
+ * Fixed models for the real subscription round-trip leg of the matrix (per user decision).
+ *
+ * The grid leg picks a model from `grid_models_list`; the subscription leg (codex/claude answering
+ * on its OWN login) uses a fixed, cheap alias that is enabled in the tmux launch. codex and claude
+ * expose no model-list CLI, so these are explicit and tuneable in one place.
+ */
+import type { AgentEngine } from '../engines/types.js'
+
+export const SUBSCRIPTION_MODEL: Partial<Record<AgentEngine, string>> = {
+ codex: 'gpt-5.5',
+ claude: 'claude-sonnet-5',
+}
diff --git a/cli/src/e2e/gridArgv.ts b/cli/src/e2e/gridArgv.ts
new file mode 100644
index 000000000..92fe3fe1f
--- /dev/null
+++ b/cli/src/e2e/gridArgv.ts
@@ -0,0 +1,85 @@
+/**
+ * argv-package — the deterministic spawn spec the real-matrix uses to reproduce EXACTLY what the
+ * harness enters when an agent switches to a grid and back to its own login.
+ *
+ * It does NOT re-implement the launch. It reuses the harness's own builders
+ * (`buildGridEngineLaunch` for the grid env/argv, `buildEngineCommandArgv` for the final command,
+ * `gridEnvVarNames` for what to clear on the way home), so the matrix spawns the same argv the
+ * daemon would, field for field — no drift, no self-rolled copy.
+ */
+import { type GridLaunchOverride, buildGridEngineLaunch, gridEnvVarNames } from '../lib/gridLaunch.js'
+import { buildEngineCommandArgv } from '../lib/engineLaunch.js'
+import type { AgentEngine } from '../engines/types.js'
+
+/** A config file the launch writes into a private dir the engine is pointed at. */
+export interface SpawnConfigFile {
+ name: string
+ content: string
+}
+
+/** Resolved spawn for one transition: what the daemon would hand `tmux respawn-pane`. */
+export interface SpawnSpec {
+ /** Full engine argv, `command[0]` is the engine binary (matches `buildEngineCommandArgv`). */
+ command: string[]
+ /** Environment layered over the engine's inherited env (where the grid key goes). */
+ env: Record<string, string>
+ /** Env vars that must be cleared so a stale grid/vendor credential is not inherited. */
+ clearEnv: string[]
+ /** A private config dir the engine reads its provider from (e.g. opencode), if any. */
+ configDir?: {
+ /** The variable the engine reads it by; absent for one handed a flag instead (Pi's `-e`, [arg]). */
+ envVar?: string
+ arg?: string
+ pointAt?: string
+ files: SpawnConfigFile[]
+ }
+}
+
+export interface GridSpawnError {
+ ok: false
+ error: string
+ detail: string
+}
+
+function refuse(error: string, detail: string): GridSpawnError {
+ return { ok: false, error, detail }
+}
+
+/** Spawn spec to move `engine` onto `grid` (optionally resuming `resumeId`). */
+export function gridSpawn(
+ engine: AgentEngine,
+ grid: GridLaunchOverride,
+ resumeId?: string,
+): SpawnSpec | GridSpawnError {
+ const built = buildGridEngineLaunch(engine, grid, { hermesSystemManaged: false })
+ if (!built.ok) {
+ return { ok: built.ok, error: built.error, detail: built.detail }
+ }
+ const command = buildEngineCommandArgv(engine, {
+ ...(resumeId ? { resumeSessionId: resumeId } : {}),
+ extraArgs: built.launch.args,
+ })
+ return {
+ command,
+ env: built.launch.env,
+ clearEnv: gridEnvVarNames(engine),
+ ...(built.launch.configDir
+ ? {
+ configDir: {
+ ...(built.launch.configDir.envVar ? { envVar: built.launch.configDir.envVar } : {}),
+ ...(built.launch.configDir.arg ? { arg: built.launch.configDir.arg } : {}),
+ ...(built.launch.configDir.pointAt ? { pointAt: built.launch.configDir.pointAt } : {}),
+ files: built.launch.configDir.files.map((f) => ({ name: f.name, content: f.content })),
+ },
+ }
+ : {}),
+ }
+}
+
+/** Spawn spec to return `engine` to its own vendor login (clear the grid, nothing else set). */
+export function backHomeSpawn(engine: AgentEngine, resumeId?: string): SpawnSpec {
+ const command = buildEngineCommandArgv(engine, {
+ ...(resumeId ? { resumeSessionId: resumeId } : {}),
+ })
+ return { command, env: {}, clearEnv: gridEnvVarNames(engine) }
+}
diff --git a/cli/src/e2e/gridSwitchDriver.spec.ts b/cli/src/e2e/gridSwitchDriver.spec.ts
new file mode 100644
index 000000000..b25506e4c
--- /dev/null
+++ b/cli/src/e2e/gridSwitchDriver.spec.ts
@@ -0,0 +1,189 @@
+import http from 'node:http'
+import type { AddressInfo } from 'node:net'
+import { afterEach, describe, expect, it } from 'vitest'
+import type { Frame, LocalClientSink } from '../backendSocket.js'
+import { attachLocalWsServer, type LocalWsBackend, type LocalWsServer } from '../localWsServer.js'
+import { GridSwitchDriver, LOCAL_WS_PATH, runGridSwitchFlow } from './gridSwitchDriver.js'
+
+const machineId = 'machine-abc'
+const agentId = 'session-1'
+const GRID_MODEL = 'grid:gpt-5-mini' // a model id the daemon's grid_models_list would report
+
+/**
+ * A local backend that answers the SAME RPCs the real daemon answers, using the SAME frame
+ * convention (`{ type, payload: { requestId, ...} }`). This is the seam the driver talks to —
+ * in a live run it is the actual BackendSocket in the daemon; here it stands in so the test
+ * is offline and costs $0, while still exercising the real WebSocket transport.
+ */
+class FakeDaemon implements LocalWsBackend {
+ sink: LocalClientSink | null = null
+ connId: string | null = null
+ received: Array<{ type: string; payload: Record<string, unknown> }> = []
+ retargetCalls: Array<{ clear: boolean; model?: string; gridName?: string }> = []
+
+ registerLocalClient(connId: string, sink: LocalClientSink): boolean {
+ this.connId = connId
+ this.sink = sink
+ return true
+ }
+
+ async unregisterLocalClient(connId: string): Promise<void> {
+ this.received.push({ type: '__unregister', payload: { connId } })
+ }
+
+ send(type: string, payload: Record<string, unknown>): void {
+ this.sink?.sendFrame({ type, payload } as Frame)
+ }
+
+ handleLocalFrame(_connId: string, frame: Frame): void {
+ const type = frame.type as string
+ const payload = (frame.payload as Record<string, unknown>) ?? {}
+ this.received.push({ type, payload })
+ const requestId = payload.requestId as string | undefined
+ switch (type) {
+ case 'machine_select': {
+ this.send('connected', {
+ requestId,
+ machineId,
+ transport: 'local',
+ e2ee: false,
+ } as unknown as Record<string, unknown>)
+ break
+ }
+ case 'grid_models_list': {
+ this.send('grid_models_list', {
+ requestId,
+ gridName: 'someone-7f3a91c4',
+ models: [{ id: GRID_MODEL, node: 'node-x' }],
+ grids: [
+ {
+ name: 'shared-lab',
+ own: false,
+ models: [{ id: 'shared:opus', node: 'node-y' }],
+ },
+ ],
+ localModelEngines: ['codex', 'claude'],
+ gridCli: { version: '1.0.0' },
+ } as unknown as Record<string, unknown>)
+ break
+ }
+ case 'agent_retarget': {
+ const clear = payload.clearGrid === true
+ const model = typeof payload.gridModel === 'string' ? payload.gridModel : undefined
+ const gridName = typeof payload.gridName === 'string' ? payload.gridName : undefined
+ this.retargetCalls.push({ clear, model, gridName })
+ // Absent gridModel + clear==false is an invalid frame, exactly as the daemon refuses it.
+ if (!clear && !model) {
+ this.send('agent_retarget', { requestId, error: 'INVALID_GRID', detail: 'grid is required' })
+ break
+ }
+ this.send('agent_retarget', { requestId, retargeted: true })
+ break
+ }
+ default:
+ this.send(type, { requestId, error: 'UNSUPPORTED' })
+ }
+ }
+
+ async handleLocalBinary(): Promise<void> {}
+}
+
+describe('GridSwitchDriver (talks to the daemon like the app)', () => {
+ let server: http.Server | null = null
+ let local: LocalWsServer | null = null
+ let daemon: FakeDaemon | null = null
+
+ afterEach(async () => {
+ await local?.close()
+ await new Promise<void>((resolve) => server?.close(() => resolve()) ?? resolve())
+ local = null
+ server = null
+ daemon = null
+ })
+
+ async function startDaemon(localCredential?: string): Promise<string> {
+ daemon = new FakeDaemon()
+ server = http.createServer((_req, res) => {
+ res.statusCode = 404
+ res.end()
+ })
+ local = attachLocalWsServer(server, { machineId, backend: daemon, ...(localCredential ? { localCredential } : {}) })
+ await new Promise<void>((resolve) => server!.listen(0, '127.0.0.1', resolve))
+ const port = (server.address() as AddressInfo).port
+ return `ws://127.0.0.1:${port}${LOCAL_WS_PATH}`
+ }
+
+ it('performs the full app journey and records every value/step', async () => {
+ const url = await startDaemon()
+ const outcome = await runGridSwitchFlow(url, { machineId, agentId })
+
+ // 1) Handshake completes over the real JSON frame.
+ expect(outcome.connected).toMatchObject({ machineId, transport: 'local', e2ee: false })
+
+ // 2) grid_models_list returns the picker values the app would render.
+ expect(outcome.models.gridName).toBe('someone-7f3a91c4')
+ expect(outcome.models.models[0].id).toBe(GRID_MODEL)
+ expect(outcome.models.grids[0].name).toBe('shared-lab')
+ expect(outcome.models.localModelEngines?.has('codex')).toBe(true)
+
+ // 3) Default pick = first model; moving onto the grid succeeds.
+ expect(outcome.ontoGrid.ok).toBe(true)
+ expect(outcome.ontoGrid.retargeted).toBe(true)
+
+ // 4) Coming back to own login succeeds.
+ expect(outcome.backHome.ok).toBe(true)
+ expect(outcome.backHome.retargeted).toBe(true)
+
+ // 5) The daemon saw the EXACT payloads the desktop app sends.
+ const retarget = daemon!.received.filter((r) => r.type === 'agent_retarget')
+ expect(retarget).toHaveLength(2)
+ expect(retarget[0].payload).toMatchObject({ agentId, gridModel: GRID_MODEL }) // onto grid
+ expect(retarget[1].payload).toMatchObject({ agentId, clearGrid: true }) // back home
+ expect(retarget[1].payload.gridModel).toBeUndefined()
+
+ // 6) The trace is the human-readable step/value log asked for.
+ const steps = outcome.trace.map((s) => s.step)
+ expect(steps).toEqual(['machine_select', 'grid_models_list', 'agent_retarget(grid)', 'agent_retarget(clear)'])
+ })
+
+ it('replays an explicit grid name/model exactly as the app sends them', async () => {
+ const url = await startDaemon()
+ const driver = new GridSwitchDriver(url, machineId)
+ try {
+ await driver.machineSelect()
+ await driver.gridModels()
+ const res = await driver.retargetToGrid(agentId, 'shared:opus', 'shared-lab')
+ expect(res.ok).toBe(true)
+ const call = daemon!.retargetCalls.find((c) => !c.clear)!
+ expect(call).toMatchObject({ model: 'shared:opus', gridName: 'shared-lab' })
+ } finally {
+ driver.close()
+ }
+ })
+
+ it('surfaces a daemon refusal as an error result with detail', async () => {
+ const url = await startDaemon()
+ const driver = new GridSwitchDriver(url, machineId)
+ try {
+ await driver.machineSelect()
+ // No gridModel, no clearGrid -> daemon refuses with INVALID_GRID.
+ const res = await driver.retargetToGrid(agentId, '', undefined)
+ expect(res.ok).toBe(false)
+ expect(res.error).toBe('INVALID_GRID')
+ expect(res.detail).toBe('grid is required')
+ } finally {
+ driver.close()
+ }
+ })
+
+ // A daemon since guest machine sharing (bc7f56ff9, CLI 0.4.21) takes a TCP upgrade only with its local
+ // credential: grid-dev's run would have been refused (401) right after reading the machine.
+ it('connects to a daemon that requires the local credential, with the headers it is given', async () => {
+ const credential = 'k'.repeat(43)
+ const url = await startDaemon(credential)
+ await expect(runGridSwitchFlow(url, { machineId, agentId })).rejects.toThrow(/401/)
+ const outcome = await runGridSwitchFlow(url, { machineId, agentId, headers: { 'X-Harness-Local': credential } })
+ expect(outcome.connected).toMatchObject({ machineId, transport: 'local' })
+ expect(outcome.backHome.ok).toBe(true)
+ })
+})
diff --git a/cli/src/e2e/gridSwitchDriver.ts b/cli/src/e2e/gridSwitchDriver.ts
new file mode 100644
index 000000000..8f7e0b001
--- /dev/null
+++ b/cli/src/e2e/gridSwitchDriver.ts
@@ -0,0 +1,332 @@
+/**
+ * GridSwitchDriver — an E2E driver that talks to the harness daemon the SAME way the desktop
+ * app does, so it can be used to reproduce the real value / step flow of switching an agent
+ * between a private grid and its own vendor login, without a human clicking the UI.
+ *
+ * Transport: a plain WebSocket to the daemon's local socket (`<origin>/api/local-ws`, the
+ * `LOCAL_WS_PATH` the app connects to via `ws_conn.request`). The protocol is:
+ * client -> { type: 'machine_select', payload: { machineId, localProtocolVersion: 1 } }
+ * client <- { type: 'connected', payload: { machineId, transport: 'local', e2ee: false } }
+ * client -> { type: <cmd>, payload: { ...cmd args, requestId } }
+ * client <- { type: <cmd>, payload: { requestId, ...result } } (errors carry { error, detail? })
+ *
+ * The commands replayed here are the exact ones the app sends (`desktop/lib/state/app_state.dart`):
+ * - `grid_models_list` -> models/grids/gridName for the pane header picker
+ * - `agent_retarget` (grid) -> { agentId, gridModel, gridName? } move ONTO a grid
+ * - `agent_retarget` (clearGrid) -> { agentId, clearGrid: true } move back to own login
+ *
+ * Every reply is threaded back by matching `requestId`, and each exchange is recorded into a
+ * `GridSwitchTrace` — the step/value log that answers "khi tôi thao tác nó làm gì, value nào".
+ */
+import { WebSocket } from 'ws'
+
+export const LOCAL_WS_PATH = '/api/local-ws'
+export const LOCAL_WS_PROTOCOL_VERSION = 1
+
+/** One recorded exchange in the trace. */
+export interface GridSwitchTraceStep {
+ step: string
+ /** The frame the driver (acting as the app) sent. */
+ request: Record<string, unknown>
+ /** The daemon's reply, already decoded. */
+ reply: Record<string, unknown>
+}
+
+export interface GridModel {
+ id: string
+ node: string
+ grid?: string
+}
+
+export interface GridSection {
+ name: string
+ own: boolean
+ models: GridModel[]
+}
+
+export interface GridModelsAnswer {
+ gridName: string | null
+ models: GridModel[]
+ grids: GridSection[]
+ localModelEngines: Set<string> | null
+}
+
+/** The full grid object `agent_retarget` accepts in place of a picker model id. */
+export interface GridOverride {
+ networkId: string
+ networkName: string
+ baseUrl: string
+ apiKey: string
+}
+
+export interface GridSwitchResult {
+ ok: boolean
+ retargeted: boolean
+ error?: string
+ detail?: string
+}
+
+/** A tiny request/reply WS client matching the backend frame convention. */
+export class LocalWsClient {
+ private ws: WebSocket | null = null
+ private pending = new Map<string, { resolve: (v: Record<string, unknown>) => void; reject: (e: Error) => void }>()
+ private idSeq = 0
+ /** Events the daemon pushes without a matching request (e.g. `connected`, `announce`). */
+ private pushes: ((frame: Record<string, unknown>) => void) | null = null
+
+ onPush(cb: ((frame: Record<string, unknown>) => void) | null): void {
+ this.pushes = cb
+ }
+
+ /** [headers]: the daemon's local credential (lib/localCredential.ts), which its TCP upgrade requires. */
+ connect(url: string, headers: Record<string, string> = {}): Promise<void> {
+ return new Promise((resolve, reject) => {
+ const ws = new WebSocket(url, { headers })
+ this.ws = ws
+ ws.on('open', () => resolve())
+ ws.on('error', (err) => reject(err))
+ ws.on('message', (raw) => {
+ let frame: Record<string, unknown>
+ try {
+ frame = JSON.parse(raw.toString()) as Record<string, unknown>
+ } catch {
+ return
+ }
+ this.dispatch(frame)
+ })
+ ws.on('close', () => {
+ for (const [, { reject }] of this.pending) reject(new Error('socket closed'))
+ this.pending.clear()
+ })
+ })
+ }
+
+ private dispatch(frame: Record<string, unknown>): void {
+ const payload = (frame.payload ?? {}) as Record<string, unknown>
+ const requestId = typeof payload.requestId === 'string' ? payload.requestId : undefined
+ if (requestId !== undefined && this.pending.has(requestId)) {
+ const entry = this.pending.get(requestId)!
+ this.pending.delete(requestId)
+ entry.resolve(payload)
+ return
+ }
+ if (this.pushes) this.pushes(frame)
+ }
+
+ /** Send a command with a fresh requestId and await the matching reply. */
+ request(type: string, payload: Record<string, unknown>): Promise<Record<string, unknown>> {
+ const requestId = `req-${++this.idSeq}`
+ const full = { ...payload, requestId }
+ return new Promise((resolve, reject) => {
+ if (!this.ws || this.ws.readyState !== WebSocket.OPEN) {
+ reject(new Error('socket not open'))
+ return
+ }
+ this.pending.set(requestId, { resolve, reject })
+ this.ws.send(JSON.stringify({ type, payload: full }))
+ })
+ }
+
+ /**
+ * Send a command whose ack is delivered as an UNREQUESTED push rather than a requestId reply
+ * (`machine_select` -> the server's own `connected` frame). Resolves with the pushed payload.
+ */
+ sendAndAwaitPush(type: string, payload: Record<string, unknown>, awaitType: string): Promise<Record<string, unknown>> {
+ return new Promise((resolve, reject) => {
+ if (!this.ws || this.ws.readyState !== WebSocket.OPEN) {
+ reject(new Error('socket not open'))
+ return
+ }
+ const timer = setTimeout(() => reject(new Error(`timeout waiting for ${awaitType}`)), 8000)
+ const prev = this.pushes
+ this.pushes = (frame) => {
+ if (frame.type === awaitType) {
+ clearTimeout(timer)
+ this.pushes = prev
+ const p = (frame.payload ?? {}) as Record<string, unknown>
+ resolve(p)
+ }
+ prev?.(frame)
+ }
+ this.ws.send(JSON.stringify({ type, payload }))
+ })
+ }
+
+ close(): void {
+ try {
+ this.ws?.close()
+ } catch {
+ /* ignore */
+ }
+ this.ws = null
+ }
+}
+
+/**
+ * The driver. `daemonUrl` is the local socket the real daemon listens on, e.g.
+ * `ws://127.0.0.1:${hookPort}/api/local-ws`. Point it at any running daemon, or at a
+ * spec's in-process server.
+ */
+export class GridSwitchDriver {
+ readonly trace: GridSwitchTraceStep[] = []
+ private client = new LocalWsClient()
+
+ /** [headers] go on the socket's upgrade: a real daemon's local credential (harnessLocal.ts `localHeaders`). */
+ constructor(private readonly daemonUrl: string, private readonly machineId: string, private readonly headers: Record<string, string> = {}) {}
+
+ private record(step: string, request: Record<string, unknown>, reply: Record<string, unknown>): void {
+ this.trace.push({ step, request, reply })
+ }
+
+ /** 1. Open the socket + select this machine. The ack is the server's `connected` push frame. */
+ async machineSelect(): Promise<Record<string, unknown>> {
+ await this.client.connect(this.daemonUrl, this.headers)
+ const connected = await this.client.sendAndAwaitPush(
+ 'machine_select',
+ { machineId: this.machineId, localProtocolVersion: LOCAL_WS_PROTOCOL_VERSION },
+ 'connected',
+ )
+ this.record('machine_select', { machineId: this.machineId }, connected)
+ return connected
+ }
+
+ /** 2. Ask the daemon for the models its grid serves (the pane header picker). */
+ async gridModels(): Promise<GridModelsAnswer> {
+ const reply = await this.client.request('grid_models_list', {})
+ this.record('grid_models_list', {}, reply)
+ const models = ((reply.models as unknown) ?? []) as GridModel[]
+ const grids = (((reply.grids as unknown) ?? []) as Array<Record<string, unknown>>).map((g) => ({
+ name: g.name as string,
+ own: g.own === true,
+ models: ((g.models as unknown) ?? []) as GridModel[],
+ }))
+ const localEngines = reply.localModelEngines as unknown
+ return {
+ gridName: (reply.gridName as string) ?? null,
+ models,
+ grids,
+ localModelEngines: Array.isArray(localEngines)
+ ? new Set((localEngines as string[]).map((e) => e.toLowerCase()))
+ : null,
+ }
+ }
+
+ /**
+ * 3. The app's picker choice: move the agent ONTO a grid model. With `override`, the full grid
+ * object is sent instead (the daemon accepts both): a relay the daemon's own `grid` sign-in does
+ * not know — a local relay checkout under test — with the key recorded out of the trace.
+ */
+ async retargetToGrid(agentId: string, gridModel: string, gridName?: string, override?: GridOverride): Promise<GridSwitchResult> {
+ const payload: Record<string, unknown> = override ? { agentId, grid: { ...override, model: gridModel } } : { agentId, gridModel }
+ if (gridName !== undefined && !override) payload.gridName = gridName
+ const reply = await this.client.request('agent_retarget', payload)
+ const recorded = override ? { ...payload, grid: { ...(payload.grid as object), apiKey: '<redacted>' } } : payload
+ this.record('agent_retarget(grid)', recorded, reply)
+ return normalizeResult(reply)
+ }
+
+ /** 4. The "put it back" action: return the agent to its own vendor login. */
+ async clearGrid(agentId: string): Promise<GridSwitchResult> {
+ const payload = { agentId, clearGrid: true }
+ const reply = await this.client.request('agent_retarget', payload)
+ this.record('agent_retarget(clear)', payload, reply)
+ return normalizeResult(reply)
+ }
+
+ /**
+ * 0. New Harness: create a fresh agent for this run, the way the app's "New agent" does
+ * (`agent_create` with an engine and an absolute cwd). The test never borrows someone's pane.
+ */
+ async createAgent(engine: string, cwd: string, name: string, permissionMode = 'auto'): Promise<{ ok: true; agentId: string; pane: string | null } | { ok: false; error: string; detail?: string }> {
+ // `auto`, the app's default, so the agent is launched exactly as a person's is (see
+ // runGridSwitchTrace for what `full` cost). The daemon validates the mode per engine
+ // (INVALID_PERMISSION_MODE), so a wrong one fails loudly here instead of quietly launching the
+ // engine in some other mode.
+ const payload = { engine, cwd, name, permissionMode }
+ const reply = await this.client.request('agent_create', payload)
+ this.record('agent_create', payload, reply)
+ if (reply.error) return { ok: false, error: reply.error as string, detail: reply.detail as string | undefined }
+ const agent = reply.agent as { id?: string; tmuxPane?: string } | undefined
+ if (!agent?.id) return { ok: false, error: 'NO_AGENT_ID', detail: 'agent_create replied without agent.id' }
+ // The row already names its pane (`tmuxPane: "%N"`); the registry is only a fallback.
+ const pane = typeof agent.tmuxPane === 'string' && /^%\d+$/.test(agent.tmuxPane) ? agent.tmuxPane : null
+ return { ok: true, agentId: agent.id, pane }
+ }
+
+ /** 5. Clean up: delete the agent this run created (its pane goes with it). */
+ async deleteAgent(agentId: string): Promise<boolean> {
+ const payload = { agentId }
+ const reply = await this.client.request('agent_delete', payload)
+ this.record('agent_delete', payload, reply)
+ return reply.deleted === true
+ }
+
+ close(): void {
+ this.client.close()
+ }
+}
+
+function normalizeResult(payload: Record<string, unknown>): GridSwitchResult {
+ if (payload.error) {
+ return { ok: false, retargeted: false, error: payload.error as string, detail: payload.detail as string | undefined }
+ }
+ return { ok: true, retargeted: payload.retargeted === true }
+}
+
+/**
+ * Which grid model to move onto: the SMALLEST one served on any grid the daemon lists (own grid
+ * first), so the real switch costs as little as possible. `models` at the top level is only the own
+ * grid's list and is often empty — the models people actually share live under `grids[]`.
+ */
+export function pickGridModel(answer: GridModelsAnswer): { model: string; gridName: string } | null {
+ // The own grid first: the top-level `models` (its list, under `gridName`), then every section
+ // flagged `own`, then the rest. Ties on size keep the earlier one.
+ const own: GridSection[] = answer.gridName ? [{ name: answer.gridName, own: true, models: answer.models }] : []
+ const sections = [...own, ...answer.grids.filter((g) => g.own), ...answer.grids.filter((g) => !g.own)]
+ let best: { model: string; gridName: string; size: number } | null = null
+ for (const g of sections) {
+ for (const m of g.models) {
+ const size = paramsB(m.id)
+ if (!best || size < best.size) best = { model: m.id, gridName: g.name, size }
+ }
+ }
+ return best ? { model: best.model, gridName: best.gridName } : null
+}
+
+/** Parameter count in billions parsed from a model id (`Qwen3.8-27B` → 27, `…-35B-A3B` → 35); unknown → Infinity. */
+function paramsB(id: string): number {
+ const m = /(\d+(?:\.\d+)?)B(?![A-Za-z])/i.exec(id)
+ return m ? Number(m[1]) : Number.POSITIVE_INFINITY
+}
+
+/** Run the whole flow in one go and hand back the trace + results. */
+export interface GridFlowOutcome {
+ connected: Record<string, unknown>
+ models: GridModelsAnswer
+ ontoGrid: GridSwitchResult
+ backHome: GridSwitchResult
+ trace: GridSwitchTraceStep[]
+}
+
+/**
+ * Replay the full app journey against a daemon URL:
+ * machine_select -> grid_models_list -> agent_retarget(onto grid) -> agent_retarget(clear)
+ * `agentId` and `machineId` are whatever agent/session the caller wants to move.
+ */
+export async function runGridSwitchFlow(
+ daemonUrl: string,
+ opts: { machineId: string; agentId: string; gridModel?: string; gridName?: string; headers?: Record<string, string> },
+): Promise<GridFlowOutcome> {
+ const driver = new GridSwitchDriver(daemonUrl, opts.machineId, opts.headers)
+ try {
+ const connected = await driver.machineSelect()
+ const models = await driver.gridModels()
+ const picked = pickGridModel(models)
+ const pick = opts.gridModel ?? picked?.model ?? models.models[0]?.id
+ const ontoGrid = await driver.retargetToGrid(opts.agentId, pick, opts.gridName ?? (opts.gridModel ? undefined : picked?.gridName))
+ const backHome = await driver.clearGrid(opts.agentId)
+ return { connected, models, ontoGrid, backHome, trace: driver.trace }
+ } finally {
+ driver.close()
+ }
+}
diff --git a/cli/src/e2e/harnessLocal.spec.ts b/cli/src/e2e/harnessLocal.spec.ts
new file mode 100644
index 000000000..23c42c6e1
--- /dev/null
+++ b/cli/src/e2e/harnessLocal.spec.ts
@@ -0,0 +1,43 @@
+import { createServer, type Server } from 'node:http'
+import type { AddressInfo } from 'node:net'
+import { afterEach, describe, expect, it } from 'vitest'
+import { startHookServer, type HookServerHandlers } from '../hookServer.js'
+import { machineId } from './harnessLocal.js'
+
+// The e2e reads the daemon it drives through the daemon's own local API, which since guest machine sharing
+// (bc7f56ff9) answers the full /api/status only to a caller that shows the local credential. grid-dev's run
+// stopped at "daemon /api/status has no machineId" the day its daemon updated to 0.4.21 (2026-10-19), because
+// the e2e asked without it. Against the real server, so a change to that gate fails here, not on grid-dev.
+let server: Server | null = null
+afterEach(async () => {
+ delete process.env.HARNESS_PORT
+ delete process.env.HARNESS_MACHINE_ID
+ await new Promise<void>((done) => (server ? server.close(() => done()) : done()))
+ server = null
+})
+
+async function daemon(): Promise<void> {
+ // A pinned id skips the daemon altogether: this test would then pass without asking it.
+ delete process.env.HARNESS_MACHINE_ID
+ const started = await startHookServer(0, {
+ onStatus: () => ({ version: '9.9.9', pid: 1, computerId: 'c1', machineId: 'm1', signedIn: true, localWs: { path: '/api/local-ws' }, localSocket: null }),
+ } as Partial<HookServerHandlers> as HookServerHandlers)
+ server = started.server
+ process.env.HARNESS_PORT = String(started.port)
+}
+
+describe('the e2e reads the machine it drives', () => {
+ it('from a daemon that answers the full status only with the local credential', async () => {
+ await daemon()
+ await expect(machineId()).resolves.toBe('m1')
+ })
+
+ it('says the credential could not be read when the daemon gives only its public status', async () => {
+ delete process.env.HARNESS_MACHINE_ID
+ // What a daemon answers a caller without its credential (hookServer.ts `publicStatus`).
+ server = createServer((_req, res) => { res.setHeader('content-type', 'application/json'); res.end(JSON.stringify({ credentialRequired: true, version: '9.9.9', pid: 1, computerId: 'c1' })) })
+ await new Promise<void>((done) => server!.listen(0, '127.0.0.1', done))
+ process.env.HARNESS_PORT = String((server.address() as AddressInfo).port)
+ await expect(machineId()).rejects.toThrow(/wants its local credential, and this process could not read it/)
+ })
+})
diff --git a/cli/src/e2e/harnessLocal.ts b/cli/src/e2e/harnessLocal.ts
new file mode 100644
index 000000000..8c8fafd2f
--- /dev/null
+++ b/cli/src/e2e/harnessLocal.ts
@@ -0,0 +1,103 @@
+/**
+ * harnessLocal — what the e2e reads from THIS machine's harness install, the same places the
+ * daemon itself uses (`env.ts`), so nothing has to be passed in: the daemon's port, this
+ * computer's id, and the tmux pane an agent runs in (the registry row's runtime).
+ */
+import { readFileSync } from 'node:fs'
+import { join } from 'node:path'
+import { execFile } from 'node:child_process'
+import { promisify } from 'node:util'
+import { env } from '../config/env.js'
+import { localCredentialHeaders } from '../lib/localCredential.js'
+import type { GridOverride } from './gridSwitchDriver.js'
+
+const execFileP = promisify(execFile)
+
+export function daemonPort(): number {
+ return Number(process.env.HARNESS_PORT ?? env.PORT)
+}
+
+/**
+ * What every request to the daemon's loopback TCP API carries: its local credential, read from its data
+ * folder as every other client of it reads it (lib/localCredential.ts). Without it the daemon answers
+ * `/api/status` with who it is but not its machine, and refuses `/api/local-ws` (401), since guest machine
+ * sharing (bc7f56ff9, CLI 0.4.21): grid-dev's run stopped there on 2026-10-19.
+ */
+export function localHeaders(): Record<string, string> {
+ return localCredentialHeaders(env.ADAPTER_DATA_DIR)
+}
+
+export function daemonUrl(): string {
+ return `ws://127.0.0.1:${daemonPort()}/api/local-ws`
+}
+
+/**
+ * The machine id the RUNNING daemon answers `machine_select` for — read from its own
+ * `/api/status`, the same way `harness status` does. Not `computer-id`: that is this box's
+ * identity, while the daemon binds a backend-issued machine id, and a `machine_select` for any
+ * other id is relayed as a remote machine (`NO_PEER_LINK`).
+ */
+export async function machineId(): Promise<string> {
+ if (process.env.HARNESS_MACHINE_ID) return process.env.HARNESS_MACHINE_ID
+ const res = await fetch(`http://127.0.0.1:${daemonPort()}/api/status`, { headers: localHeaders(), signal: AbortSignal.timeout(3_000) })
+ if (!res.ok) throw new Error(`daemon /api/status answered ${res.status} — is \`harness\` running on port ${daemonPort()}?`)
+ const status = (await res.json()) as { machineId?: unknown; credentialRequired?: unknown }
+ // The public answer: this process could not show the credential (another OS user, or another data folder).
+ if (status.credentialRequired === true) {
+ throw new Error(`daemon on port ${daemonPort()} wants its local credential, and this process could not read it from ${env.ADAPTER_DATA_DIR}/hook-credential — run as the daemon's user, with its ADAPTER_DATA_DIR`)
+ }
+ if (typeof status.machineId !== 'string' || !status.machineId) throw new Error('daemon /api/status has no machineId')
+ return status.machineId
+}
+
+interface RegistryRow {
+ agentId?: string
+ tmuxPane?: string
+ runtimes?: Array<{ backend?: string; paneId?: string }>
+}
+
+/** The `%N` tmux pane of `agentId`, from the registry the daemon writes; null until it is registered. */
+export function paneOf(agentId: string): string | null {
+ let rows: RegistryRow[]
+ try {
+ const raw = JSON.parse(readFileSync(join(env.ADAPTER_DATA_DIR, 'registry.json'), 'utf8')) as unknown
+ rows = Array.isArray(raw) ? (raw as RegistryRow[]) : []
+ } catch {
+ return null
+ }
+ const row = rows.find((r) => r.agentId === agentId)
+ if (!row) return null
+ const pane = row.runtimes?.find((r) => r.backend === 'tmux')?.paneId ?? row.tmuxPane
+ return pane && /^%\d+$/.test(pane) ? pane : null
+}
+
+/** Poll `paneOf` until the daemon has registered the agent's pane (a fresh agent takes a moment). */
+export async function waitForPane(agentId: string, timeoutMs = 30_000): Promise<string | null> {
+ const start = Date.now()
+ while (Date.now() - start < timeoutMs) {
+ const pane = paneOf(agentId)
+ if (pane) return pane
+ await new Promise((r) => setTimeout(r, 500))
+ }
+ return null
+}
+
+/**
+ * A grid the daemon is NOT signed into — a local relay checkout under test — described the way
+ * `agent_retarget` takes it. Resolved from another `GRID_HOME` through the public `grid` CLI:
+ * `grid info --env <name>` gives the relay URL + key, `grid ls` the network id. Set
+ * E2E_GRID_HOME (+ E2E_GRID_CONTROL_PLANE_URL) and E2E_GRID_NAME to use it.
+ */
+export async function gridOverrideFromEnv(): Promise<{ override: GridOverride; name: string } | null> {
+ const home = process.env.E2E_GRID_HOME
+ const name = process.env.E2E_GRID_NAME
+ if (!home || !name) return null
+ const env = { ...process.env, GRID_HOME: home, ...(process.env.E2E_GRID_CONTROL_PLANE_URL ? { GRID_CONTROL_PLANE_URL: process.env.E2E_GRID_CONTROL_PLANE_URL } : {}) }
+ const { stdout: envOut } = await execFileP('grid', ['info', '--env', name], { env, timeout: 30_000 })
+ const baseUrl = /OPENAI_BASE_URL='([^']+)'/.exec(envOut)?.[1]
+ const apiKey = /OPENAI_API_KEY='([^']+)'/.exec(envOut)?.[1]
+ const { stdout: ls } = await execFileP('grid', ['ls'], { env, timeout: 30_000 })
+ const networkId = ls.split('\n').map((l) => l.trim().split(/\s+/)).find((cols) => cols[0] === name)?.[1]
+ if (!baseUrl || !apiKey || !networkId) throw new Error(`could not resolve grid ${name} from GRID_HOME=${home}`)
+ return { override: { networkId, networkName: name, baseUrl, apiKey }, name }
+}
diff --git a/cli/src/e2e/matrix.ts b/cli/src/e2e/matrix.ts
new file mode 100644
index 000000000..31cbe5f95
--- /dev/null
+++ b/cli/src/e2e/matrix.ts
@@ -0,0 +1,118 @@
+/**
+ * matrix — turn a planned trigger run into a concrete, auditable matrix entry: the exact spawn
+ * argv (from the harness's real builders), the login/grid presence NOTIFYs, and a namespaced
+ * session for every artifact. A matrix run is "the real test" — subscription -> grid -> back home.
+ *
+ * `--dry-run` (default) does everything except touch a live agent: it builds argv, checks
+ * login/grid availability, names the session, and emits the report. It is the $0 offline path that
+ * still proves the trigger + argv + naming + notify decision. Live switching adds a daemon + agent.
+ */
+import { type GridLaunchOverride, parseGridLaunchOverride } from '../lib/gridLaunch.js'
+import type { AgentEngine } from '../engines/types.js'
+import { gridSpawn, backHomeSpawn } from './gridArgv.js'
+import { sessionName } from './sessionName.js'
+import type { PlannedRun } from './trigger.js'
+import { SUBSCRIPTION_MODEL } from './fixedModels.js'
+import { plannedLegs, type LegOutcome } from './smokeChecks.js'
+import type { GridSwitchTraceStep } from './gridSwitchDriver.js'
+
+/** A placeholder override for dry-run: proves the argv builder without a real grid endpoint. */
+export function probeGridOverride(): GridLaunchOverride {
+ const raw = parseGridLaunchOverride({
+ networkId: 'grid-probe',
+ networkName: 'probe-grid',
+ baseUrl: 'https://grid.invalid/probe/relay/v1',
+ apiKey: 'probe-key',
+ model: 'grid:gpt-5-mini',
+ })
+ return raw.state === 'ok' ? raw.override : (undefined as never)
+}
+
+export interface EngineStatus {
+ binary: boolean
+ login: boolean
+ grid: boolean
+ /** What the engine itself said about its login — e.g. `apikey`, or a grid token wearing one. */
+ loginNote?: string
+}
+
+export interface MatrixEntry {
+ engine: AgentEngine
+ session: string
+ version: string
+ testcase: string
+ gridModel: string | null
+ gridSpawn: { command: string[]; envKeys: string[]; clearEnv: string[] } | { error: string; detail: string }
+ backHome: { command: string[]; clearEnv: string[] }
+ /** Fixed model used for the real subscription round-trip leg (per user decision). */
+ subscriptionModel: string | null
+ status: EngineStatus
+ notify: string[]
+ /**
+ * The smoke checks per leg (tool / mcp / recall on subscription and grid).
+ * Dry run: every check `not-run`. Live (`runGridSwitchTrace.ts`): the real outcomes + pane tails.
+ */
+ legs: LegOutcome[]
+ /** Live runs only: the daemon exchanges, in order. */
+ trace?: GridSwitchTraceStep[]
+}
+
+export const TESTCASE = 'grid-switch'
+
+export interface MatrixDeps {
+ status: (engine: AgentEngine) => Promise<EngineStatus>
+ grid: () => Promise<GridLaunchOverride | null> // null -> no grid resolved (live) -> NOTIFY
+ at?: Date
+}
+
+const dryGrid = probeGridOverride()
+
+/**
+ * Build one dry-run audit entry for a planned run. Live switching (daemon + agent) is layered on
+ * top of the same argv/session/status later; this is the decision + spawn + notify surface.
+ */
+export async function buildMatrixEntry(plan: PlannedRun, deps: MatrixDeps): Promise<MatrixEntry> {
+ const session = sessionName({
+ engine: plan.engine,
+ version: plan.latest ?? 'UNKNOWN',
+ testcase: TESTCASE,
+ gridModel: plan.latest ? dryGrid.model : undefined,
+ at: deps.at,
+ })
+ const status = await deps.status(plan.engine)
+ const notify: string[] = []
+
+ if (!status.binary) notify.push('ENGINE_MISSING: binary not found')
+ if (!status.login) notify.push('NOT_LOGGED_IN: engine needs a subscription login')
+ if (!status.grid) notify.push('GRID_UNAVAILABLE: machine not signed in to a grid (or no model)')
+
+ const grid = (await deps.grid()) ?? dryGrid
+ const build = gridSpawn(plan.engine, grid, 'SESSION_PLACEHOLDER')
+ const backHome = backHomeSpawn(plan.engine, 'SESSION_PLACEHOLDER')
+
+ const spawnResult: MatrixEntry['gridSpawn'] = isSpawnError(build)
+ ? { error: build.error, detail: build.detail }
+ : {
+ command: build.command,
+ envKeys: Object.keys(build.env),
+ clearEnv: build.clearEnv,
+ }
+
+ return {
+ engine: plan.engine,
+ session,
+ version: plan.latest ?? 'UNKNOWN',
+ testcase: TESTCASE,
+ gridModel: dryGrid.model ?? null,
+ gridSpawn: spawnResult,
+ backHome: { command: backHome.command, clearEnv: backHome.clearEnv },
+ subscriptionModel: SUBSCRIPTION_MODEL[plan.engine] ?? null,
+ status,
+ notify,
+ legs: plannedLegs(),
+ }
+}
+
+function isSpawnError(spec: Awaited<ReturnType<typeof gridSpawn>>): spec is { ok: false; error: string; detail: string } {
+ return 'ok' in spec && spec.ok === false
+}
diff --git a/cli/src/e2e/npmPackages.ts b/cli/src/e2e/npmPackages.ts
new file mode 100644
index 000000000..43167812d
--- /dev/null
+++ b/cli/src/e2e/npmPackages.ts
@@ -0,0 +1,10 @@
+/**
+ * The npm package that ships each engine, used by the npm version source to read its `latest`
+ * dist-tag. Kept separate so the real registry reader and any spec both mean the same package.
+ */
+import type { AgentEngine } from '../engines/types.js'
+
+export const NPM_PACKAGE: Partial<Record<AgentEngine, string>> = {
+ codex: '@openai/codex',
+ claude: '@anthropic-ai/claude-code',
+}
diff --git a/cli/src/e2e/paneProbe.ts b/cli/src/e2e/paneProbe.ts
new file mode 100644
index 000000000..e68f70322
--- /dev/null
+++ b/cli/src/e2e/paneProbe.ts
@@ -0,0 +1,332 @@
+/**
+ * paneProbe — run the smoke checks against a LIVE agent pane, the way a person would: type the
+ * prompt into the tmux pane, wait for the marker to show up in what the pane prints.
+ *
+ * tmux only. The harness runs every agent in a tmux pane (`tmuxBackend.ts`), and `respawn-pane`
+ * keeps the pane id across a grid switch, so one `%N` follows the agent through all three legs.
+ *
+ * Pure-ish: the tmux calls are injected (`Tmux`) so the sequencing — settle, type, poll, tail — is
+ * unit-testable offline; `realTmux` is the one that shells out.
+ */
+import { execFile } from 'node:child_process'
+import { promisify } from 'node:util'
+import { scenarioFor, type CheckOutcome, type Leg, type LegOutcome, type LogKind, type SmokeCheck } from './smokeChecks.js'
+import type { ToolResult } from './sessionTools.js'
+
+const execFileP = promisify(execFile)
+let typeSequence = 0
+
+export interface Tmux {
+ /** The pane's visible text plus scrollback tail (`capture-pane -p -S -<lines>`). */
+ capture(pane: string, lines: number): Promise<string>
+ /** Type `text` literally, then Enter. */
+ type(pane: string, text: string): Promise<void>
+ /** Press one key by tmux name (`1`, `Enter`, `Escape`) — for a dialog, not for a prompt. */
+ key(pane: string, key: string): Promise<void>
+ sleep(ms: number): Promise<void>
+ now(): number
+}
+
+export const realTmux: Tmux = {
+ async capture(pane, lines) {
+ const { stdout } = await execFileP('tmux', ['capture-pane', '-p', '-t', pane, '-S', `-${lines}`], { timeout: 5000 })
+ return stdout
+ },
+ async type(pane, text) {
+ // Not `send-keys -l`: a whole line arriving as one keystroke burst trips the TUI's paste
+ // detector (codex 0.155.1 kept the text in the composer and swallowed the Enter). The harness's
+ // own injection (`lib/tmux.ts sendToTmux`) pastes from a named buffer, lets it land, then sends
+ // Enter on its own — same sequence here.
+ const buffer = `e2e-probe-${process.pid}-${++typeSequence}`
+ await execFileP('tmux', ['set-buffer', '-b', buffer, '--', text], { timeout: 5000 })
+ await execFileP('tmux', ['paste-buffer', '-t', pane, '-b', buffer, '-d'], { timeout: 5000 })
+ await new Promise((r) => setTimeout(r, 400))
+ await execFileP('tmux', ['send-keys', '-t', pane, 'Enter'], { timeout: 5000 })
+ },
+ async key(pane, key) {
+ await execFileP('tmux', ['send-keys', '-t', pane, key], { timeout: 5000 })
+ },
+ sleep: (ms) => new Promise((r) => setTimeout(r, ms)),
+ now: () => Date.now(),
+}
+
+/**
+ * Startup dialogs a fresh coding tool puts in front of its prompt. A check typed into one of these
+ * is not a check: the text is swallowed and Enter picks whatever is highlighted (measured: codex
+ * 0.155's "Update available" took the Enter as "1. Update now" and ran `npm install -g`). Each is
+ * answered the way a person running a test would — and each answer is recorded in the leg's
+ * evidence, since a tool that greets every resume with a dialog is itself a finding.
+ */
+export const STARTUP_DIALOGS: ReadonlyArray<{ id: string; match: RegExp; keys: string[] }> = [
+ // codex: "Do you trust the contents of this directory?" 1. Yes, continue
+ { id: 'codex-trust', match: /Do you trust the contents of this directory/, keys: ['1', 'Enter'] },
+ // codex: "Update available! x -> y" 1. Update now / 2. Skip / 3. Skip until next version
+ { id: 'codex-update', match: /Update available!/, keys: ['2', 'Enter'] },
+ // codex 0.156: "Hooks need review — 2 hooks are new or changed" with the cursor on "Review hooks".
+ // The hooks are the harness's own (how it tracks the agent), so "Trust all and continue" is what a
+ // person using the harness answers too — Down once from "Review hooks", then Enter.
+ { id: 'codex-hooks-trust', match: /Hooks need review/, keys: ['Down', 'Enter'] },
+ // codex: "GPT-5.5 retires on <date>. Switch to <next> to continue" 1. Try new model / 2. Use existing
+ // model. Keep the model the run was configured with: a test that quietly changes its own model is
+ // no longer the test that was asked for.
+ { id: 'codex-model-retire', match: /retires on [^\n]*\n?[^\n]*Switch to|1\. Try new model/, keys: ['2', 'Enter'] },
+ // claude: "Do you trust the files in this folder?" — Enter accepts the highlighted "Yes, proceed"
+ { id: 'claude-trust', match: /Do you trust the files in this folder/, keys: ['Enter'] },
+ // claude 2.1.281's rewording — "Quick safety check: Is this a project you created or one you
+ // trust?" — and its default moved to **"No, exit"**: the old Enter would now quit the engine.
+ // Down to "Yes, I trust this folder", then Enter. (The run pre-trusts its own folder, so this is
+ // the fallback for when that does not take.)
+ { id: 'claude-trust-v2', match: /Yes, I trust this folder/, keys: ['Down', 'Enter'] },
+ // claude: a project .mcp.json must be approved — Enter accepts the highlighted "Use this and all future MCP servers"
+ { id: 'claude-mcp-approve', match: /MCP server[s]? (found|configured) in \.mcp\.json|Use this and all future MCP servers|Use this MCP server/, keys: ['Enter'] },
+ // claude, on `--dangerously-skip-permissions` (permission mode `full`): a full-screen Bypass
+ // Permissions warning whose DEFAULT is "No, exit" — Enter alone exits 1 and leaves the pane a
+ // shell. Down first, then Enter, picks "Yes, I accept". This is not a fallback in practice: with
+ // `bypassPermissionsModeAccepted` already written to ~/.claude.json, claude 2.1.281 still drew the
+ // warning, and answering it here is what let the run's first leg happen at all.
+ { id: 'claude-bypass-accept', match: /running in Bypass Permissions mode/, keys: ['Down', 'Enter'] },
+ // claude, MID-TURN, on any gateway (measured on grid.autonomous.ai AND a local relay): an "auto mode
+ // classifier … your requests go through <host>, which isn't compatible" notice that blocks the turn
+ // until a key is pressed. A finding about the relay's Anthropic contract, and a modal to get past.
+ { id: 'claude-automode-notice', match: /isn't compatible with this update|ask your gateway to/, keys: ['Enter'] },
+]
+
+/**
+ * The tool saying its account is out of usage — the same words in both, read out of the binaries
+ * rather than guessed:
+ *
+ * codex "You've hit your usage limit. … try again at Sep 24th, 2026 6:02 PM."
+ * claude "You've hit your session limit · resets 6pm" (also weekly / monthly / team budget)
+ * "You're out of usage credits. /model to switch models."
+ *
+ * Without this, a check on an exhausted account waited out its 90s, came back `stuck`, and the run
+ * went to the reviewer as if it were a bug to diagnose — for a condition the pane had already stated
+ * in plain words. "Approaching usage limit" (claude's early warning) is deliberately not matched.
+ */
+export const QUOTA_EXHAUSTED = /You['\u2019]ve hit your [^\n]{0,40}?(limit|budget)|You['\u2019]re out of usage credits|usage limit reached/i
+
+/** When the tool says the quota comes back, in its own words — or null if it did not say. */
+export function quotaResets(text: string): string | null {
+ const m = text.match(/(resets? (?:at |in |on )?[^\n\u00b7|.]{1,40}|try again (?:at|in) [^\n|]{1,50}?)(?:\.(?:\s|$)|\n|$)/im)
+ return m ? m[1].trim() : null
+}
+
+/**
+ * The dialog that is actually waiting — the LOWEST one on screen, not the first in the list. A fresh
+ * codex stacks them (the hooks review, then the model notice under it), and the earlier text stays
+ * in the scrollback after it is answered; picking by list order answered a screen that was already
+ * gone and sent its keys into the one that was not. One answer per dialog per round of dismissals.
+ */
+function activeDialog(screen: string, answered: readonly string[]): (typeof STARTUP_DIALOGS)[number] | undefined {
+ let best: (typeof STARTUP_DIALOGS)[number] | undefined
+ let at = -1
+ for (const d of STARTUP_DIALOGS) {
+ if (answered.includes(d.id)) continue
+ const all = [...screen.matchAll(new RegExp(d.match.source, d.match.flags.includes('g') ? d.match.flags : d.match.flags + 'g'))]
+ const last = all.length ? all[all.length - 1].index! : -1
+ if (last > at) { at = last; best = d }
+ }
+ return best
+}
+
+/**
+ * Answer any startup dialog on screen, then wait for the pane to settle again. Returns the ids of
+ * the dialogs answered (evidence). Bounded: a dialog that comes back after being answered is left
+ * alone after `max` rounds so the check that follows records it as stuck, with the dialog on the tail.
+ */
+export async function dismissStartupDialogs(tmux: Tmux, pane: string, opts: ProbeOptions = {}, max = 5): Promise<string[]> {
+ const o = { ...DEFAULTS, ...opts }
+ const answered: string[] = []
+ for (let round = 0; round < max; round++) {
+ const screen = await tmux.capture(pane, o.tailLines)
+ const dialog = activeDialog(screen, answered)
+ if (!dialog) break
+ for (const k of dialog.keys) {
+ await tmux.key(pane, k)
+ await tmux.sleep(300)
+ }
+ answered.push(dialog.id)
+ await waitForPaneSettle(tmux, pane, opts)
+ }
+ return answered
+}
+
+export interface ProbeOptions {
+ /** How long one check may take before it is `stuck`. Grid models can be slow; default 90s. */
+ checkTimeoutMs?: number
+ /** The tool / MCP log lines right now (`workspace.ts readLog`); a step with a `log` needs one new matching line. */
+ readLog?: (kind: LogKind) => string[]
+ /** A workspace file's content right now, or null when it does not exist (`workspace.ts readWorkspaceFile`). */
+ readFile?: (relPath: string) => string | null
+ /** What the engine's session file says `tool` did for the step tagged `ref` (`sessionTools.ts toolResult`). */
+ toolResult?: (ref: string, tool: string) => ToolResult
+ /** Whose wording the requests use (smokeChecks.ts scenarioFor): each names that engine's tool. */
+ engine?: string
+ /** The pane is "ready" once its screen stops changing for this long. A resumed TUI needs it. */
+ settleMs?: number
+ /** Give up waiting for the pane to settle after this long (the TUI itself may be stuck). */
+ settleTimeoutMs?: number
+ pollMs?: number
+ tailLines?: number
+}
+
+const DEFAULTS: Required<ProbeOptions> = {
+ checkTimeoutMs: 90_000,
+ readLog: () => [],
+ readFile: () => null,
+ toolResult: () => ({ called: false, answered: false }),
+ engine: 'claude',
+ settleMs: 2_000,
+ settleTimeoutMs: 60_000,
+ pollMs: 1_000,
+ tailLines: 40,
+}
+
+/** Wait until two consecutive captures match — the TUI has drawn and is waiting on input. */
+export async function waitForPaneSettle(tmux: Tmux, pane: string, opts: ProbeOptions = {}): Promise<boolean> {
+ const o = { ...DEFAULTS, ...opts }
+ const start = tmux.now()
+ let last = await tmux.capture(pane, o.tailLines)
+ while (tmux.now() - start < o.settleTimeoutMs) {
+ await tmux.sleep(o.settleMs)
+ const next = await tmux.capture(pane, o.tailLines)
+ if (next === last) return true
+ last = next
+ }
+ return false
+}
+
+/** A short tag typed after the prompt so THIS check's echo can be told from earlier ones. */
+export function checkRef(now: number): string {
+ return `ref-${now.toString(36).slice(-4).padStart(4, '0')}${Math.floor(Math.random() * 1296).toString(36).padStart(2, '0')}`
+}
+
+/**
+ * Type one check into the pane and wait for its marker. Evidence (`tail`) comes back either way.
+ *
+ * The marker only counts AFTER the echo of this check's own `ref` tag. Counting occurrences was
+ * wrong: a resume redraws the whole transcript, so the previous leg's answer is on screen again,
+ * and as the new turn scrolls the old one out the count stays flat — a real `ANSWER_42` on the
+ * back-home leg was recorded as stuck (measured, codex 0.155.1). The tag pins the search to the
+ * new turn instead; a wrapped tag (never echoed in one piece) falls back to "one more hit than before".
+ */
+export async function runCheck(tmux: Tmux, pane: string, check: SmokeCheck, opts: ProbeOptions = {}): Promise<CheckOutcome> {
+ const o = { ...DEFAULTS, ...opts }
+ const before = await tmux.capture(pane, o.tailLines)
+ const logBefore = check.log ? o.readLog(check.log).length : 0
+ // read: the file's token, made fresh for this run and written nowhere else (workspace.ts).
+ const token = check.answerFromFile ? (o.readFile(check.answerFromFile) ?? '').trim() : ''
+ if (check.answerFromFile && !token) {
+ return { id: check.id, status: 'stuck', elapsedMs: 0, tail: before.trimEnd(), note: `the workspace has no ${check.answerFromFile} — the run's setup, not the tool` }
+ }
+ const start = tmux.now()
+ const ref = checkRef(start)
+ await tmux.type(pane, `${check.prompt} (${ref})`)
+ // Make sure it landed. A tool that has only just started can drop input while it is still
+ // loading — measured on grid-dev: claude's first question of a run never reached it (its own
+ // transcript has every later question and not that one). The tag is on screen within a second
+ // when the text arrived, in the composer or in the transcript; if it is nowhere, type it again.
+ for (let retry = 0; retry < 2; retry++) {
+ let landed = false
+ for (let waited = 0; waited < 5_000 && !landed; waited += o.pollMs) {
+ await tmux.sleep(o.pollMs)
+ landed = (await tmux.capture(pane, o.tailLines)).includes(ref)
+ }
+ if (landed) break
+ await tmux.type(pane, `${check.prompt} (${ref})`)
+ }
+
+ // The proof, never the reply: a log line for exactly these numbers, the file on disk as asked,
+ // the read file's token on the pane after this prompt (nobody guesses `kiwi-4821-tulip`),
+ // or the engine's own record of the tool coming back with results.
+ const proven = (afterEcho: string | null, tail: string): { logLines?: string[] } | null => {
+ if (check.log) {
+ const logLines = o.readLog(check.log).slice(logBefore)
+ return logLines.some((l) => l.includes(check.logPattern!)) ? { logLines } : null
+ }
+ if (check.sessionTool) return o.toolResult(ref, check.sessionTool).answered ? {} : null
+ if (check.file) {
+ const content = o.readFile(check.file.path)
+ if (content === null) return null
+ const f = check.file
+ if (f.equals !== undefined && content.trim() !== f.equals) return null
+ if (f.contains !== undefined && !content.includes(f.contains)) return null
+ if (f.lacks !== undefined && content.includes(f.lacks)) return null
+ return {}
+ }
+ if (token) return (afterEcho ?? (before.includes(token) ? '' : tail)).includes(token) ? {} : null
+ return null
+ }
+
+ let tail = before
+ const dialogs: string[] = []
+ while (tmux.now() - start < o.checkTimeoutMs) {
+ await tmux.sleep(o.pollMs)
+ tail = await tmux.capture(pane, o.tailLines)
+ const echo = tail.lastIndexOf(ref)
+ const afterEcho = echo >= 0 ? tail.slice(echo + ref.length) : null
+ // A modal can land in the middle of a turn (claude's gateway notice); answer it and keep waiting.
+ // Only one that appeared after THIS prompt: startup screens answered before it stay in the
+ // scrollback, and answering them again would send their keys into the running turn.
+ const modal = activeDialog(afterEcho ?? tail, dialogs)
+ if (modal) {
+ for (const k of modal.keys) { await tmux.key(pane, k); await tmux.sleep(300) }
+ dialogs.push(modal.id)
+ continue
+ }
+ // Out of usage: said once, right after our prompt. Only text after THIS prompt's echo counts
+ // (or, when the echo scrolled away, only if the screen did not already say it before we typed).
+ const answer = afterEcho ?? (QUOTA_EXHAUSTED.test(before) ? '' : tail)
+ if (QUOTA_EXHAUSTED.test(answer)) {
+ const resets = quotaResets(answer)
+ return { id: check.id, ref, status: 'no-quota', elapsedMs: tmux.now() - start, tail: tail.trimEnd(), ...(resets ? { resets } : {}), ...(dialogs.length ? { dialogs } : {}) }
+ }
+ const proof = proven(afterEcho, tail)
+ if (proof) {
+ const elapsedMs = tmux.now() - start
+ // Proven — but the turn may still be running (the model reporting what it did). Let the pane
+ // go quiet before the next request, the way a person waits for an answer before typing.
+ await waitForPaneSettle(tmux, pane, { ...opts, settleTimeoutMs: 30_000 })
+ return { id: check.id, ref, status: 'ok', elapsedMs, tail: (await tmux.capture(pane, o.tailLines)).trimEnd(), ...proof, ...(dialogs.length ? { dialogs } : {}) }
+ }
+ }
+ // Not proven in time: say what was missing, with what was actually there.
+ const logLines = check.log ? o.readLog(check.log).slice(logBefore) : undefined
+ let note: string
+ if (check.log) note = `no ${check.log} log line for "${check.logPattern}"${logLines && logLines.length ? ` (got: ${logLines.join(' | ')})` : ''}`
+ else if (check.sessionTool) {
+ const used = o.toolResult(ref, check.sessionTool)
+ note = !used.called ? `${check.sessionTool} was never called` : `${check.sessionTool} came back without results${used.output ? `: ${JSON.stringify(used.output)}` : ''}`
+ } else if (check.file) {
+ const content = o.readFile(check.file.path)
+ note = content === null ? `${check.file.path} does not exist` : `${check.file.path} is ${JSON.stringify(content.trim().slice(0, 120))}`
+ } else note = `${check.answerFromFile}'s token never appeared after the question`
+ return { id: check.id, ref, status: 'stuck', elapsedMs: tmux.now() - start, tail: tail.trimEnd(), note, ...(logLines ? { logLines } : {}), ...(dialogs.length ? { dialogs } : {}) }
+}
+
+/**
+ * Run this leg's steps of the scenario, in order. Stops at the first stuck step: a pane that is
+ * not answering will not answer the next prompt either, and the queued text would only muddy the evidence.
+ */
+export async function probeLeg(tmux: Tmux, pane: string, leg: Leg, opts: ProbeOptions = {}): Promise<LegOutcome> {
+ const steps = scenarioFor(opts.engine ?? 'claude')[leg]
+ const checks: CheckOutcome[] = []
+ const settled = await waitForPaneSettle(tmux, pane, opts)
+ if (!settled) {
+ const tail = (await tmux.capture(pane, opts.tailLines ?? DEFAULTS.tailLines)).trimEnd()
+ // Nothing typed: the TUI never stopped redrawing, so the first check is stuck before it began.
+ checks.push({ id: steps[0].id, status: 'stuck', elapsedMs: 0, tail })
+ for (const c of steps.slice(1)) checks.push({ id: c.id, status: 'not-run' })
+ return { leg, checks }
+ }
+ const dialogs = await dismissStartupDialogs(tmux, pane, opts)
+ for (const [i, check] of steps.entries()) {
+ const outcome = await runCheck(tmux, pane, check, opts)
+ checks.push(outcome)
+ if (outcome.status === 'stuck' || outcome.status === 'no-quota') {
+ for (const rest of steps.slice(i + 1)) checks.push({ id: rest.id, status: 'not-run' })
+ break
+ }
+ }
+ return dialogs.length ? { leg, checks, dialogs } : { leg, checks }
+}
diff --git a/cli/src/e2e/plan.ts b/cli/src/e2e/plan.ts
new file mode 100644
index 000000000..0362308ab
--- /dev/null
+++ b/cli/src/e2e/plan.ts
@@ -0,0 +1,23 @@
+/**
+ * `harness e2e plan` — the trigger's decision, printed: for each engine under test, the installed
+ * version, the latest released one, and whether a run is due. The cron runner (autonomous-grid-cli,
+ * e2e/run.sh) reads the JSON form and runs `harness e2e grid-switch` per engine it names.
+ *
+ * Env: E2E_VERSION_SOURCE=mock|npm (default mock until the npm reader is validated), E2E_ENGINES=codex,claude.
+ */
+import type { AgentEngine } from '../engines/types.js'
+import { versionSourceFromEnv } from './versionSource.js'
+import { InstalledCurrentVersion } from './currentVersion.js'
+import { planMatrixRuns } from './trigger.js'
+
+export async function planCommand(json: boolean): Promise<void> {
+ const engines = (process.env.E2E_ENGINES ?? 'codex,claude').split(',').map((e) => e.trim() as AgentEngine).filter(Boolean)
+ const plans = await planMatrixRuns(versionSourceFromEnv(), new InstalledCurrentVersion(), engines)
+ if (json) {
+ console.log(JSON.stringify({ generatedAt: new Date().toISOString(), plans }, null, 2))
+ return
+ }
+ for (const p of plans) {
+ console.log(`${p.engine.padEnd(8)} current=${String(p.current).padEnd(12)} latest=${String(p.latest).padEnd(12)} ${p.plan ? 'RUN' : 'skip'} (${p.reason})`)
+ }
+}
diff --git a/cli/src/e2e/runGridSwitchTrace.ts b/cli/src/e2e/runGridSwitchTrace.ts
new file mode 100644
index 000000000..e2a2fbd4a
--- /dev/null
+++ b/cli/src/e2e/runGridSwitchTrace.ts
@@ -0,0 +1,324 @@
+/**
+ * Run the grid-switch journey for REAL: create a fresh agent through the daemon (like "New agent"
+ * in the app), run it subscription -> grid -> back home with the scenario on every leg, delete
+ * it, and leave a review bundle the `e2e-watchdog` agent (autonomous-grid-cli, e2e/) reads.
+ *
+ * agent_create — a new codex/claude in its own tmux pane, in a scratch folder; nobody's pane is borrowed
+ * subscription — checks on it as it is (its own login): the baseline
+ * grid — agent_retarget onto a grid model (pane respawned, session resumed), checks again
+ * agent_delete — gone at the end, unless E2E_KEEP_AGENT=1 (the reviewer, autonomous-grid-cli's
+ * e2e-watchdog, reads the bundle afterwards; keep the agent when it should type into the pane)
+ *
+ * Usage (zero-config; everything is read from this machine's harness install):
+ * harness e2e grid-switch (or: npx tsx src/e2e/runGridSwitchTrace.ts)
+ *
+ * Env (all optional):
+ * HARNESS_ENGINE=codex|claude engine under test (default codex)
+ * HARNESS_GRID_MODEL / HARNESS_GRID_NAME pin the grid model (default: smallest listed — pin the fastest
+ * one from `grid stats <grid> --verbose` when it matters)
+ * HARNESS_AGENT_ID + HARNESS_PANE_ID reuse an existing agent instead of creating one (not deleted)
+ * HARNESS_PORT / HARNESS_MACHINE_ID override what env.ts / computer-id say
+ * E2E_GRID_HOME + E2E_GRID_NAME (+ E2E_GRID_CONTROL_PLANE_URL) move onto a grid from ANOTHER grid home —
+ * a local relay checkout under test — instead of the daemon's own
+ * E2E_KEEP_AGENT=1 leave the created agent running for inspection
+ *
+ * The daemon is the one `harness` started (`ws://127.0.0.1:${PORT}/api/local-ws`) — the exact
+ * transport and commands the desktop app uses.
+ */
+import { mkdirSync, realpathSync, writeFileSync, cpSync, existsSync, readFileSync } from 'node:fs'
+import { join } from 'node:path'
+import { execFile } from 'node:child_process'
+import { promisify } from 'node:util'
+import { GridSwitchDriver, pickGridModel, type GridSwitchTraceStep } from './gridSwitchDriver.js'
+import { probeLeg, realTmux, waitForPaneSettle } from './paneProbe.js'
+import { firstStuck, plannedLegs, quotaHit, LEGS, type LegOutcome, type LogKind } from './smokeChecks.js'
+
+const LAST_LEG = LEGS[LEGS.length - 1]
+import { prepareWorkspace, readLog, readWorkspaceFile, removeCodexMcp } from './workspace.js'
+import { sessionName, workspaceDirName } from './sessionName.js'
+import { TESTCASE } from './matrix.js'
+import { outDir, runDir } from './artifacts.js'
+import { env } from '../config/env.js'
+import { daemonUrl, localHeaders, machineId, waitForPane, gridOverrideFromEnv } from './harnessLocal.js'
+import { parseVersionLine } from './currentVersion.js'
+import { folderTrust } from '../engines/launchPrep.js'
+import { toolResult, useByRef } from './sessionTools.js'
+
+const execFileP = promisify(execFile)
+
+function mkdirAll(dir: string): string {
+ mkdirSync(dir, { recursive: true })
+ return dir
+}
+
+export interface GridSwitchRunResult {
+ session: string
+ reportPath: string
+ runDir: string
+ stuck: ReturnType<typeof firstStuck>
+ error: string | null
+}
+
+/**
+ * The live journey, as a function: `harness e2e grid-switch` and the cron runner call this; the
+ * env described at the top of this file is the only configuration.
+ */
+export async function runGridSwitch(): Promise<GridSwitchRunResult> {
+const engine = process.env.HARNESS_ENGINE ?? 'codex'
+const url = daemonUrl()
+const machine = await machineId()
+const version = await (async () => {
+ try {
+ const { stdout } = await execFileP(engine, ['--version'], { timeout: 10_000 })
+ return parseVersionLine(stdout) ?? 'UNKNOWN'
+ } catch {
+ return 'UNKNOWN'
+ }
+})()
+
+console.error(`[grid-e2e] dialing ${url} · machine ${machine} · ${engine} ${version}`)
+
+const legs: LegOutcome[] = []
+const planned = plannedLegs()
+let pane: string | null = process.env.HARNESS_PANE_ID ?? null
+let workspace: string | null = null
+// A subscription model answers in seconds; a local model on a grid may take a minute. Only the
+// grid leg gets the long budget, so a stuck check on the tool's own login is known in 45s, not 90.
+const CHECK_TIMEOUT_MS: Record<LegOutcome['leg'], number> = { subscription: 120_000, grid: 120_000 }
+
+async function checks(leg: LegOutcome['leg']): Promise<void> {
+ const readLogs = workspace ? (kind: LogKind) => readLog(workspace!, kind) : undefined
+ const readFile = workspace ? (rel: string) => readWorkspaceFile(workspace!, rel) : undefined
+ const usedTool = workspace ? (ref: string, tool: string) => toolResult(engine, workspace!, ref, tool) : undefined
+ const outcome = pane
+ ? await probeLeg(realTmux, pane, leg, { engine, checkTimeoutMs: CHECK_TIMEOUT_MS[leg], ...(readLogs ? { readLog: readLogs } : {}), ...(readFile ? { readFile } : {}), ...(usedTool ? { toolResult: usedTool } : {}) })
+ : planned.find((l) => l.leg === leg)!
+ legs.push(outcome)
+ console.error(`[grid-e2e] ${leg}: ${outcome.checks.map((c) => `${c.id}=${c.status}`).join(' ')}${outcome.dialogs ? ` (answered: ${outcome.dialogs.join(', ')})` : ''}`)
+ // The marker shows before the tool's turn is fully over (hooks still reporting); a retarget on a
+ // turn in flight is refused AGENT_BUSY. Let the pane go quiet first, the way a person would —
+ // but only when a retarget follows: after the last leg the agent is deleted, not moved.
+ if (pane && leg !== LAST_LEG) {
+ await waitForPaneSettle(realTmux, pane, { settleMs: 3_000, settleTimeoutMs: 60_000 })
+ await new Promise((r) => setTimeout(r, 3_000))
+ }
+}
+
+const driver = new GridSwitchDriver(url, machine, localHeaders())
+let trace: GridSwitchTraceStep[] = []
+let gridModel: string | null = null
+let agentId: string | null = process.env.HARNESS_AGENT_ID ?? null
+// codex's MCP server is registered globally (`codex mcp add`, see workspace.ts). Tracked so the run
+// removes exactly what it added, and removes it even when the run stops early.
+let registeredCodexMcp = false
+// How the MCP server got registered for this engine, and in which permission mode the agent runs —
+// both in the trace, because an MCP step that never fired is read very differently once you know
+// the server was never registered or the engine was launched in a mode that refuses it.
+let mcpSetup: { config: string | null; note: string } | null = null
+// The app's own default (`kDefaultPermissionMode` in desktop/lib/core/permission_modes.dart), and
+// on purpose not overridable: the run creates agents exactly as a person does. It ran in `full`
+// until 2026-09-25, and that alone made codex 0.157.0 fail a switch the app does fine — `full`
+// launches codex with no `-c`, which starts codex's background daemon, and the daemon kept the
+// conversation after the switch ("This conversation is open in another app"). `auto`'s
+// `--approve-for-me` is three `-c` overrides to codex, and codex never starts the daemon for those.
+const permissionMode = 'auto'
+const created = !agentId
+let error: string | null = null
+// Named before anything runs so the scratch folder and the trace share it; the grid model is
+// stamped on once known.
+const startedAt = new Date()
+try {
+ await driver.machineSelect()
+
+ // Which grid and model — settled BEFORE the agent exists, and asked of the daemon only when the
+ // run did not pin them. `grid_models_list` starts one Python `grid` per grid the account can see,
+ // all at once (~125 MB each); on grid-dev that was ten of them, 1.2 GB, landing while claude
+ // started, and the 1.5 GB container OOM-killed claude. Pinned (HARNESS_GRID_NAME +
+ // HARNESS_GRID_MODEL, or a relay under test), the list is never needed; unpinned, it is asked
+ // while nothing else is running.
+ // A relay under test (E2E_GRID_HOME + E2E_GRID_NAME) replaces the daemon's own grid; the model
+ // then has to be pinned (HARNESS_GRID_MODEL), since that grid's list is not in `grid_models_list`.
+ const local = await gridOverrideFromEnv()
+ const pinned = !!process.env.HARNESS_GRID_MODEL && (!!process.env.HARNESS_GRID_NAME || !!local)
+ const models = pinned ? null : await driver.gridModels()
+ const picked = local || !models ? null : pickGridModel(models)
+ gridModel = process.env.HARNESS_GRID_MODEL ?? picked?.model ?? null
+ const gridName = local ? local.name : (process.env.HARNESS_GRID_NAME ?? (process.env.HARNESS_GRID_MODEL ? undefined : picked?.gridName))
+ console.error(`[grid-e2e] grid model: ${gridModel ?? '(none)'}${gridName ? ` on ${gridName}` : ''}${local ? ` (relay under test: ${local.override.baseUrl})` : ''}${pinned ? ' (pinned — no model list asked)' : ''}`)
+
+ if (created) {
+ // realpath: $TMPDIR is a symlink on macOS (`/var/…` → `/private/var/…`) and the engine keys its
+ // trust on the resolved path — a trust written for the symlink is never matched.
+ const cwd = realpathSync(mkdirAll(join(outDir(), 'agents', workspaceDirName(sessionName({ engine, version, testcase: TESTCASE, at: startedAt })))))
+ // The person's project: the notes the file steps work on, an MCP server registered for this engine, and its log.
+ const laid = prepareWorkspace(cwd, engine)
+ workspace = cwd
+ console.error(`[grid-e2e] workspace ${cwd} · mcp ${laid.mcpConfig ?? '(none)'} — ${laid.mcpNote}`)
+ // codex's registration is global (`codex mcp add`), so it is this run's to take back out.
+ if (engine === 'codex' && laid.mcpConfig) registeredCodexMcp = true
+ mcpSetup = { config: laid.mcpConfig, note: laid.mcpNote }
+ // The scratch folder is this run's own, so answer the engine's "trust this folder?" the way the
+ // daemon does for a workspace it made — otherwise the first check would be typed into that dialog.
+ try {
+ if (engine === 'claude' || engine === 'codex') folderTrust(engine)?.record(cwd)
+ } catch (err) {
+ console.error(`[grid-e2e] pre-trust ${cwd}: ${(err as Error).message}`)
+ }
+ // REGISTRATION_FAILED right after the daemon starts is not the test's to report: the saved
+ // registry still holds an agent from before the restart (a container, a reboot), tmux numbers
+ // the new pane %0 again, and the stale row owns that key until discovery confirms it gone —
+ // measured on grid-dev: refused at 07:41:11, the stale row forgotten at 07:41:20. Wait it out.
+ let made = await driver.createAgent(engine, cwd, `e2e ${engine} ${version}`, permissionMode)
+ for (let retry = 1; !made.ok && made.error === 'REGISTRATION_FAILED' && retry <= 3; retry++) {
+ console.error(`[grid-e2e] agent_create REGISTRATION_FAILED — a stale agent from before the daemon restart still holds the pane; retry ${retry}/3 in 15s`)
+ await new Promise((r) => setTimeout(r, 15_000))
+ made = await driver.createAgent(engine, cwd, `e2e ${engine} ${version}`, permissionMode)
+ }
+ if (!made.ok) throw new Error(`${made.error}: ${made.detail ?? ''}`)
+ agentId = made.agentId
+ pane = made.pane ?? (await waitForPane(agentId))
+ console.error(`[grid-e2e] created agent ${agentId}${pane ? ` in pane ${pane}` : ' (pane not registered yet: checks not-run)'}`)
+ // A fresh codex/claude has a trust / first-run screen before its prompt; let it draw.
+ if (pane) await waitForPaneSettle(realTmux, pane, { settleMs: 3_000, settleTimeoutMs: 90_000 })
+ }
+
+
+ // Out of usage ends the journey where it happened: an account that cannot answer on its own login
+ // tells us nothing about a switch, and switching it anyway only spends the grid's time too.
+ await checks('subscription')
+ if (!quotaHit(legs)) {
+ if (!gridModel) throw new Error('GRID_UNAVAILABLE: grid_models_list returned no model')
+ const onto = await driver.retargetToGrid(agentId!, gridModel, gridName, local?.override)
+ if (!onto.ok) throw new Error(`${onto.error}: ${onto.detail ?? ''}`)
+ await checks('grid')
+ }
+
+} catch (err) {
+ error = (err as Error).message
+ console.error(`[grid-e2e] stopped: ${error}`)
+} finally {
+ trace = [...driver.trace]
+}
+
+// How each step got done, and by which model: read from the engine's own session file now that the
+// run is over. Reported, never judged — a step's pass/fail is its result on disk (smokeChecks.ts).
+if (workspace) {
+ const refs = legs.flatMap((l) => l.checks.map((c) => c.ref).filter((r): r is string => !!r))
+ const used = useByRef(engine, workspace, refs)
+ for (const l of legs) for (const c of l.checks) {
+ const u = c.ref ? used[c.ref] : undefined
+ if (u) { c.tools = u.tools; c.models = u.models }
+ }
+}
+
+const session = sessionName({ engine, version, testcase: TESTCASE, gridModel: gridModel ?? undefined, at: startedAt })
+const report = {
+ generatedAt: new Date().toISOString(),
+ mode: 'live',
+ engines: [
+ {
+ engine,
+ session,
+ version,
+ testcase: TESTCASE,
+ gridModel,
+ agentId,
+ pane,
+ /** The folder the agent worked in: notes/, out/, the MCP server and config, and .e2e/calc-mcp.log with every MCP call. */
+ workspace,
+ /** The mode the engine was launched in — always the app's default, `auto`. */
+ permissionMode,
+ /** Where this engine's MCP server was registered, and how — null config means the MCP steps could not pass. */
+ mcp: mcpSetup,
+ /** true: this run made the agent. It stays alive through the review (so the reviewer can type into the pane), then is deleted unless E2E_KEEP_AGENT=1. */
+ createdByRun: created,
+ keptAlive: !created || process.env.E2E_KEEP_AGENT === '1',
+ notify: error ? [error] : [],
+ /** Set when the tool said its account is out of usage: { leg, check, resets }. The pipeline
+ * reads this before anything else — no review, one plain message, and a retry later. */
+ quota: quotaHit(legs),
+ // Every leg, run or not, so a table built from this always has its three rows.
+ legs: [...legs, ...planned.filter((p) => !legs.some((l) => l.leg === p.leg))],
+ trace,
+ },
+ ],
+}
+// The durable review bundle: everything a person (or the watchdog, later) needs to look back at this run.
+const bundle = runDir(session)
+const reportPath = join(bundle, 'trace.json')
+;(report.engines[0] as Record<string, unknown>).runDir = bundle
+writeFileSync(reportPath, JSON.stringify(report, null, 2))
+if (workspace) {
+ for (const rel of ['notes', 'out', 'tools', '.e2e', '.codex', '.mcp.json']) {
+ const src = join(workspace, rel)
+ if (existsSync(src)) cpSync(src, join(bundle, 'workspace', rel), { recursive: true })
+ }
+}
+
+for (const step of trace) {
+ console.log(`\n▶ ${step.step}`)
+ console.log(` request: ${JSON.stringify(step.request)}`)
+ console.log(` reply: ${JSON.stringify(step.reply)}`)
+}
+const stuck = firstStuck(legs)
+console.log(`\n=== ${session} ===`)
+console.log(stuck ? `STUCK AT ${stuck.leg} / ${stuck.check} (${stuck.status})` : error ? `STOPPED: ${error}` : 'clean')
+console.log(`Wrote trace → ${reportPath}`)
+
+// Everything to look back at: the daemon's own lines for this agent, the watchdog's raw events, and
+// a manifest that says how to reopen each piece.
+try {
+ const daemonLog = join(env.ADAPTER_DATA_DIR, 'harness.log')
+ if (existsSync(daemonLog) && agentId) {
+ const short = agentId.slice(0, 8)
+ const lines = readFileSync(daemonLog, 'utf8').split('\n').filter((l) => l.includes(agentId!) || l.includes(short))
+ // The [turn]/[hooks]/[recap] lines are keyed by the ENGINE session id; find it from the attach line.
+ const sid = /session=([0-9a-f]{8})/.exec(lines.join('\n'))?.[1]
+ const all = sid ? readFileSync(daemonLog, 'utf8').split('\n').filter((l) => l.includes(agentId!) || l.includes(short) || l.includes(sid)) : lines
+ writeFileSync(join(bundle, 'daemon.log'), all.join('\n') + '\n')
+ }
+} catch (err) {
+ console.error(`[grid-e2e] daemon log slice: ${(err as Error).message}`)
+}
+writeFileSync(join(bundle, 'manifest.json'), JSON.stringify({
+ session,
+ engine,
+ version,
+ agentId,
+ pane,
+ workspace,
+ gridModel,
+ reportPath,
+ // Filled by the reviewer's runner (autonomous-grid-cli e2e/run.sh): { model, sessionID, reopen }.
+ watchdog: null,
+ files: { trace: 'trace.json', daemonLog: 'daemon.log', watchdogEvents: 'watchdog.jsonl', watchdogReview: 'watchdog.md', workspace: 'workspace/', evidence: 'evidence/' },
+ generatedAt: new Date().toISOString(),
+}, null, 2))
+console.log(`Review bundle → ${bundle}`)
+
+// The run is over; the agent this run made can go. Kept only on request — for a person, or for the
+// reviewer to type into the pane (E2E_KEEP_AGENT=1; delete it in the app afterwards).
+if (created && agentId && process.env.E2E_KEEP_AGENT !== '1') {
+ try {
+ await driver.deleteAgent(agentId)
+ console.log(`Deleted agent ${agentId}${pane ? ` (pane ${pane})` : ''}`)
+ } catch (err) {
+ console.log(`Could not delete agent ${agentId}: ${(err as Error).message} — delete it in the app`)
+ }
+} else if (created && agentId) {
+ console.log(`Kept agent ${agentId}${pane ? ` in pane ${pane}` : ''} (E2E_KEEP_AGENT=1)`)
+}
+// Take the global codex entry back out. Not when the agent is being kept: someone who asked to keep
+// the pane wants to type into it, and an agent whose MCP server has been unregistered is not the
+// agent the run left behind.
+if (registeredCodexMcp && process.env.E2E_KEEP_AGENT !== '1') {
+ console.log(removeCodexMcp() ? 'Unregistered the codex MCP server' : 'Could not unregister the codex MCP server — remove it with `codex mcp remove e2e_calc`')
+}
+driver.close()
+return { session, reportPath, runDir: bundle, stuck, error }
+}
+
+// `npx tsx src/e2e/runGridSwitchTrace.ts` still works; `harness e2e grid-switch` is the same call.
+if (process.argv[1] && /runGridSwitchTrace\.(ts|js)$/.test(process.argv[1])) {
+ runGridSwitch().then((r) => process.exit(r.error || r.stuck ? 1 : 0))
+}
diff --git a/cli/src/e2e/runMatrix.ts b/cli/src/e2e/runMatrix.ts
new file mode 100644
index 000000000..d575be72e
--- /dev/null
+++ b/cli/src/e2e/runMatrix.ts
@@ -0,0 +1,144 @@
+/**
+ * runMatrix — CLI entry: the cronjob's action. Reads current vs latest for each engine, plans the
+ * matrix runs, emits namespaced audit entries (argv + login/grid NOTIFYs + the planned smoke
+ * checks per leg) and writes a report. The live journey is `harness e2e grid-switch`; the reviewer
+ * (opencode `e2e-watchdog`) lives with the relay in autonomous-grid-cli and reads the run afterwards.
+ *
+ * Env:
+ * E2E_VERSION_SOURCE=mock|npm (default mock until the real reader is validated)
+ * E2E_ENGINES=codex,claude engines under test
+ *
+ * Dry-run: proves trigger + argv + naming + notify + the check plan with zero network/accounts;
+ * every check is `not-run`. The live journey with real checks is `runGridSwitchTrace.ts`.
+ *
+ * Usage: npx tsx src/e2e/runMatrix.ts
+ */
+import { writeFileSync } from 'node:fs'
+import { join } from 'node:path'
+import { execFile } from 'node:child_process'
+import { promisify } from 'node:util'
+import type { AgentEngine } from '../engines/types.js'
+import { versionSourceFromEnv } from './versionSource.js'
+import { InstalledCurrentVersion, ENGINE_BIN } from './currentVersion.js'
+import { planMatrixRuns } from './trigger.js'
+import { buildMatrixEntry, type EngineStatus } from './matrix.js'
+import { outDir } from './artifacts.js'
+
+const execFileP = promisify(execFile)
+
+const ENGINES: AgentEngine[] = (process.env.E2E_ENGINES ?? 'codex,claude')
+ .split(',')
+ .map((e) => e.trim() as AgentEngine)
+ .filter(Boolean)
+
+/**
+ * Codex's own answer, not the presence of a file.
+ *
+ * `~/.codex/auth.json` existing proved nothing: on a machine whose codex was pointed at a grid, that
+ * file holds `auth_mode: "apikey"` with a grid token, `codex login status` says "Logged in using an
+ * API key - open_5fb***_grid", and every own-account turn comes back 401 from api.openai.com. A
+ * matrix that called that "logged in" sent the run off to fail for a reason it had already been
+ * told. The mode is carried out with the answer so the report can name it.
+ */
+async function codexLogin(): Promise<{ loggedIn: boolean; mode: string | null }> {
+ try {
+ const { stdout } = await execFileP('codex', ['login', 'status'], { timeout: 15_000 })
+ const line = stdout.trim().split('\n')[0] ?? ''
+ if (!/logged in/i.test(line)) return { loggedIn: false, mode: line || null }
+ // An API key here is a key for api.openai.com. A grid token (`…_grid`) is not one, however much
+ // it looks like a login — it is the one case that passes every offline check and still 401s.
+ const apiKey = /api key/i.test(line)
+ const gridKey = apiKey && /_grid\b/i.test(line)
+ return { loggedIn: !gridKey, mode: gridKey ? `grid token in apikey mode (${line})` : apiKey ? 'apikey' : 'chatgpt' }
+ } catch (err) {
+ return { loggedIn: false, mode: `codex login status failed: ${(err as Error).message.split('\n')[0]}` }
+ }
+}
+
+async function claudeLoggedIn(): Promise<boolean> {
+ try {
+ const { stdout } = await execFileP('claude', ['auth', 'status'], { timeout: 15_000 })
+ return (JSON.parse(stdout) as { loggedIn?: boolean }).loggedIn === true
+ } catch {
+ return false
+ }
+}
+
+function hasBinary(bin: string): boolean {
+ // Hard to verify PATH without a shell tool here; report presence by the binary name only.
+ return !!bin
+}
+
+async function statusOf(engine: AgentEngine): Promise<EngineStatus> {
+ const bin = ENGINE_BIN[engine] ?? engine
+ const binary = hasBinary(bin)
+ let login = false
+ let note: string | null = null
+ if (binary) {
+ switch (engine) {
+ case 'codex': {
+ const codex = await codexLogin()
+ login = codex.loggedIn
+ note = codex.mode
+ break
+ }
+ case 'claude':
+ // Not a file check: on macOS Claude Code keeps its login in the Keychain, so the only honest
+ // answer is the tool's own (`claude auth status` → { loggedIn }).
+ login = await claudeLoggedIn()
+ break
+ default:
+ login = false
+ }
+ }
+ // grid status filled by caller when it resolves
+ return { binary, login, grid: true, ...(note ? { loginNote: note } : {}) }
+}
+
+async function main(): Promise<number> {
+ const source = versionSourceFromEnv()
+ const current = new InstalledCurrentVersion()
+ const plans = await planMatrixRuns(source, current, ENGINES)
+
+ const toRun = plans.filter((p) => p.plan)
+ console.log('=== trigger plan (current vs latest) ===')
+ for (const p of plans) {
+ console.log(` ${p.engine.padEnd(8)} current=${String(p.current).padEnd(12)} latest=${String(p.latest).padEnd(12)} reason=${p.reason} run=${p.plan}`)
+ }
+ if (toRun.length === 0) {
+ console.log('Nothing to run — all engines at latest.')
+ return 0
+ }
+
+ const entries = []
+ for (const plan of toRun) {
+ const entry = await buildMatrixEntry(plan, {
+ status: statusOf,
+ grid: async () => null,
+ })
+ entries.push(entry)
+ }
+
+ const report = { generatedAt: new Date().toISOString(), mode: 'dry-run', engines: entries }
+ const reportPath = join(outDir(), `report-${Date.now()}.json`)
+ writeFileSync(reportPath, JSON.stringify(report, null, 2))
+ console.log(`\nWrote report → ${reportPath}`)
+
+ for (const e of entries) {
+ console.log(`\n▶ ${e.session}`)
+ console.log(` status: ${JSON.stringify(e.status)}`)
+ for (const n of e.notify) console.log(` ⚠ NOTIFY: ${n}`)
+ if ('error' in e.gridSpawn) {
+ console.log(` gridSpawn ERROR: ${e.gridSpawn.error} — ${e.gridSpawn.detail}`)
+ } else {
+ console.log(` grid command: ${e.gridSpawn.command.join(' ')}`)
+ console.log(` grid env: ${e.gridSpawn.envKeys.join(', ') || '(none)'}`)
+ console.log(` clearEnv: ${e.gridSpawn.clearEnv.join(', ') || '(none)'}`)
+ }
+ console.log(` checks: ${e.legs.map((l) => `${l.leg}[${l.checks.map((c) => c.id).join(',')}]`).join(' → ')} (dry-run: not-run)`)
+ }
+
+ return 0
+}
+
+main().then((code) => process.exit(code))
diff --git a/cli/src/e2e/sessionName.ts b/cli/src/e2e/sessionName.ts
new file mode 100644
index 000000000..ab3902767
--- /dev/null
+++ b/cli/src/e2e/sessionName.ts
@@ -0,0 +1,40 @@
+/**
+ * One namespaced name per matrix run, stamped identically on the trace, the watchdog session and
+ * the report, so every artifact of a run is grep-able and the audit answers: what version of which
+ * coding-agent, which test, which grid model, when.
+ *
+ * Convention: `<engine>@<version>-><testcase>@<grid-model>--<timestamp>`
+ * e.g. `[email protected]>grid-switch@grid-gpt-5-mini--20260921T1805Z`
+ */
+export function sessionName(opts: {
+ engine: string
+ version: string
+ testcase: string
+ gridModel?: string
+ at?: Date
+}): string {
+ const model = opts.gridModel && opts.gridModel.trim() ? sanitize(opts.gridModel) : 'none'
+ const ts = (opts.at ?? new Date())
+ .toISOString()
+ .replace(/[-:]/g, '')
+ .replace(/\.\d{3}Z$/, 'Z')
+ return `${opts.engine}@${sanitize(opts.version)}->${sanitize(opts.testcase)}@${model}--${ts}`
+}
+
+function sanitize(v: string): string {
+ return v.replace(/[^A-Za-z0-9._@-]/g, '-')
+}
+
+/**
+ * The agent's working folder, named after the session but with nothing a tool might reinterpret.
+ *
+ * The session name carries `->` and `@` (readable, grep-able, and fine as a bundle folder), and it
+ * used to be the agent's cwd too. claude 2.1.274 turns the `->` in that path into `-` when it builds a
+ * Write path, so its files landed in a sibling folder that does not exist for anyone else — every
+ * write step failed as "out/hello-1.txt does not exist" with the file sitting next door (grid-dev,
+ * 2026-09-25; 2.1.273 did not). A person's project folder does not look like that, so the agent's
+ * does not either: letters, digits, `.`, `_` and `-` only.
+ */
+export function workspaceDirName(session: string): string {
+ return session.replace(/->/g, '_to_').replace(/[^A-Za-z0-9._-]/g, '-')
+}
diff --git a/cli/src/e2e/sessionTools.ts b/cli/src/e2e/sessionTools.ts
new file mode 100644
index 000000000..697dcc064
--- /dev/null
+++ b/cli/src/e2e/sessionTools.ts
@@ -0,0 +1,218 @@
+/**
+ * sessionTools — which of the engine's own tools each step went through, and which model answered
+ * it, read afterwards from the engine's session file.
+ *
+ * A step passes on its proof (a log line, the file on disk, the read token — smokeChecks.ts); this
+ * says HOW the engine got there, which is the other half of "does the tool still work on the grid":
+ * claude may read with `Read` or with `Bash cat`, codex writes with `apply_patch` and, in code mode,
+ * reaches everything through `exec`. Every step's prompt carries its `(ref-…)` tag, so the tool
+ * calls between one tagged prompt and the next belong to that step.
+ *
+ * claude ~/.claude/projects/<cwd with every non-alphanumeric as "-">/*.jsonl
+ * user message text holds the ref · assistant content[] `tool_use` → name
+ * codex ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl (the one holding the refs)
+ * payload `message` (role user) holds the ref · `function_call` / `custom_tool_call` →
+ * name, and for code mode's `exec` the `tools.<name>(` calls inside its JavaScript
+ */
+import { readdirSync, readFileSync, statSync } from 'node:fs'
+import { homedir } from 'node:os'
+import { join } from 'node:path'
+
+export interface StepUse {
+ /** The engine's tools this step called, in order, consecutive repeats folded. */
+ tools: string[]
+ /** The models that answered this step — claude writes it on every reply, codex on every turn.
+ * How "claude never ran on Fable" is checked rather than assumed. */
+ models: string[]
+}
+
+/** ref → what the engine used for that step. */
+export function useByRef(engine: string, cwd: string, refs: readonly string[], home = homedir()): Record<string, StepUse> {
+ const out: Record<string, StepUse> = {}
+ if (!refs.length) return out
+ const files = engine === 'claude' ? claudeSessionFiles(cwd, home) : engine === 'codex' ? codexSessionFiles(refs, home) : []
+ for (const file of files) {
+ let current: string | null = null
+ // codex writes a turn's `turn_context` (its model) BEFORE the user message that starts the turn,
+ // so its model belongs to the NEXT ref; claude writes the model on each reply, after the ref.
+ let turnModel: string | null = null
+ const addModel = (use: StepUse, model: string | null): void => { if (model && !use.models.includes(model)) use.models.push(model) }
+ for (const line of readText(file).split('\n')) {
+ let rec: Record<string, unknown>
+ try { rec = JSON.parse(line) as Record<string, unknown> } catch { continue }
+ const model = modelOf(engine, rec)
+ if (engine === 'codex' && model) { turnModel = model; continue }
+ const found = userRef(engine, rec, refs)
+ if (found) {
+ current = found
+ out[current] ??= { tools: [], models: [] }
+ if (engine === 'codex') addModel(out[current], turnModel)
+ continue
+ }
+ if (!current) continue
+ const use = out[current]
+ for (const tool of toolCalls(engine, rec)) if (use.tools[use.tools.length - 1] !== tool) use.tools.push(tool)
+ if (engine === 'claude') addModel(use, model)
+ }
+ }
+ return out
+}
+
+function modelOf(engine: string, rec: Record<string, unknown>): string | null {
+ if (engine === 'claude') {
+ const m = rec.type === 'assistant' ? (rec.message as { model?: unknown } | undefined)?.model : undefined
+ return typeof m === 'string' && m !== '<synthetic>' ? m : null
+ }
+ // codex: a top-level `turn_context` record per turn, the model in its payload (measured on a
+ // 0.156.1 rollout — not a payload type, which is where the other records keep theirs).
+ const p = rec.payload as { model?: unknown } | undefined
+ return rec.type === 'turn_context' && typeof p?.model === 'string' ? p.model : null
+}
+
+function readText(file: string): string {
+ try { return readFileSync(file, 'utf8') } catch { return '' }
+}
+
+function textOf(content: unknown): string {
+ if (typeof content === 'string') return content
+ if (Array.isArray(content)) return content.map((c) => (c && typeof c === 'object' && typeof (c as { text?: unknown }).text === 'string' ? (c as { text: string }).text : '')).join(' ')
+ return ''
+}
+
+function userRef(engine: string, rec: Record<string, unknown>, refs: readonly string[]): string | null {
+ let text = ''
+ if (engine === 'claude' && rec.type === 'user') text = textOf((rec.message as { content?: unknown } | undefined)?.content)
+ if (engine === 'codex') {
+ const p = rec.payload as { type?: string; role?: string; content?: unknown } | undefined
+ if (p?.type === 'message' && p.role === 'user') text = textOf(p.content)
+ }
+ return text ? refs.find((r) => text.includes(r)) ?? null : null
+}
+
+function toolCalls(engine: string, rec: Record<string, unknown>): string[] {
+ if (engine === 'claude') {
+ if (rec.type !== 'assistant') return []
+ const content = (rec.message as { content?: unknown } | undefined)?.content
+ return Array.isArray(content)
+ ? content.filter((c) => c && typeof c === 'object' && (c as { type?: unknown }).type === 'tool_use').map((c) => String((c as { name?: unknown }).name))
+ : []
+ }
+ const p = rec.payload as { type?: string; name?: string; namespace?: string; input?: unknown; arguments?: unknown } | undefined
+ if (!p?.type) return []
+ if (p.type === 'function_call' || p.type === 'custom_tool_call') {
+ const name = codexCallName(p)
+ if (name === 'exec' && typeof p.input === 'string') {
+ // Code mode: one `exec` of JavaScript that calls the real tools as `tools.<name>(…)`.
+ const inner = [...p.input.matchAll(/tools\.([A-Za-z0-9_]+)\s*\(/g)].map((m) => `exec→${m[1]}`)
+ return inner.length ? inner : ['exec']
+ }
+ return [name]
+ }
+ if (p.type === 'local_shell_call') return ['local_shell']
+ return []
+}
+
+/** A namespaced call arrives as name `sub` + namespace `mcp__e2e_calc` (codex 0.156.1 on the grid). */
+function codexCallName(p: { name?: string; namespace?: string }): string {
+ const ns = p.namespace && p.name && !p.name.startsWith(p.namespace) ? p.namespace : ''
+ return ns ? `${ns}${ns.endsWith('__') ? '' : '__'}${p.name}` : String(p.name)
+}
+
+export interface ToolResult {
+ /** `tool` was called after this step's prompt. */
+ called: boolean
+ /** One of those calls came back with results: not an error, and a link in what it returned. */
+ answered: boolean
+ /** What the first call returned, cut short — the probe quotes it when the step is not proven. */
+ output?: string
+}
+
+/**
+ * Whether `tool` was called for the step tagged `ref` and came back with results — the proof for a
+ * tool whose server is not on this machine (the harness's web search runs on the grid), so its own
+ * log cannot be read the way `e2e_calc`'s is. "Results" is a link in what the tool returned: a web
+ * search that answered has one, and an error or an approval refusal ("MCP tool call requires
+ * approval") does not.
+ *
+ * claude assistant `tool_use` { id, name } → user `tool_result` { tool_use_id, is_error, content }
+ * codex `function_call` { name, namespace, call_id } → `function_call_output` { call_id, output }
+ */
+export function toolResult(engine: string, cwd: string, ref: string, tool: string, home = homedir()): ToolResult {
+ const files = engine === 'claude' ? claudeSessionFiles(cwd, home) : engine === 'codex' ? codexSessionFiles([ref], home) : []
+ const calls = new Set<string>()
+ const outputs: string[] = []
+ let errored = false
+ for (const file of files) {
+ let inStep = false
+ for (const line of readText(file).split('\n')) {
+ let rec: Record<string, unknown>
+ try { rec = JSON.parse(line) as Record<string, unknown> } catch { continue }
+ const found = userRef(engine, rec, [ref])
+ if (found) { inStep = true; continue }
+ if (!inStep) continue
+ // The next step's prompt — another `(ref-…)` — ends this one. Not any user line: both engines
+ // also record context and hook text in the user role, in the middle of a turn.
+ if (isOtherStep(engine, rec)) { inStep = false; continue }
+ if (engine === 'claude') {
+ const content = (rec.message as { content?: unknown } | undefined)?.content
+ if (!Array.isArray(content)) continue
+ for (const c of content as Array<Record<string, unknown>>) {
+ if (rec.type === 'assistant' && c?.type === 'tool_use' && c.name === tool) calls.add(String(c.id))
+ if (rec.type === 'user' && c?.type === 'tool_result' && calls.has(String(c.tool_use_id))) {
+ outputs.push(textOf(c.content))
+ if (c.is_error === true) errored = true
+ }
+ }
+ } else {
+ const p = rec.payload as { type?: string; name?: string; namespace?: string; call_id?: string; output?: unknown } | undefined
+ if (p?.type === 'function_call' && codexCallName(p) === tool && p.call_id) calls.add(p.call_id)
+ if (p?.type === 'function_call_output' && p.call_id && calls.has(p.call_id)) outputs.push(textOf(p.output))
+ }
+ }
+ }
+ const answered = !errored && outputs.some((o) => /https?:\/\//.test(o))
+ return { called: calls.size > 0, answered, ...(outputs.length ? { output: outputs[0].replace(/\s+/g, ' ').slice(0, 160) } : {}) }
+}
+
+/** A user line carrying a step's `(ref-…)` tag (paneProbe.ts checkRef) — the probe's next prompt. */
+function isOtherStep(engine: string, rec: Record<string, unknown>): boolean {
+ let text = ''
+ if (engine === 'claude' && rec.type === 'user') text = textOf((rec.message as { content?: unknown } | undefined)?.content)
+ if (engine === 'codex') {
+ const p = rec.payload as { type?: string; role?: string; content?: unknown } | undefined
+ if (p?.type === 'message' && p.role === 'user') text = textOf(p.content)
+ }
+ return /\(ref-[0-9a-z]{6}\)/.test(text)
+}
+
+function claudeSessionFiles(cwd: string, home: string): string[] {
+ const dir = join(home, '.claude', 'projects', cwd.replace(/[^A-Za-z0-9]/g, '-'))
+ try {
+ return readdirSync(dir).filter((f) => f.endsWith('.jsonl')).map((f) => join(dir, f))
+ } catch {
+ return []
+ }
+}
+
+/** The rollout files of the last day that mention one of this run's refs. */
+function codexSessionFiles(refs: readonly string[], home: string): string[] {
+ const root = join(home, '.codex', 'sessions')
+ const since = Date.now() - 24 * 60 * 60 * 1000
+ const hits: string[] = []
+ const walk = (dir: string, depth: number): void => {
+ let entries: string[]
+ try { entries = readdirSync(dir) } catch { return }
+ for (const e of entries) {
+ const p = join(dir, e)
+ let st
+ try { st = statSync(p) } catch { continue }
+ if (st.isDirectory() && depth < 3) walk(p, depth + 1)
+ else if (st.isFile() && e.startsWith('rollout-') && e.endsWith('.jsonl') && st.mtimeMs >= since) {
+ const text = readText(p)
+ if (refs.some((r) => text.includes(r))) hits.push(p)
+ }
+ }
+ }
+ walk(root, 0)
+ return hits
+}
diff --git a/cli/src/e2e/smokeChecks.ts b/cli/src/e2e/smokeChecks.ts
new file mode 100644
index 000000000..7d171ae75
--- /dev/null
+++ b/cli/src/e2e/smokeChecks.ts
@@ -0,0 +1,183 @@
+/**
+ * smokeChecks — the scenario: what a person does with a coding tool, on its own account and then on
+ * a grid model in the same session: read a file, create one, edit one, use an MCP server. Four
+ * plain requests per side — worded the way a person types them, no reply formats.
+ *
+ * Every step is proven by something the model cannot produce by talking:
+ *
+ * read the file holds a token made fresh for this run (`workspace.ts`), never written anywhere
+ * else — not in the prompt, not in a log, different on each side. It showing up on the
+ * pane after the question means the file was read; nobody guesses `kiwi-4821-tulip`
+ * write the file exists on disk with exactly the requested text
+ * edit the file on disk changed the requested word, and only that
+ * mcp own account: the workspace's MCP server logs the call for EXACTLY these numbers
+ * (.e2e/calc-mcp.log)
+ * web grid: the web search the harness gives every agent on a grid (`mcp__harness__web_search`,
+ * gridWebMcp.ts) was called and came back with results — read from the engine's session
+ * file (sessionTools.ts `toolResult`), since that server runs on the grid, not here
+ *
+ * So a model that answers from its head, or claims to have done something it did not, fails the
+ * step: the reply is never the proof. Each side uses its own files and numbers (…-1 / …-2), so the
+ * grid side cannot pass on what the first side left behind.
+ *
+ * No shell step (dropped 2026-10-06): the file steps already go through the engine's shell where the
+ * engine works that way (codex reads and writes through `exec_command`), and a script of our own
+ * proved nothing a person's file work does not.
+ *
+ * No delete step either (dropped the same day it was added): codex counts `rm -f` as dangerous and
+ * asks for approval (codex-rs shell-command is_dangerous_command.rs), so the step passed or failed on
+ * whether the model happened to type `-f`, not on the switch.
+ *
+ * Memory across the switch is deliberately not a step (dropped 2026-09-24): its only honest proof
+ * would be a fact that exists nowhere but the conversation, and the question is whether the tools
+ * work on the grid.
+ */
+
+export type Leg = 'subscription' | 'grid'
+
+/** The journey: the tool's own account, then the grid in the same session. */
+export const LEGS: readonly Leg[] = ['subscription', 'grid']
+
+/** Which log a step must leave a line in: the MCP server's (`workspace.ts`). */
+export type LogKind = 'mcp'
+
+export type CheckId = 'read' | 'write' | 'edit' | 'mcp' | 'web'
+
+/** The harness's web search on a grid, as the engines name it (`harnessWebTools.ts`). */
+export const WEB_SEARCH_TOOL = 'mcp__harness__web_search'
+
+export interface SmokeCheck {
+ id: CheckId
+ /** Typed into the agent's pane, verbatim. Plain, engine-agnostic wording (codex and claude). */
+ prompt: string
+ /** mcp: the log that must gain a line containing `logPattern`. */
+ log?: LogKind
+ logPattern?: string
+ /** read: a workspace file whose (per-run, random) content must show up after the prompt. */
+ answerFromFile?: string
+ /** write / edit: what the workspace file must look like when the step is done. */
+ file?: { path: string; equals?: string; contains?: string; lacks?: string }
+ /** web: the engine's tool that must have been called for this step and come back with results. */
+ sessionTool?: string
+ /** What a miss usually means — the watchdog's first hypothesis, not its conclusion. */
+ onMiss: string
+}
+
+const MISS = {
+ read: 'the file was not read: the read tool (or the shell read) fails on this leg — the answer never showed the file\'s token',
+ write: 'the file was not created with the requested text: the write tool fails on this leg (codex: apply_patch Add File; claude: Write)',
+ edit: 'the file was not changed as asked: the edit tool fails on this leg (codex: apply_patch Update File; claude: Edit)',
+ mcp: 'the MCP server was not called: the respawn dropped the MCP config, the resumed session did not reload servers, or it answered without calling it (no log line)',
+ web: 'the web search was not called, or came back without results: the harness web MCP was not given to the engine on the grid, the engine could not read the tool (a namespace tool on llama.cpp / LM Studio), or it answered from its head',
+}
+
+/**
+ * Plain requests, the way a person types them, and the same for every engine. A step passes on its
+ * RESULT — the file really read, written, edited; the MCP server really called — never on
+ * which of the engine's file tools got it there. Which tools it used (claude `Read` or
+ * `Bash cat`, codex `apply_patch` or a shell write) is read back from the session file and REPORTED
+ * (sessionTools.ts, the message's Tools line), never judged: an engine is free to reach a result its
+ * own way, and a person asking for it would be.
+ *
+ * That report is still worth reading. It is how it was seen, on the first real runs, that claude
+ * does file work through Bash unless asked otherwise, and that codex on a grid model writes files
+ * through the shell where on its own account it uses apply_patch (a grid model is not in codex's
+ * model catalog, so codex does not offer it apply_patch).
+ */
+const files = (n: 1 | 2): SmokeCheck[] => [
+ { id: 'read', prompt: `Read notes/info-${n}.txt and tell me what it says.`, answerFromFile: `notes/info-${n}.txt`, onMiss: MISS.read },
+ { id: 'write', prompt: `Create the file out/hello-${n}.txt with this text: hello from step ${n}`, file: { path: `out/hello-${n}.txt`, equals: `hello from step ${n}` }, onMiss: MISS.write },
+ { id: 'edit', prompt: `In notes/todo-${n}.txt, change pending to done.`, file: { path: `notes/todo-${n}.txt`, contains: 'status: done', lacks: 'pending' }, onMiss: MISS.edit },
+]
+
+/**
+ * The three file requests on each side (own files per side, so nothing carries over), then an MCP
+ * call: the workspace's own server on the tool's account, the harness's web search on the grid —
+ * the MCP a person on a grid actually has, and the one codex 0.160's namespace tools broke on
+ * llama.cpp (autonomous-grid-cli#41). The own account has no harness web MCP (it is given only on a
+ * grid, gridLaunch.ts), so that side keeps `e2e_calc`.
+ */
+export const SCENARIO: Record<Leg, readonly SmokeCheck[]> = {
+ subscription: [
+ ...files(1),
+ { id: 'mcp', prompt: 'Use the MCP server e2e_calc to add 30 and 12.', log: 'mcp', logPattern: 'add 30 12 = 42', onMiss: MISS.mcp },
+ ],
+ grid: [
+ ...files(2),
+ { id: 'web', prompt: 'Search the web for the latest stable version of Node.js and tell me the version number.', sessionTool: WEB_SEARCH_TOOL, onMiss: MISS.web },
+ ],
+}
+
+/** The scenario an engine is given — the same plain requests for every engine (see above). */
+export function scenarioFor(_engine: string): Record<Leg, readonly SmokeCheck[]> {
+ return SCENARIO
+}
+
+/** Every step of every leg, for callers that want the flat list (the skill, the tests). */
+export const SMOKE_CHECKS: readonly SmokeCheck[] = LEGS.flatMap((leg) => SCENARIO[leg])
+
+/**
+ * `no-quota`: the tool answered with its own "out of usage" message. Not a finding about the
+ * switch, and not a bug anyone should read an analysis of — the account simply could not run the
+ * test. It stops the journey, skips the reviewer, and the pipeline retries once the quota is back.
+ */
+export type CheckStatus = 'not-run' | 'ok' | 'stuck' | 'no-quota'
+
+/** One step's outcome on one leg — the unit the watchdog reads. */
+export interface CheckOutcome {
+ id: SmokeCheck['id']
+ status: CheckStatus
+ elapsedMs?: number
+ /** The pane's last lines after the step — evidence for the reviewer, present on ok AND stuck so legs can be diffed. */
+ tail?: string
+ /** Log lines the step added (tool / MCP) — the proof it went through the tool. Empty on a step that answered from memory. */
+ logLines?: string[]
+ /** Why it is not `ok` when the pane looked fine: the marker showed but the tool/MCP log gained no line. */
+ note?: string
+ /** Modals that landed DURING this step and were answered (claude's gateway notice) — a finding in itself. */
+ dialogs?: string[]
+ /** On `no-quota`: when the tool said it comes back, in its own words ("resets 6pm", "try again at …"). */
+ resets?: string
+ /** The engine's own tools this step went through, read from its session file (sessionTools.ts):
+ * `Read`, `Write`, `Edit`, `Bash`, `apply_patch`, `exec_command`, `mcp__e2e_calc__add`, … */
+ tools?: string[]
+ /** The models that answered this step (sessionTools.ts) — own account: the pinned one; grid: the grid's. */
+ models?: string[]
+ /** The `(ref-…)` tag this step's prompt carried — how its tools and models are found afterwards. */
+ ref?: string
+}
+
+export interface LegOutcome {
+ leg: Leg
+ checks: CheckOutcome[]
+ /** Startup dialogs the probe had to answer before the first step (`codex-update`, `codex-trust`, …). */
+ dialogs?: string[]
+}
+
+/** The dry-run shape: every leg, every step, nothing run yet. */
+export function plannedLegs(): LegOutcome[] {
+ return LEGS.map((leg) => ({
+ leg,
+ checks: SCENARIO[leg].map((c) => ({ id: c.id, status: 'not-run' as const })),
+ }))
+}
+
+/** The (leg, step) where the account ran out of usage, or null. Checked before "what stuck". */
+export function quotaHit(legs: readonly LegOutcome[]): { leg: Leg; check: SmokeCheck['id']; resets: string | null } | null {
+ for (const l of legs) {
+ for (const c of l.checks) {
+ if (c.status === 'no-quota') return { leg: l.leg, check: c.id, resets: c.resets ?? null }
+ }
+ }
+ return null
+}
+
+/** The first (leg, step) that did not pass — "what stuck", or null when the run is clean. */
+export function firstStuck(legs: readonly LegOutcome[]): { leg: Leg; check: SmokeCheck['id']; status: CheckStatus } | null {
+ for (const l of legs) {
+ for (const c of l.checks) {
+ if (c.status !== 'ok') return { leg: l.leg, check: c.id, status: c.status }
+ }
+ }
+ return null
+}
diff --git a/cli/src/e2e/trigger.ts b/cli/src/e2e/trigger.ts
new file mode 100644
index 000000000..9bd62ee03
--- /dev/null
+++ b/cli/src/e2e/trigger.ts
@@ -0,0 +1,74 @@
+/**
+ * trigger — decide which engines need a matrix run right now.
+ *
+ * The rule: run for an engine when its INSTALLED (current) version differs from the tool's LATEST
+ * released version. "Current" is read from the installed binary (`--version`/cli_version);
+ * "latest" comes from a pluggable `VersionSource` (mock during development, npm when validated).
+ *
+ * This is the cronjob's decision step, kept pure (no I/O beyond the two injected providers) so it
+ * is deterministic and unit-testable offline.
+ */
+import type { AgentEngine } from '../engines/types.js'
+import { type VersionSource } from './versionSource.js'
+
+export interface VersionProvider {
+ currentVersion(engine: AgentEngine): Promise<string | null>
+}
+
+export interface PlannedRun {
+ engine: AgentEngine
+ current: string | null
+ latest: string | null
+ /** Why a run is (or is not) planned — for the audit log + NOTIFY. */
+ reason: 'upgrade' | 'downgrade' | 'differs' | 'unknown-current' | 'no-latest' | 'unchanged'
+ plan: boolean
+}
+
+/** Compare current vs latest for `engines` and return the run plan for each. */
+export async function planMatrixRuns(
+ source: VersionSource,
+ currentProvider: VersionProvider,
+ engines: readonly AgentEngine[],
+): Promise<PlannedRun[]> {
+ const runs: PlannedRun[] = []
+ for (const engine of engines) {
+ const latest = await source.latestVersion(engine)
+ const current = await currentProvider.currentVersion(engine)
+
+ if (latest === null) {
+ runs.push({ engine, current, latest, reason: 'no-latest', plan: false })
+ continue
+ }
+ if (current === null) {
+ // Installed version unreadable but a latest is known: verify rather than assume.
+ runs.push({ engine, current, latest, reason: 'unknown-current', plan: true })
+ continue
+ }
+ const cmp = compareVersions(current, latest)
+ runs.push({
+ engine,
+ current,
+ latest,
+ reason: cmp === 0 ? 'unchanged' : cmp < 0 ? 'upgrade' : 'downgrade',
+ plan: cmp !== 0,
+ })
+ }
+ return runs
+}
+
+/**
+ * Naive dotted-numeric compare (0.155.0 < 0.155.1, 1.18.31 < 1.18.32). Handles prerelease-ish
+ * suffixes by numeric prefix only; good enough to notice "a newer/older stable is out".
+ */
+export function compareVersions(a: string, b: string): number {
+ const pa = (a.match(/\d+/g) ?? []).map(Number)
+ const pb = (b.match(/\d+/g) ?? []).map(Number)
+ const n = Math.max(pa.length, pb.length)
+ for (let i = 0; i < n; i++) {
+ const x = pa[i] ?? 0
+ const y = pb[i] ?? 0
+ if (x < y) return -1
+ if (x > y) return 1
+ }
+ return 0
+}
diff --git a/cli/src/e2e/versionSource.ts b/cli/src/e2e/versionSource.ts
new file mode 100644
index 000000000..798fdf592
--- /dev/null
+++ b/cli/src/e2e/versionSource.ts
@@ -0,0 +1,85 @@
+/**
+ * Version sources for the E2E trigger.
+ *
+ * The trigger runs a matrix run when the INSTALLED (current) version of a coding-agent tool
+ * differs from the tool's LATEST released version. "Latest" is read through a pluggable
+ * `VersionSource` so we can develop/test offline against a MOCK (no network, $0), then swap in a
+ * real implementation without changing the trigger logic.
+ */
+import type { AgentEngine } from '../engines/types.js'
+
+export interface VersionSource {
+ /** The latest released version string for `engine`, or null when it cannot be determined. */
+ latestVersion(engine: AgentEngine): Promise<string | null>
+}
+
+/** Fixture "latest" versions, keyed by engine. Defaults make the offline trigger readable. */
+const MOCK_LATEST: Partial<Record<AgentEngine, string>> = {
+ codex: '0.156.0',
+ claude: '2.1.279',
+}
+
+/**
+ * Offline source used during development and in CI. Lets the whole trigger + matrix decision run
+ * with no network and no accounts. Point at fixtures in a spec to exercise each branch.
+ */
+export class MockVersionSource implements VersionSource {
+ constructor(private readonly overrides: Partial<Record<AgentEngine, string>> = {}) {}
+
+ async latestVersion(engine: AgentEngine): Promise<string | null> {
+ return this.overrides[engine] ?? MOCK_LATEST[engine] ?? null
+ }
+}
+
+/**
+ * Real source: each engine's STABLE release channel on npm — and only that.
+ *
+ * "Stable" is not the same dist-tag everywhere, which is why this is a table rather than `/latest`
+ * for all (dist-tags read 2026-09-24):
+ *
+ * @openai/codex latest 0.156.1 ← stable alpha 0.158.0-alpha.8 (pre-releases)
+ * @anthropic-ai/claude-code stable 2.1.273 ← stable latest/next 2.1.281 (the fast channel)
+ *
+ * Reading `latest` for claude tested its fast channel, eight releases ahead of what Claude Code
+ * itself calls stable. And a version with a pre-release suffix (`-alpha.8`, `-beta.1`, `-rc.2`) is
+ * refused whatever tag it came from: a pipeline that must never test a nightly should not depend on
+ * a vendor never mis-tagging one.
+ */
+import { NPM_PACKAGE } from './npmPackages.js'
+
+export const STABLE_TAG: Partial<Record<AgentEngine, string>> = { codex: 'latest', claude: 'stable' }
+
+/** `0.156.1` yes; `0.158.0-alpha.8`, `2.1.0-beta` no. */
+export function isStableVersion(version: string): boolean {
+ return /^\d+\.\d+\.\d+$/.test(version)
+}
+
+export class NpmVersionSource implements VersionSource {
+ constructor(private readonly fetchImpl: typeof fetch = fetch) {}
+
+ async latestVersion(engine: AgentEngine): Promise<string | null> {
+ const pkg = NPM_PACKAGE[engine]
+ if (!pkg) return null
+ try {
+ // No `Accept: application/vnd.npm.install-v1+json` here, however natural it looks: the
+ // abbreviated-metadata type is offered for the PACKAGE document, and asking for it on the
+ // `/latest` dist-tag endpoint gets a 406 with an empty body. That failure is silent — the
+ // catch below turns it into `latest=null`, which the trigger reads as "nothing new", so the
+ // whole pipeline would sit quiet forever while looking perfectly healthy. Measured:
+ // curl -o /dev/null -w '%{http_code}' -H 'Accept: application/vnd.npm.install-v1+json' \
+ // https://registry.npmjs.org/@openai/codex/latest -> 406
+ // curl -s https://registry.npmjs.org/@openai/codex/latest | jq .version -> "0.156.1"
+ const res = await this.fetchImpl(`https://registry.npmjs.org/${pkg}/${STABLE_TAG[engine] ?? 'latest'}`)
+ if (!res.ok) return null
+ const body = (await res.json()) as { version?: string }
+ return body.version && isStableVersion(body.version) ? body.version : null
+ } catch {
+ return null
+ }
+ }
+}
+
+/** Pick the source from `E2E_VERSION_SOURCE` (default `mock` until the real one is validated). */
+export function versionSourceFromEnv(env: Record<string, string | undefined> = process.env): VersionSource {
+ return env.E2E_VERSION_SOURCE === 'npm' ? new NpmVersionSource() : new MockVersionSource()
+}
diff --git a/cli/src/e2e/workspace.ts b/cli/src/e2e/workspace.ts
new file mode 100644
index 000000000..04d60e244
--- /dev/null
+++ b/cli/src/e2e/workspace.ts
@@ -0,0 +1,338 @@
+/**
+ * workspace — the folder the test agent is created in, laid out like a person's real project:
+ * the notes the file steps read and edit, an MCP server registered for the engine
+ * (`e2e_calc`), and the log it writes to. The scenario (`smokeChecks.ts`) names them; the probe
+ * reads the files and the log.
+ *
+ * <cwd>/notes/info-N.txt read: a token made fresh for this run
+ * <cwd>/notes/todo-N.txt edit: `status: pending`
+ * <cwd>/tools/calc-mcp.mjs the MCP server
+ * <cwd>/.mcp.json claude: project-level MCP server
+ * ~/.codex/config.toml codex: registered with `codex mcp add` — see below
+ * <cwd>/.e2e/calc-mcp.log one line per MCP tools/call
+ *
+ * The two engines differ, and both shapes here were measured against the installed CLIs rather than
+ * assumed:
+ *
+ * * claude reads `<cwd>/.mcp.json` — `claude mcp list` in the workspace lists `e2e_calc` and a
+ * headless run really reaches the server. The agent runs in the app's default mode (`auto`), so
+ * the project's `.claude/settings.json` allows the steps' tools the way a person who answered
+ * "don't ask again" has them (`allowClaudeProjectTools`).
+ * * codex has NO project-level MCP config. A `<cwd>/.codex/config.toml` is never read (from inside
+ * such a workspace, `codex mcp list` does not list the server), so registration goes through
+ * codex's own `codex mcp add`, which writes `~/.codex/config.toml`, with its tools always allowed
+ * (`approveCodexMcpTools`) — and `removeCodexMcp` takes the entry out again when the run is over.
+ */
+import { mkdirSync, readFileSync, writeFileSync, existsSync, renameSync, realpathSync } from 'node:fs'
+import { execFileSync } from 'node:child_process'
+import { randomInt } from 'node:crypto'
+import { homedir } from 'node:os'
+import { join } from 'node:path'
+import type { LogKind } from './smokeChecks.js'
+
+// Written from this string, not copied from beside this file: the published CLI is ONE bundled
+// `cli.js` (build-bundle.mjs), and files next to the source never reach an installed machine. The
+// original under `workspace/` is the readable copy; the spec pins them equal.
+export const CALC_MCP_MJS = `#!/usr/bin/env node
+/**
+ * e2e_calc — the smallest MCP server that can prove "the tool still uses MCP after a switch".
+ *
+ * stdio transport, newline-delimited JSON-RPC, two tools: add(a, b) and sub(a, b). Every call is
+ * appended to the log file given as argv[2] (\`<ts> <tool> <a> <b> = <result>\`), so the e2e has hard
+ * evidence the model went THROUGH the server rather than doing the arithmetic in its head.
+ * No SDK on purpose: nothing to install in the scratch workspace the test agent runs in.
+ */
+import { appendFileSync } from 'node:fs'
+import { createInterface } from 'node:readline'
+
+const logFile = process.argv[2]
+const TOOLS = [
+ { name: 'add', description: 'Add two numbers: a + b', inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] } },
+ { name: 'sub', description: 'Subtract two numbers: a - b', inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] } },
+]
+
+function send(msg) {
+ process.stdout.write(JSON.stringify(msg) + '\\n')
+}
+
+function log(line) {
+ if (logFile) appendFileSync(logFile, \`\${new Date().toISOString()} \${line}\\n\`)
+}
+
+const rl = createInterface({ input: process.stdin })
+rl.on('line', (line) => {
+ if (!line.trim()) return
+ let req
+ try { req = JSON.parse(line) } catch { return }
+ const { id, method, params } = req
+ if (id === undefined) return // notification (notifications/initialized, …)
+ switch (method) {
+ case 'initialize':
+ send({ jsonrpc: '2.0', id, result: { protocolVersion: params?.protocolVersion ?? '2024-11-05', capabilities: { tools: {} }, serverInfo: { name: 'e2e_calc', version: '1.0.0' } } })
+ return
+ case 'ping':
+ send({ jsonrpc: '2.0', id, result: {} })
+ return
+ case 'tools/list':
+ send({ jsonrpc: '2.0', id, result: { tools: TOOLS } })
+ return
+ case 'tools/call': {
+ const name = params?.name
+ const a = Number(params?.arguments?.a)
+ const b = Number(params?.arguments?.b)
+ if ((name !== 'add' && name !== 'sub') || Number.isNaN(a) || Number.isNaN(b)) {
+ send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: \`error: unknown tool or bad arguments (\${name})\` }], isError: true } })
+ return
+ }
+ const result = name === 'add' ? a + b : a - b
+ log(\`\${name} \${a} \${b} = \${result}\`)
+ send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: String(result) }] } })
+ return
+ }
+ default:
+ send({ jsonrpc: '2.0', id, error: { code: -32601, message: \`method not found: \${method}\` } })
+ }
+})
+`
+
+export const MCP_SERVER_NAME = 'e2e_calc'
+/** The tools `CALC_MCP_MJS` serves. */
+export const CALC_MCP_TOOLS = ['add', 'sub'] as const
+
+/**
+ * What a person using claude in this project has long since answered "Yes, and don't ask again"
+ * to: the file tools, reading a file through the shell, the project's MCP server and
+ * the harness's web search on a grid — kept by claude in the project's `.claude/settings.json`
+ * (`permissions.allow`), with the `.mcp.json` server approved in the same file
+ * (`enabledMcpjsonServers`). Measured on claude 2.1.274 + Haiku in `auto`: without it a shell step
+ * stopped on "This command requires approval".
+ */
+export function allowClaudeProjectTools(cwd: string): void {
+ mkdirSync(join(cwd, '.claude'), { recursive: true })
+ const settings = {
+ permissions: {
+ allow: [
+ 'Read', 'Write', 'Edit',
+ 'Bash(cat:*)',
+ `mcp__${MCP_SERVER_NAME}`,
+ 'mcp__harness',
+ ],
+ },
+ enabledMcpjsonServers: [MCP_SERVER_NAME],
+ }
+ writeFileSync(join(cwd, '.claude', 'settings.json'), JSON.stringify(settings, null, 2) + '\n')
+}
+
+export function logPath(cwd: string, _kind: LogKind): string {
+ return join(cwd, '.e2e', 'calc-mcp.log')
+}
+
+export interface WorkspaceLayout {
+ mcpServer: string
+ /** Where this engine's MCP registration ended up, or null when there is none. */
+ mcpConfig: string | null
+ /** How it got there — or, when `mcpConfig` is null, why the MCP steps cannot pass. */
+ mcpNote: string
+}
+
+/**
+ * Register the workspace's MCP server with codex, through codex's own command.
+ *
+ * `remove` first so a stale entry from an interrupted run (pointing at a workspace that no longer
+ * exists) cannot survive as a server codex fails to start. Both calls are best-effort: a codex too
+ * old for `mcp add` must leave a run without MCP rather than without a run, and the caller reports
+ * that in the trace.
+ */
+export function registerCodexMcp(mcpServer: string, mcpLog: string, bin = 'codex'): { ok: true } | { ok: false; detail: string } {
+ const run = (args: string[]): string => execFileSync(bin, args, { encoding: 'utf8', timeout: 60_000, stdio: ['ignore', 'pipe', 'pipe'] })
+ try { run(['mcp', 'remove', MCP_SERVER_NAME]) } catch { /* not registered — the normal case */ }
+ try {
+ run(['mcp', 'add', MCP_SERVER_NAME, '--', 'node', mcpServer, mcpLog])
+ } catch (err) {
+ const e = err as { stderr?: string; message?: string }
+ return { ok: false, detail: (e.stderr || e.message || String(err)).trim().split('\n')[0] }
+ }
+ // Registered is not the same as visible: ask codex what it sees, so a run never starts believing
+ // in a server the engine will not load.
+ try {
+ if (!run(['mcp', 'list']).includes(MCP_SERVER_NAME)) return { ok: false, detail: `codex mcp add reported success but 'codex mcp list' does not show ${MCP_SERVER_NAME}` }
+ } catch { /* `mcp list` can fail on its own (a broken unrelated server); the add is what matters */ }
+ return { ok: true }
+}
+
+/**
+ * Mark the server's tools as always allowed, the entry codex writes itself when a person answers a
+ * tool's approval with "always allow" — how every MCP tool on a working developer's machine ends up
+ * (`[mcp_servers.<server>.tools.<tool>] approval_mode = "approve"`). Without it a tool with no
+ * `readOnlyHint` needs approval on every call (codex-rs/core/src/mcp_tool_call.rs
+ * `requires_mcp_tool_approval`), and in `auto` that approval is codex's reviewer, which asks for a
+ * model named `codex-auto-review` — one only OpenAI serves: on the grid it answered 503 and codex
+ * refused the call (openai/codex#24879). Nested under the server's table, so `codex mcp remove`
+ * takes it out with the server.
+ */
+export function approveCodexMcpTools(configPath: string, tools: readonly string[]): void {
+ const lines = tools.map((t) => `\n[mcp_servers.${MCP_SERVER_NAME}.tools.${t}]\napproval_mode = "approve"\n`).join('')
+ writeFileSync(configPath, lines, { flag: 'a' })
+}
+
+/** Take the entry back out — the run's own cleanup, so the next run starts from a clean config. */
+export function removeCodexMcp(bin = 'codex'): boolean {
+ try {
+ execFileSync(bin, ['mcp', 'remove', MCP_SERVER_NAME], { encoding: 'utf8', timeout: 60_000, stdio: ['ignore', 'pipe', 'pipe'] })
+ return true
+ } catch {
+ return false
+ }
+}
+
+/**
+ * Lay the workspace out for `engine`. Idempotent. Returns what was written, for the trace.
+ *
+ * `codexBin` exists so a test can watch the registration without running the real `codex mcp add`:
+ * that command writes the developer's own `~/.codex/config.toml`, and a unit test has no business
+ * leaving an entry there pointing at a temp folder it is about to delete.
+ */
+export function prepareWorkspace(cwd: string, engine: string, opts: { codexBin?: string; codexConfig?: string } = {}): WorkspaceLayout {
+ mkdirSync(join(cwd, 'tools'), { recursive: true })
+ mkdirSync(join(cwd, '.e2e'), { recursive: true })
+ writeFileSync(logPath(cwd, 'mcp'), '', { flag: 'a' })
+
+ // read / write / edit (smokeChecks.ts): one set of files per side, so the grid side can never pass
+ // (named `info-N.txt`, not `secret-N.txt`: gpt-6-luna refused outright to read a file called
+ // "secret" — "I can't provide the contents of a file named notes/secret-1.txt" — and a test that
+ // provokes a refusal measures the refusal, not the switch)
+ // on what the first side left. The token is made HERE, fresh for every run, and written nowhere
+ // but its file — not in a prompt, not in a log — so the only way to name it is to read it.
+ mkdirSync(join(cwd, 'notes'), { recursive: true })
+ mkdirSync(join(cwd, 'out'), { recursive: true })
+ for (const n of [1, 2]) {
+ writeFileSync(join(cwd, 'notes', `info-${n}.txt`), `${freshToken()}\n`)
+ writeFileSync(join(cwd, 'notes', `todo-${n}.txt`), `# todo ${n}\nstatus: pending\n`)
+ }
+
+ // The MCP server runs from the workspace too, so the engine's config points at a path that exists
+ // on every machine the e2e runs on.
+ const mcpServer = join(cwd, 'tools', 'calc-mcp.mjs')
+ writeFileSync(mcpServer, CALC_MCP_MJS)
+ const mcpLog = logPath(cwd, 'mcp')
+ let mcpConfig: string | null = null
+ let mcpNote = `${engine} has no MCP registration here: the MCP steps cannot pass`
+ if (engine === 'codex') {
+ const registered = registerCodexMcp(mcpServer, mcpLog, opts.codexBin ?? 'codex')
+ if (registered.ok) {
+ mcpConfig = opts.codexConfig ?? join(homedir(), '.codex', 'config.toml')
+ approveCodexMcpTools(mcpConfig, CALC_MCP_TOOLS)
+ mcpNote = `registered with \`codex mcp add\`, tools ${CALC_MCP_TOOLS.join('/')} always allowed (codex does not read a project-level .codex/config.toml)`
+ } else {
+ mcpNote = `codex mcp add failed: ${registered.detail}`
+ }
+ } else if (engine === 'claude') {
+ mcpConfig = join(cwd, '.mcp.json')
+ writeFileSync(mcpConfig, JSON.stringify({ mcpServers: { [MCP_SERVER_NAME]: { command: 'node', args: [mcpServer, mcpLog] } } }, null, 2) + '\n')
+ preApproveClaudeMcp(cwd, [MCP_SERVER_NAME])
+ allowClaudeProjectTools(cwd)
+ mcpNote = 'project .mcp.json; the steps\' tools always allowed in .claude/settings.json'
+ }
+ return { mcpServer, mcpConfig, mcpNote }
+}
+
+/**
+ * Claude Code loads a project's `.mcp.json` servers only once the person approved them; the answer
+ * is kept in `~/.claude.json` as `projects[<cwd>].enabledMcpjsonServers`. For a workspace this run
+ * made, that answer is the run's to give — same posture and same file handling as the daemon's
+ * `preTrustClaudeProject` (lib/claudeTrust.ts): only add, never touch a shape we did not expect.
+ *
+ * Measured against claude 2.1.x, this is NOT what carries the MCP steps, and the file says so rather
+ * than leaving the next reader to re-measure: claude rewrites `~/.claude.json` at startup and the
+ * key is gone afterwards (`claude mcp list` still says "Pending approval"). What actually reaches
+ * the server is the `full` permission mode the agent is created in. Kept because it costs one write
+ * and is the right answer if a future claude honours it; never relied on.
+ */
+export function preApproveClaudeMcp(cwd: string, names: string[], home = homedir()): 'approved' | 'already' | 'skipped' {
+ const file = join(home, '.claude.json')
+ if (!existsSync(file)) return 'skipped'
+ let config: unknown
+ try {
+ config = JSON.parse(readFileSync(file, 'utf8'))
+ } catch {
+ return 'skipped'
+ }
+ const isObj = (v: unknown): v is Record<string, unknown> => typeof v === 'object' && v !== null && !Array.isArray(v)
+ if (!isObj(config)) return 'skipped'
+ if (config.projects !== undefined && !isObj(config.projects)) return 'skipped'
+ const projects = (config.projects ?? {}) as Record<string, unknown>
+ const existing = projects[cwd]
+ if (existing !== undefined && !isObj(existing)) return 'skipped'
+ const current = Array.isArray(existing?.enabledMcpjsonServers) ? (existing!.enabledMcpjsonServers as unknown[]) : []
+ if (names.every((n) => current.includes(n))) return 'already'
+ projects[cwd] = { allowedTools: [], ...(existing ?? {}), enabledMcpjsonServers: [...current, ...names.filter((n) => !current.includes(n))] }
+ config.projects = projects
+ // Atomic, through a symlink (dotfile managers link ~/.claude.json), like the daemon does.
+ const target = realpathSync(file)
+ const tmp = `${target}.e2e-${process.pid}.tmp`
+ writeFileSync(tmp, JSON.stringify(config, null, 2), { mode: 0o600 })
+ renameSync(tmp, target)
+ return 'approved'
+}
+
+/**
+ * Accept Claude Code's Bypass Permissions warning ahead of time.
+ *
+ * `--dangerously-skip-permissions` does not go straight to the prompt the first time: an
+ * interactive claude draws a full-screen "WARNING: Claude Code running in Bypass Permissions mode"
+ * with **"No, exit" selected**, and on Enter it exits 1 — the pane becomes a shell and every check
+ * of the run is stuck with no engine behind it. (A headless `claude -p` never shows it, which is
+ * why the flag looks harmless until it runs in a pane.) The answer is remembered in `~/.claude.json`
+ * as `bypassPermissionsModeAccepted`, a key read out of claude 2.1.281's own binary rather than
+ * guessed — so the run gives it the same way a person would, once, before the agent starts.
+ *
+ * It is NOT sufficient on its own, and the next reader should not have to find that out in a failed
+ * run: on the pass that proved this, the flag was written (`accepted`) and the warning still came up
+ * — `paneProbe`'s `claude-bypass-accept` answered it, and the leg passed. So the dialog handler is
+ * what carries the run and this write is the belt: cheap, correct, and it may be what keeps the
+ * screen away on a machine that reads the flag earlier.
+ */
+export function preAcceptClaudeBypassMode(home = homedir()): 'accepted' | 'already' | 'skipped' {
+ const file = join(home, '.claude.json')
+ if (!existsSync(file)) return 'skipped'
+ let config: unknown
+ try {
+ config = JSON.parse(readFileSync(file, 'utf8'))
+ } catch {
+ return 'skipped'
+ }
+ if (typeof config !== 'object' || config === null || Array.isArray(config)) return 'skipped'
+ const conf = config as Record<string, unknown>
+ if (conf.bypassPermissionsModeAccepted === true) return 'already'
+ conf.bypassPermissionsModeAccepted = true
+ const target = realpathSync(file)
+ const tmp = `${target}.e2e-${process.pid}.tmp`
+ writeFileSync(tmp, JSON.stringify(conf, null, 2), { mode: 0o600 })
+ renameSync(tmp, target)
+ return 'accepted'
+}
+
+const WORDS = ['kiwi', 'tulip', 'cobalt', 'maple', 'otter', 'saffron', 'glacier', 'lantern', 'pepper', 'quartz', 'harbor', 'violet', 'ember', 'falcon', 'juniper', 'meadow']
+
+/** `kiwi-4821-tulip`: two words and four digits, ~2.5 million combinations, new every run. */
+export function freshToken(): string {
+ return `${WORDS[randomInt(WORDS.length)]}-${randomInt(1000, 10000)}-${WORDS[randomInt(WORDS.length)]}`
+}
+
+/** A workspace file's content, or null when it does not exist — what the probe checks write/edit/read against. */
+export function readWorkspaceFile(cwd: string, relPath: string): string | null {
+ try {
+ return readFileSync(join(cwd, relPath), 'utf8')
+ } catch {
+ return null
+ }
+}
+
+/** The log's lines right now (empty when it does not exist yet). */
+export function readLog(cwd: string, kind: LogKind): string[] {
+ try {
+ return readFileSync(logPath(cwd, kind), 'utf8').split('\n').filter(Boolean)
+ } catch {
+ return []
+ }
+}
diff --git a/cli/src/e2e/workspace/calc-mcp.mjs b/cli/src/e2e/workspace/calc-mcp.mjs
new file mode 100755
index 000000000..a6981ba82
--- /dev/null
+++ b/cli/src/e2e/workspace/calc-mcp.mjs
@@ -0,0 +1,60 @@
+#!/usr/bin/env node
+/**
+ * e2e_calc — the smallest MCP server that can prove "the tool still uses MCP after a switch".
+ *
+ * stdio transport, newline-delimited JSON-RPC, two tools: add(a, b) and sub(a, b). Every call is
+ * appended to the log file given as argv[2] (`<ts> <tool> <a> <b> = <result>`), so the e2e has hard
+ * evidence the model went THROUGH the server rather than doing the arithmetic in its head.
+ * No SDK on purpose: nothing to install in the scratch workspace the test agent runs in.
+ */
+import { appendFileSync } from 'node:fs'
+import { createInterface } from 'node:readline'
+
+const logFile = process.argv[2]
+const TOOLS = [
+ { name: 'add', description: 'Add two numbers: a + b', inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] } },
+ { name: 'sub', description: 'Subtract two numbers: a - b', inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] } },
+]
+
+function send(msg) {
+ process.stdout.write(JSON.stringify(msg) + '\n')
+}
+
+function log(line) {
+ if (logFile) appendFileSync(logFile, `${new Date().toISOString()} ${line}\n`)
+}
+
+const rl = createInterface({ input: process.stdin })
+rl.on('line', (line) => {
+ if (!line.trim()) return
+ let req
+ try { req = JSON.parse(line) } catch { return }
+ const { id, method, params } = req
+ if (id === undefined) return // notification (notifications/initialized, …)
+ switch (method) {
+ case 'initialize':
+ send({ jsonrpc: '2.0', id, result: { protocolVersion: params?.protocolVersion ?? '2024-11-05', capabilities: { tools: {} }, serverInfo: { name: 'e2e_calc', version: '1.0.0' } } })
+ return
+ case 'ping':
+ send({ jsonrpc: '2.0', id, result: {} })
+ return
+ case 'tools/list':
+ send({ jsonrpc: '2.0', id, result: { tools: TOOLS } })
+ return
+ case 'tools/call': {
+ const name = params?.name
+ const a = Number(params?.arguments?.a)
+ const b = Number(params?.arguments?.b)
+ if ((name !== 'add' && name !== 'sub') || Number.isNaN(a) || Number.isNaN(b)) {
+ send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: `error: unknown tool or bad arguments (${name})` }], isError: true } })
+ return
+ }
+ const result = name === 'add' ? a + b : a - b
+ log(`${name} ${a} ${b} = ${result}`)
+ send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: String(result) }] } })
+ return
+ }
+ default:
+ send({ jsonrpc: '2.0', id, error: { code: -32601, message: `method not found: ${method}` } })
+ }
+})
diff --git a/cli/src/lib/nativeProcessControl.spec.ts b/cli/src/lib/nativeProcessControl.spec.ts
index 2b4d618b8..351e631bd 100644
--- a/cli/src/lib/nativeProcessControl.spec.ts
+++ b/cli/src/lib/nativeProcessControl.spec.ts
@@ -8,7 +8,9 @@ import { NativeEvidenceBudget } from '../engines/kit/nativeEvidence.js'
import { controlDescriptors, nativeProcessControl, parseNativeProcessControl } from './nativeProcessControl.js'
import { env } from '../config/env.js'
-const host = vi.hoisted(() => ({ root: '', exec: vi.fn(), helper: vi.fn(), before: undefined as undefined | ((operation: string, path: string) => void) }))
+const host = vi.hoisted(() => ({ root: '', exec: vi.fn(), helper: vi.fn(), before: undefined as undefined | ((operation: string, path: string) => void),
+ // A filesystem lists a directory in its own order (a tmpfs, ext4's hash): CI listed task 99 before 42.
+ listDescending: false }))
vi.mock('node:os', async original => ({ ...await original<object>(), platform: () => process.platform }))
vi.mock('node:child_process', async original => ({ ...await original<object>(), execFile: host.exec }))
vi.mock('./nativeProcessImages.js', async original => ({ ...await original<object>(), bundledProcessImageHelper: host.helper }))
@@ -22,6 +24,16 @@ vi.mock('node:fs', async original => {
return Reflect.apply(fs[method], fs, [mapped(path), ...args])
}) as never
}
+ const opendir = result.opendirSync
+ result.opendirSync = ((path: unknown, ...args: unknown[]) => {
+ const directory = Reflect.apply(opendir, fs, [path, ...args]) as import('node:fs').Dir
+ if (!host.listDescending) return directory
+ const listed: import('node:fs').Dirent[] = []
+ for (let next = directory.readSync(); next; next = directory.readSync()) listed.push(next)
+ directory.closeSync()
+ listed.sort((a, b) => Number(b.name) - Number(a.name))
+ return { readSync: () => listed.shift() ?? null, closeSync: () => {} }
+ }) as never
return result
})
const platform = Object.getOwnPropertyDescriptor(process, 'platform')!
@@ -175,6 +187,15 @@ describe('joined Linux control proof', () => {
file(join(host.root, 'proc/45/stat'), `45 (exited) Z 42 ${Array(17).fill(0).join(' ')} 100\n`)
expect((await nativeProcessControl([42, 43, 44], 42, budget())).children).toEqual([43, 44])
})
+ it('gives the children in thread order, however the directory lists its threads', async () => {
+ processRow(43, 42); processRow(44, 42)
+ file(join(host.root, 'proc/42/task/42/children'), '43 ')
+ file(join(host.root, 'proc/42/task/99/children'), '44 ')
+ host.listDescending = true
+ try {
+ expect((await nativeProcessControl([42, 43, 44], 42, budget())).children).toEqual([43, 44])
+ } finally { host.listDescending = false }
+ })
it.each(['descriptor', 'generation', 'image', 'child', 'task'])('holds a changing %s in the joined observation', async mode => {
let calls = 0
host.before = (operation, path) => {
diff --git a/cli/src/lib/nativeProcessControl.ts b/cli/src/lib/nativeProcessControl.ts
index dda25e527..c88fa66e7 100644
--- a/cli/src/lib/nativeProcessControl.ts
+++ b/cli/src/lib/nativeProcessControl.ts
@@ -141,7 +141,9 @@ function linuxControl(pids: readonly number[], parent: number, budget: NativeEvi
if (!parent) return []
const tasks = entries(`/proc/${parent}/task`, 128), ids: number[] = []
if (!tasks.length) return nativeUnavailable('the native task pool is empty')
- for (const task of tasks) {
+ // In thread order, as /proc lists them: a directory read elsewhere (a test's tmpfs, a hashed ext4) lists in
+ // its own order, and the children came back in it (CI, 2026-10-20: [44, 43] for [43, 44]).
+ for (const task of [...tasks].sort((a, b) => Number(a) - Number(b))) {
const text = bytes(`/proc/${parent}/task/${task}/children`, 1024)
if (!/^(?:\d+\s+)*$/.test(text)) return nativeUnavailable('the native child pool is malformed')
for (const value of text.trim().split(/\s+/).filter(Boolean)) {
diff --git a/docs/e2e-findings-2026-09-22.md b/docs/e2e-findings-2026-09-22.md
new file mode 100644
index 000000000..dc568036c
--- /dev/null
+++ b/docs/e2e-findings-2026-09-22.md
@@ -0,0 +1,39 @@
+# E2E grid-switch — finding 2026-09-22: codex resume onto a grid fails after an MCP call
+
+**Symptom.** A codex 0.155.x session that has called an MCP tool on its own login, moved onto a grid
+model (any node: `Qwen/Qwen3.8-27B`, `DeepSeek-V4-Flash-0731`) and resumed, never answers again:
+`■ stream disconnected before completion: transient provider recovery exhausted`. A FRESH session
+on the same grid model answers fine — which is why "switching in the app works" and this only shows
+up in the real scenario (use the person's tool + MCP, then switch, keep talking).
+
+**Root cause (isolated with a curl replay, no harness involved).** The relay answers the resumed
+request with `event: response.failed` / `error.code: server_error "transient provider recovery
+exhausted"`. The replayed history contains two Responses item types codex emits when it loads a deferred
+MCP tool — `tool_search_call` / `tool_search_output`. Not new in 0.155: on by default since
+`rust-v0.150.0` (upstream #29486, 2026-06-22; features `tool_search` / `tool_search_always_defer_mcp_tools`
+are "Removed" = baked in). It surfaced today because the scenario (use an MCP tool on the vendor, THEN
+switch) had never been run — the e2e must run on a schedule, not only when a version changes. Removing ONLY those two
+items from `input` makes the same request complete (`RESUME_40+2` — the model even recalls the
+conversation). `reasoning` items (with `encrypted_content`) and `function_call` items are harmless.
+
+| replayed request | relay result |
+|-------------------------------------------|-----------------------------------|
+| as recorded (resume history) | `response.failed` server_error |
+| minus `tool_search_call/_output` | `completed`, answers + recalls |
+| minus `reasoning` only | `response.failed` |
+| minus `reasoning` + `tool_search_*` | `completed` |
+
+**Where to fix.** The grid relay, when forwarding a Responses request to a non-OpenAI provider:
+drop (or translate) `tool_search_call` / `tool_search_output` input items — they are discovery
+bookkeeping; the real call is the `function_call` that follows. Codex cannot be told to stop
+emitting them.
+
+**Repro.** `cli/src/e2e/runGridSwitchTrace.ts` (live, creates its own agent) reproduces it on every
+run at `grid / recall`; the isolated replay is a `POST <grid>/relay/v1/responses` with the recorded
+`input` minus the two item types (script kept in the run's scratch folder during the session).
+
+**Side notes from the same runs.**
+- codex 0.155.0 → 0.155.1 got upgraded on this Mac by run 3 (the probe's Enter landed on the
+ "Update now" dialog; now answered with Skip before any check).
+- The codex subscription hit its usage limit during these runs (`You've hit your usage limit`);
+ the subscription leg cannot run again until it resets.
diff --git a/docs/e2e-grid-matrix-plan.md b/docs/e2e-grid-matrix-plan.md
new file mode 100644
index 000000000..96229f9d8
--- /dev/null
+++ b/docs/e2e-grid-matrix-plan.md
@@ -0,0 +1,183 @@
+# E2E Grid-Switch Matrix — Plan
+
+Goal: when a NEW version of a coding-agent tool (codex / claude) is acknowledged,
+verify it against the **real** harness grid-switch journey (not a mock): an agent on its own
+subscription is moved onto a grid model, resumed, checked that it still answers / uses tools /
+has its MCP servers / spawns a subagent, then moved back to its own login and checked again — with every step/value logged under a
+namespaced session so we can audit what each agent read and did, catch hallucination / wrong
+decisions / wrong actions, and produce a review from a free-model watchdog.
+
+---
+
+## 1. Two layers of environment (decide A/B/C)
+
+| Layer | What runs | Needs | Sandboxable? |
+|---|---|---|---|
+| **Controller** (drive + watchdog) | `gridSwitchDriver` + `paneProbe` (Node) + the opencode `e2e-watchdog` agent (free model) | Node, tmux, opencode | ⚠️ needs the pane: runs on the machine under test |
+| **Under-test** (the machine being tested) | harness daemon + tmux + real codex/claude + grid sign-in | tmux, interactive login shell, **codex auth**, **claude login**, **grid sign-in** | ⚠️ only if you provision all login/state inside |
+
+> **DECIDED: Option A — code-managed, ZERO-CONFIG, deterministic scripts on the real machine.**
+> All logic lives in repo code (argv-package, matrix script, watchdog agent, session naming).
+> Every parameter is auto-derived in code (self-discovering) — daemon port from `env.PORT`, machineId
+> from the daemon, engine version from `--version`/`cli_version`, grid model picked automatically from
+> available, grid endpoint/credential from the machine's grid login, session name from timestamp.
+> No config file, no manual flags, no drift → consistent wherever the same prerequisites are met.
+> Under-test reuses the already-running harness machine (codex auth/tmux/grid sign-in). No workflow
+> container is required for consistency; a container is OPTIONAL only for the watchdog (read-only
+> trace, no credentials) if blast-radius isolation is wanted later.
+
+- **A ✔ (chosen) — code-managed scripts; under-test on the real machine.**
+ Fastest, least manual deploy. Real machine already has `~/.codex/auth.json`, tmux, zsh.
+ Watchdog runs on the machine (or optionally in an isolated container) and only reads the trace.
+- **B — everything in a container sandbox.**
+ More setup: must mount codex/claude/grid auth + tmux + harness and keep login state fresh
+ for every matrix run. Rejected: too much manual ops.
+- **C — split relay: loopback local relay for the relay/convert part ($0), separate real
+ subscription→grid run on the real machine.**
+
+---
+
+## 2. Session naming — one name per run, auditable end-to-end
+
+Convention: `session = <engine>@<version>-><testcase>@<grid-model>--<timestamp>`
+
+Examples:
+```
[email protected]>grid-switch@grid:gpt-5-mini--20260921T1805Z
[email protected]>back-home@opus--20260921T1810Z
+claude@UNKNOWN->grid-switch@none--20260921T1820Z # engine not logged in -> still nameable + notified
+```
+
+The SAME name is stamped on everything from one run:
+```
[email protected]>[email protected] # driver step/value log
+report-…json / [email protected]>…json # what the watchdog reads; its review is written back into it
[email protected]>...md # watchdog review output
+```
+`<engine>@UNKNOWN` is used when a tool is present but not logged in, so the run still has an
+audit name and a NOTIFY instead of a hard failure.
+
+---
+
+## 3. argv-package — the real command the harness enters
+
+Spawn is `tmux respawn-pane -k` with the engine argv from `buildEngineCommandArgv`
+(`cli/src/lib/engineLaunch.ts`) + env/clearEnv from `launchOverrides.ts` → `gridLaunch.ts`.
+
+- **codex → grid:** `codex [resume <session>] -c model_provider="grid" -c model_providers.grid.name=…
+ -c model_providers.grid.base_url=<grid>/relay/v1 -c model_providers.grid.env_key=…
+ -c model_providers.grid.wire_api="responses" -c model_providers.grid.supports_websockets="true"
+ [-c web_search="disabled"]`; env `OPENAI_API_KEY=<grid token>`; clear vendor creds.
+- **claude → grid:** `claude --resume <session> [mode]`; env `ANTHROPIC_BASE_URL=<grid anthropic relay>`,
+ `ANTHROPIC_MODEL=<model>`; clear `ANTHROPIC_API_KEY`.
+- **back to own login:** clearEnv of grid vars + respawn with `ownLoginProviderArgs` (codex re-reads
+ `config.toml`; claude returns to its own `ANTHROPIC_*`).
+
+The E2E matrix builds these argv itself (same builder) so it spawns exactly like the harness.
+
+---
+
+## 3b. Real-test models (fixed per user decision)
+
+- **Subscription leg** (codex/claude answering on their OWN login) uses a fixed cheap model,
+ enabled in the tmux launch (`cli/src/e2e/fixedModels.ts`):
+ - codex → `gpt-5.5`
+ - claude → `claude-sonnet-5`
+ (codex/claude expose no model-list CLI, so these are explicit and tuneable in one place.)
+- **Grid leg** picks the smallest available model automatically from `grid_models_list`.
+
+---
+
+## 4. Real-matrix flow (per engine × version)
+
+```
+1. prep: daemon + real agent in a tmux pane + engine login present
+2. for each engine (codex/claude) × version:
+ a. agent answers on its OWN subscription (real round-trip)
+ b. driver -> agent_retarget grid (payload gridModel/gridName)
+ c. harness respawns with grid argv; driver PINGS the engine; wait for answer from the GRID
+ d. verify the answer came from the grid, not the old subscription
+ e. driver -> agent_retarget clearGrid; verify back on own login
+ f. write trace-<session>.json + per-engine log
+3. engine not logged in -> NOTIFY (no hard fail); still name the session @UNKNOWN
+ grid not sign-in -> NOTIFY (per decision)
+```
+
+Handles: refusal codes (AGENT_BUSY, RETARGET_UNSUPPORTED_BACKEND, GRID_UNAVAILABLE, …) as
+surfaced results, and surfaces every step in the trace so a reviewer can audit what happened.
+
+---
+
+## 5. Where each piece lives (decided 2026-09-22)
+
+Two repos, split by who owns the code under test:
+
+- **autonomous-harness** (this repo) — the **driver**: `harness e2e plan` (trigger: installed vs
+ latest) and `harness e2e grid-switch` (`cli/src/e2e/`): creates a fresh agent through the daemon,
+ lays out the person's project (`tools/calc.sh` + MCP `e2e_calc`, `workspace.ts`), runs the
+ conversation subscription → grid → back-home (`smokeChecks.ts` scenario: tool / mcp / recall,
+ proven by the tool/MCP logs, `paneProbe.ts`), moves it with `agent_retarget` (own grid, or a
+ relay under test via the `grid` override), and leaves a **review bundle** in
+ `~/.harness/cli/data/e2e/runs/<session>/` (trace, daemon log slice, workspace logs, manifest).
+- **autonomous-grid-cli** (`e2e/`) — the **reviewer and the fix loop**, next to the relay it fixes:
+ `agents/e2e-watchdog.md` (system prompt) + `skills/e2e-watchdog/SKILL.md` (quick view, report
+ fields, clean-run shape, find/reproduce, relay bisect, fix + PR rules, report shape),
+ `install.sh` (sync into `~/.config/opencode`), `run.sh` (the cron entry: plan → grid-switch →
+ `opencode run --agent e2e-watchdog … "Run report: <bundle>/trace.json. What got stuck?"` → bundle
+ gains `watchdog.jsonl` + `watchdog.md` + the opencode session id to reopen), and `stack/` (a
+ relay checkout under test: strict node, logging proxy, `matrix.sh`, the recorded requests).
+- **The grid server** runs it: harness daemon + tmux + codex/claude logins + opencode (free model),
+ cron `e2e/run.sh`. The e2e-watchdog reads the relay code in place, tests it, opens the PR there.
+
+Verified 2026-09-22 on this Mac: the relay-commit matrix (`matrix.sh f5bbb4e 00fa887 22519fa
+d6c5489 main` × recorded codex requests R/C/T) fails and passes exactly at the commit that fixed
+each shape; `main` still fails T (`tool_search_*`, codex 0.155.1) — the open ticket for the agent.
+
+---
+
+## 6. Cost / accounts (honest)
+
+- **Watchdog:** $0 (opencode free model, verified).
+- **Real matrix** (`subscription -> grid -> back`): each grid ping costs real tokens against the
+ codex/claude subscription + grid model. Small per run, but real. **Accepted** per earlier
+ direction; the offline driver spec (`gridSwitchDriver.spec.ts`) remains the $0 CI path.
+- **Accounts needed:** codex (present ✅), claude (login pending ❌ -> notify), grid sign-in (pending ❌ -> notify).
+
+---
+
+## 7. Machine state today (verified)
+
+| Item | Status |
+|---|---|
+| codex binary + auth | ✅ `~/.codex/auth.json` present |
+| claude binary | ✅ present; login ❌ (no `~/.claude/.credentials.json`) -> NOTIFY |
+| tmux / sqlite3 / zsh | ✅ present |
+| opencode binary + free models | ✅ `opencode/big-pickle` verified cost 0 (reviewer only) |
+| grid sign-in | ❓ not confirmed -> check+notify |
+
+---
+
+## 8. Operational loop (manual once, then fully automated)
+
+**Manual once (user):** install harness + codex + claude on the server; sign in
+codex / claude / grid. This is the ONLY manual step.
+
+**Then automated (no manual action):**
+1. boot harness daemon if not running
+2. detect current versions (codex/claude/grid) from `--version`/`cli_version`
+3. update to latest when a new version is acknowledged
+4. run the matrix E2E per version: subscription -> switch grid -> real ping -> verify -> back home
+ (session naming for auditability)
+5. missing login / not signed in -> NOTIFY, no hard fail
+6. watchdog (opencode `e2e-watchdog` agent + skill, free model) reads the trace -> what stuck, reproduced, PASS/FAIL + notice
+
+---
+
+## 9. Build order (pending env choice A/B/C)
+
+1. **argv-package** — expose the exact grid/back-home command builder the matrix reuses.
+2. **real-matrix script** — driver + switch + ping + verify + notify, with session naming.
+3. **watchdog** — opencode `e2e-watchdog` agent + skill, smoke checks per leg, pane probe. ✔
+4. CI/scheduling wrapper for when a new version is acknowledged (per-version, not blanket).
+
+**BLOCKED ON:** env choice (A/B/C) + confirming grid sign-in + claude login (or notify-only).