The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
harness e2e on main, with the daemon's local credential: grid-switch stops failing on 'no machineId' (#1441)
lingrd6bb09d3f9e0a
diff --git a/cli/src/cli.ts b/cli/src/cli.tsindex b51b70d6a..78979da13 100644--- a/cli/src/cli.ts+++ b/cli/src/cli.ts@@ -161,6 +161,8 @@ Machine:harness help agents the agent commands, states, exit codes and rules on one screenharness settings [<name> on|off] what agents may do (agents-start-agents, pull-request-line, agent-messages): your account's experiments in Settings ▸ Experimental, each off by defaultharness machines delete <id> remove ANOTHER machine (refuses this one; use \`harness logout\`)+ harness e2e plan which coding tools have a new version out (the grid-switch e2e trigger)+ harness e2e grid-switch move a fresh agent subscription -> grid -> back home with its tool + MCP; leaves a review bundleharness remote from a Harness terminal tile: open a terminal on another of your machines and move this tile to itharness version print the installed version (v${VERSION})harness update [--force] update to the latest build now (it also self-updates in the background;@@ -2750,6 +2752,20 @@ switch (cmd) {color: process.stdout.isTTY === true,})break+ case 'e2e':+ // The grid-switch e2e: a fresh agent moved subscription -> grid -> back home with the person's+ // tool + MCP, leaving a review bundle. Env-driven (see src/e2e/runGridSwitchTrace.ts); the+ // relay-side reviewer (opencode `e2e-watchdog`) lives with the relay, in autonomous-grid-cli.+ // Shipped with the CLI, so it is always the daemon's own version that drives the daemon.+ if (args[0] === 'grid-switch') {+ import('./e2e/runGridSwitchTrace.js')+ .then(({ runGridSwitch }) => runGridSwitch())+ .then((r) => { process.exitCode = r.error || r.stuck ? 1 : 0 })+ .catch(onError)+ } else if (args[0] === 'plan') {+ import('./e2e/plan.js').then(({ planCommand }) => planCommand(flags.includes('--json'))).catch(onError)+ } else { console.error(`Unknown command: e2e ${args[0] ?? ''} (grid-switch | plan)`); usage(1) }+ breakcase 'machines':if (!args[0]) machinesListCommand(flags.includes('--json')).catch(onError)else if (args[0] === 'list') machinesListCommand(flags.includes('--json')).catch(onError)diff --git a/cli/src/e2e/artifacts.ts b/cli/src/e2e/artifacts.tsnew file mode 100644index 000000000..f9570872c--- /dev/null+++ b/cli/src/e2e/artifacts.ts@@ -0,0 +1,35 @@+/**+ * artifacts — where a run leaves what people and the watchdog read AFTERWARDS.+ *+ * Two places, on purpose:+ * - `outDir()`: scratch under $TMPDIR (`grid-matrix-out/`) — the agent's workspace, dry-run reports.+ * - `runDir(session)`: the durable review bundle, under the harness's own data dir+ * (`<ADAPTER_DATA_DIR>/e2e/runs/<session>/`), one folder per run:+ * trace.json the report (legs, trace, pane tails, watchdog verdict)+ * manifest.json ids + how to reopen everything (opencode session, daemon log, workspace)+ * daemon.log the daemon's lines for this run's agent/session, cut from harness.log+ * watchdog.jsonl the opencode agent's raw event stream (every tool call it made)+ * watchdog.md its review, as text+ * workspace/ tools/, .codex/ or .mcp.json, .e2e/*.log — the tool/MCP call logs+ * evidence/ what the watchdog saved while reproducing (proxy req/res, bisect table)+ */+import { mkdirSync } from 'node:fs'+import { join } from 'node:path'+import { tmpdir } from 'node:os'+import { env } from '../config/env.js'++export function outDir(): string {+ const dir = join(tmpdir(), 'grid-matrix-out')+ mkdirSync(dir, { recursive: true })+ return dir+}++export function runsRoot(): string {+ return join(env.ADAPTER_DATA_DIR, 'e2e', 'runs')+}++export function runDir(session: string): string {+ const dir = join(runsRoot(), session.replace(/[^A-Za-z0-9._@>-]/g, '-'))+ mkdirSync(join(dir, 'evidence'), { recursive: true })+ return dir+}diff --git a/cli/src/e2e/currentVersion.ts b/cli/src/e2e/currentVersion.tsnew file mode 100644index 000000000..0b5b5e49e--- /dev/null+++ b/cli/src/e2e/currentVersion.ts@@ -0,0 +1,46 @@+/**+ * current version provider — reads the INSTALLED version of each coding-agent tool on the server.+ * The harness already captures this as `cli_version` on a registered agent; here we read it from+ * each binary's `--version` for the trigger to compare against latest.+ */+import { execFile } from 'node:child_process'+import { promisify } from 'node:util'+import type { AgentEngine } from '../engines/types.js'+import type { VersionProvider } from './trigger.js'++const execFileP = promisify(execFile)++/** How to ask each engine binary its version. */+const VERSION_COMMAND: Partial<Record<AgentEngine, string[]>> = {+ codex: ['--version'],+ claude: ['--version'],+}++/** Nice display names for logs/notifies. */+export const ENGINE_BIN: Partial<Record<AgentEngine, string>> = {+ codex: 'codex',+ claude: 'claude',+}++/** Extract the dotted version from a `--version` string like `codex-cli 0.155.0` / `2.0.34`. */+export function parseVersionLine(line: string): string | null {+ const m = line.match(/\b\d+\.\d+(?:\.\d+)*\b/)+ return m ? m[0] : null+}++/** Reads installed versions from the real binaries on PATH. */+export class InstalledCurrentVersion implements VersionProvider {+ constructor(private readonly bin: Partial<Record<AgentEngine, string>> = ENGINE_BIN) {}++ async currentVersion(engine: AgentEngine): Promise<string | null> {+ const argv = VERSION_COMMAND[engine]+ const binary = this.bin[engine]+ if (!argv || !binary) return null+ try {+ const { stdout } = await execFileP(binary, argv, { timeout: 10_000 })+ return parseVersionLine(stdout) ?? null+ } catch {+ return null+ }+ }+}diff --git a/cli/src/e2e/e2e.spec.ts b/cli/src/e2e/e2e.spec.tsnew file mode 100644index 000000000..c670046db--- /dev/null+++ b/cli/src/e2e/e2e.spec.ts@@ -0,0 +1,664 @@+import { describe, expect, it } from 'vitest'+import { gridSpawn, backHomeSpawn } from './gridArgv.js'+import { MockVersionSource, NpmVersionSource } from './versionSource.js'+import { compareVersions, planMatrixRuns } from './trigger.js'+import { sessionName, workspaceDirName } from './sessionName.js'+import { buildMatrixEntry } from './matrix.js'+import type { AgentEngine } from '../engines/types.js'+import { SMOKE_CHECKS, SCENARIO, LEGS, firstStuck, plannedLegs, quotaHit, scenarioFor, type CheckId } from './smokeChecks.js'+import { pickGridModel } from './gridSwitchDriver.js'+import { prepareWorkspace, readLog, logPath, readWorkspaceFile, preAcceptClaudeBypassMode, CALC_MCP_MJS } from './workspace.js'+import { toolResult, useByRef } from './sessionTools.js'+import { execFileSync } from 'node:child_process'+import { probeLeg, runCheck, dismissStartupDialogs, STARTUP_DIALOGS, type Tmux } from './paneProbe.js'+import { mkdtempSync, mkdirSync, readFileSync, writeFileSync, chmodSync, existsSync, rmSync } from 'node:fs'+import { join } from 'node:path'+import { tmpdir } from 'node:os'++const GRID = {+ networkId: 'grid-1',+ networkName: 'someone-grid',+ baseUrl: 'https://grid.autonomous.ai/grid-1/relay/v1',+ apiKey: 'sk-probe',+ model: 'grid:gpt-5-mini',+}++describe('argv-package (spawn exactly like the harness)', () => {+ it('codex grid: provider argv + wire_api + clearEnv', () => {+ const spec = gridSpawn('codex', GRID, 'S1')+ expect('ok' in spec ? false : true).toBe(true) // not an error spec+ const s = spec as { command: string[]; env: Record<string, string>; clearEnv: string[] }+ expect(s.command[0]).toBe('codex')+ expect(s.command).toContain('resume') // codex resume <id>+ expect(s.command).toContain('S1')+ expect(s.command.join(' ')).toContain('model_provider="grid"')+ expect(s.command.join(' ')).toContain('model_providers.grid.wire_api="responses"')+ expect(s.command.join(' ')).toContain('https://grid.autonomous.ai/grid-1/relay/v1')+ expect(Object.keys(s.env).length).toBeGreaterThan(0) // the grid key goes in env, not argv+ expect(s.clearEnv.length).toBeGreaterThan(0)+ })++ it('claude grid: --resume + ANTHROPIC_BASE_URL env', () => {+ const spec = gridSpawn('claude', GRID, 'S1') as { command: string[]; env: Record<string, string> }+ expect(spec.command[0]).toBe('claude')+ expect(spec.command).toContain('--resume')+ expect(spec.env.ANTHROPIC_BASE_URL).toContain('grid.autonomous.ai/grid-1')+ })++ it('back home clears the grid env vars', () => {+ const spec = backHomeSpawn('claude', 'S1') as { clearEnv: string[] }+ expect(spec.clearEnv).toContain('ANTHROPIC_BASE_URL')+ })+})++describe('version source (mock first, npm later)', () => {+ it('mock returns fixtures offline', async () => {+ const src = new MockVersionSource()+ expect(await src.latestVersion('codex')).toBe('0.156.0')+ expect(await src.latestVersion('claude')).toBe('2.1.279')+ })++ it('npm impl reads the registry via fetch (stubbed)', async () => {+ let called = ''+ let sentInit: RequestInit | undefined+ const fetchImpl = (async (url: string, init?: RequestInit) => {+ called = url+ sentInit = init+ return { ok: true, json: async () => ({ version: '9.9.9' }) } as unknown as Response+ }) as typeof fetch+ const src = new NpmVersionSource(fetchImpl)+ expect(await src.latestVersion('codex')).toBe('9.9.9')+ expect(called).toContain('@openai/codex')+ // No Accept header: `/latest` answers 406 to the abbreviated-metadata type, and the failure is+ // silent (latest=null reads as "nothing new"), so the request is pinned bare. See versionSource.+ expect(sentInit).toBeUndefined()+ })++ it('reads each engine\'s STABLE channel, and never a pre-release', async () => {+ const asked: string[] = []+ const answer = (version: string) => (async (url: string) => {+ asked.push(url)+ return { ok: true, json: async () => ({ version }) } as unknown as Response+ }) as typeof fetch+ expect(await new NpmVersionSource(answer('2.1.273')).latestVersion('claude')).toBe('2.1.273')+ expect(await new NpmVersionSource(answer('0.156.1')).latestVersion('codex')).toBe('0.156.1')+ // claude's stable is its own `stable` tag, not `latest` (its fast channel); codex's is `latest`.+ expect(asked).toEqual(['https://registry.npmjs.org/@anthropic-ai/claude-code/stable', 'https://registry.npmjs.org/@openai/codex/latest'])+ // Whatever tag it came from, a pre-release is never "the new version".+ for (const v of ['0.158.0-alpha.8', '2.1.300-beta.1', '1.0.0-rc.2', '0.1.2505172116-nightly']) {+ expect(await new NpmVersionSource(answer(v)).latestVersion('codex'), v).toBeNull()+ }+ })++ // A stub cannot catch a registry that refuses our headers, and that exact bug once kept the whole+ // pipeline quiet. This talks to npm for real, so it is opt-in: `RUN_REAL_NPM=1 vitest run src/e2e`.+ it.runIf(process.env.RUN_REAL_NPM === '1')('npm impl really answers for both engines', async () => {+ const src = new NpmVersionSource()+ for (const engine of ['codex', 'claude'] as const) {+ expect(await src.latestVersion(engine)).toMatch(/^\d+\.\d+\.\d+/)+ }+ }, 30_000)+})++describe('claude bypass-permissions warning (what killed a run once)', () => {+ it('accepts it in a config copy, is idempotent, and leaves an unreadable one alone', () => {+ const home = mkdtempSync(join(tmpdir(), 'wd-home-'))+ try {+ expect(preAcceptClaudeBypassMode(home)).toBe('skipped') // no ~/.claude.json at all+ writeFileSync(join(home, '.claude.json'), JSON.stringify({ projects: { '/x': { allowedTools: [] } } }))+ expect(preAcceptClaudeBypassMode(home)).toBe('accepted')+ const after = JSON.parse(readFileSync(join(home, '.claude.json'), 'utf8'))+ expect(after.bypassPermissionsModeAccepted).toBe(true)+ expect(after.projects['/x']).toEqual({ allowedTools: [] }) // nothing else touched+ expect(preAcceptClaudeBypassMode(home)).toBe('already')+ writeFileSync(join(home, '.claude.json'), 'not json')+ expect(preAcceptClaudeBypassMode(home)).toBe('skipped')+ } finally {+ rmSync(home, { recursive: true, force: true })+ }+ })++ it('the probe answers the warning by moving off the default "No, exit"', () => {+ const dialog = STARTUP_DIALOGS.find((d) => d.id === 'claude-bypass-accept')!+ expect(dialog.match.test(' WARNING: Claude Code running in Bypass Permissions mode')).toBe(true)+ expect(dialog.keys).toEqual(['Down', 'Enter'])+ })+})++describe('trigger (current vs latest)', () => {+ it('runs on upgrade, not on unchanged', async () => {+ const src = new MockVersionSource({ codex: '0.156.0', claude: '2.1.279' })+ const cur = {+ currentVersion: async (e: AgentEngine) => (e === 'codex' ? '0.155.1' : '2.1.279'),+ }+ const plans = await planMatrixRuns(src, cur, ['codex', 'claude'])+ const codex = plans.find((p) => p.engine === 'codex')!+ const claude = plans.find((p) => p.engine === 'claude')!+ expect(codex.plan).toBe(true)+ expect(codex.reason).toBe('upgrade')+ expect(claude.plan).toBe(false)+ expect(claude.reason).toBe('unchanged')+ })++ it('compares dotted numeric versions', () => {+ expect(compareVersions('0.155.0', '0.155.1')).toBe(-1)+ expect(compareVersions('0.156.0', '0.155.1')).toBe(1)+ expect(compareVersions('2.1.279', '2.1.279')).toBe(0)+ })+})++describe('workspace folder name (what claude 2.1.274 tripped on)', () => {+ it('keeps the session readable but drops every character a tool might rewrite', () => {+ const session = sessionName({ engine: 'claude', version: '2.1.274', testcase: 'grid-switch', at: new Date('2026-09-24T20:22:39Z') })+ expect(session).toContain('->') // the bundle keeps its readable name+ const dir = workspaceDirName(session)+ expect(dir).toBe('claude-2.1.274_to_grid-switch-none--20260924T202239Z')+ expect(dir).toMatch(/^[A-Za-z0-9._-]+$/)+ })+})++describe('session naming', () => {+ it('is namespaced and grep-able', () => {+ const name = sessionName({ engine: 'codex', version: '0.156.0', testcase: 'grid-switch', gridModel: 'grid:gpt-5-mini', at: new Date('2026-09-21T18:05:00Z') })+ expect(name).toMatch(/^codex@0\.156\.0->grid-switch@grid-gpt-5-mini--/)+ })+})++describe('matrix entry (dry run)', () => {+ it('emits argv + notify when not logged in', async () => {+ const plan = { engine: 'claude' as AgentEngine, current: '2.1.278', latest: '2.1.279', reason: 'upgrade' as const, plan: true }+ const entry = await buildMatrixEntry(plan, {+ status: async () => ({ binary: true, login: false, grid: true }),+ grid: async () => null,+ at: new Date('2026-09-21T18:05:00Z'),+ })+ expect(entry.session).toContain('[email protected]')+ expect(entry.notify.join(' ')).toMatch(/NOT_LOGGED_IN/)+ expect(entry.status.login).toBe(false)+ expect(entry.subscriptionModel).toBe('claude-sonnet-5')+ // Dry run: the journey is planned, nothing typed yet.+ expect(entry.legs.map((l) => l.leg)).toEqual(['subscription', 'grid'])+ expect(entry.legs.flatMap((l) => l.checks).every((c) => c.status === 'not-run')).toBe(true)+ })++ it('uses the fixed models for the subscription leg', async () => {+ const src = new MockVersionSource({ codex: '0.156.0' })+ const plans = await planMatrixRuns(src, { currentVersion: async () => '0.155.0' }, ['codex'])+ const entry = await buildMatrixEntry(plans[0], {+ status: async () => ({ binary: true, login: true, grid: true }),+ grid: async () => null,+ })+ expect(entry.subscriptionModel).toBe('gpt-5.5')+ })+})++describe('scenario (what a person does, on each side of the switch)', () => {+ it('four plain requests on each side — read, write, edit, then MCP — with nothing shared', () => {+ expect(LEGS).toEqual(['subscription', 'grid'])+ // The own account calls the workspace's MCP server; the grid, the harness's web search.+ expect(SCENARIO.subscription.map((c) => c.id)).toEqual(['read', 'write', 'edit', 'mcp'])+ expect(SCENARIO.grid.map((c) => c.id)).toEqual(['read', 'write', 'edit', 'web'])+ expect(SCENARIO.grid[3].sessionTool).toBe('mcp__harness__web_search')+ // Nothing the first side leaves behind can pass the second: its own numbers and its own files.+ const patterns = SMOKE_CHECKS.filter((c) => c.logPattern).map((c) => c.logPattern)+ expect(new Set(patterns).size).toBe(patterns.length)+ const files = SMOKE_CHECKS.map((c) => c.answerFromFile ?? c.file?.path).filter(Boolean)+ expect(new Set(files).size).toBe(files.length)+ expect(plannedLegs().flatMap((l) => l.checks).every((c) => c.status === 'not-run')).toBe(true)+ })++ it('every prompt is a plain request, and never contains its own proof', () => {+ for (const c of SMOKE_CHECKS) {+ expect(c.prompt).not.toMatch(/reply with exactly|<result>|RECALL_|TOOL_|MCP_/i)+ if (c.logPattern) expect(c.prompt).not.toContain(c.logPattern) // "add 40 2" is asked; "= 42" is only the log's+ expect(!!c.log || !!c.answerFromFile || !!c.file || !!c.sessionTool).toBe(true) // proven by the machine, never by the reply+ // The result is what is tested, not the route: no request tells the engine which tool to use.+ expect(c.prompt).not.toMatch(/\b(Bash|Read|Write|Edit) tool\b|apply_patch|in the shell/)+ }+ // And every engine gets the same words.+ expect(scenarioFor('codex')).toEqual(scenarioFor('claude'))+ })++ it('firstStuck names the first leg/step that did not pass', () => {+ const legs = plannedLegs()+ legs[0].checks.forEach((c) => (c.status = 'ok'))+ legs[1].checks[0].status = 'ok'+ legs[1].checks[1].status = 'stuck'+ expect(firstStuck(legs)).toEqual({ leg: 'grid', check: 'write', status: 'stuck' })+ legs.forEach((l) => l.checks.forEach((c) => (c.status = 'ok')))+ expect(firstStuck(legs)).toBeNull()+ })+})++/** A pane that answers each typed prompt with `reply(prompt)` on the next capture. */+function fakePane(reply: (prompt: string) => string | null): Tmux & { typed: string[] } {+ let screen = '> '+ let clock = 0+ const typed: string[] = []+ return {+ typed,+ async capture() {+ return screen+ },+ async type(_pane, text) {+ typed.push(text)+ screen += `\n> ${text}`+ const r = reply(text)+ if (r !== null) screen += `\n${r}`+ },+ async key(_pane, key) {+ typed.push(`<${key}>`)+ // Any key answers a dialog: the dialog text leaves the screen.+ screen = screen.replace(/Update available![^\n]*\n?/, '').replace(/Do you trust[^\n]*\n?/, '')+ },+ async sleep(ms) {+ clock += ms+ },+ now: () => clock,+ }+}++/** An MCP log that gains a line whenever the fake tool "uses" the server. */+function fakeLogs() {+ const lines: Record<'mcp', string[]> = { mcp: [] }+ return { lines, readLog: (k: 'mcp') => [...lines[k]] }+}++/**+ * A fake coding tool on a fake machine: it does what each request asks — reads the file, writes+ * it, edits it, calls the MCP server (a log line), searches the web (a result in its+ * session, as `toolResult` would read it). With `lie`, it only SAYS it did those steps, which is the+ * case the proofs exist for; with `empty`, its web search comes back with nothing.+ */+function fakeAgent(opts: { lie?: CheckId[]; empty?: boolean } = {}) {+ const files: Record<string, string> = {+ 'notes/info-1.txt': 'kiwi-4821-tulip\n', 'notes/info-2.txt': 'otter-1234-ember\n',+ 'notes/todo-1.txt': '# todo 1\nstatus: pending\n', 'notes/todo-2.txt': '# todo 2\nstatus: pending\n',+ }+ const logs = fakeLogs()+ const searched: Record<string, string> = {}+ const lie = new Set(opts.lie ?? [])+ const reply = (p: string): string | null => {+ let m: RegExpMatchArray | null+ if ((m = p.match(/Read (notes\/info-\d\.txt)/))) return lie.has('read') ? 'It says hello.' : `It says: ${files[m[1]].trim()}`+ if ((m = p.match(/Search the web .*\((ref-[0-9a-z]{6})\)/))) {+ if (!lie.has('web')) searched[m[1]] = opts.empty ? 'No results.' : 'Node.js 24.11.0 — https://nodejs.org/en/download'+ return 'The latest stable Node.js is 24.11.0.'+ }+ if ((m = p.match(/Create the file (\S+) with this text: (.*) \(ref-/))) {+ if (!lie.has('write')) files[m[1]] = `${m[2]}\n`+ return 'Done.'+ }+ if ((m = p.match(/In (\S+), change pending to done/))) {+ if (!lie.has('edit')) files[m[1]] = files[m[1]].replace('pending', 'done')+ return 'Updated.'+ }+ if ((m = p.match(/e2e_calc to add (\d+) and (\d+)/))) {+ if (!lie.has('mcp')) logs.lines.mcp.push(`t add ${m[1]} ${m[2]} = ${+m[1] + +m[2]}`)+ return `${+m[1] + +m[2]}`+ }+ return null+ }+ const pane = fakePane(reply)+ const toolResult = (ref: string, tool: string) => {+ const out = tool === 'mcp__harness__web_search' ? searched[ref] : undefined+ return { called: out !== undefined, answered: !!out && out.includes('https://'), ...(out ? { output: out } : {}) }+ }+ const probe = { settleMs: 1, pollMs: 1, checkTimeoutMs: 5_000, readLog: logs.readLog, readFile: (f: string) => files[f] ?? null, toolResult }+ return { pane, files, logs, probe }+}++const READ_STEP = SCENARIO.subscription[0]+const MCP_STEP = SCENARIO.subscription[3]+const WEB_STEP = SCENARIO.grid[3]++describe('pane probe (live steps on a tmux pane)', () => {+ it('a tool that really does the four requests passes both sides, and each proof is recorded', async () => {+ const { pane, probe, files } = fakeAgent()+ const own = await probeLeg(pane, '%1', 'subscription', probe)+ const grid = await probeLeg(pane, '%1', 'grid', probe)+ expect(own.checks.map((c) => `${c.id}=${c.status}`)).toEqual(['read=ok', 'write=ok', 'edit=ok', 'mcp=ok'])+ expect(grid.checks.map((c) => `${c.id}=${c.status}`)).toEqual(['read=ok', 'write=ok', 'edit=ok', 'web=ok'])+ expect(own.checks[3].logLines).toEqual(['t add 30 12 = 42'])+ expect(files['out/hello-2.txt']).toBe('hello from step 2\n')+ expect(files['notes/todo-1.txt']).toContain('status: done')+ // Every step carries the tag its prompt was typed with — how its tools are found afterwards.+ expect([...own.checks, ...grid.checks].every((c) => /^ref-[0-9a-z]{6}$/.test(c.ref ?? ''))).toBe(true)+ })++ it('saying is not doing: each step fails when the tool only claims it, with what was missing', async () => {+ const notes: Record<CheckId, RegExp> = {+ read: /notes\/info-1\.txt's token never appeared/,+ write: /out\/hello-1\.txt does not exist/,+ edit: /notes\/todo-1\.txt is "# todo 1\\nstatus: pending"/,+ mcp: /no mcp log line for "add 30 12 = 42"/,+ web: /mcp__harness__web_search was never called/,+ }+ for (const step of [...SCENARIO.subscription, WEB_STEP]) {+ const { pane, probe } = fakeAgent({ lie: [step.id] })+ const out = await runCheck(pane, '%1', step, probe)+ expect(out.status, step.id).toBe('stuck')+ expect(out.note, step.id).toMatch(notes[step.id])+ }+ })++ it('web: a search that came back with no results is not a search that worked', async () => {+ const { pane, probe } = fakeAgent({ empty: true })+ const out = await runCheck(pane, '%1', WEB_STEP, probe)+ expect(out.status).toBe('stuck')+ expect(out.note).toBe('mcp__harness__web_search came back without results: "No results."')+ })++ it('read: the token must come after THIS question — one already on screen is no proof', async () => {+ const { pane, probe } = fakeAgent({ lie: ['read'] })+ await pane.type('%1', 'kiwi-4821-tulip') // on screen before the question was asked+ expect((await runCheck(pane, '%1', READ_STEP, probe)).status).toBe('stuck')+ })++ it('write: the exact text, not merely a file', async () => {+ const { pane, probe, files } = fakeAgent({ lie: ['write'] })+ files['out/hello-1.txt'] = 'hello from step one\n'+ const out = await runCheck(pane, '%1', SCENARIO.subscription[1], probe)+ expect(out.status).toBe('stuck')+ expect(out.note).toBe('out/hello-1.txt is "hello from step one"')+ })++ it('read with no info file in the workspace is the run\'s setup failing, said as such', async () => {+ const { pane, probe } = fakeAgent()+ const out = await runCheck(pane, '%1', READ_STEP, { ...probe, readFile: () => null })+ expect(out.status).toBe('stuck')+ expect(out.note).toMatch(/the run's setup, not the tool/)+ expect(pane.typed).toHaveLength(0) // nothing typed for a step that cannot be proven+ })++ it('out of usage is recognised at once, with its reset time, instead of waiting out the step', async () => {+ // What each tool really prints — codex and claude, from their own binaries.+ const cases: Array<[string, string | null]> = [+ ["■ You've hit your usage limit. Upgrade to Pro, or try again at Sep 24th, 2026 6:02 PM.", 'try again at Sep 24th, 2026 6:02 PM'],+ ["You've hit your session limit · resets 6pm", 'resets 6pm'],+ ["You’ve hit your weekly limit · resets Mon 9am", 'resets Mon 9am'],+ ["You're out of usage credits. /model to switch models.", null],+ ]+ for (const [said, resets] of cases) {+ const pane = fakePane(() => said)+ const out = await runCheck(pane, '%1', MCP_STEP, { settleMs: 1, pollMs: 1, checkTimeoutMs: 120_000 })+ expect(out.status, said).toBe('no-quota')+ expect(out.resets ?? null, said).toBe(resets)+ expect(out.elapsedMs!, said).toBeLessThan(10) // one poll, not the whole budget+ }+ // Claude's early warning is not the limit: the step carries on and passes.+ const logs = fakeLogs()+ const warned = fakePane(() => { logs.lines.mcp.push('t add 30 12 = 42'); return 'Approaching usage limit · resets 6pm\n42' })+ expect((await runCheck(warned, '%1', MCP_STEP, { settleMs: 1, pollMs: 1, readLog: logs.readLog })).status).toBe('ok')+ })++ it('an old limit message already on screen does not count against a new request', async () => {+ const { pane, probe } = fakeAgent()+ await pane.type('%1', "You've hit your session limit · resets 6pm") // yesterday's, still in the scrollback+ pane.typed.length = 0+ expect((await runCheck(pane, '%1', MCP_STEP, probe)).status).toBe('ok')+ })++ it('out of usage ends the leg: nothing more is typed into an account that cannot answer', async () => {+ const pane = fakePane(() => "You've hit your usage limit. try again at 6:02 PM.")+ const leg = await probeLeg(pane, '%1', 'subscription', { settleMs: 1, pollMs: 1, readFile: () => 'kiwi-4821-tulip\n' })+ expect(leg.checks.map((c) => `${c.id}=${c.status}`)).toEqual(['read=no-quota', 'write=not-run', 'edit=not-run', 'mcp=not-run'])+ expect(pane.typed).toHaveLength(1)+ expect(quotaHit([leg])).toEqual({ leg: 'subscription', check: 'read', resets: 'try again at 6:02 PM' })+ })++ it('answers codex 0.156\'s stacked startup screens — the one on top first, each once', async () => {+ const { pane, probe } = fakeAgent()+ // What the pane held on grid-dev: the hooks review, and the model notice drawn under it.+ await pane.type('%1', 'Hooks need review\n 2 hooks are new or changed.\n› Review hooks\n Trust all and continue')+ await pane.type('%1', 'GPT-5.5 retires on October 14, 2026. Switch to GPT-5.6 Sol to continue working in Codex.\n› 1. Try new model\n 2. Use existing model')+ pane.typed.length = 0+ const answered = await dismissStartupDialogs(pane, '%1', { settleMs: 1, pollMs: 1 })+ expect(answered).toEqual(['codex-model-retire', 'codex-hooks-trust'])+ expect(pane.typed).toEqual(['<2>', '<Enter>', '<Down>', '<Enter>'])+ // And the step typed after them is not disturbed by their text still being in the scrollback.+ pane.typed.length = 0+ expect((await runCheck(pane, '%1', MCP_STEP, probe)).status).toBe('ok')+ expect(pane.typed).toHaveLength(1)+ })++ it('types the request again when a tool that is still loading dropped it', async () => {+ const { pane, probe } = fakeAgent()+ // The first paste vanishes, the way claude's first question did on grid-dev; the second lands.+ const realType = pane.type.bind(pane)+ let dropped = false+ pane.type = async (id, text) => { if (!dropped) { dropped = true; pane.typed.push(text); return } await realType(id, text) }+ // The real per-step budget: the 5s spent seeing whether the first paste landed comes out of it.+ const out = await runCheck(pane, '%1', MCP_STEP, { ...probe, pollMs: 100, checkTimeoutMs: 120_000 })+ expect(out.status).toBe('ok')+ expect(pane.typed).toHaveLength(2) // typed, not seen, typed again+ expect(pane.typed[0]).toBe(pane.typed[1]) // the same request with the same tag+ })++ it('answers claude 2.1.281\'s reworded folder trust by moving off its "No, exit" default', async () => {+ const pane = fakePane(() => null)+ await pane.type('%1', 'Quick safety check: Is this a project you created or one you trust?\n ❯ No, exit\n Yes, I trust this folder')+ pane.typed.length = 0+ expect(await dismissStartupDialogs(pane, '%1', { settleMs: 1, pollMs: 1 })).toEqual(['claude-trust-v2'])+ expect(pane.typed).toEqual(['<Down>', '<Enter>'])+ })++ it('a step that is not proven stops the leg there: nothing more is typed after it', async () => {+ const { pane, probe } = fakeAgent({ lie: ['write'] })+ const leg = await probeLeg(pane, '%1', 'subscription', { ...probe, checkTimeoutMs: 5 })+ expect(leg.checks.map((c) => `${c.id}=${c.status}`)).toEqual(['read=ok', 'write=stuck', 'edit=not-run', 'mcp=not-run'])+ expect(pane.typed).toHaveLength(2)+ })++ it('answers a startup dialog (codex update → Skip) before typing the first request, and records it', async () => {+ const { pane, probe } = fakeAgent()+ // What a fresh codex shows before its prompt; a request typed here would pick "1. Update now".+ await pane.type('%1', 'Update available! 0.155.0 -> 0.155.1 › 1. Update now 2. Skip Press enter to continue')+ pane.typed.length = 0+ const leg = await probeLeg(pane, '%1', 'subscription', probe)+ expect(pane.typed.slice(0, 2)).toEqual(['<2>', '<Enter>'])+ expect(pane.typed[2]).toContain(READ_STEP.prompt)+ expect(leg.dialogs).toEqual(['codex-update'])+ expect(leg.checks.every((c) => c.status === 'ok')).toBe(true)+ expect(await dismissStartupDialogs(pane, '%1', { settleMs: 1 })).toEqual([]) // nothing left on screen+ })+})++describe('session tools and models (read from the engine\'s own session file)', () => {+ // Records shaped exactly like the real ones — a codex 0.156.1 rollout and a claude 2.1.273+ // session from grid-dev, 2026-09-24 — trimmed to the fields that matter.+ it('codex: code-mode exec calls, classic calls, a namespaced MCP call, and each turn\'s model', () => {+ const home = mkdtempSync(join(tmpdir(), 'wd-sess-'))+ try {+ const dir = join(home, '.codex', 'sessions', '2026', '09', '24')+ mkdirSync(dir, { recursive: true })+ const rec = (o: object) => JSON.stringify(o)+ const user = (t: string) => rec({ type: 'response_item', payload: { type: 'message', role: 'user', content: [{ type: 'input_text', text: t }] } })+ writeFileSync(join(dir, 'rollout-x.jsonl'), [+ rec({ type: 'turn_context', payload: { model: 'gpt-6-luna' } }), // written BEFORE the turn's message+ user('Run tools/calc.sh add 40 2 and tell me the result. (ref-aaaaaa)'),+ rec({ type: 'response_item', payload: { type: 'custom_tool_call', name: 'exec', input: 'const r = await tools.exec_command({cmd:"tools/calc.sh add 40 2"})' } }),+ rec({ type: 'turn_context', payload: { model: 'DeepSeek-V4-Flash-0731' } }),+ user('Use the MCP server e2e_calc to add 60 and 18. (ref-bbbbbb)'),+ rec({ type: 'response_item', payload: { type: 'function_call', name: 'sub', namespace: 'mcp__e2e_calc', arguments: '{}' } }),+ rec({ type: 'response_item', payload: { type: 'function_call', name: 'apply_patch', arguments: '{}' } }),+ ].join('\n'))+ expect(useByRef('codex', '/unused', ['ref-aaaaaa', 'ref-bbbbbb'], home)).toEqual({+ 'ref-aaaaaa': { tools: ['exec→exec_command'], models: ['gpt-6-luna'] },+ 'ref-bbbbbb': { tools: ['mcp__e2e_calc__sub', 'apply_patch'], models: ['DeepSeek-V4-Flash-0731'] },+ })+ } finally {+ rmSync(home, { recursive: true, force: true })+ }+ })++ it('claude: tool_use names per step, ToolSearch included, and the model of every reply', () => {+ const home = mkdtempSync(join(tmpdir(), 'wd-sess-'))+ try {+ const cwd = '/tmp/grid-matrix-out/agents/[email protected]>grid-switch@none--20260924T081321Z'+ const dir = join(home, '.claude', 'projects', '-tmp-grid-matrix-out-agents-claude-2-1-273--grid-switch-none--20260924T081321Z')+ mkdirSync(dir, { recursive: true })+ const rec = (o: object) => JSON.stringify(o)+ const reply = (model: string, ...tools: string[]) => rec({ type: 'assistant', message: { model, content: tools.map((name) => ({ type: 'tool_use', name, input: {} })) } })+ writeFileSync(join(dir, 's.jsonl'), [+ rec({ type: 'user', message: { role: 'user', content: 'Read notes/info-1.txt and tell me what it says. (ref-cccccc)' } }),+ reply('claude-haiku-4-5-20251001', 'Read'),+ rec({ type: 'user', message: { role: 'user', content: [{ type: 'tool_result', content: 'kiwi-4821-tulip' }] } }),+ reply('claude-haiku-4-5-20251001'),+ rec({ type: 'user', message: { role: 'user', content: 'Use the MCP server e2e_calc to add 30 and 12. (ref-dddddd)' } }),+ reply('DeepSeek-V4-Flash-0731', 'ToolSearch'),+ reply('DeepSeek-V4-Flash-0731', 'mcp__e2e_calc__add'),+ reply('<synthetic>'),+ ].join('\n'))+ expect(useByRef('claude', cwd, ['ref-cccccc', 'ref-dddddd'], home)).toEqual({+ 'ref-cccccc': { tools: ['Read'], models: ['claude-haiku-4-5-20251001'] },+ 'ref-dddddd': { tools: ['ToolSearch', 'mcp__e2e_calc__add'], models: ['DeepSeek-V4-Flash-0731'] },+ })+ } finally {+ rmSync(home, { recursive: true, force: true })+ }+ })++ it('codex: the web search of THIS step, and whether it came back with results', () => {+ const home = mkdtempSync(join(tmpdir(), 'wd-sess-'))+ try {+ const dir = join(home, '.codex', 'sessions', '2026', '10', '06')+ mkdirSync(dir, { recursive: true })+ const rec = (o: object) => JSON.stringify(o)+ const item = (payload: object) => rec({ type: 'response_item', payload })+ const user = (t: string) => item({ type: 'message', role: 'user', content: [{ type: 'input_text', text: t }] })+ // The output shape codex 0.160.0 recorded for an MCP call on the grid (grid-dev, 2026-10-01).+ const out = (callId: string, text: string) => item({ type: 'function_call_output', call_id: callId, output: [{ type: 'input_text', text: 'Wall time: 0.4 seconds\nOutput:' }, { type: 'input_text', text }] })+ writeFileSync(join(dir, 'rollout-x.jsonl'), [+ user('Delete the file notes/old-2.txt. (ref-aaaaaa)'),+ user('<environment_context>a context line in the user role, mid-step</environment_context>'),+ user('Search the web for the latest stable version of Node.js and tell me the version number. (ref-bbbbbb)'),+ user('<environment_context>a context line in the user role, mid-step</environment_context>'),+ item({ type: 'function_call', name: 'web_search', namespace: 'mcp__harness', arguments: '{"query":"node"}', call_id: 'c1' }),+ out('c1', 'Node.js 24.11.0 (LTS) https://nodejs.org/en/download'),+ user('Search again. (ref-cccccc)'),+ item({ type: 'function_call', name: 'web_search', namespace: 'mcp__harness', arguments: '{}', call_id: 'c2' }),+ out('c2', 'MCP tool call requires approval, which the policy denies'),+ ].join('\n'))+ expect(toolResult('codex', '/unused', 'ref-aaaaaa', 'mcp__harness__web_search', home)).toEqual({ called: false, answered: false })+ expect(toolResult('codex', '/unused', 'ref-bbbbbb', 'mcp__harness__web_search', home)).toMatchObject({ called: true, answered: true })+ // Called, refused: no results, and the refusal is what the probe quotes.+ expect(toolResult('codex', '/unused', 'ref-cccccc', 'mcp__harness__web_search', home)).toEqual({+ called: true, answered: false, output: 'Wall time: 0.4 seconds Output: MCP tool call requires approval, which the policy denies',+ })+ } finally {+ rmSync(home, { recursive: true, force: true })+ }+ })++ it('claude: the web search tool_use, its tool_result, and an error result', () => {+ const home = mkdtempSync(join(tmpdir(), 'wd-sess-'))+ try {+ const cwd = '/tmp/ws'+ const dir = join(home, '.claude', 'projects', '-tmp-ws')+ mkdirSync(dir, { recursive: true })+ const rec = (o: object) => JSON.stringify(o)+ const prompt = (t: string) => rec({ type: 'user', message: { role: 'user', content: t } })+ const call = (id: string) => rec({ type: 'assistant', message: { model: 'DeepSeek-V4-Flash-0731', content: [{ type: 'tool_use', id, name: 'mcp__harness__web_search', input: { query: 'node' } }] } })+ const result = (id: string, text: string, isError = false) => rec({ type: 'user', message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: id, is_error: isError, content: [{ type: 'text', text }] }] } })+ writeFileSync(join(dir, 's.jsonl'), [+ prompt('Search the web for the latest stable version of Node.js and tell me the version number. (ref-dddddd)'),+ call('t1'), result('t1', 'Node.js 24.11.0 https://nodejs.org'),+ prompt('Search again. (ref-eeeeee)'),+ call('t2'), result('t2', 'see https://example.com — upstream timed out', true),+ ].join('\n'))+ expect(toolResult('claude', cwd, 'ref-dddddd', 'mcp__harness__web_search', home)).toMatchObject({ called: true, answered: true })+ expect(toolResult('claude', cwd, 'ref-eeeeee', 'mcp__harness__web_search', home)).toMatchObject({ called: true, answered: false })+ } finally {+ rmSync(home, { recursive: true, force: true })+ }+ })+})++describe('grid model pick (live)', () => {+ it('takes the smallest model on any listed grid, own grid first, even when the top-level list is empty', () => {+ const pick = pickGridModel({+ gridName: 'mine',+ models: [],+ localModelEngines: null,+ grids: [+ { name: 'mine', own: true, models: [] },+ { name: 'autonomous.ai', own: false, models: [+ { id: 'DeepSeek-V4-Flash-0731', node: 'a' },+ { id: 'Qwen/Qwen3.8-27B', node: 'b' },+ { id: 'Qwen3.6-35B-A3B', node: 'c' },+ { id: 'gemma-4-31B-it', node: 'd' },+ ] },+ ],+ })+ expect(pick).toEqual({ model: 'Qwen/Qwen3.8-27B', gridName: 'autonomous.ai' })+ expect(pickGridModel({ gridName: null, models: [], grids: [], localModelEngines: null })).toBeNull()+ })+})++describe('workspace (the person\'s project the agent is created in)', () => {+ it('lays out the notes, the MCP config for the engine, and the log the MCP step is proven by', () => {+ const cwd = mkdtempSync(join(tmpdir(), 'wd-ws-'))+ try {+ // A stand-in for the codex binary: it records the argv it was called with and answers+ // `mcp list` the way a codex that accepted the registration would.+ const codexBin = join(cwd, 'fake-codex')+ const argvLog = join(cwd, 'fake-codex.log')+ writeFileSync(codexBin, `#!/bin/sh\necho "$@" >> ${JSON.stringify(argvLog)}\n[ "$2" = list ] && echo "e2e_calc node /x/calc-mcp.mjs"\nexit 0\n`)+ chmodSync(codexBin, 0o755)+ // The tool approvals are appended to codex's config: a copy here, never the developer's own.+ const codexConfig = join(cwd, 'codex-config.toml')+ writeFileSync(codexConfig, 'model = "gpt-6-luna"\n')+ const laid = prepareWorkspace(cwd, 'codex', { codexBin, codexConfig })+ // The file steps' files: a fresh unguessable secret per side, a todo to edit.+ const secrets = [1, 2].map((n) => readWorkspaceFile(cwd, `notes/info-${n}.txt`)?.trim() ?? '')+ for (const t of secrets) expect(t).toMatch(/^[a-z]+-\d{4}-[a-z]+$/)+ expect(secrets[0]).not.toBe(secrets[1])+ expect(readWorkspaceFile(cwd, 'notes/todo-1.txt')).toContain('status: pending')+ expect(existsSync(join(cwd, 'out'))).toBe(true)+ expect(readWorkspaceFile(cwd, 'out/hello-1.txt')).toBeNull() // written by the step, never by us+ // Fresh per run: two workspaces never share a secret.+ const other = mkdtempSync(join(tmpdir(), 'wd-ws-'))+ prepareWorkspace(other, 'claude')+ expect(readWorkspaceFile(other, 'notes/info-1.txt')).not.toBe(readWorkspaceFile(cwd, 'notes/info-1.txt'))+ rmSync(other, { recursive: true, force: true })+ // The registration is codex's own command, with the server and its log as argv — and a stale+ // entry is removed before the add, so an interrupted run cannot leave one behind.+ const calls = readFileSync(argvLog, 'utf8').trim().split('\n')+ expect(calls[0]).toBe('mcp remove e2e_calc')+ expect(calls[1]).toBe(`mcp add e2e_calc -- node ${laid.mcpServer} ${logPath(cwd, 'mcp')}`)+ expect(calls[2]).toBe('mcp list')+ // The string shipped in the bundle is the file in workspace/, byte for byte.+ expect(CALC_MCP_MJS).toBe(readFileSync(join(__dirname, 'workspace', 'calc-mcp.mjs'), 'utf8'))+ expect(existsSync(join(cwd, 'tools', 'calc-mcp.mjs'))).toBe(true)+ // codex has no project-level MCP config — a `<cwd>/.codex/config.toml` is never read (measured:+ // `codex mcp list` inside such a workspace does not list the server), so the workspace must not+ // pretend otherwise. Registration goes through `codex mcp add`, which this test does not run:+ // it would write the developer's own ~/.codex/config.toml. What is pinned is that the run says+ // out loud which of the two happened.+ expect(existsSync(join(cwd, '.codex', 'config.toml'))).toBe(false)+ expect(laid.mcpConfig).toBe(codexConfig)+ expect(laid.mcpNote).toMatch(/codex mcp add/)+ // Its tools always allowed, as on a developer's machine: codex then never sends them to its+ // reviewer, whose `codex-auto-review` model the grid does not serve (openai/codex#24879).+ const toml = readFileSync(codexConfig, 'utf8')+ expect(toml.startsWith('model = "gpt-6-luna"\n')).toBe(true) // appended, nothing rewritten+ for (const tool of ['add', 'sub']) expect(toml).toContain(`[mcp_servers.e2e_calc.tools.${tool}]\napproval_mode = "approve"\n`)+ // The MCP server answers tools/call and leaves its line in the MCP log.+ const rpc = ['{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}', '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"sub","arguments":{"a":60,"b":18}}}'].join('\n') + '\n'+ const out = execFileSync('node', [laid.mcpServer, logPath(cwd, 'mcp')], { input: rpc }).toString()+ expect(out).toContain('"text":"42"')+ expect(readLog(cwd, 'mcp').at(-1)).toContain('sub 60 18 = 42')+ // claude gets .mcp.json instead.+ const claude = prepareWorkspace(cwd, 'claude')+ expect(JSON.parse(readFileSync(join(cwd, '.mcp.json'), 'utf8')).mcpServers.e2e_calc.command).toBe('node')+ expect(claude.mcpConfig).toBe(join(cwd, '.mcp.json'))+ // …and the project settings a person who said "don't ask again" has: the steps' tools allowed.+ const settings = JSON.parse(readFileSync(join(cwd, '.claude', 'settings.json'), 'utf8'))+ expect(settings.permissions.allow).toEqual(expect.arrayContaining(['Write', 'Edit', 'mcp__e2e_calc', 'mcp__harness']))+ expect(settings.enabledMcpjsonServers).toEqual(['e2e_calc'])+ } finally {+ rmSync(cwd, { recursive: true, force: true })+ }+ })+})diff --git a/cli/src/e2e/fixedModels.ts b/cli/src/e2e/fixedModels.tsnew file mode 100644index 000000000..c2a714510--- /dev/null+++ b/cli/src/e2e/fixedModels.ts@@ -0,0 +1,13 @@+/**+ * Fixed models for the real subscription round-trip leg of the matrix (per user decision).+ *+ * The grid leg picks a model from `grid_models_list`; the subscription leg (codex/claude answering+ * on its OWN login) uses a fixed, cheap alias that is enabled in the tmux launch. codex and claude+ * expose no model-list CLI, so these are explicit and tuneable in one place.+ */+import type { AgentEngine } from '../engines/types.js'++export const SUBSCRIPTION_MODEL: Partial<Record<AgentEngine, string>> = {+ codex: 'gpt-5.5',+ claude: 'claude-sonnet-5',+}diff --git a/cli/src/e2e/gridArgv.ts b/cli/src/e2e/gridArgv.tsnew file mode 100644index 000000000..92fe3fe1f--- /dev/null+++ b/cli/src/e2e/gridArgv.ts@@ -0,0 +1,85 @@+/**+ * argv-package — the deterministic spawn spec the real-matrix uses to reproduce EXACTLY what the+ * harness enters when an agent switches to a grid and back to its own login.+ *+ * It does NOT re-implement the launch. It reuses the harness's own builders+ * (`buildGridEngineLaunch` for the grid env/argv, `buildEngineCommandArgv` for the final command,+ * `gridEnvVarNames` for what to clear on the way home), so the matrix spawns the same argv the+ * daemon would, field for field — no drift, no self-rolled copy.+ */+import { type GridLaunchOverride, buildGridEngineLaunch, gridEnvVarNames } from '../lib/gridLaunch.js'+import { buildEngineCommandArgv } from '../lib/engineLaunch.js'+import type { AgentEngine } from '../engines/types.js'++/** A config file the launch writes into a private dir the engine is pointed at. */+export interface SpawnConfigFile {+ name: string+ content: string+}++/** Resolved spawn for one transition: what the daemon would hand `tmux respawn-pane`. */+export interface SpawnSpec {+ /** Full engine argv, `command[0]` is the engine binary (matches `buildEngineCommandArgv`). */+ command: string[]+ /** Environment layered over the engine's inherited env (where the grid key goes). */+ env: Record<string, string>+ /** Env vars that must be cleared so a stale grid/vendor credential is not inherited. */+ clearEnv: string[]+ /** A private config dir the engine reads its provider from (e.g. opencode), if any. */+ configDir?: {+ /** The variable the engine reads it by; absent for one handed a flag instead (Pi's `-e`, [arg]). */+ envVar?: string+ arg?: string+ pointAt?: string+ files: SpawnConfigFile[]+ }+}++export interface GridSpawnError {+ ok: false+ error: string+ detail: string+}++function refuse(error: string, detail: string): GridSpawnError {+ return { ok: false, error, detail }+}++/** Spawn spec to move `engine` onto `grid` (optionally resuming `resumeId`). */+export function gridSpawn(+ engine: AgentEngine,+ grid: GridLaunchOverride,+ resumeId?: string,+): SpawnSpec | GridSpawnError {+ const built = buildGridEngineLaunch(engine, grid, { hermesSystemManaged: false })+ if (!built.ok) {+ return { ok: built.ok, error: built.error, detail: built.detail }+ }+ const command = buildEngineCommandArgv(engine, {+ ...(resumeId ? { resumeSessionId: resumeId } : {}),+ extraArgs: built.launch.args,+ })+ return {+ command,+ env: built.launch.env,+ clearEnv: gridEnvVarNames(engine),+ ...(built.launch.configDir+ ? {+ configDir: {+ ...(built.launch.configDir.envVar ? { envVar: built.launch.configDir.envVar } : {}),+ ...(built.launch.configDir.arg ? { arg: built.launch.configDir.arg } : {}),+ ...(built.launch.configDir.pointAt ? { pointAt: built.launch.configDir.pointAt } : {}),+ files: built.launch.configDir.files.map((f) => ({ name: f.name, content: f.content })),+ },+ }+ : {}),+ }+}++/** Spawn spec to return `engine` to its own vendor login (clear the grid, nothing else set). */+export function backHomeSpawn(engine: AgentEngine, resumeId?: string): SpawnSpec {+ const command = buildEngineCommandArgv(engine, {+ ...(resumeId ? { resumeSessionId: resumeId } : {}),+ })+ return { command, env: {}, clearEnv: gridEnvVarNames(engine) }+}diff --git a/cli/src/e2e/gridSwitchDriver.spec.ts b/cli/src/e2e/gridSwitchDriver.spec.tsnew file mode 100644index 000000000..b25506e4c--- /dev/null+++ b/cli/src/e2e/gridSwitchDriver.spec.ts@@ -0,0 +1,189 @@+import http from 'node:http'+import type { AddressInfo } from 'node:net'+import { afterEach, describe, expect, it } from 'vitest'+import type { Frame, LocalClientSink } from '../backendSocket.js'+import { attachLocalWsServer, type LocalWsBackend, type LocalWsServer } from '../localWsServer.js'+import { GridSwitchDriver, LOCAL_WS_PATH, runGridSwitchFlow } from './gridSwitchDriver.js'++const machineId = 'machine-abc'+const agentId = 'session-1'+const GRID_MODEL = 'grid:gpt-5-mini' // a model id the daemon's grid_models_list would report++/**+ * A local backend that answers the SAME RPCs the real daemon answers, using the SAME frame+ * convention (`{ type, payload: { requestId, ...} }`). This is the seam the driver talks to —+ * in a live run it is the actual BackendSocket in the daemon; here it stands in so the test+ * is offline and costs $0, while still exercising the real WebSocket transport.+ */+class FakeDaemon implements LocalWsBackend {+ sink: LocalClientSink | null = null+ connId: string | null = null+ received: Array<{ type: string; payload: Record<string, unknown> }> = []+ retargetCalls: Array<{ clear: boolean; model?: string; gridName?: string }> = []++ registerLocalClient(connId: string, sink: LocalClientSink): boolean {+ this.connId = connId+ this.sink = sink+ return true+ }++ async unregisterLocalClient(connId: string): Promise<void> {+ this.received.push({ type: '__unregister', payload: { connId } })+ }++ send(type: string, payload: Record<string, unknown>): void {+ this.sink?.sendFrame({ type, payload } as Frame)+ }++ handleLocalFrame(_connId: string, frame: Frame): void {+ const type = frame.type as string+ const payload = (frame.payload as Record<string, unknown>) ?? {}+ this.received.push({ type, payload })+ const requestId = payload.requestId as string | undefined+ switch (type) {+ case 'machine_select': {+ this.send('connected', {+ requestId,+ machineId,+ transport: 'local',+ e2ee: false,+ } as unknown as Record<string, unknown>)+ break+ }+ case 'grid_models_list': {+ this.send('grid_models_list', {+ requestId,+ gridName: 'someone-7f3a91c4',+ models: [{ id: GRID_MODEL, node: 'node-x' }],+ grids: [+ {+ name: 'shared-lab',+ own: false,+ models: [{ id: 'shared:opus', node: 'node-y' }],+ },+ ],+ localModelEngines: ['codex', 'claude'],+ gridCli: { version: '1.0.0' },+ } as unknown as Record<string, unknown>)+ break+ }+ case 'agent_retarget': {+ const clear = payload.clearGrid === true+ const model = typeof payload.gridModel === 'string' ? payload.gridModel : undefined+ const gridName = typeof payload.gridName === 'string' ? payload.gridName : undefined+ this.retargetCalls.push({ clear, model, gridName })+ // Absent gridModel + clear==false is an invalid frame, exactly as the daemon refuses it.+ if (!clear && !model) {+ this.send('agent_retarget', { requestId, error: 'INVALID_GRID', detail: 'grid is required' })+ break+ }+ this.send('agent_retarget', { requestId, retargeted: true })+ break+ }+ default:+ this.send(type, { requestId, error: 'UNSUPPORTED' })+ }+ }++ async handleLocalBinary(): Promise<void> {}+}++describe('GridSwitchDriver (talks to the daemon like the app)', () => {+ let server: http.Server | null = null+ let local: LocalWsServer | null = null+ let daemon: FakeDaemon | null = null++ afterEach(async () => {+ await local?.close()+ await new Promise<void>((resolve) => server?.close(() => resolve()) ?? resolve())+ local = null+ server = null+ daemon = null+ })++ async function startDaemon(localCredential?: string): Promise<string> {+ daemon = new FakeDaemon()+ server = http.createServer((_req, res) => {+ res.statusCode = 404+ res.end()+ })+ local = attachLocalWsServer(server, { machineId, backend: daemon, ...(localCredential ? { localCredential } : {}) })+ await new Promise<void>((resolve) => server!.listen(0, '127.0.0.1', resolve))+ const port = (server.address() as AddressInfo).port+ return `ws://127.0.0.1:${port}${LOCAL_WS_PATH}`+ }++ it('performs the full app journey and records every value/step', async () => {+ const url = await startDaemon()+ const outcome = await runGridSwitchFlow(url, { machineId, agentId })++ // 1) Handshake completes over the real JSON frame.+ expect(outcome.connected).toMatchObject({ machineId, transport: 'local', e2ee: false })++ // 2) grid_models_list returns the picker values the app would render.+ expect(outcome.models.gridName).toBe('someone-7f3a91c4')+ expect(outcome.models.models[0].id).toBe(GRID_MODEL)+ expect(outcome.models.grids[0].name).toBe('shared-lab')+ expect(outcome.models.localModelEngines?.has('codex')).toBe(true)++ // 3) Default pick = first model; moving onto the grid succeeds.+ expect(outcome.ontoGrid.ok).toBe(true)+ expect(outcome.ontoGrid.retargeted).toBe(true)++ // 4) Coming back to own login succeeds.+ expect(outcome.backHome.ok).toBe(true)+ expect(outcome.backHome.retargeted).toBe(true)++ // 5) The daemon saw the EXACT payloads the desktop app sends.+ const retarget = daemon!.received.filter((r) => r.type === 'agent_retarget')+ expect(retarget).toHaveLength(2)+ expect(retarget[0].payload).toMatchObject({ agentId, gridModel: GRID_MODEL }) // onto grid+ expect(retarget[1].payload).toMatchObject({ agentId, clearGrid: true }) // back home+ expect(retarget[1].payload.gridModel).toBeUndefined()++ // 6) The trace is the human-readable step/value log asked for.+ const steps = outcome.trace.map((s) => s.step)+ expect(steps).toEqual(['machine_select', 'grid_models_list', 'agent_retarget(grid)', 'agent_retarget(clear)'])+ })++ it('replays an explicit grid name/model exactly as the app sends them', async () => {+ const url = await startDaemon()+ const driver = new GridSwitchDriver(url, machineId)+ try {+ await driver.machineSelect()+ await driver.gridModels()+ const res = await driver.retargetToGrid(agentId, 'shared:opus', 'shared-lab')+ expect(res.ok).toBe(true)+ const call = daemon!.retargetCalls.find((c) => !c.clear)!+ expect(call).toMatchObject({ model: 'shared:opus', gridName: 'shared-lab' })+ } finally {+ driver.close()+ }+ })++ it('surfaces a daemon refusal as an error result with detail', async () => {+ const url = await startDaemon()+ const driver = new GridSwitchDriver(url, machineId)+ try {+ await driver.machineSelect()+ // No gridModel, no clearGrid -> daemon refuses with INVALID_GRID.+ const res = await driver.retargetToGrid(agentId, '', undefined)+ expect(res.ok).toBe(false)+ expect(res.error).toBe('INVALID_GRID')+ expect(res.detail).toBe('grid is required')+ } finally {+ driver.close()+ }+ })++ // A daemon since guest machine sharing (bc7f56ff9, CLI 0.4.21) takes a TCP upgrade only with its local+ // credential: grid-dev's run would have been refused (401) right after reading the machine.+ it('connects to a daemon that requires the local credential, with the headers it is given', async () => {+ const credential = 'k'.repeat(43)+ const url = await startDaemon(credential)+ await expect(runGridSwitchFlow(url, { machineId, agentId })).rejects.toThrow(/401/)+ const outcome = await runGridSwitchFlow(url, { machineId, agentId, headers: { 'X-Harness-Local': credential } })+ expect(outcome.connected).toMatchObject({ machineId, transport: 'local' })+ expect(outcome.backHome.ok).toBe(true)+ })+})diff --git a/cli/src/e2e/gridSwitchDriver.ts b/cli/src/e2e/gridSwitchDriver.tsnew file mode 100644index 000000000..8f7e0b001--- /dev/null+++ b/cli/src/e2e/gridSwitchDriver.ts@@ -0,0 +1,332 @@+/**+ * GridSwitchDriver — an E2E driver that talks to the harness daemon the SAME way the desktop+ * app does, so it can be used to reproduce the real value / step flow of switching an agent+ * between a private grid and its own vendor login, without a human clicking the UI.+ *+ * Transport: a plain WebSocket to the daemon's local socket (`<origin>/api/local-ws`, the+ * `LOCAL_WS_PATH` the app connects to via `ws_conn.request`). The protocol is:+ * client -> { type: 'machine_select', payload: { machineId, localProtocolVersion: 1 } }+ * client <- { type: 'connected', payload: { machineId, transport: 'local', e2ee: false } }+ * client -> { type: <cmd>, payload: { ...cmd args, requestId } }+ * client <- { type: <cmd>, payload: { requestId, ...result } } (errors carry { error, detail? })+ *+ * The commands replayed here are the exact ones the app sends (`desktop/lib/state/app_state.dart`):+ * - `grid_models_list` -> models/grids/gridName for the pane header picker+ * - `agent_retarget` (grid) -> { agentId, gridModel, gridName? } move ONTO a grid+ * - `agent_retarget` (clearGrid) -> { agentId, clearGrid: true } move back to own login+ *+ * Every reply is threaded back by matching `requestId`, and each exchange is recorded into a+ * `GridSwitchTrace` — the step/value log that answers "khi tôi thao tác nó làm gì, value nào".+ */+import { WebSocket } from 'ws'++export const LOCAL_WS_PATH = '/api/local-ws'+export const LOCAL_WS_PROTOCOL_VERSION = 1++/** One recorded exchange in the trace. */+export interface GridSwitchTraceStep {+ step: string+ /** The frame the driver (acting as the app) sent. */+ request: Record<string, unknown>+ /** The daemon's reply, already decoded. */+ reply: Record<string, unknown>+}++export interface GridModel {+ id: string+ node: string+ grid?: string+}++export interface GridSection {+ name: string+ own: boolean+ models: GridModel[]+}++export interface GridModelsAnswer {+ gridName: string | null+ models: GridModel[]+ grids: GridSection[]+ localModelEngines: Set<string> | null+}++/** The full grid object `agent_retarget` accepts in place of a picker model id. */+export interface GridOverride {+ networkId: string+ networkName: string+ baseUrl: string+ apiKey: string+}++export interface GridSwitchResult {+ ok: boolean+ retargeted: boolean+ error?: string+ detail?: string+}++/** A tiny request/reply WS client matching the backend frame convention. */+export class LocalWsClient {+ private ws: WebSocket | null = null+ private pending = new Map<string, { resolve: (v: Record<string, unknown>) => void; reject: (e: Error) => void }>()+ private idSeq = 0+ /** Events the daemon pushes without a matching request (e.g. `connected`, `announce`). */+ private pushes: ((frame: Record<string, unknown>) => void) | null = null++ onPush(cb: ((frame: Record<string, unknown>) => void) | null): void {+ this.pushes = cb+ }++ /** [headers]: the daemon's local credential (lib/localCredential.ts), which its TCP upgrade requires. */+ connect(url: string, headers: Record<string, string> = {}): Promise<void> {+ return new Promise((resolve, reject) => {+ const ws = new WebSocket(url, { headers })+ this.ws = ws+ ws.on('open', () => resolve())+ ws.on('error', (err) => reject(err))+ ws.on('message', (raw) => {+ let frame: Record<string, unknown>+ try {+ frame = JSON.parse(raw.toString()) as Record<string, unknown>+ } catch {+ return+ }+ this.dispatch(frame)+ })+ ws.on('close', () => {+ for (const [, { reject }] of this.pending) reject(new Error('socket closed'))+ this.pending.clear()+ })+ })+ }++ private dispatch(frame: Record<string, unknown>): void {+ const payload = (frame.payload ?? {}) as Record<string, unknown>+ const requestId = typeof payload.requestId === 'string' ? payload.requestId : undefined+ if (requestId !== undefined && this.pending.has(requestId)) {+ const entry = this.pending.get(requestId)!+ this.pending.delete(requestId)+ entry.resolve(payload)+ return+ }+ if (this.pushes) this.pushes(frame)+ }++ /** Send a command with a fresh requestId and await the matching reply. */+ request(type: string, payload: Record<string, unknown>): Promise<Record<string, unknown>> {+ const requestId = `req-${++this.idSeq}`+ const full = { ...payload, requestId }+ return new Promise((resolve, reject) => {+ if (!this.ws || this.ws.readyState !== WebSocket.OPEN) {+ reject(new Error('socket not open'))+ return+ }+ this.pending.set(requestId, { resolve, reject })+ this.ws.send(JSON.stringify({ type, payload: full }))+ })+ }++ /**+ * Send a command whose ack is delivered as an UNREQUESTED push rather than a requestId reply+ * (`machine_select` -> the server's own `connected` frame). Resolves with the pushed payload.+ */+ sendAndAwaitPush(type: string, payload: Record<string, unknown>, awaitType: string): Promise<Record<string, unknown>> {+ return new Promise((resolve, reject) => {+ if (!this.ws || this.ws.readyState !== WebSocket.OPEN) {+ reject(new Error('socket not open'))+ return+ }+ const timer = setTimeout(() => reject(new Error(`timeout waiting for ${awaitType}`)), 8000)+ const prev = this.pushes+ this.pushes = (frame) => {+ if (frame.type === awaitType) {+ clearTimeout(timer)+ this.pushes = prev+ const p = (frame.payload ?? {}) as Record<string, unknown>+ resolve(p)+ }+ prev?.(frame)+ }+ this.ws.send(JSON.stringify({ type, payload }))+ })+ }++ close(): void {+ try {+ this.ws?.close()+ } catch {+ /* ignore */+ }+ this.ws = null+ }+}++/**+ * The driver. `daemonUrl` is the local socket the real daemon listens on, e.g.+ * `ws://127.0.0.1:${hookPort}/api/local-ws`. Point it at any running daemon, or at a+ * spec's in-process server.+ */+export class GridSwitchDriver {+ readonly trace: GridSwitchTraceStep[] = []+ private client = new LocalWsClient()++ /** [headers] go on the socket's upgrade: a real daemon's local credential (harnessLocal.ts `localHeaders`). */+ constructor(private readonly daemonUrl: string, private readonly machineId: string, private readonly headers: Record<string, string> = {}) {}++ private record(step: string, request: Record<string, unknown>, reply: Record<string, unknown>): void {+ this.trace.push({ step, request, reply })+ }++ /** 1. Open the socket + select this machine. The ack is the server's `connected` push frame. */+ async machineSelect(): Promise<Record<string, unknown>> {+ await this.client.connect(this.daemonUrl, this.headers)+ const connected = await this.client.sendAndAwaitPush(+ 'machine_select',+ { machineId: this.machineId, localProtocolVersion: LOCAL_WS_PROTOCOL_VERSION },+ 'connected',+ )+ this.record('machine_select', { machineId: this.machineId }, connected)+ return connected+ }++ /** 2. Ask the daemon for the models its grid serves (the pane header picker). */+ async gridModels(): Promise<GridModelsAnswer> {+ const reply = await this.client.request('grid_models_list', {})+ this.record('grid_models_list', {}, reply)+ const models = ((reply.models as unknown) ?? []) as GridModel[]+ const grids = (((reply.grids as unknown) ?? []) as Array<Record<string, unknown>>).map((g) => ({+ name: g.name as string,+ own: g.own === true,+ models: ((g.models as unknown) ?? []) as GridModel[],+ }))+ const localEngines = reply.localModelEngines as unknown+ return {+ gridName: (reply.gridName as string) ?? null,+ models,+ grids,+ localModelEngines: Array.isArray(localEngines)+ ? new Set((localEngines as string[]).map((e) => e.toLowerCase()))+ : null,+ }+ }++ /**+ * 3. The app's picker choice: move the agent ONTO a grid model. With `override`, the full grid+ * object is sent instead (the daemon accepts both): a relay the daemon's own `grid` sign-in does+ * not know — a local relay checkout under test — with the key recorded out of the trace.+ */+ async retargetToGrid(agentId: string, gridModel: string, gridName?: string, override?: GridOverride): Promise<GridSwitchResult> {+ const payload: Record<string, unknown> = override ? { agentId, grid: { ...override, model: gridModel } } : { agentId, gridModel }+ if (gridName !== undefined && !override) payload.gridName = gridName+ const reply = await this.client.request('agent_retarget', payload)+ const recorded = override ? { ...payload, grid: { ...(payload.grid as object), apiKey: '<redacted>' } } : payload+ this.record('agent_retarget(grid)', recorded, reply)+ return normalizeResult(reply)+ }++ /** 4. The "put it back" action: return the agent to its own vendor login. */+ async clearGrid(agentId: string): Promise<GridSwitchResult> {+ const payload = { agentId, clearGrid: true }+ const reply = await this.client.request('agent_retarget', payload)+ this.record('agent_retarget(clear)', payload, reply)+ return normalizeResult(reply)+ }++ /**+ * 0. New Harness: create a fresh agent for this run, the way the app's "New agent" does+ * (`agent_create` with an engine and an absolute cwd). The test never borrows someone's pane.+ */+ async createAgent(engine: string, cwd: string, name: string, permissionMode = 'auto'): Promise<{ ok: true; agentId: string; pane: string | null } | { ok: false; error: string; detail?: string }> {+ // `auto`, the app's default, so the agent is launched exactly as a person's is (see+ // runGridSwitchTrace for what `full` cost). The daemon validates the mode per engine+ // (INVALID_PERMISSION_MODE), so a wrong one fails loudly here instead of quietly launching the+ // engine in some other mode.+ const payload = { engine, cwd, name, permissionMode }+ const reply = await this.client.request('agent_create', payload)+ this.record('agent_create', payload, reply)+ if (reply.error) return { ok: false, error: reply.error as string, detail: reply.detail as string | undefined }+ const agent = reply.agent as { id?: string; tmuxPane?: string } | undefined+ if (!agent?.id) return { ok: false, error: 'NO_AGENT_ID', detail: 'agent_create replied without agent.id' }+ // The row already names its pane (`tmuxPane: "%N"`); the registry is only a fallback.+ const pane = typeof agent.tmuxPane === 'string' && /^%\d+$/.test(agent.tmuxPane) ? agent.tmuxPane : null+ return { ok: true, agentId: agent.id, pane }+ }++ /** 5. Clean up: delete the agent this run created (its pane goes with it). */+ async deleteAgent(agentId: string): Promise<boolean> {+ const payload = { agentId }+ const reply = await this.client.request('agent_delete', payload)+ this.record('agent_delete', payload, reply)+ return reply.deleted === true+ }++ close(): void {+ this.client.close()+ }+}++function normalizeResult(payload: Record<string, unknown>): GridSwitchResult {+ if (payload.error) {+ return { ok: false, retargeted: false, error: payload.error as string, detail: payload.detail as string | undefined }+ }+ return { ok: true, retargeted: payload.retargeted === true }+}++/**+ * Which grid model to move onto: the SMALLEST one served on any grid the daemon lists (own grid+ * first), so the real switch costs as little as possible. `models` at the top level is only the own+ * grid's list and is often empty — the models people actually share live under `grids[]`.+ */+export function pickGridModel(answer: GridModelsAnswer): { model: string; gridName: string } | null {+ // The own grid first: the top-level `models` (its list, under `gridName`), then every section+ // flagged `own`, then the rest. Ties on size keep the earlier one.+ const own: GridSection[] = answer.gridName ? [{ name: answer.gridName, own: true, models: answer.models }] : []+ const sections = [...own, ...answer.grids.filter((g) => g.own), ...answer.grids.filter((g) => !g.own)]+ let best: { model: string; gridName: string; size: number } | null = null+ for (const g of sections) {+ for (const m of g.models) {+ const size = paramsB(m.id)+ if (!best || size < best.size) best = { model: m.id, gridName: g.name, size }+ }+ }+ return best ? { model: best.model, gridName: best.gridName } : null+}++/** Parameter count in billions parsed from a model id (`Qwen3.8-27B` → 27, `…-35B-A3B` → 35); unknown → Infinity. */+function paramsB(id: string): number {+ const m = /(\d+(?:\.\d+)?)B(?![A-Za-z])/i.exec(id)+ return m ? Number(m[1]) : Number.POSITIVE_INFINITY+}++/** Run the whole flow in one go and hand back the trace + results. */+export interface GridFlowOutcome {+ connected: Record<string, unknown>+ models: GridModelsAnswer+ ontoGrid: GridSwitchResult+ backHome: GridSwitchResult+ trace: GridSwitchTraceStep[]+}++/**+ * Replay the full app journey against a daemon URL:+ * machine_select -> grid_models_list -> agent_retarget(onto grid) -> agent_retarget(clear)+ * `agentId` and `machineId` are whatever agent/session the caller wants to move.+ */+export async function runGridSwitchFlow(+ daemonUrl: string,+ opts: { machineId: string; agentId: string; gridModel?: string; gridName?: string; headers?: Record<string, string> },+): Promise<GridFlowOutcome> {+ const driver = new GridSwitchDriver(daemonUrl, opts.machineId, opts.headers)+ try {+ const connected = await driver.machineSelect()+ const models = await driver.gridModels()+ const picked = pickGridModel(models)+ const pick = opts.gridModel ?? picked?.model ?? models.models[0]?.id+ const ontoGrid = await driver.retargetToGrid(opts.agentId, pick, opts.gridName ?? (opts.gridModel ? undefined : picked?.gridName))+ const backHome = await driver.clearGrid(opts.agentId)+ return { connected, models, ontoGrid, backHome, trace: driver.trace }+ } finally {+ driver.close()+ }+}diff --git a/cli/src/e2e/harnessLocal.spec.ts b/cli/src/e2e/harnessLocal.spec.tsnew file mode 100644index 000000000..23c42c6e1--- /dev/null+++ b/cli/src/e2e/harnessLocal.spec.ts@@ -0,0 +1,43 @@+import { createServer, type Server } from 'node:http'+import type { AddressInfo } from 'node:net'+import { afterEach, describe, expect, it } from 'vitest'+import { startHookServer, type HookServerHandlers } from '../hookServer.js'+import { machineId } from './harnessLocal.js'++// The e2e reads the daemon it drives through the daemon's own local API, which since guest machine sharing+// (bc7f56ff9) answers the full /api/status only to a caller that shows the local credential. grid-dev's run+// stopped at "daemon /api/status has no machineId" the day its daemon updated to 0.4.21 (2026-10-19), because+// the e2e asked without it. Against the real server, so a change to that gate fails here, not on grid-dev.+let server: Server | null = null+afterEach(async () => {+ delete process.env.HARNESS_PORT+ delete process.env.HARNESS_MACHINE_ID+ await new Promise<void>((done) => (server ? server.close(() => done()) : done()))+ server = null+})++async function daemon(): Promise<void> {+ // A pinned id skips the daemon altogether: this test would then pass without asking it.+ delete process.env.HARNESS_MACHINE_ID+ const started = await startHookServer(0, {+ onStatus: () => ({ version: '9.9.9', pid: 1, computerId: 'c1', machineId: 'm1', signedIn: true, localWs: { path: '/api/local-ws' }, localSocket: null }),+ } as Partial<HookServerHandlers> as HookServerHandlers)+ server = started.server+ process.env.HARNESS_PORT = String(started.port)+}++describe('the e2e reads the machine it drives', () => {+ it('from a daemon that answers the full status only with the local credential', async () => {+ await daemon()+ await expect(machineId()).resolves.toBe('m1')+ })++ it('says the credential could not be read when the daemon gives only its public status', async () => {+ delete process.env.HARNESS_MACHINE_ID+ // What a daemon answers a caller without its credential (hookServer.ts `publicStatus`).+ server = createServer((_req, res) => { res.setHeader('content-type', 'application/json'); res.end(JSON.stringify({ credentialRequired: true, version: '9.9.9', pid: 1, computerId: 'c1' })) })+ await new Promise<void>((done) => server!.listen(0, '127.0.0.1', done))+ process.env.HARNESS_PORT = String((server.address() as AddressInfo).port)+ await expect(machineId()).rejects.toThrow(/wants its local credential, and this process could not read it/)+ })+})diff --git a/cli/src/e2e/harnessLocal.ts b/cli/src/e2e/harnessLocal.tsnew file mode 100644index 000000000..8c8fafd2f--- /dev/null+++ b/cli/src/e2e/harnessLocal.ts@@ -0,0 +1,103 @@+/**+ * harnessLocal — what the e2e reads from THIS machine's harness install, the same places the+ * daemon itself uses (`env.ts`), so nothing has to be passed in: the daemon's port, this+ * computer's id, and the tmux pane an agent runs in (the registry row's runtime).+ */+import { readFileSync } from 'node:fs'+import { join } from 'node:path'+import { execFile } from 'node:child_process'+import { promisify } from 'node:util'+import { env } from '../config/env.js'+import { localCredentialHeaders } from '../lib/localCredential.js'+import type { GridOverride } from './gridSwitchDriver.js'++const execFileP = promisify(execFile)++export function daemonPort(): number {+ return Number(process.env.HARNESS_PORT ?? env.PORT)+}++/**+ * What every request to the daemon's loopback TCP API carries: its local credential, read from its data+ * folder as every other client of it reads it (lib/localCredential.ts). Without it the daemon answers+ * `/api/status` with who it is but not its machine, and refuses `/api/local-ws` (401), since guest machine+ * sharing (bc7f56ff9, CLI 0.4.21): grid-dev's run stopped there on 2026-10-19.+ */+export function localHeaders(): Record<string, string> {+ return localCredentialHeaders(env.ADAPTER_DATA_DIR)+}++export function daemonUrl(): string {+ return `ws://127.0.0.1:${daemonPort()}/api/local-ws`+}++/**+ * The machine id the RUNNING daemon answers `machine_select` for — read from its own+ * `/api/status`, the same way `harness status` does. Not `computer-id`: that is this box's+ * identity, while the daemon binds a backend-issued machine id, and a `machine_select` for any+ * other id is relayed as a remote machine (`NO_PEER_LINK`).+ */+export async function machineId(): Promise<string> {+ if (process.env.HARNESS_MACHINE_ID) return process.env.HARNESS_MACHINE_ID+ const res = await fetch(`http://127.0.0.1:${daemonPort()}/api/status`, { headers: localHeaders(), signal: AbortSignal.timeout(3_000) })+ if (!res.ok) throw new Error(`daemon /api/status answered ${res.status} — is \`harness\` running on port ${daemonPort()}?`)+ const status = (await res.json()) as { machineId?: unknown; credentialRequired?: unknown }+ // The public answer: this process could not show the credential (another OS user, or another data folder).+ if (status.credentialRequired === true) {+ throw new Error(`daemon on port ${daemonPort()} wants its local credential, and this process could not read it from ${env.ADAPTER_DATA_DIR}/hook-credential — run as the daemon's user, with its ADAPTER_DATA_DIR`)+ }+ if (typeof status.machineId !== 'string' || !status.machineId) throw new Error('daemon /api/status has no machineId')+ return status.machineId+}++interface RegistryRow {+ agentId?: string+ tmuxPane?: string+ runtimes?: Array<{ backend?: string; paneId?: string }>+}++/** The `%N` tmux pane of `agentId`, from the registry the daemon writes; null until it is registered. */+export function paneOf(agentId: string): string | null {+ let rows: RegistryRow[]+ try {+ const raw = JSON.parse(readFileSync(join(env.ADAPTER_DATA_DIR, 'registry.json'), 'utf8')) as unknown+ rows = Array.isArray(raw) ? (raw as RegistryRow[]) : []+ } catch {+ return null+ }+ const row = rows.find((r) => r.agentId === agentId)+ if (!row) return null+ const pane = row.runtimes?.find((r) => r.backend === 'tmux')?.paneId ?? row.tmuxPane+ return pane && /^%\d+$/.test(pane) ? pane : null+}++/** Poll `paneOf` until the daemon has registered the agent's pane (a fresh agent takes a moment). */+export async function waitForPane(agentId: string, timeoutMs = 30_000): Promise<string | null> {+ const start = Date.now()+ while (Date.now() - start < timeoutMs) {+ const pane = paneOf(agentId)+ if (pane) return pane+ await new Promise((r) => setTimeout(r, 500))+ }+ return null+}++/**+ * A grid the daemon is NOT signed into — a local relay checkout under test — described the way+ * `agent_retarget` takes it. Resolved from another `GRID_HOME` through the public `grid` CLI:+ * `grid info --env <name>` gives the relay URL + key, `grid ls` the network id. Set+ * E2E_GRID_HOME (+ E2E_GRID_CONTROL_PLANE_URL) and E2E_GRID_NAME to use it.+ */+export async function gridOverrideFromEnv(): Promise<{ override: GridOverride; name: string } | null> {+ const home = process.env.E2E_GRID_HOME+ const name = process.env.E2E_GRID_NAME+ if (!home || !name) return null+ const env = { ...process.env, GRID_HOME: home, ...(process.env.E2E_GRID_CONTROL_PLANE_URL ? { GRID_CONTROL_PLANE_URL: process.env.E2E_GRID_CONTROL_PLANE_URL } : {}) }+ const { stdout: envOut } = await execFileP('grid', ['info', '--env', name], { env, timeout: 30_000 })+ const baseUrl = /OPENAI_BASE_URL='([^']+)'/.exec(envOut)?.[1]+ const apiKey = /OPENAI_API_KEY='([^']+)'/.exec(envOut)?.[1]+ const { stdout: ls } = await execFileP('grid', ['ls'], { env, timeout: 30_000 })+ const networkId = ls.split('\n').map((l) => l.trim().split(/\s+/)).find((cols) => cols[0] === name)?.[1]+ if (!baseUrl || !apiKey || !networkId) throw new Error(`could not resolve grid ${name} from GRID_HOME=${home}`)+ return { override: { networkId, networkName: name, baseUrl, apiKey }, name }+}diff --git a/cli/src/e2e/matrix.ts b/cli/src/e2e/matrix.tsnew file mode 100644index 000000000..31cbe5f95--- /dev/null+++ b/cli/src/e2e/matrix.ts@@ -0,0 +1,118 @@+/**+ * matrix — turn a planned trigger run into a concrete, auditable matrix entry: the exact spawn+ * argv (from the harness's real builders), the login/grid presence NOTIFYs, and a namespaced+ * session for every artifact. A matrix run is "the real test" — subscription -> grid -> back home.+ *+ * `--dry-run` (default) does everything except touch a live agent: it builds argv, checks+ * login/grid availability, names the session, and emits the report. It is the $0 offline path that+ * still proves the trigger + argv + naming + notify decision. Live switching adds a daemon + agent.+ */+import { type GridLaunchOverride, parseGridLaunchOverride } from '../lib/gridLaunch.js'+import type { AgentEngine } from '../engines/types.js'+import { gridSpawn, backHomeSpawn } from './gridArgv.js'+import { sessionName } from './sessionName.js'+import type { PlannedRun } from './trigger.js'+import { SUBSCRIPTION_MODEL } from './fixedModels.js'+import { plannedLegs, type LegOutcome } from './smokeChecks.js'+import type { GridSwitchTraceStep } from './gridSwitchDriver.js'++/** A placeholder override for dry-run: proves the argv builder without a real grid endpoint. */+export function probeGridOverride(): GridLaunchOverride {+ const raw = parseGridLaunchOverride({+ networkId: 'grid-probe',+ networkName: 'probe-grid',+ baseUrl: 'https://grid.invalid/probe/relay/v1',+ apiKey: 'probe-key',+ model: 'grid:gpt-5-mini',+ })+ return raw.state === 'ok' ? raw.override : (undefined as never)+}++export interface EngineStatus {+ binary: boolean+ login: boolean+ grid: boolean+ /** What the engine itself said about its login — e.g. `apikey`, or a grid token wearing one. */+ loginNote?: string+}++export interface MatrixEntry {+ engine: AgentEngine+ session: string+ version: string+ testcase: string+ gridModel: string | null+ gridSpawn: { command: string[]; envKeys: string[]; clearEnv: string[] } | { error: string; detail: string }+ backHome: { command: string[]; clearEnv: string[] }+ /** Fixed model used for the real subscription round-trip leg (per user decision). */+ subscriptionModel: string | null+ status: EngineStatus+ notify: string[]+ /**+ * The smoke checks per leg (tool / mcp / recall on subscription and grid).+ * Dry run: every check `not-run`. Live (`runGridSwitchTrace.ts`): the real outcomes + pane tails.+ */+ legs: LegOutcome[]+ /** Live runs only: the daemon exchanges, in order. */+ trace?: GridSwitchTraceStep[]+}++export const TESTCASE = 'grid-switch'++export interface MatrixDeps {+ status: (engine: AgentEngine) => Promise<EngineStatus>+ grid: () => Promise<GridLaunchOverride | null> // null -> no grid resolved (live) -> NOTIFY+ at?: Date+}++const dryGrid = probeGridOverride()++/**+ * Build one dry-run audit entry for a planned run. Live switching (daemon + agent) is layered on+ * top of the same argv/session/status later; this is the decision + spawn + notify surface.+ */+export async function buildMatrixEntry(plan: PlannedRun, deps: MatrixDeps): Promise<MatrixEntry> {+ const session = sessionName({+ engine: plan.engine,+ version: plan.latest ?? 'UNKNOWN',+ testcase: TESTCASE,+ gridModel: plan.latest ? dryGrid.model : undefined,+ at: deps.at,+ })+ const status = await deps.status(plan.engine)+ const notify: string[] = []++ if (!status.binary) notify.push('ENGINE_MISSING: binary not found')+ if (!status.login) notify.push('NOT_LOGGED_IN: engine needs a subscription login')+ if (!status.grid) notify.push('GRID_UNAVAILABLE: machine not signed in to a grid (or no model)')++ const grid = (await deps.grid()) ?? dryGrid+ const build = gridSpawn(plan.engine, grid, 'SESSION_PLACEHOLDER')+ const backHome = backHomeSpawn(plan.engine, 'SESSION_PLACEHOLDER')++ const spawnResult: MatrixEntry['gridSpawn'] = isSpawnError(build)+ ? { error: build.error, detail: build.detail }+ : {+ command: build.command,+ envKeys: Object.keys(build.env),+ clearEnv: build.clearEnv,+ }++ return {+ engine: plan.engine,+ session,+ version: plan.latest ?? 'UNKNOWN',+ testcase: TESTCASE,+ gridModel: dryGrid.model ?? null,+ gridSpawn: spawnResult,+ backHome: { command: backHome.command, clearEnv: backHome.clearEnv },+ subscriptionModel: SUBSCRIPTION_MODEL[plan.engine] ?? null,+ status,+ notify,+ legs: plannedLegs(),+ }+}++function isSpawnError(spec: Awaited<ReturnType<typeof gridSpawn>>): spec is { ok: false; error: string; detail: string } {+ return 'ok' in spec && spec.ok === false+}diff --git a/cli/src/e2e/npmPackages.ts b/cli/src/e2e/npmPackages.tsnew file mode 100644index 000000000..43167812d--- /dev/null+++ b/cli/src/e2e/npmPackages.ts@@ -0,0 +1,10 @@+/**+ * The npm package that ships each engine, used by the npm version source to read its `latest`+ * dist-tag. Kept separate so the real registry reader and any spec both mean the same package.+ */+import type { AgentEngine } from '../engines/types.js'++export const NPM_PACKAGE: Partial<Record<AgentEngine, string>> = {+ codex: '@openai/codex',+ claude: '@anthropic-ai/claude-code',+}diff --git a/cli/src/e2e/paneProbe.ts b/cli/src/e2e/paneProbe.tsnew file mode 100644index 000000000..e68f70322--- /dev/null+++ b/cli/src/e2e/paneProbe.ts@@ -0,0 +1,332 @@+/**+ * paneProbe — run the smoke checks against a LIVE agent pane, the way a person would: type the+ * prompt into the tmux pane, wait for the marker to show up in what the pane prints.+ *+ * tmux only. The harness runs every agent in a tmux pane (`tmuxBackend.ts`), and `respawn-pane`+ * keeps the pane id across a grid switch, so one `%N` follows the agent through all three legs.+ *+ * Pure-ish: the tmux calls are injected (`Tmux`) so the sequencing — settle, type, poll, tail — is+ * unit-testable offline; `realTmux` is the one that shells out.+ */+import { execFile } from 'node:child_process'+import { promisify } from 'node:util'+import { scenarioFor, type CheckOutcome, type Leg, type LegOutcome, type LogKind, type SmokeCheck } from './smokeChecks.js'+import type { ToolResult } from './sessionTools.js'++const execFileP = promisify(execFile)+let typeSequence = 0++export interface Tmux {+ /** The pane's visible text plus scrollback tail (`capture-pane -p -S -<lines>`). */+ capture(pane: string, lines: number): Promise<string>+ /** Type `text` literally, then Enter. */+ type(pane: string, text: string): Promise<void>+ /** Press one key by tmux name (`1`, `Enter`, `Escape`) — for a dialog, not for a prompt. */+ key(pane: string, key: string): Promise<void>+ sleep(ms: number): Promise<void>+ now(): number+}++export const realTmux: Tmux = {+ async capture(pane, lines) {+ const { stdout } = await execFileP('tmux', ['capture-pane', '-p', '-t', pane, '-S', `-${lines}`], { timeout: 5000 })+ return stdout+ },+ async type(pane, text) {+ // Not `send-keys -l`: a whole line arriving as one keystroke burst trips the TUI's paste+ // detector (codex 0.155.1 kept the text in the composer and swallowed the Enter). The harness's+ // own injection (`lib/tmux.ts sendToTmux`) pastes from a named buffer, lets it land, then sends+ // Enter on its own — same sequence here.+ const buffer = `e2e-probe-${process.pid}-${++typeSequence}`+ await execFileP('tmux', ['set-buffer', '-b', buffer, '--', text], { timeout: 5000 })+ await execFileP('tmux', ['paste-buffer', '-t', pane, '-b', buffer, '-d'], { timeout: 5000 })+ await new Promise((r) => setTimeout(r, 400))+ await execFileP('tmux', ['send-keys', '-t', pane, 'Enter'], { timeout: 5000 })+ },+ async key(pane, key) {+ await execFileP('tmux', ['send-keys', '-t', pane, key], { timeout: 5000 })+ },+ sleep: (ms) => new Promise((r) => setTimeout(r, ms)),+ now: () => Date.now(),+}++/**+ * Startup dialogs a fresh coding tool puts in front of its prompt. A check typed into one of these+ * is not a check: the text is swallowed and Enter picks whatever is highlighted (measured: codex+ * 0.155's "Update available" took the Enter as "1. Update now" and ran `npm install -g`). Each is+ * answered the way a person running a test would — and each answer is recorded in the leg's+ * evidence, since a tool that greets every resume with a dialog is itself a finding.+ */+export const STARTUP_DIALOGS: ReadonlyArray<{ id: string; match: RegExp; keys: string[] }> = [+ // codex: "Do you trust the contents of this directory?" 1. Yes, continue+ { id: 'codex-trust', match: /Do you trust the contents of this directory/, keys: ['1', 'Enter'] },+ // codex: "Update available! x -> y" 1. Update now / 2. Skip / 3. Skip until next version+ { id: 'codex-update', match: /Update available!/, keys: ['2', 'Enter'] },+ // codex 0.156: "Hooks need review — 2 hooks are new or changed" with the cursor on "Review hooks".+ // The hooks are the harness's own (how it tracks the agent), so "Trust all and continue" is what a+ // person using the harness answers too — Down once from "Review hooks", then Enter.+ { id: 'codex-hooks-trust', match: /Hooks need review/, keys: ['Down', 'Enter'] },+ // codex: "GPT-5.5 retires on <date>. Switch to <next> to continue" 1. Try new model / 2. Use existing+ // model. Keep the model the run was configured with: a test that quietly changes its own model is+ // no longer the test that was asked for.+ { id: 'codex-model-retire', match: /retires on [^\n]*\n?[^\n]*Switch to|1\. Try new model/, keys: ['2', 'Enter'] },+ // claude: "Do you trust the files in this folder?" — Enter accepts the highlighted "Yes, proceed"+ { id: 'claude-trust', match: /Do you trust the files in this folder/, keys: ['Enter'] },+ // claude 2.1.281's rewording — "Quick safety check: Is this a project you created or one you+ // trust?" — and its default moved to **"No, exit"**: the old Enter would now quit the engine.+ // Down to "Yes, I trust this folder", then Enter. (The run pre-trusts its own folder, so this is+ // the fallback for when that does not take.)+ { id: 'claude-trust-v2', match: /Yes, I trust this folder/, keys: ['Down', 'Enter'] },+ // claude: a project .mcp.json must be approved — Enter accepts the highlighted "Use this and all future MCP servers"+ { id: 'claude-mcp-approve', match: /MCP server[s]? (found|configured) in \.mcp\.json|Use this and all future MCP servers|Use this MCP server/, keys: ['Enter'] },+ // claude, on `--dangerously-skip-permissions` (permission mode `full`): a full-screen Bypass+ // Permissions warning whose DEFAULT is "No, exit" — Enter alone exits 1 and leaves the pane a+ // shell. Down first, then Enter, picks "Yes, I accept". This is not a fallback in practice: with+ // `bypassPermissionsModeAccepted` already written to ~/.claude.json, claude 2.1.281 still drew the+ // warning, and answering it here is what let the run's first leg happen at all.+ { id: 'claude-bypass-accept', match: /running in Bypass Permissions mode/, keys: ['Down', 'Enter'] },+ // claude, MID-TURN, on any gateway (measured on grid.autonomous.ai AND a local relay): an "auto mode+ // classifier … your requests go through <host>, which isn't compatible" notice that blocks the turn+ // until a key is pressed. A finding about the relay's Anthropic contract, and a modal to get past.+ { id: 'claude-automode-notice', match: /isn't compatible with this update|ask your gateway to/, keys: ['Enter'] },+]++/**+ * The tool saying its account is out of usage — the same words in both, read out of the binaries+ * rather than guessed:+ *+ * codex "You've hit your usage limit. … try again at Sep 24th, 2026 6:02 PM."+ * claude "You've hit your session limit · resets 6pm" (also weekly / monthly / team budget)+ * "You're out of usage credits. /model to switch models."+ *+ * Without this, a check on an exhausted account waited out its 90s, came back `stuck`, and the run+ * went to the reviewer as if it were a bug to diagnose — for a condition the pane had already stated+ * in plain words. "Approaching usage limit" (claude's early warning) is deliberately not matched.+ */+export const QUOTA_EXHAUSTED = /You['\u2019]ve hit your [^\n]{0,40}?(limit|budget)|You['\u2019]re out of usage credits|usage limit reached/i++/** When the tool says the quota comes back, in its own words — or null if it did not say. */+export function quotaResets(text: string): string | null {+ const m = text.match(/(resets? (?:at |in |on )?[^\n\u00b7|.]{1,40}|try again (?:at|in) [^\n|]{1,50}?)(?:\.(?:\s|$)|\n|$)/im)+ return m ? m[1].trim() : null+}++/**+ * The dialog that is actually waiting — the LOWEST one on screen, not the first in the list. A fresh+ * codex stacks them (the hooks review, then the model notice under it), and the earlier text stays+ * in the scrollback after it is answered; picking by list order answered a screen that was already+ * gone and sent its keys into the one that was not. One answer per dialog per round of dismissals.+ */+function activeDialog(screen: string, answered: readonly string[]): (typeof STARTUP_DIALOGS)[number] | undefined {+ let best: (typeof STARTUP_DIALOGS)[number] | undefined+ let at = -1+ for (const d of STARTUP_DIALOGS) {+ if (answered.includes(d.id)) continue+ const all = [...screen.matchAll(new RegExp(d.match.source, d.match.flags.includes('g') ? d.match.flags : d.match.flags + 'g'))]+ const last = all.length ? all[all.length - 1].index! : -1+ if (last > at) { at = last; best = d }+ }+ return best+}++/**+ * Answer any startup dialog on screen, then wait for the pane to settle again. Returns the ids of+ * the dialogs answered (evidence). Bounded: a dialog that comes back after being answered is left+ * alone after `max` rounds so the check that follows records it as stuck, with the dialog on the tail.+ */+export async function dismissStartupDialogs(tmux: Tmux, pane: string, opts: ProbeOptions = {}, max = 5): Promise<string[]> {+ const o = { ...DEFAULTS, ...opts }+ const answered: string[] = []+ for (let round = 0; round < max; round++) {+ const screen = await tmux.capture(pane, o.tailLines)+ const dialog = activeDialog(screen, answered)+ if (!dialog) break+ for (const k of dialog.keys) {+ await tmux.key(pane, k)+ await tmux.sleep(300)+ }+ answered.push(dialog.id)+ await waitForPaneSettle(tmux, pane, opts)+ }+ return answered+}++export interface ProbeOptions {+ /** How long one check may take before it is `stuck`. Grid models can be slow; default 90s. */+ checkTimeoutMs?: number+ /** The tool / MCP log lines right now (`workspace.ts readLog`); a step with a `log` needs one new matching line. */+ readLog?: (kind: LogKind) => string[]+ /** A workspace file's content right now, or null when it does not exist (`workspace.ts readWorkspaceFile`). */+ readFile?: (relPath: string) => string | null+ /** What the engine's session file says `tool` did for the step tagged `ref` (`sessionTools.ts toolResult`). */+ toolResult?: (ref: string, tool: string) => ToolResult+ /** Whose wording the requests use (smokeChecks.ts scenarioFor): each names that engine's tool. */+ engine?: string+ /** The pane is "ready" once its screen stops changing for this long. A resumed TUI needs it. */+ settleMs?: number+ /** Give up waiting for the pane to settle after this long (the TUI itself may be stuck). */+ settleTimeoutMs?: number+ pollMs?: number+ tailLines?: number+}++const DEFAULTS: Required<ProbeOptions> = {+ checkTimeoutMs: 90_000,+ readLog: () => [],+ readFile: () => null,+ toolResult: () => ({ called: false, answered: false }),+ engine: 'claude',+ settleMs: 2_000,+ settleTimeoutMs: 60_000,+ pollMs: 1_000,+ tailLines: 40,+}++/** Wait until two consecutive captures match — the TUI has drawn and is waiting on input. */+export async function waitForPaneSettle(tmux: Tmux, pane: string, opts: ProbeOptions = {}): Promise<boolean> {+ const o = { ...DEFAULTS, ...opts }+ const start = tmux.now()+ let last = await tmux.capture(pane, o.tailLines)+ while (tmux.now() - start < o.settleTimeoutMs) {+ await tmux.sleep(o.settleMs)+ const next = await tmux.capture(pane, o.tailLines)+ if (next === last) return true+ last = next+ }+ return false+}++/** A short tag typed after the prompt so THIS check's echo can be told from earlier ones. */+export function checkRef(now: number): string {+ return `ref-${now.toString(36).slice(-4).padStart(4, '0')}${Math.floor(Math.random() * 1296).toString(36).padStart(2, '0')}`+}++/**+ * Type one check into the pane and wait for its marker. Evidence (`tail`) comes back either way.+ *+ * The marker only counts AFTER the echo of this check's own `ref` tag. Counting occurrences was+ * wrong: a resume redraws the whole transcript, so the previous leg's answer is on screen again,+ * and as the new turn scrolls the old one out the count stays flat — a real `ANSWER_42` on the+ * back-home leg was recorded as stuck (measured, codex 0.155.1). The tag pins the search to the+ * new turn instead; a wrapped tag (never echoed in one piece) falls back to "one more hit than before".+ */+export async function runCheck(tmux: Tmux, pane: string, check: SmokeCheck, opts: ProbeOptions = {}): Promise<CheckOutcome> {+ const o = { ...DEFAULTS, ...opts }+ const before = await tmux.capture(pane, o.tailLines)+ const logBefore = check.log ? o.readLog(check.log).length : 0+ // read: the file's token, made fresh for this run and written nowhere else (workspace.ts).+ const token = check.answerFromFile ? (o.readFile(check.answerFromFile) ?? '').trim() : ''+ if (check.answerFromFile && !token) {+ return { id: check.id, status: 'stuck', elapsedMs: 0, tail: before.trimEnd(), note: `the workspace has no ${check.answerFromFile} — the run's setup, not the tool` }+ }+ const start = tmux.now()+ const ref = checkRef(start)+ await tmux.type(pane, `${check.prompt} (${ref})`)+ // Make sure it landed. A tool that has only just started can drop input while it is still+ // loading — measured on grid-dev: claude's first question of a run never reached it (its own+ // transcript has every later question and not that one). The tag is on screen within a second+ // when the text arrived, in the composer or in the transcript; if it is nowhere, type it again.+ for (let retry = 0; retry < 2; retry++) {+ let landed = false+ for (let waited = 0; waited < 5_000 && !landed; waited += o.pollMs) {+ await tmux.sleep(o.pollMs)+ landed = (await tmux.capture(pane, o.tailLines)).includes(ref)+ }+ if (landed) break+ await tmux.type(pane, `${check.prompt} (${ref})`)+ }++ // The proof, never the reply: a log line for exactly these numbers, the file on disk as asked,+ // the read file's token on the pane after this prompt (nobody guesses `kiwi-4821-tulip`),+ // or the engine's own record of the tool coming back with results.+ const proven = (afterEcho: string | null, tail: string): { logLines?: string[] } | null => {+ if (check.log) {+ const logLines = o.readLog(check.log).slice(logBefore)+ return logLines.some((l) => l.includes(check.logPattern!)) ? { logLines } : null+ }+ if (check.sessionTool) return o.toolResult(ref, check.sessionTool).answered ? {} : null+ if (check.file) {+ const content = o.readFile(check.file.path)+ if (content === null) return null+ const f = check.file+ if (f.equals !== undefined && content.trim() !== f.equals) return null+ if (f.contains !== undefined && !content.includes(f.contains)) return null+ if (f.lacks !== undefined && content.includes(f.lacks)) return null+ return {}+ }+ if (token) return (afterEcho ?? (before.includes(token) ? '' : tail)).includes(token) ? {} : null+ return null+ }++ let tail = before+ const dialogs: string[] = []+ while (tmux.now() - start < o.checkTimeoutMs) {+ await tmux.sleep(o.pollMs)+ tail = await tmux.capture(pane, o.tailLines)+ const echo = tail.lastIndexOf(ref)+ const afterEcho = echo >= 0 ? tail.slice(echo + ref.length) : null+ // A modal can land in the middle of a turn (claude's gateway notice); answer it and keep waiting.+ // Only one that appeared after THIS prompt: startup screens answered before it stay in the+ // scrollback, and answering them again would send their keys into the running turn.+ const modal = activeDialog(afterEcho ?? tail, dialogs)+ if (modal) {+ for (const k of modal.keys) { await tmux.key(pane, k); await tmux.sleep(300) }+ dialogs.push(modal.id)+ continue+ }+ // Out of usage: said once, right after our prompt. Only text after THIS prompt's echo counts+ // (or, when the echo scrolled away, only if the screen did not already say it before we typed).+ const answer = afterEcho ?? (QUOTA_EXHAUSTED.test(before) ? '' : tail)+ if (QUOTA_EXHAUSTED.test(answer)) {+ const resets = quotaResets(answer)+ return { id: check.id, ref, status: 'no-quota', elapsedMs: tmux.now() - start, tail: tail.trimEnd(), ...(resets ? { resets } : {}), ...(dialogs.length ? { dialogs } : {}) }+ }+ const proof = proven(afterEcho, tail)+ if (proof) {+ const elapsedMs = tmux.now() - start+ // Proven — but the turn may still be running (the model reporting what it did). Let the pane+ // go quiet before the next request, the way a person waits for an answer before typing.+ await waitForPaneSettle(tmux, pane, { ...opts, settleTimeoutMs: 30_000 })+ return { id: check.id, ref, status: 'ok', elapsedMs, tail: (await tmux.capture(pane, o.tailLines)).trimEnd(), ...proof, ...(dialogs.length ? { dialogs } : {}) }+ }+ }+ // Not proven in time: say what was missing, with what was actually there.+ const logLines = check.log ? o.readLog(check.log).slice(logBefore) : undefined+ let note: string+ if (check.log) note = `no ${check.log} log line for "${check.logPattern}"${logLines && logLines.length ? ` (got: ${logLines.join(' | ')})` : ''}`+ else if (check.sessionTool) {+ const used = o.toolResult(ref, check.sessionTool)+ note = !used.called ? `${check.sessionTool} was never called` : `${check.sessionTool} came back without results${used.output ? `: ${JSON.stringify(used.output)}` : ''}`+ } else if (check.file) {+ const content = o.readFile(check.file.path)+ note = content === null ? `${check.file.path} does not exist` : `${check.file.path} is ${JSON.stringify(content.trim().slice(0, 120))}`+ } else note = `${check.answerFromFile}'s token never appeared after the question`+ return { id: check.id, ref, status: 'stuck', elapsedMs: tmux.now() - start, tail: tail.trimEnd(), note, ...(logLines ? { logLines } : {}), ...(dialogs.length ? { dialogs } : {}) }+}++/**+ * Run this leg's steps of the scenario, in order. Stops at the first stuck step: a pane that is+ * not answering will not answer the next prompt either, and the queued text would only muddy the evidence.+ */+export async function probeLeg(tmux: Tmux, pane: string, leg: Leg, opts: ProbeOptions = {}): Promise<LegOutcome> {+ const steps = scenarioFor(opts.engine ?? 'claude')[leg]+ const checks: CheckOutcome[] = []+ const settled = await waitForPaneSettle(tmux, pane, opts)+ if (!settled) {+ const tail = (await tmux.capture(pane, opts.tailLines ?? DEFAULTS.tailLines)).trimEnd()+ // Nothing typed: the TUI never stopped redrawing, so the first check is stuck before it began.+ checks.push({ id: steps[0].id, status: 'stuck', elapsedMs: 0, tail })+ for (const c of steps.slice(1)) checks.push({ id: c.id, status: 'not-run' })+ return { leg, checks }+ }+ const dialogs = await dismissStartupDialogs(tmux, pane, opts)+ for (const [i, check] of steps.entries()) {+ const outcome = await runCheck(tmux, pane, check, opts)+ checks.push(outcome)+ if (outcome.status === 'stuck' || outcome.status === 'no-quota') {+ for (const rest of steps.slice(i + 1)) checks.push({ id: rest.id, status: 'not-run' })+ break+ }+ }+ return dialogs.length ? { leg, checks, dialogs } : { leg, checks }+}diff --git a/cli/src/e2e/plan.ts b/cli/src/e2e/plan.tsnew file mode 100644index 000000000..0362308ab--- /dev/null+++ b/cli/src/e2e/plan.ts@@ -0,0 +1,23 @@+/**+ * `harness e2e plan` — the trigger's decision, printed: for each engine under test, the installed+ * version, the latest released one, and whether a run is due. The cron runner (autonomous-grid-cli,+ * e2e/run.sh) reads the JSON form and runs `harness e2e grid-switch` per engine it names.+ *+ * Env: E2E_VERSION_SOURCE=mock|npm (default mock until the npm reader is validated), E2E_ENGINES=codex,claude.+ */+import type { AgentEngine } from '../engines/types.js'+import { versionSourceFromEnv } from './versionSource.js'+import { InstalledCurrentVersion } from './currentVersion.js'+import { planMatrixRuns } from './trigger.js'++export async function planCommand(json: boolean): Promise<void> {+ const engines = (process.env.E2E_ENGINES ?? 'codex,claude').split(',').map((e) => e.trim() as AgentEngine).filter(Boolean)+ const plans = await planMatrixRuns(versionSourceFromEnv(), new InstalledCurrentVersion(), engines)+ if (json) {+ console.log(JSON.stringify({ generatedAt: new Date().toISOString(), plans }, null, 2))+ return+ }+ for (const p of plans) {+ console.log(`${p.engine.padEnd(8)} current=${String(p.current).padEnd(12)} latest=${String(p.latest).padEnd(12)} ${p.plan ? 'RUN' : 'skip'} (${p.reason})`)+ }+}diff --git a/cli/src/e2e/runGridSwitchTrace.ts b/cli/src/e2e/runGridSwitchTrace.tsnew file mode 100644index 000000000..e2a2fbd4a--- /dev/null+++ b/cli/src/e2e/runGridSwitchTrace.ts@@ -0,0 +1,324 @@+/**+ * Run the grid-switch journey for REAL: create a fresh agent through the daemon (like "New agent"+ * in the app), run it subscription -> grid -> back home with the scenario on every leg, delete+ * it, and leave a review bundle the `e2e-watchdog` agent (autonomous-grid-cli, e2e/) reads.+ *+ * agent_create — a new codex/claude in its own tmux pane, in a scratch folder; nobody's pane is borrowed+ * subscription — checks on it as it is (its own login): the baseline+ * grid — agent_retarget onto a grid model (pane respawned, session resumed), checks again+ * agent_delete — gone at the end, unless E2E_KEEP_AGENT=1 (the reviewer, autonomous-grid-cli's+ * e2e-watchdog, reads the bundle afterwards; keep the agent when it should type into the pane)+ *+ * Usage (zero-config; everything is read from this machine's harness install):+ * harness e2e grid-switch (or: npx tsx src/e2e/runGridSwitchTrace.ts)+ *+ * Env (all optional):+ * HARNESS_ENGINE=codex|claude engine under test (default codex)+ * HARNESS_GRID_MODEL / HARNESS_GRID_NAME pin the grid model (default: smallest listed — pin the fastest+ * one from `grid stats <grid> --verbose` when it matters)+ * HARNESS_AGENT_ID + HARNESS_PANE_ID reuse an existing agent instead of creating one (not deleted)+ * HARNESS_PORT / HARNESS_MACHINE_ID override what env.ts / computer-id say+ * E2E_GRID_HOME + E2E_GRID_NAME (+ E2E_GRID_CONTROL_PLANE_URL) move onto a grid from ANOTHER grid home —+ * a local relay checkout under test — instead of the daemon's own+ * E2E_KEEP_AGENT=1 leave the created agent running for inspection+ *+ * The daemon is the one `harness` started (`ws://127.0.0.1:${PORT}/api/local-ws`) — the exact+ * transport and commands the desktop app uses.+ */+import { mkdirSync, realpathSync, writeFileSync, cpSync, existsSync, readFileSync } from 'node:fs'+import { join } from 'node:path'+import { execFile } from 'node:child_process'+import { promisify } from 'node:util'+import { GridSwitchDriver, pickGridModel, type GridSwitchTraceStep } from './gridSwitchDriver.js'+import { probeLeg, realTmux, waitForPaneSettle } from './paneProbe.js'+import { firstStuck, plannedLegs, quotaHit, LEGS, type LegOutcome, type LogKind } from './smokeChecks.js'++const LAST_LEG = LEGS[LEGS.length - 1]+import { prepareWorkspace, readLog, readWorkspaceFile, removeCodexMcp } from './workspace.js'+import { sessionName, workspaceDirName } from './sessionName.js'+import { TESTCASE } from './matrix.js'+import { outDir, runDir } from './artifacts.js'+import { env } from '../config/env.js'+import { daemonUrl, localHeaders, machineId, waitForPane, gridOverrideFromEnv } from './harnessLocal.js'+import { parseVersionLine } from './currentVersion.js'+import { folderTrust } from '../engines/launchPrep.js'+import { toolResult, useByRef } from './sessionTools.js'++const execFileP = promisify(execFile)++function mkdirAll(dir: string): string {+ mkdirSync(dir, { recursive: true })+ return dir+}++export interface GridSwitchRunResult {+ session: string+ reportPath: string+ runDir: string+ stuck: ReturnType<typeof firstStuck>+ error: string | null+}++/**+ * The live journey, as a function: `harness e2e grid-switch` and the cron runner call this; the+ * env described at the top of this file is the only configuration.+ */+export async function runGridSwitch(): Promise<GridSwitchRunResult> {+const engine = process.env.HARNESS_ENGINE ?? 'codex'+const url = daemonUrl()+const machine = await machineId()+const version = await (async () => {+ try {+ const { stdout } = await execFileP(engine, ['--version'], { timeout: 10_000 })+ return parseVersionLine(stdout) ?? 'UNKNOWN'+ } catch {+ return 'UNKNOWN'+ }+})()++console.error(`[grid-e2e] dialing ${url} · machine ${machine} · ${engine} ${version}`)++const legs: LegOutcome[] = []+const planned = plannedLegs()+let pane: string | null = process.env.HARNESS_PANE_ID ?? null+let workspace: string | null = null+// A subscription model answers in seconds; a local model on a grid may take a minute. Only the+// grid leg gets the long budget, so a stuck check on the tool's own login is known in 45s, not 90.+const CHECK_TIMEOUT_MS: Record<LegOutcome['leg'], number> = { subscription: 120_000, grid: 120_000 }++async function checks(leg: LegOutcome['leg']): Promise<void> {+ const readLogs = workspace ? (kind: LogKind) => readLog(workspace!, kind) : undefined+ const readFile = workspace ? (rel: string) => readWorkspaceFile(workspace!, rel) : undefined+ const usedTool = workspace ? (ref: string, tool: string) => toolResult(engine, workspace!, ref, tool) : undefined+ const outcome = pane+ ? await probeLeg(realTmux, pane, leg, { engine, checkTimeoutMs: CHECK_TIMEOUT_MS[leg], ...(readLogs ? { readLog: readLogs } : {}), ...(readFile ? { readFile } : {}), ...(usedTool ? { toolResult: usedTool } : {}) })+ : planned.find((l) => l.leg === leg)!+ legs.push(outcome)+ console.error(`[grid-e2e] ${leg}: ${outcome.checks.map((c) => `${c.id}=${c.status}`).join(' ')}${outcome.dialogs ? ` (answered: ${outcome.dialogs.join(', ')})` : ''}`)+ // The marker shows before the tool's turn is fully over (hooks still reporting); a retarget on a+ // turn in flight is refused AGENT_BUSY. Let the pane go quiet first, the way a person would —+ // but only when a retarget follows: after the last leg the agent is deleted, not moved.+ if (pane && leg !== LAST_LEG) {+ await waitForPaneSettle(realTmux, pane, { settleMs: 3_000, settleTimeoutMs: 60_000 })+ await new Promise((r) => setTimeout(r, 3_000))+ }+}++const driver = new GridSwitchDriver(url, machine, localHeaders())+let trace: GridSwitchTraceStep[] = []+let gridModel: string | null = null+let agentId: string | null = process.env.HARNESS_AGENT_ID ?? null+// codex's MCP server is registered globally (`codex mcp add`, see workspace.ts). Tracked so the run+// removes exactly what it added, and removes it even when the run stops early.+let registeredCodexMcp = false+// How the MCP server got registered for this engine, and in which permission mode the agent runs —+// both in the trace, because an MCP step that never fired is read very differently once you know+// the server was never registered or the engine was launched in a mode that refuses it.+let mcpSetup: { config: string | null; note: string } | null = null+// The app's own default (`kDefaultPermissionMode` in desktop/lib/core/permission_modes.dart), and+// on purpose not overridable: the run creates agents exactly as a person does. It ran in `full`+// until 2026-09-25, and that alone made codex 0.157.0 fail a switch the app does fine — `full`+// launches codex with no `-c`, which starts codex's background daemon, and the daemon kept the+// conversation after the switch ("This conversation is open in another app"). `auto`'s+// `--approve-for-me` is three `-c` overrides to codex, and codex never starts the daemon for those.+const permissionMode = 'auto'+const created = !agentId+let error: string | null = null+// Named before anything runs so the scratch folder and the trace share it; the grid model is+// stamped on once known.+const startedAt = new Date()+try {+ await driver.machineSelect()++ // Which grid and model — settled BEFORE the agent exists, and asked of the daemon only when the+ // run did not pin them. `grid_models_list` starts one Python `grid` per grid the account can see,+ // all at once (~125 MB each); on grid-dev that was ten of them, 1.2 GB, landing while claude+ // started, and the 1.5 GB container OOM-killed claude. Pinned (HARNESS_GRID_NAME ++ // HARNESS_GRID_MODEL, or a relay under test), the list is never needed; unpinned, it is asked+ // while nothing else is running.+ // A relay under test (E2E_GRID_HOME + E2E_GRID_NAME) replaces the daemon's own grid; the model+ // then has to be pinned (HARNESS_GRID_MODEL), since that grid's list is not in `grid_models_list`.+ const local = await gridOverrideFromEnv()+ const pinned = !!process.env.HARNESS_GRID_MODEL && (!!process.env.HARNESS_GRID_NAME || !!local)+ const models = pinned ? null : await driver.gridModels()+ const picked = local || !models ? null : pickGridModel(models)+ gridModel = process.env.HARNESS_GRID_MODEL ?? picked?.model ?? null+ const gridName = local ? local.name : (process.env.HARNESS_GRID_NAME ?? (process.env.HARNESS_GRID_MODEL ? undefined : picked?.gridName))+ console.error(`[grid-e2e] grid model: ${gridModel ?? '(none)'}${gridName ? ` on ${gridName}` : ''}${local ? ` (relay under test: ${local.override.baseUrl})` : ''}${pinned ? ' (pinned — no model list asked)' : ''}`)++ if (created) {+ // realpath: $TMPDIR is a symlink on macOS (`/var/…` → `/private/var/…`) and the engine keys its+ // trust on the resolved path — a trust written for the symlink is never matched.+ const cwd = realpathSync(mkdirAll(join(outDir(), 'agents', workspaceDirName(sessionName({ engine, version, testcase: TESTCASE, at: startedAt })))))+ // The person's project: the notes the file steps work on, an MCP server registered for this engine, and its log.+ const laid = prepareWorkspace(cwd, engine)+ workspace = cwd+ console.error(`[grid-e2e] workspace ${cwd} · mcp ${laid.mcpConfig ?? '(none)'} — ${laid.mcpNote}`)+ // codex's registration is global (`codex mcp add`), so it is this run's to take back out.+ if (engine === 'codex' && laid.mcpConfig) registeredCodexMcp = true+ mcpSetup = { config: laid.mcpConfig, note: laid.mcpNote }+ // The scratch folder is this run's own, so answer the engine's "trust this folder?" the way the+ // daemon does for a workspace it made — otherwise the first check would be typed into that dialog.+ try {+ if (engine === 'claude' || engine === 'codex') folderTrust(engine)?.record(cwd)+ } catch (err) {+ console.error(`[grid-e2e] pre-trust ${cwd}: ${(err as Error).message}`)+ }+ // REGISTRATION_FAILED right after the daemon starts is not the test's to report: the saved+ // registry still holds an agent from before the restart (a container, a reboot), tmux numbers+ // the new pane %0 again, and the stale row owns that key until discovery confirms it gone —+ // measured on grid-dev: refused at 07:41:11, the stale row forgotten at 07:41:20. Wait it out.+ let made = await driver.createAgent(engine, cwd, `e2e ${engine} ${version}`, permissionMode)+ for (let retry = 1; !made.ok && made.error === 'REGISTRATION_FAILED' && retry <= 3; retry++) {+ console.error(`[grid-e2e] agent_create REGISTRATION_FAILED — a stale agent from before the daemon restart still holds the pane; retry ${retry}/3 in 15s`)+ await new Promise((r) => setTimeout(r, 15_000))+ made = await driver.createAgent(engine, cwd, `e2e ${engine} ${version}`, permissionMode)+ }+ if (!made.ok) throw new Error(`${made.error}: ${made.detail ?? ''}`)+ agentId = made.agentId+ pane = made.pane ?? (await waitForPane(agentId))+ console.error(`[grid-e2e] created agent ${agentId}${pane ? ` in pane ${pane}` : ' (pane not registered yet: checks not-run)'}`)+ // A fresh codex/claude has a trust / first-run screen before its prompt; let it draw.+ if (pane) await waitForPaneSettle(realTmux, pane, { settleMs: 3_000, settleTimeoutMs: 90_000 })+ }+++ // Out of usage ends the journey where it happened: an account that cannot answer on its own login+ // tells us nothing about a switch, and switching it anyway only spends the grid's time too.+ await checks('subscription')+ if (!quotaHit(legs)) {+ if (!gridModel) throw new Error('GRID_UNAVAILABLE: grid_models_list returned no model')+ const onto = await driver.retargetToGrid(agentId!, gridModel, gridName, local?.override)+ if (!onto.ok) throw new Error(`${onto.error}: ${onto.detail ?? ''}`)+ await checks('grid')+ }++} catch (err) {+ error = (err as Error).message+ console.error(`[grid-e2e] stopped: ${error}`)+} finally {+ trace = [...driver.trace]+}++// How each step got done, and by which model: read from the engine's own session file now that the+// run is over. Reported, never judged — a step's pass/fail is its result on disk (smokeChecks.ts).+if (workspace) {+ const refs = legs.flatMap((l) => l.checks.map((c) => c.ref).filter((r): r is string => !!r))+ const used = useByRef(engine, workspace, refs)+ for (const l of legs) for (const c of l.checks) {+ const u = c.ref ? used[c.ref] : undefined+ if (u) { c.tools = u.tools; c.models = u.models }+ }+}++const session = sessionName({ engine, version, testcase: TESTCASE, gridModel: gridModel ?? undefined, at: startedAt })+const report = {+ generatedAt: new Date().toISOString(),+ mode: 'live',+ engines: [+ {+ engine,+ session,+ version,+ testcase: TESTCASE,+ gridModel,+ agentId,+ pane,+ /** The folder the agent worked in: notes/, out/, the MCP server and config, and .e2e/calc-mcp.log with every MCP call. */+ workspace,+ /** The mode the engine was launched in — always the app's default, `auto`. */+ permissionMode,+ /** Where this engine's MCP server was registered, and how — null config means the MCP steps could not pass. */+ mcp: mcpSetup,+ /** true: this run made the agent. It stays alive through the review (so the reviewer can type into the pane), then is deleted unless E2E_KEEP_AGENT=1. */+ createdByRun: created,+ keptAlive: !created || process.env.E2E_KEEP_AGENT === '1',+ notify: error ? [error] : [],+ /** Set when the tool said its account is out of usage: { leg, check, resets }. The pipeline+ * reads this before anything else — no review, one plain message, and a retry later. */+ quota: quotaHit(legs),+ // Every leg, run or not, so a table built from this always has its three rows.+ legs: [...legs, ...planned.filter((p) => !legs.some((l) => l.leg === p.leg))],+ trace,+ },+ ],+}+// The durable review bundle: everything a person (or the watchdog, later) needs to look back at this run.+const bundle = runDir(session)+const reportPath = join(bundle, 'trace.json')+;(report.engines[0] as Record<string, unknown>).runDir = bundle+writeFileSync(reportPath, JSON.stringify(report, null, 2))+if (workspace) {+ for (const rel of ['notes', 'out', 'tools', '.e2e', '.codex', '.mcp.json']) {+ const src = join(workspace, rel)+ if (existsSync(src)) cpSync(src, join(bundle, 'workspace', rel), { recursive: true })+ }+}++for (const step of trace) {+ console.log(`\n▶ ${step.step}`)+ console.log(` request: ${JSON.stringify(step.request)}`)+ console.log(` reply: ${JSON.stringify(step.reply)}`)+}+const stuck = firstStuck(legs)+console.log(`\n=== ${session} ===`)+console.log(stuck ? `STUCK AT ${stuck.leg} / ${stuck.check} (${stuck.status})` : error ? `STOPPED: ${error}` : 'clean')+console.log(`Wrote trace → ${reportPath}`)++// Everything to look back at: the daemon's own lines for this agent, the watchdog's raw events, and+// a manifest that says how to reopen each piece.+try {+ const daemonLog = join(env.ADAPTER_DATA_DIR, 'harness.log')+ if (existsSync(daemonLog) && agentId) {+ const short = agentId.slice(0, 8)+ const lines = readFileSync(daemonLog, 'utf8').split('\n').filter((l) => l.includes(agentId!) || l.includes(short))+ // The [turn]/[hooks]/[recap] lines are keyed by the ENGINE session id; find it from the attach line.+ const sid = /session=([0-9a-f]{8})/.exec(lines.join('\n'))?.[1]+ const all = sid ? readFileSync(daemonLog, 'utf8').split('\n').filter((l) => l.includes(agentId!) || l.includes(short) || l.includes(sid)) : lines+ writeFileSync(join(bundle, 'daemon.log'), all.join('\n') + '\n')+ }+} catch (err) {+ console.error(`[grid-e2e] daemon log slice: ${(err as Error).message}`)+}+writeFileSync(join(bundle, 'manifest.json'), JSON.stringify({+ session,+ engine,+ version,+ agentId,+ pane,+ workspace,+ gridModel,+ reportPath,+ // Filled by the reviewer's runner (autonomous-grid-cli e2e/run.sh): { model, sessionID, reopen }.+ watchdog: null,+ files: { trace: 'trace.json', daemonLog: 'daemon.log', watchdogEvents: 'watchdog.jsonl', watchdogReview: 'watchdog.md', workspace: 'workspace/', evidence: 'evidence/' },+ generatedAt: new Date().toISOString(),+}, null, 2))+console.log(`Review bundle → ${bundle}`)++// The run is over; the agent this run made can go. Kept only on request — for a person, or for the+// reviewer to type into the pane (E2E_KEEP_AGENT=1; delete it in the app afterwards).+if (created && agentId && process.env.E2E_KEEP_AGENT !== '1') {+ try {+ await driver.deleteAgent(agentId)+ console.log(`Deleted agent ${agentId}${pane ? ` (pane ${pane})` : ''}`)+ } catch (err) {+ console.log(`Could not delete agent ${agentId}: ${(err as Error).message} — delete it in the app`)+ }+} else if (created && agentId) {+ console.log(`Kept agent ${agentId}${pane ? ` in pane ${pane}` : ''} (E2E_KEEP_AGENT=1)`)+}+// Take the global codex entry back out. Not when the agent is being kept: someone who asked to keep+// the pane wants to type into it, and an agent whose MCP server has been unregistered is not the+// agent the run left behind.+if (registeredCodexMcp && process.env.E2E_KEEP_AGENT !== '1') {+ console.log(removeCodexMcp() ? 'Unregistered the codex MCP server' : 'Could not unregister the codex MCP server — remove it with `codex mcp remove e2e_calc`')+}+driver.close()+return { session, reportPath, runDir: bundle, stuck, error }+}++// `npx tsx src/e2e/runGridSwitchTrace.ts` still works; `harness e2e grid-switch` is the same call.+if (process.argv[1] && /runGridSwitchTrace\.(ts|js)$/.test(process.argv[1])) {+ runGridSwitch().then((r) => process.exit(r.error || r.stuck ? 1 : 0))+}diff --git a/cli/src/e2e/runMatrix.ts b/cli/src/e2e/runMatrix.tsnew file mode 100644index 000000000..d575be72e--- /dev/null+++ b/cli/src/e2e/runMatrix.ts@@ -0,0 +1,144 @@+/**+ * runMatrix — CLI entry: the cronjob's action. Reads current vs latest for each engine, plans the+ * matrix runs, emits namespaced audit entries (argv + login/grid NOTIFYs + the planned smoke+ * checks per leg) and writes a report. The live journey is `harness e2e grid-switch`; the reviewer+ * (opencode `e2e-watchdog`) lives with the relay in autonomous-grid-cli and reads the run afterwards.+ *+ * Env:+ * E2E_VERSION_SOURCE=mock|npm (default mock until the real reader is validated)+ * E2E_ENGINES=codex,claude engines under test+ *+ * Dry-run: proves trigger + argv + naming + notify + the check plan with zero network/accounts;+ * every check is `not-run`. The live journey with real checks is `runGridSwitchTrace.ts`.+ *+ * Usage: npx tsx src/e2e/runMatrix.ts+ */+import { writeFileSync } from 'node:fs'+import { join } from 'node:path'+import { execFile } from 'node:child_process'+import { promisify } from 'node:util'+import type { AgentEngine } from '../engines/types.js'+import { versionSourceFromEnv } from './versionSource.js'+import { InstalledCurrentVersion, ENGINE_BIN } from './currentVersion.js'+import { planMatrixRuns } from './trigger.js'+import { buildMatrixEntry, type EngineStatus } from './matrix.js'+import { outDir } from './artifacts.js'++const execFileP = promisify(execFile)++const ENGINES: AgentEngine[] = (process.env.E2E_ENGINES ?? 'codex,claude')+ .split(',')+ .map((e) => e.trim() as AgentEngine)+ .filter(Boolean)++/**+ * Codex's own answer, not the presence of a file.+ *+ * `~/.codex/auth.json` existing proved nothing: on a machine whose codex was pointed at a grid, that+ * file holds `auth_mode: "apikey"` with a grid token, `codex login status` says "Logged in using an+ * API key - open_5fb***_grid", and every own-account turn comes back 401 from api.openai.com. A+ * matrix that called that "logged in" sent the run off to fail for a reason it had already been+ * told. The mode is carried out with the answer so the report can name it.+ */+async function codexLogin(): Promise<{ loggedIn: boolean; mode: string | null }> {+ try {+ const { stdout } = await execFileP('codex', ['login', 'status'], { timeout: 15_000 })+ const line = stdout.trim().split('\n')[0] ?? ''+ if (!/logged in/i.test(line)) return { loggedIn: false, mode: line || null }+ // An API key here is a key for api.openai.com. A grid token (`…_grid`) is not one, however much+ // it looks like a login — it is the one case that passes every offline check and still 401s.+ const apiKey = /api key/i.test(line)+ const gridKey = apiKey && /_grid\b/i.test(line)+ return { loggedIn: !gridKey, mode: gridKey ? `grid token in apikey mode (${line})` : apiKey ? 'apikey' : 'chatgpt' }+ } catch (err) {+ return { loggedIn: false, mode: `codex login status failed: ${(err as Error).message.split('\n')[0]}` }+ }+}++async function claudeLoggedIn(): Promise<boolean> {+ try {+ const { stdout } = await execFileP('claude', ['auth', 'status'], { timeout: 15_000 })+ return (JSON.parse(stdout) as { loggedIn?: boolean }).loggedIn === true+ } catch {+ return false+ }+}++function hasBinary(bin: string): boolean {+ // Hard to verify PATH without a shell tool here; report presence by the binary name only.+ return !!bin+}++async function statusOf(engine: AgentEngine): Promise<EngineStatus> {+ const bin = ENGINE_BIN[engine] ?? engine+ const binary = hasBinary(bin)+ let login = false+ let note: string | null = null+ if (binary) {+ switch (engine) {+ case 'codex': {+ const codex = await codexLogin()+ login = codex.loggedIn+ note = codex.mode+ break+ }+ case 'claude':+ // Not a file check: on macOS Claude Code keeps its login in the Keychain, so the only honest+ // answer is the tool's own (`claude auth status` → { loggedIn }).+ login = await claudeLoggedIn()+ break+ default:+ login = false+ }+ }+ // grid status filled by caller when it resolves+ return { binary, login, grid: true, ...(note ? { loginNote: note } : {}) }+}++async function main(): Promise<number> {+ const source = versionSourceFromEnv()+ const current = new InstalledCurrentVersion()+ const plans = await planMatrixRuns(source, current, ENGINES)++ const toRun = plans.filter((p) => p.plan)+ console.log('=== trigger plan (current vs latest) ===')+ for (const p of plans) {+ console.log(` ${p.engine.padEnd(8)} current=${String(p.current).padEnd(12)} latest=${String(p.latest).padEnd(12)} reason=${p.reason} run=${p.plan}`)+ }+ if (toRun.length === 0) {+ console.log('Nothing to run — all engines at latest.')+ return 0+ }++ const entries = []+ for (const plan of toRun) {+ const entry = await buildMatrixEntry(plan, {+ status: statusOf,+ grid: async () => null,+ })+ entries.push(entry)+ }++ const report = { generatedAt: new Date().toISOString(), mode: 'dry-run', engines: entries }+ const reportPath = join(outDir(), `report-${Date.now()}.json`)+ writeFileSync(reportPath, JSON.stringify(report, null, 2))+ console.log(`\nWrote report → ${reportPath}`)++ for (const e of entries) {+ console.log(`\n▶ ${e.session}`)+ console.log(` status: ${JSON.stringify(e.status)}`)+ for (const n of e.notify) console.log(` ⚠ NOTIFY: ${n}`)+ if ('error' in e.gridSpawn) {+ console.log(` gridSpawn ERROR: ${e.gridSpawn.error} — ${e.gridSpawn.detail}`)+ } else {+ console.log(` grid command: ${e.gridSpawn.command.join(' ')}`)+ console.log(` grid env: ${e.gridSpawn.envKeys.join(', ') || '(none)'}`)+ console.log(` clearEnv: ${e.gridSpawn.clearEnv.join(', ') || '(none)'}`)+ }+ console.log(` checks: ${e.legs.map((l) => `${l.leg}[${l.checks.map((c) => c.id).join(',')}]`).join(' → ')} (dry-run: not-run)`)+ }++ return 0+}++main().then((code) => process.exit(code))diff --git a/cli/src/e2e/sessionName.ts b/cli/src/e2e/sessionName.tsnew file mode 100644index 000000000..ab3902767--- /dev/null+++ b/cli/src/e2e/sessionName.ts@@ -0,0 +1,40 @@+/**+ * One namespaced name per matrix run, stamped identically on the trace, the watchdog session and+ * the report, so every artifact of a run is grep-able and the audit answers: what version of which+ * coding-agent, which test, which grid model, when.+ *+ * Convention: `<engine>@<version>-><testcase>@<grid-model>--<timestamp>`+ * e.g. `[email protected]>grid-switch@grid-gpt-5-mini--20260921T1805Z`+ */+export function sessionName(opts: {+ engine: string+ version: string+ testcase: string+ gridModel?: string+ at?: Date+}): string {+ const model = opts.gridModel && opts.gridModel.trim() ? sanitize(opts.gridModel) : 'none'+ const ts = (opts.at ?? new Date())+ .toISOString()+ .replace(/[-:]/g, '')+ .replace(/\.\d{3}Z$/, 'Z')+ return `${opts.engine}@${sanitize(opts.version)}->${sanitize(opts.testcase)}@${model}--${ts}`+}++function sanitize(v: string): string {+ return v.replace(/[^A-Za-z0-9._@-]/g, '-')+}++/**+ * The agent's working folder, named after the session but with nothing a tool might reinterpret.+ *+ * The session name carries `->` and `@` (readable, grep-able, and fine as a bundle folder), and it+ * used to be the agent's cwd too. claude 2.1.274 turns the `->` in that path into `-` when it builds a+ * Write path, so its files landed in a sibling folder that does not exist for anyone else — every+ * write step failed as "out/hello-1.txt does not exist" with the file sitting next door (grid-dev,+ * 2026-09-25; 2.1.273 did not). A person's project folder does not look like that, so the agent's+ * does not either: letters, digits, `.`, `_` and `-` only.+ */+export function workspaceDirName(session: string): string {+ return session.replace(/->/g, '_to_').replace(/[^A-Za-z0-9._-]/g, '-')+}diff --git a/cli/src/e2e/sessionTools.ts b/cli/src/e2e/sessionTools.tsnew file mode 100644index 000000000..697dcc064--- /dev/null+++ b/cli/src/e2e/sessionTools.ts@@ -0,0 +1,218 @@+/**+ * sessionTools — which of the engine's own tools each step went through, and which model answered+ * it, read afterwards from the engine's session file.+ *+ * A step passes on its proof (a log line, the file on disk, the read token — smokeChecks.ts); this+ * says HOW the engine got there, which is the other half of "does the tool still work on the grid":+ * claude may read with `Read` or with `Bash cat`, codex writes with `apply_patch` and, in code mode,+ * reaches everything through `exec`. Every step's prompt carries its `(ref-…)` tag, so the tool+ * calls between one tagged prompt and the next belong to that step.+ *+ * claude ~/.claude/projects/<cwd with every non-alphanumeric as "-">/*.jsonl+ * user message text holds the ref · assistant content[] `tool_use` → name+ * codex ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl (the one holding the refs)+ * payload `message` (role user) holds the ref · `function_call` / `custom_tool_call` →+ * name, and for code mode's `exec` the `tools.<name>(` calls inside its JavaScript+ */+import { readdirSync, readFileSync, statSync } from 'node:fs'+import { homedir } from 'node:os'+import { join } from 'node:path'++export interface StepUse {+ /** The engine's tools this step called, in order, consecutive repeats folded. */+ tools: string[]+ /** The models that answered this step — claude writes it on every reply, codex on every turn.+ * How "claude never ran on Fable" is checked rather than assumed. */+ models: string[]+}++/** ref → what the engine used for that step. */+export function useByRef(engine: string, cwd: string, refs: readonly string[], home = homedir()): Record<string, StepUse> {+ const out: Record<string, StepUse> = {}+ if (!refs.length) return out+ const files = engine === 'claude' ? claudeSessionFiles(cwd, home) : engine === 'codex' ? codexSessionFiles(refs, home) : []+ for (const file of files) {+ let current: string | null = null+ // codex writes a turn's `turn_context` (its model) BEFORE the user message that starts the turn,+ // so its model belongs to the NEXT ref; claude writes the model on each reply, after the ref.+ let turnModel: string | null = null+ const addModel = (use: StepUse, model: string | null): void => { if (model && !use.models.includes(model)) use.models.push(model) }+ for (const line of readText(file).split('\n')) {+ let rec: Record<string, unknown>+ try { rec = JSON.parse(line) as Record<string, unknown> } catch { continue }+ const model = modelOf(engine, rec)+ if (engine === 'codex' && model) { turnModel = model; continue }+ const found = userRef(engine, rec, refs)+ if (found) {+ current = found+ out[current] ??= { tools: [], models: [] }+ if (engine === 'codex') addModel(out[current], turnModel)+ continue+ }+ if (!current) continue+ const use = out[current]+ for (const tool of toolCalls(engine, rec)) if (use.tools[use.tools.length - 1] !== tool) use.tools.push(tool)+ if (engine === 'claude') addModel(use, model)+ }+ }+ return out+}++function modelOf(engine: string, rec: Record<string, unknown>): string | null {+ if (engine === 'claude') {+ const m = rec.type === 'assistant' ? (rec.message as { model?: unknown } | undefined)?.model : undefined+ return typeof m === 'string' && m !== '<synthetic>' ? m : null+ }+ // codex: a top-level `turn_context` record per turn, the model in its payload (measured on a+ // 0.156.1 rollout — not a payload type, which is where the other records keep theirs).+ const p = rec.payload as { model?: unknown } | undefined+ return rec.type === 'turn_context' && typeof p?.model === 'string' ? p.model : null+}++function readText(file: string): string {+ try { return readFileSync(file, 'utf8') } catch { return '' }+}++function textOf(content: unknown): string {+ if (typeof content === 'string') return content+ if (Array.isArray(content)) return content.map((c) => (c && typeof c === 'object' && typeof (c as { text?: unknown }).text === 'string' ? (c as { text: string }).text : '')).join(' ')+ return ''+}++function userRef(engine: string, rec: Record<string, unknown>, refs: readonly string[]): string | null {+ let text = ''+ if (engine === 'claude' && rec.type === 'user') text = textOf((rec.message as { content?: unknown } | undefined)?.content)+ if (engine === 'codex') {+ const p = rec.payload as { type?: string; role?: string; content?: unknown } | undefined+ if (p?.type === 'message' && p.role === 'user') text = textOf(p.content)+ }+ return text ? refs.find((r) => text.includes(r)) ?? null : null+}++function toolCalls(engine: string, rec: Record<string, unknown>): string[] {+ if (engine === 'claude') {+ if (rec.type !== 'assistant') return []+ const content = (rec.message as { content?: unknown } | undefined)?.content+ return Array.isArray(content)+ ? content.filter((c) => c && typeof c === 'object' && (c as { type?: unknown }).type === 'tool_use').map((c) => String((c as { name?: unknown }).name))+ : []+ }+ const p = rec.payload as { type?: string; name?: string; namespace?: string; input?: unknown; arguments?: unknown } | undefined+ if (!p?.type) return []+ if (p.type === 'function_call' || p.type === 'custom_tool_call') {+ const name = codexCallName(p)+ if (name === 'exec' && typeof p.input === 'string') {+ // Code mode: one `exec` of JavaScript that calls the real tools as `tools.<name>(…)`.+ const inner = [...p.input.matchAll(/tools\.([A-Za-z0-9_]+)\s*\(/g)].map((m) => `exec→${m[1]}`)+ return inner.length ? inner : ['exec']+ }+ return [name]+ }+ if (p.type === 'local_shell_call') return ['local_shell']+ return []+}++/** A namespaced call arrives as name `sub` + namespace `mcp__e2e_calc` (codex 0.156.1 on the grid). */+function codexCallName(p: { name?: string; namespace?: string }): string {+ const ns = p.namespace && p.name && !p.name.startsWith(p.namespace) ? p.namespace : ''+ return ns ? `${ns}${ns.endsWith('__') ? '' : '__'}${p.name}` : String(p.name)+}++export interface ToolResult {+ /** `tool` was called after this step's prompt. */+ called: boolean+ /** One of those calls came back with results: not an error, and a link in what it returned. */+ answered: boolean+ /** What the first call returned, cut short — the probe quotes it when the step is not proven. */+ output?: string+}++/**+ * Whether `tool` was called for the step tagged `ref` and came back with results — the proof for a+ * tool whose server is not on this machine (the harness's web search runs on the grid), so its own+ * log cannot be read the way `e2e_calc`'s is. "Results" is a link in what the tool returned: a web+ * search that answered has one, and an error or an approval refusal ("MCP tool call requires+ * approval") does not.+ *+ * claude assistant `tool_use` { id, name } → user `tool_result` { tool_use_id, is_error, content }+ * codex `function_call` { name, namespace, call_id } → `function_call_output` { call_id, output }+ */+export function toolResult(engine: string, cwd: string, ref: string, tool: string, home = homedir()): ToolResult {+ const files = engine === 'claude' ? claudeSessionFiles(cwd, home) : engine === 'codex' ? codexSessionFiles([ref], home) : []+ const calls = new Set<string>()+ const outputs: string[] = []+ let errored = false+ for (const file of files) {+ let inStep = false+ for (const line of readText(file).split('\n')) {+ let rec: Record<string, unknown>+ try { rec = JSON.parse(line) as Record<string, unknown> } catch { continue }+ const found = userRef(engine, rec, [ref])+ if (found) { inStep = true; continue }+ if (!inStep) continue+ // The next step's prompt — another `(ref-…)` — ends this one. Not any user line: both engines+ // also record context and hook text in the user role, in the middle of a turn.+ if (isOtherStep(engine, rec)) { inStep = false; continue }+ if (engine === 'claude') {+ const content = (rec.message as { content?: unknown } | undefined)?.content+ if (!Array.isArray(content)) continue+ for (const c of content as Array<Record<string, unknown>>) {+ if (rec.type === 'assistant' && c?.type === 'tool_use' && c.name === tool) calls.add(String(c.id))+ if (rec.type === 'user' && c?.type === 'tool_result' && calls.has(String(c.tool_use_id))) {+ outputs.push(textOf(c.content))+ if (c.is_error === true) errored = true+ }+ }+ } else {+ const p = rec.payload as { type?: string; name?: string; namespace?: string; call_id?: string; output?: unknown } | undefined+ if (p?.type === 'function_call' && codexCallName(p) === tool && p.call_id) calls.add(p.call_id)+ if (p?.type === 'function_call_output' && p.call_id && calls.has(p.call_id)) outputs.push(textOf(p.output))+ }+ }+ }+ const answered = !errored && outputs.some((o) => /https?:\/\//.test(o))+ return { called: calls.size > 0, answered, ...(outputs.length ? { output: outputs[0].replace(/\s+/g, ' ').slice(0, 160) } : {}) }+}++/** A user line carrying a step's `(ref-…)` tag (paneProbe.ts checkRef) — the probe's next prompt. */+function isOtherStep(engine: string, rec: Record<string, unknown>): boolean {+ let text = ''+ if (engine === 'claude' && rec.type === 'user') text = textOf((rec.message as { content?: unknown } | undefined)?.content)+ if (engine === 'codex') {+ const p = rec.payload as { type?: string; role?: string; content?: unknown } | undefined+ if (p?.type === 'message' && p.role === 'user') text = textOf(p.content)+ }+ return /\(ref-[0-9a-z]{6}\)/.test(text)+}++function claudeSessionFiles(cwd: string, home: string): string[] {+ const dir = join(home, '.claude', 'projects', cwd.replace(/[^A-Za-z0-9]/g, '-'))+ try {+ return readdirSync(dir).filter((f) => f.endsWith('.jsonl')).map((f) => join(dir, f))+ } catch {+ return []+ }+}++/** The rollout files of the last day that mention one of this run's refs. */+function codexSessionFiles(refs: readonly string[], home: string): string[] {+ const root = join(home, '.codex', 'sessions')+ const since = Date.now() - 24 * 60 * 60 * 1000+ const hits: string[] = []+ const walk = (dir: string, depth: number): void => {+ let entries: string[]+ try { entries = readdirSync(dir) } catch { return }+ for (const e of entries) {+ const p = join(dir, e)+ let st+ try { st = statSync(p) } catch { continue }+ if (st.isDirectory() && depth < 3) walk(p, depth + 1)+ else if (st.isFile() && e.startsWith('rollout-') && e.endsWith('.jsonl') && st.mtimeMs >= since) {+ const text = readText(p)+ if (refs.some((r) => text.includes(r))) hits.push(p)+ }+ }+ }+ walk(root, 0)+ return hits+}diff --git a/cli/src/e2e/smokeChecks.ts b/cli/src/e2e/smokeChecks.tsnew file mode 100644index 000000000..7d171ae75--- /dev/null+++ b/cli/src/e2e/smokeChecks.ts@@ -0,0 +1,183 @@+/**+ * smokeChecks — the scenario: what a person does with a coding tool, on its own account and then on+ * a grid model in the same session: read a file, create one, edit one, use an MCP server. Four+ * plain requests per side — worded the way a person types them, no reply formats.+ *+ * Every step is proven by something the model cannot produce by talking:+ *+ * read the file holds a token made fresh for this run (`workspace.ts`), never written anywhere+ * else — not in the prompt, not in a log, different on each side. It showing up on the+ * pane after the question means the file was read; nobody guesses `kiwi-4821-tulip`+ * write the file exists on disk with exactly the requested text+ * edit the file on disk changed the requested word, and only that+ * mcp own account: the workspace's MCP server logs the call for EXACTLY these numbers+ * (.e2e/calc-mcp.log)+ * web grid: the web search the harness gives every agent on a grid (`mcp__harness__web_search`,+ * gridWebMcp.ts) was called and came back with results — read from the engine's session+ * file (sessionTools.ts `toolResult`), since that server runs on the grid, not here+ *+ * So a model that answers from its head, or claims to have done something it did not, fails the+ * step: the reply is never the proof. Each side uses its own files and numbers (…-1 / …-2), so the+ * grid side cannot pass on what the first side left behind.+ *+ * No shell step (dropped 2026-10-06): the file steps already go through the engine's shell where the+ * engine works that way (codex reads and writes through `exec_command`), and a script of our own+ * proved nothing a person's file work does not.+ *+ * No delete step either (dropped the same day it was added): codex counts `rm -f` as dangerous and+ * asks for approval (codex-rs shell-command is_dangerous_command.rs), so the step passed or failed on+ * whether the model happened to type `-f`, not on the switch.+ *+ * Memory across the switch is deliberately not a step (dropped 2026-09-24): its only honest proof+ * would be a fact that exists nowhere but the conversation, and the question is whether the tools+ * work on the grid.+ */++export type Leg = 'subscription' | 'grid'++/** The journey: the tool's own account, then the grid in the same session. */+export const LEGS: readonly Leg[] = ['subscription', 'grid']++/** Which log a step must leave a line in: the MCP server's (`workspace.ts`). */+export type LogKind = 'mcp'++export type CheckId = 'read' | 'write' | 'edit' | 'mcp' | 'web'++/** The harness's web search on a grid, as the engines name it (`harnessWebTools.ts`). */+export const WEB_SEARCH_TOOL = 'mcp__harness__web_search'++export interface SmokeCheck {+ id: CheckId+ /** Typed into the agent's pane, verbatim. Plain, engine-agnostic wording (codex and claude). */+ prompt: string+ /** mcp: the log that must gain a line containing `logPattern`. */+ log?: LogKind+ logPattern?: string+ /** read: a workspace file whose (per-run, random) content must show up after the prompt. */+ answerFromFile?: string+ /** write / edit: what the workspace file must look like when the step is done. */+ file?: { path: string; equals?: string; contains?: string; lacks?: string }+ /** web: the engine's tool that must have been called for this step and come back with results. */+ sessionTool?: string+ /** What a miss usually means — the watchdog's first hypothesis, not its conclusion. */+ onMiss: string+}++const MISS = {+ read: 'the file was not read: the read tool (or the shell read) fails on this leg — the answer never showed the file\'s token',+ write: 'the file was not created with the requested text: the write tool fails on this leg (codex: apply_patch Add File; claude: Write)',+ edit: 'the file was not changed as asked: the edit tool fails on this leg (codex: apply_patch Update File; claude: Edit)',+ mcp: 'the MCP server was not called: the respawn dropped the MCP config, the resumed session did not reload servers, or it answered without calling it (no log line)',+ web: 'the web search was not called, or came back without results: the harness web MCP was not given to the engine on the grid, the engine could not read the tool (a namespace tool on llama.cpp / LM Studio), or it answered from its head',+}++/**+ * Plain requests, the way a person types them, and the same for every engine. A step passes on its+ * RESULT — the file really read, written, edited; the MCP server really called — never on+ * which of the engine's file tools got it there. Which tools it used (claude `Read` or+ * `Bash cat`, codex `apply_patch` or a shell write) is read back from the session file and REPORTED+ * (sessionTools.ts, the message's Tools line), never judged: an engine is free to reach a result its+ * own way, and a person asking for it would be.+ *+ * That report is still worth reading. It is how it was seen, on the first real runs, that claude+ * does file work through Bash unless asked otherwise, and that codex on a grid model writes files+ * through the shell where on its own account it uses apply_patch (a grid model is not in codex's+ * model catalog, so codex does not offer it apply_patch).+ */+const files = (n: 1 | 2): SmokeCheck[] => [+ { id: 'read', prompt: `Read notes/info-${n}.txt and tell me what it says.`, answerFromFile: `notes/info-${n}.txt`, onMiss: MISS.read },+ { id: 'write', prompt: `Create the file out/hello-${n}.txt with this text: hello from step ${n}`, file: { path: `out/hello-${n}.txt`, equals: `hello from step ${n}` }, onMiss: MISS.write },+ { id: 'edit', prompt: `In notes/todo-${n}.txt, change pending to done.`, file: { path: `notes/todo-${n}.txt`, contains: 'status: done', lacks: 'pending' }, onMiss: MISS.edit },+]++/**+ * The three file requests on each side (own files per side, so nothing carries over), then an MCP+ * call: the workspace's own server on the tool's account, the harness's web search on the grid —+ * the MCP a person on a grid actually has, and the one codex 0.160's namespace tools broke on+ * llama.cpp (autonomous-grid-cli#41). The own account has no harness web MCP (it is given only on a+ * grid, gridLaunch.ts), so that side keeps `e2e_calc`.+ */+export const SCENARIO: Record<Leg, readonly SmokeCheck[]> = {+ subscription: [+ ...files(1),+ { id: 'mcp', prompt: 'Use the MCP server e2e_calc to add 30 and 12.', log: 'mcp', logPattern: 'add 30 12 = 42', onMiss: MISS.mcp },+ ],+ grid: [+ ...files(2),+ { id: 'web', prompt: 'Search the web for the latest stable version of Node.js and tell me the version number.', sessionTool: WEB_SEARCH_TOOL, onMiss: MISS.web },+ ],+}++/** The scenario an engine is given — the same plain requests for every engine (see above). */+export function scenarioFor(_engine: string): Record<Leg, readonly SmokeCheck[]> {+ return SCENARIO+}++/** Every step of every leg, for callers that want the flat list (the skill, the tests). */+export const SMOKE_CHECKS: readonly SmokeCheck[] = LEGS.flatMap((leg) => SCENARIO[leg])++/**+ * `no-quota`: the tool answered with its own "out of usage" message. Not a finding about the+ * switch, and not a bug anyone should read an analysis of — the account simply could not run the+ * test. It stops the journey, skips the reviewer, and the pipeline retries once the quota is back.+ */+export type CheckStatus = 'not-run' | 'ok' | 'stuck' | 'no-quota'++/** One step's outcome on one leg — the unit the watchdog reads. */+export interface CheckOutcome {+ id: SmokeCheck['id']+ status: CheckStatus+ elapsedMs?: number+ /** The pane's last lines after the step — evidence for the reviewer, present on ok AND stuck so legs can be diffed. */+ tail?: string+ /** Log lines the step added (tool / MCP) — the proof it went through the tool. Empty on a step that answered from memory. */+ logLines?: string[]+ /** Why it is not `ok` when the pane looked fine: the marker showed but the tool/MCP log gained no line. */+ note?: string+ /** Modals that landed DURING this step and were answered (claude's gateway notice) — a finding in itself. */+ dialogs?: string[]+ /** On `no-quota`: when the tool said it comes back, in its own words ("resets 6pm", "try again at …"). */+ resets?: string+ /** The engine's own tools this step went through, read from its session file (sessionTools.ts):+ * `Read`, `Write`, `Edit`, `Bash`, `apply_patch`, `exec_command`, `mcp__e2e_calc__add`, … */+ tools?: string[]+ /** The models that answered this step (sessionTools.ts) — own account: the pinned one; grid: the grid's. */+ models?: string[]+ /** The `(ref-…)` tag this step's prompt carried — how its tools and models are found afterwards. */+ ref?: string+}++export interface LegOutcome {+ leg: Leg+ checks: CheckOutcome[]+ /** Startup dialogs the probe had to answer before the first step (`codex-update`, `codex-trust`, …). */+ dialogs?: string[]+}++/** The dry-run shape: every leg, every step, nothing run yet. */+export function plannedLegs(): LegOutcome[] {+ return LEGS.map((leg) => ({+ leg,+ checks: SCENARIO[leg].map((c) => ({ id: c.id, status: 'not-run' as const })),+ }))+}++/** The (leg, step) where the account ran out of usage, or null. Checked before "what stuck". */+export function quotaHit(legs: readonly LegOutcome[]): { leg: Leg; check: SmokeCheck['id']; resets: string | null } | null {+ for (const l of legs) {+ for (const c of l.checks) {+ if (c.status === 'no-quota') return { leg: l.leg, check: c.id, resets: c.resets ?? null }+ }+ }+ return null+}++/** The first (leg, step) that did not pass — "what stuck", or null when the run is clean. */+export function firstStuck(legs: readonly LegOutcome[]): { leg: Leg; check: SmokeCheck['id']; status: CheckStatus } | null {+ for (const l of legs) {+ for (const c of l.checks) {+ if (c.status !== 'ok') return { leg: l.leg, check: c.id, status: c.status }+ }+ }+ return null+}diff --git a/cli/src/e2e/trigger.ts b/cli/src/e2e/trigger.tsnew file mode 100644index 000000000..9bd62ee03--- /dev/null+++ b/cli/src/e2e/trigger.ts@@ -0,0 +1,74 @@+/**+ * trigger — decide which engines need a matrix run right now.+ *+ * The rule: run for an engine when its INSTALLED (current) version differs from the tool's LATEST+ * released version. "Current" is read from the installed binary (`--version`/cli_version);+ * "latest" comes from a pluggable `VersionSource` (mock during development, npm when validated).+ *+ * This is the cronjob's decision step, kept pure (no I/O beyond the two injected providers) so it+ * is deterministic and unit-testable offline.+ */+import type { AgentEngine } from '../engines/types.js'+import { type VersionSource } from './versionSource.js'++export interface VersionProvider {+ currentVersion(engine: AgentEngine): Promise<string | null>+}++export interface PlannedRun {+ engine: AgentEngine+ current: string | null+ latest: string | null+ /** Why a run is (or is not) planned — for the audit log + NOTIFY. */+ reason: 'upgrade' | 'downgrade' | 'differs' | 'unknown-current' | 'no-latest' | 'unchanged'+ plan: boolean+}++/** Compare current vs latest for `engines` and return the run plan for each. */+export async function planMatrixRuns(+ source: VersionSource,+ currentProvider: VersionProvider,+ engines: readonly AgentEngine[],+): Promise<PlannedRun[]> {+ const runs: PlannedRun[] = []+ for (const engine of engines) {+ const latest = await source.latestVersion(engine)+ const current = await currentProvider.currentVersion(engine)++ if (latest === null) {+ runs.push({ engine, current, latest, reason: 'no-latest', plan: false })+ continue+ }+ if (current === null) {+ // Installed version unreadable but a latest is known: verify rather than assume.+ runs.push({ engine, current, latest, reason: 'unknown-current', plan: true })+ continue+ }+ const cmp = compareVersions(current, latest)+ runs.push({+ engine,+ current,+ latest,+ reason: cmp === 0 ? 'unchanged' : cmp < 0 ? 'upgrade' : 'downgrade',+ plan: cmp !== 0,+ })+ }+ return runs+}++/**+ * Naive dotted-numeric compare (0.155.0 < 0.155.1, 1.18.31 < 1.18.32). Handles prerelease-ish+ * suffixes by numeric prefix only; good enough to notice "a newer/older stable is out".+ */+export function compareVersions(a: string, b: string): number {+ const pa = (a.match(/\d+/g) ?? []).map(Number)+ const pb = (b.match(/\d+/g) ?? []).map(Number)+ const n = Math.max(pa.length, pb.length)+ for (let i = 0; i < n; i++) {+ const x = pa[i] ?? 0+ const y = pb[i] ?? 0+ if (x < y) return -1+ if (x > y) return 1+ }+ return 0+}diff --git a/cli/src/e2e/versionSource.ts b/cli/src/e2e/versionSource.tsnew file mode 100644index 000000000..798fdf592--- /dev/null+++ b/cli/src/e2e/versionSource.ts@@ -0,0 +1,85 @@+/**+ * Version sources for the E2E trigger.+ *+ * The trigger runs a matrix run when the INSTALLED (current) version of a coding-agent tool+ * differs from the tool's LATEST released version. "Latest" is read through a pluggable+ * `VersionSource` so we can develop/test offline against a MOCK (no network, $0), then swap in a+ * real implementation without changing the trigger logic.+ */+import type { AgentEngine } from '../engines/types.js'++export interface VersionSource {+ /** The latest released version string for `engine`, or null when it cannot be determined. */+ latestVersion(engine: AgentEngine): Promise<string | null>+}++/** Fixture "latest" versions, keyed by engine. Defaults make the offline trigger readable. */+const MOCK_LATEST: Partial<Record<AgentEngine, string>> = {+ codex: '0.156.0',+ claude: '2.1.279',+}++/**+ * Offline source used during development and in CI. Lets the whole trigger + matrix decision run+ * with no network and no accounts. Point at fixtures in a spec to exercise each branch.+ */+export class MockVersionSource implements VersionSource {+ constructor(private readonly overrides: Partial<Record<AgentEngine, string>> = {}) {}++ async latestVersion(engine: AgentEngine): Promise<string | null> {+ return this.overrides[engine] ?? MOCK_LATEST[engine] ?? null+ }+}++/**+ * Real source: each engine's STABLE release channel on npm — and only that.+ *+ * "Stable" is not the same dist-tag everywhere, which is why this is a table rather than `/latest`+ * for all (dist-tags read 2026-09-24):+ *+ * @openai/codex latest 0.156.1 ← stable alpha 0.158.0-alpha.8 (pre-releases)+ * @anthropic-ai/claude-code stable 2.1.273 ← stable latest/next 2.1.281 (the fast channel)+ *+ * Reading `latest` for claude tested its fast channel, eight releases ahead of what Claude Code+ * itself calls stable. And a version with a pre-release suffix (`-alpha.8`, `-beta.1`, `-rc.2`) is+ * refused whatever tag it came from: a pipeline that must never test a nightly should not depend on+ * a vendor never mis-tagging one.+ */+import { NPM_PACKAGE } from './npmPackages.js'++export const STABLE_TAG: Partial<Record<AgentEngine, string>> = { codex: 'latest', claude: 'stable' }++/** `0.156.1` yes; `0.158.0-alpha.8`, `2.1.0-beta` no. */+export function isStableVersion(version: string): boolean {+ return /^\d+\.\d+\.\d+$/.test(version)+}++export class NpmVersionSource implements VersionSource {+ constructor(private readonly fetchImpl: typeof fetch = fetch) {}++ async latestVersion(engine: AgentEngine): Promise<string | null> {+ const pkg = NPM_PACKAGE[engine]+ if (!pkg) return null+ try {+ // No `Accept: application/vnd.npm.install-v1+json` here, however natural it looks: the+ // abbreviated-metadata type is offered for the PACKAGE document, and asking for it on the+ // `/latest` dist-tag endpoint gets a 406 with an empty body. That failure is silent — the+ // catch below turns it into `latest=null`, which the trigger reads as "nothing new", so the+ // whole pipeline would sit quiet forever while looking perfectly healthy. Measured:+ // curl -o /dev/null -w '%{http_code}' -H 'Accept: application/vnd.npm.install-v1+json' \+ // https://registry.npmjs.org/@openai/codex/latest -> 406+ // curl -s https://registry.npmjs.org/@openai/codex/latest | jq .version -> "0.156.1"+ const res = await this.fetchImpl(`https://registry.npmjs.org/${pkg}/${STABLE_TAG[engine] ?? 'latest'}`)+ if (!res.ok) return null+ const body = (await res.json()) as { version?: string }+ return body.version && isStableVersion(body.version) ? body.version : null+ } catch {+ return null+ }+ }+}++/** Pick the source from `E2E_VERSION_SOURCE` (default `mock` until the real one is validated). */+export function versionSourceFromEnv(env: Record<string, string | undefined> = process.env): VersionSource {+ return env.E2E_VERSION_SOURCE === 'npm' ? new NpmVersionSource() : new MockVersionSource()+}diff --git a/cli/src/e2e/workspace.ts b/cli/src/e2e/workspace.tsnew file mode 100644index 000000000..04d60e244--- /dev/null+++ b/cli/src/e2e/workspace.ts@@ -0,0 +1,338 @@+/**+ * workspace — the folder the test agent is created in, laid out like a person's real project:+ * the notes the file steps read and edit, an MCP server registered for the engine+ * (`e2e_calc`), and the log it writes to. The scenario (`smokeChecks.ts`) names them; the probe+ * reads the files and the log.+ *+ * <cwd>/notes/info-N.txt read: a token made fresh for this run+ * <cwd>/notes/todo-N.txt edit: `status: pending`+ * <cwd>/tools/calc-mcp.mjs the MCP server+ * <cwd>/.mcp.json claude: project-level MCP server+ * ~/.codex/config.toml codex: registered with `codex mcp add` — see below+ * <cwd>/.e2e/calc-mcp.log one line per MCP tools/call+ *+ * The two engines differ, and both shapes here were measured against the installed CLIs rather than+ * assumed:+ *+ * * claude reads `<cwd>/.mcp.json` — `claude mcp list` in the workspace lists `e2e_calc` and a+ * headless run really reaches the server. The agent runs in the app's default mode (`auto`), so+ * the project's `.claude/settings.json` allows the steps' tools the way a person who answered+ * "don't ask again" has them (`allowClaudeProjectTools`).+ * * codex has NO project-level MCP config. A `<cwd>/.codex/config.toml` is never read (from inside+ * such a workspace, `codex mcp list` does not list the server), so registration goes through+ * codex's own `codex mcp add`, which writes `~/.codex/config.toml`, with its tools always allowed+ * (`approveCodexMcpTools`) — and `removeCodexMcp` takes the entry out again when the run is over.+ */+import { mkdirSync, readFileSync, writeFileSync, existsSync, renameSync, realpathSync } from 'node:fs'+import { execFileSync } from 'node:child_process'+import { randomInt } from 'node:crypto'+import { homedir } from 'node:os'+import { join } from 'node:path'+import type { LogKind } from './smokeChecks.js'++// Written from this string, not copied from beside this file: the published CLI is ONE bundled+// `cli.js` (build-bundle.mjs), and files next to the source never reach an installed machine. The+// original under `workspace/` is the readable copy; the spec pins them equal.+export const CALC_MCP_MJS = `#!/usr/bin/env node+/**+ * e2e_calc — the smallest MCP server that can prove "the tool still uses MCP after a switch".+ *+ * stdio transport, newline-delimited JSON-RPC, two tools: add(a, b) and sub(a, b). Every call is+ * appended to the log file given as argv[2] (\`<ts> <tool> <a> <b> = <result>\`), so the e2e has hard+ * evidence the model went THROUGH the server rather than doing the arithmetic in its head.+ * No SDK on purpose: nothing to install in the scratch workspace the test agent runs in.+ */+import { appendFileSync } from 'node:fs'+import { createInterface } from 'node:readline'++const logFile = process.argv[2]+const TOOLS = [+ { name: 'add', description: 'Add two numbers: a + b', inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] } },+ { name: 'sub', description: 'Subtract two numbers: a - b', inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] } },+]++function send(msg) {+ process.stdout.write(JSON.stringify(msg) + '\\n')+}++function log(line) {+ if (logFile) appendFileSync(logFile, \`\${new Date().toISOString()} \${line}\\n\`)+}++const rl = createInterface({ input: process.stdin })+rl.on('line', (line) => {+ if (!line.trim()) return+ let req+ try { req = JSON.parse(line) } catch { return }+ const { id, method, params } = req+ if (id === undefined) return // notification (notifications/initialized, …)+ switch (method) {+ case 'initialize':+ send({ jsonrpc: '2.0', id, result: { protocolVersion: params?.protocolVersion ?? '2024-11-05', capabilities: { tools: {} }, serverInfo: { name: 'e2e_calc', version: '1.0.0' } } })+ return+ case 'ping':+ send({ jsonrpc: '2.0', id, result: {} })+ return+ case 'tools/list':+ send({ jsonrpc: '2.0', id, result: { tools: TOOLS } })+ return+ case 'tools/call': {+ const name = params?.name+ const a = Number(params?.arguments?.a)+ const b = Number(params?.arguments?.b)+ if ((name !== 'add' && name !== 'sub') || Number.isNaN(a) || Number.isNaN(b)) {+ send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: \`error: unknown tool or bad arguments (\${name})\` }], isError: true } })+ return+ }+ const result = name === 'add' ? a + b : a - b+ log(\`\${name} \${a} \${b} = \${result}\`)+ send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: String(result) }] } })+ return+ }+ default:+ send({ jsonrpc: '2.0', id, error: { code: -32601, message: \`method not found: \${method}\` } })+ }+})+`++export const MCP_SERVER_NAME = 'e2e_calc'+/** The tools `CALC_MCP_MJS` serves. */+export const CALC_MCP_TOOLS = ['add', 'sub'] as const++/**+ * What a person using claude in this project has long since answered "Yes, and don't ask again"+ * to: the file tools, reading a file through the shell, the project's MCP server and+ * the harness's web search on a grid — kept by claude in the project's `.claude/settings.json`+ * (`permissions.allow`), with the `.mcp.json` server approved in the same file+ * (`enabledMcpjsonServers`). Measured on claude 2.1.274 + Haiku in `auto`: without it a shell step+ * stopped on "This command requires approval".+ */+export function allowClaudeProjectTools(cwd: string): void {+ mkdirSync(join(cwd, '.claude'), { recursive: true })+ const settings = {+ permissions: {+ allow: [+ 'Read', 'Write', 'Edit',+ 'Bash(cat:*)',+ `mcp__${MCP_SERVER_NAME}`,+ 'mcp__harness',+ ],+ },+ enabledMcpjsonServers: [MCP_SERVER_NAME],+ }+ writeFileSync(join(cwd, '.claude', 'settings.json'), JSON.stringify(settings, null, 2) + '\n')+}++export function logPath(cwd: string, _kind: LogKind): string {+ return join(cwd, '.e2e', 'calc-mcp.log')+}++export interface WorkspaceLayout {+ mcpServer: string+ /** Where this engine's MCP registration ended up, or null when there is none. */+ mcpConfig: string | null+ /** How it got there — or, when `mcpConfig` is null, why the MCP steps cannot pass. */+ mcpNote: string+}++/**+ * Register the workspace's MCP server with codex, through codex's own command.+ *+ * `remove` first so a stale entry from an interrupted run (pointing at a workspace that no longer+ * exists) cannot survive as a server codex fails to start. Both calls are best-effort: a codex too+ * old for `mcp add` must leave a run without MCP rather than without a run, and the caller reports+ * that in the trace.+ */+export function registerCodexMcp(mcpServer: string, mcpLog: string, bin = 'codex'): { ok: true } | { ok: false; detail: string } {+ const run = (args: string[]): string => execFileSync(bin, args, { encoding: 'utf8', timeout: 60_000, stdio: ['ignore', 'pipe', 'pipe'] })+ try { run(['mcp', 'remove', MCP_SERVER_NAME]) } catch { /* not registered — the normal case */ }+ try {+ run(['mcp', 'add', MCP_SERVER_NAME, '--', 'node', mcpServer, mcpLog])+ } catch (err) {+ const e = err as { stderr?: string; message?: string }+ return { ok: false, detail: (e.stderr || e.message || String(err)).trim().split('\n')[0] }+ }+ // Registered is not the same as visible: ask codex what it sees, so a run never starts believing+ // in a server the engine will not load.+ try {+ if (!run(['mcp', 'list']).includes(MCP_SERVER_NAME)) return { ok: false, detail: `codex mcp add reported success but 'codex mcp list' does not show ${MCP_SERVER_NAME}` }+ } catch { /* `mcp list` can fail on its own (a broken unrelated server); the add is what matters */ }+ return { ok: true }+}++/**+ * Mark the server's tools as always allowed, the entry codex writes itself when a person answers a+ * tool's approval with "always allow" — how every MCP tool on a working developer's machine ends up+ * (`[mcp_servers.<server>.tools.<tool>] approval_mode = "approve"`). Without it a tool with no+ * `readOnlyHint` needs approval on every call (codex-rs/core/src/mcp_tool_call.rs+ * `requires_mcp_tool_approval`), and in `auto` that approval is codex's reviewer, which asks for a+ * model named `codex-auto-review` — one only OpenAI serves: on the grid it answered 503 and codex+ * refused the call (openai/codex#24879). Nested under the server's table, so `codex mcp remove`+ * takes it out with the server.+ */+export function approveCodexMcpTools(configPath: string, tools: readonly string[]): void {+ const lines = tools.map((t) => `\n[mcp_servers.${MCP_SERVER_NAME}.tools.${t}]\napproval_mode = "approve"\n`).join('')+ writeFileSync(configPath, lines, { flag: 'a' })+}++/** Take the entry back out — the run's own cleanup, so the next run starts from a clean config. */+export function removeCodexMcp(bin = 'codex'): boolean {+ try {+ execFileSync(bin, ['mcp', 'remove', MCP_SERVER_NAME], { encoding: 'utf8', timeout: 60_000, stdio: ['ignore', 'pipe', 'pipe'] })+ return true+ } catch {+ return false+ }+}++/**+ * Lay the workspace out for `engine`. Idempotent. Returns what was written, for the trace.+ *+ * `codexBin` exists so a test can watch the registration without running the real `codex mcp add`:+ * that command writes the developer's own `~/.codex/config.toml`, and a unit test has no business+ * leaving an entry there pointing at a temp folder it is about to delete.+ */+export function prepareWorkspace(cwd: string, engine: string, opts: { codexBin?: string; codexConfig?: string } = {}): WorkspaceLayout {+ mkdirSync(join(cwd, 'tools'), { recursive: true })+ mkdirSync(join(cwd, '.e2e'), { recursive: true })+ writeFileSync(logPath(cwd, 'mcp'), '', { flag: 'a' })++ // read / write / edit (smokeChecks.ts): one set of files per side, so the grid side can never pass+ // (named `info-N.txt`, not `secret-N.txt`: gpt-6-luna refused outright to read a file called+ // "secret" — "I can't provide the contents of a file named notes/secret-1.txt" — and a test that+ // provokes a refusal measures the refusal, not the switch)+ // on what the first side left. The token is made HERE, fresh for every run, and written nowhere+ // but its file — not in a prompt, not in a log — so the only way to name it is to read it.+ mkdirSync(join(cwd, 'notes'), { recursive: true })+ mkdirSync(join(cwd, 'out'), { recursive: true })+ for (const n of [1, 2]) {+ writeFileSync(join(cwd, 'notes', `info-${n}.txt`), `${freshToken()}\n`)+ writeFileSync(join(cwd, 'notes', `todo-${n}.txt`), `# todo ${n}\nstatus: pending\n`)+ }++ // The MCP server runs from the workspace too, so the engine's config points at a path that exists+ // on every machine the e2e runs on.+ const mcpServer = join(cwd, 'tools', 'calc-mcp.mjs')+ writeFileSync(mcpServer, CALC_MCP_MJS)+ const mcpLog = logPath(cwd, 'mcp')+ let mcpConfig: string | null = null+ let mcpNote = `${engine} has no MCP registration here: the MCP steps cannot pass`+ if (engine === 'codex') {+ const registered = registerCodexMcp(mcpServer, mcpLog, opts.codexBin ?? 'codex')+ if (registered.ok) {+ mcpConfig = opts.codexConfig ?? join(homedir(), '.codex', 'config.toml')+ approveCodexMcpTools(mcpConfig, CALC_MCP_TOOLS)+ mcpNote = `registered with \`codex mcp add\`, tools ${CALC_MCP_TOOLS.join('/')} always allowed (codex does not read a project-level .codex/config.toml)`+ } else {+ mcpNote = `codex mcp add failed: ${registered.detail}`+ }+ } else if (engine === 'claude') {+ mcpConfig = join(cwd, '.mcp.json')+ writeFileSync(mcpConfig, JSON.stringify({ mcpServers: { [MCP_SERVER_NAME]: { command: 'node', args: [mcpServer, mcpLog] } } }, null, 2) + '\n')+ preApproveClaudeMcp(cwd, [MCP_SERVER_NAME])+ allowClaudeProjectTools(cwd)+ mcpNote = 'project .mcp.json; the steps\' tools always allowed in .claude/settings.json'+ }+ return { mcpServer, mcpConfig, mcpNote }+}++/**+ * Claude Code loads a project's `.mcp.json` servers only once the person approved them; the answer+ * is kept in `~/.claude.json` as `projects[<cwd>].enabledMcpjsonServers`. For a workspace this run+ * made, that answer is the run's to give — same posture and same file handling as the daemon's+ * `preTrustClaudeProject` (lib/claudeTrust.ts): only add, never touch a shape we did not expect.+ *+ * Measured against claude 2.1.x, this is NOT what carries the MCP steps, and the file says so rather+ * than leaving the next reader to re-measure: claude rewrites `~/.claude.json` at startup and the+ * key is gone afterwards (`claude mcp list` still says "Pending approval"). What actually reaches+ * the server is the `full` permission mode the agent is created in. Kept because it costs one write+ * and is the right answer if a future claude honours it; never relied on.+ */+export function preApproveClaudeMcp(cwd: string, names: string[], home = homedir()): 'approved' | 'already' | 'skipped' {+ const file = join(home, '.claude.json')+ if (!existsSync(file)) return 'skipped'+ let config: unknown+ try {+ config = JSON.parse(readFileSync(file, 'utf8'))+ } catch {+ return 'skipped'+ }+ const isObj = (v: unknown): v is Record<string, unknown> => typeof v === 'object' && v !== null && !Array.isArray(v)+ if (!isObj(config)) return 'skipped'+ if (config.projects !== undefined && !isObj(config.projects)) return 'skipped'+ const projects = (config.projects ?? {}) as Record<string, unknown>+ const existing = projects[cwd]+ if (existing !== undefined && !isObj(existing)) return 'skipped'+ const current = Array.isArray(existing?.enabledMcpjsonServers) ? (existing!.enabledMcpjsonServers as unknown[]) : []+ if (names.every((n) => current.includes(n))) return 'already'+ projects[cwd] = { allowedTools: [], ...(existing ?? {}), enabledMcpjsonServers: [...current, ...names.filter((n) => !current.includes(n))] }+ config.projects = projects+ // Atomic, through a symlink (dotfile managers link ~/.claude.json), like the daemon does.+ const target = realpathSync(file)+ const tmp = `${target}.e2e-${process.pid}.tmp`+ writeFileSync(tmp, JSON.stringify(config, null, 2), { mode: 0o600 })+ renameSync(tmp, target)+ return 'approved'+}++/**+ * Accept Claude Code's Bypass Permissions warning ahead of time.+ *+ * `--dangerously-skip-permissions` does not go straight to the prompt the first time: an+ * interactive claude draws a full-screen "WARNING: Claude Code running in Bypass Permissions mode"+ * with **"No, exit" selected**, and on Enter it exits 1 — the pane becomes a shell and every check+ * of the run is stuck with no engine behind it. (A headless `claude -p` never shows it, which is+ * why the flag looks harmless until it runs in a pane.) The answer is remembered in `~/.claude.json`+ * as `bypassPermissionsModeAccepted`, a key read out of claude 2.1.281's own binary rather than+ * guessed — so the run gives it the same way a person would, once, before the agent starts.+ *+ * It is NOT sufficient on its own, and the next reader should not have to find that out in a failed+ * run: on the pass that proved this, the flag was written (`accepted`) and the warning still came up+ * — `paneProbe`'s `claude-bypass-accept` answered it, and the leg passed. So the dialog handler is+ * what carries the run and this write is the belt: cheap, correct, and it may be what keeps the+ * screen away on a machine that reads the flag earlier.+ */+export function preAcceptClaudeBypassMode(home = homedir()): 'accepted' | 'already' | 'skipped' {+ const file = join(home, '.claude.json')+ if (!existsSync(file)) return 'skipped'+ let config: unknown+ try {+ config = JSON.parse(readFileSync(file, 'utf8'))+ } catch {+ return 'skipped'+ }+ if (typeof config !== 'object' || config === null || Array.isArray(config)) return 'skipped'+ const conf = config as Record<string, unknown>+ if (conf.bypassPermissionsModeAccepted === true) return 'already'+ conf.bypassPermissionsModeAccepted = true+ const target = realpathSync(file)+ const tmp = `${target}.e2e-${process.pid}.tmp`+ writeFileSync(tmp, JSON.stringify(conf, null, 2), { mode: 0o600 })+ renameSync(tmp, target)+ return 'accepted'+}++const WORDS = ['kiwi', 'tulip', 'cobalt', 'maple', 'otter', 'saffron', 'glacier', 'lantern', 'pepper', 'quartz', 'harbor', 'violet', 'ember', 'falcon', 'juniper', 'meadow']++/** `kiwi-4821-tulip`: two words and four digits, ~2.5 million combinations, new every run. */+export function freshToken(): string {+ return `${WORDS[randomInt(WORDS.length)]}-${randomInt(1000, 10000)}-${WORDS[randomInt(WORDS.length)]}`+}++/** A workspace file's content, or null when it does not exist — what the probe checks write/edit/read against. */+export function readWorkspaceFile(cwd: string, relPath: string): string | null {+ try {+ return readFileSync(join(cwd, relPath), 'utf8')+ } catch {+ return null+ }+}++/** The log's lines right now (empty when it does not exist yet). */+export function readLog(cwd: string, kind: LogKind): string[] {+ try {+ return readFileSync(logPath(cwd, kind), 'utf8').split('\n').filter(Boolean)+ } catch {+ return []+ }+}diff --git a/cli/src/e2e/workspace/calc-mcp.mjs b/cli/src/e2e/workspace/calc-mcp.mjsnew file mode 100755index 000000000..a6981ba82--- /dev/null+++ b/cli/src/e2e/workspace/calc-mcp.mjs@@ -0,0 +1,60 @@+#!/usr/bin/env node+/**+ * e2e_calc — the smallest MCP server that can prove "the tool still uses MCP after a switch".+ *+ * stdio transport, newline-delimited JSON-RPC, two tools: add(a, b) and sub(a, b). Every call is+ * appended to the log file given as argv[2] (`<ts> <tool> <a> <b> = <result>`), so the e2e has hard+ * evidence the model went THROUGH the server rather than doing the arithmetic in its head.+ * No SDK on purpose: nothing to install in the scratch workspace the test agent runs in.+ */+import { appendFileSync } from 'node:fs'+import { createInterface } from 'node:readline'++const logFile = process.argv[2]+const TOOLS = [+ { name: 'add', description: 'Add two numbers: a + b', inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] } },+ { name: 'sub', description: 'Subtract two numbers: a - b', inputSchema: { type: 'object', properties: { a: { type: 'number' }, b: { type: 'number' } }, required: ['a', 'b'] } },+]++function send(msg) {+ process.stdout.write(JSON.stringify(msg) + '\n')+}++function log(line) {+ if (logFile) appendFileSync(logFile, `${new Date().toISOString()} ${line}\n`)+}++const rl = createInterface({ input: process.stdin })+rl.on('line', (line) => {+ if (!line.trim()) return+ let req+ try { req = JSON.parse(line) } catch { return }+ const { id, method, params } = req+ if (id === undefined) return // notification (notifications/initialized, …)+ switch (method) {+ case 'initialize':+ send({ jsonrpc: '2.0', id, result: { protocolVersion: params?.protocolVersion ?? '2024-11-05', capabilities: { tools: {} }, serverInfo: { name: 'e2e_calc', version: '1.0.0' } } })+ return+ case 'ping':+ send({ jsonrpc: '2.0', id, result: {} })+ return+ case 'tools/list':+ send({ jsonrpc: '2.0', id, result: { tools: TOOLS } })+ return+ case 'tools/call': {+ const name = params?.name+ const a = Number(params?.arguments?.a)+ const b = Number(params?.arguments?.b)+ if ((name !== 'add' && name !== 'sub') || Number.isNaN(a) || Number.isNaN(b)) {+ send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: `error: unknown tool or bad arguments (${name})` }], isError: true } })+ return+ }+ const result = name === 'add' ? a + b : a - b+ log(`${name} ${a} ${b} = ${result}`)+ send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: String(result) }] } })+ return+ }+ default:+ send({ jsonrpc: '2.0', id, error: { code: -32601, message: `method not found: ${method}` } })+ }+})diff --git a/cli/src/lib/nativeProcessControl.spec.ts b/cli/src/lib/nativeProcessControl.spec.tsindex 2b4d618b8..351e631bd 100644--- a/cli/src/lib/nativeProcessControl.spec.ts+++ b/cli/src/lib/nativeProcessControl.spec.ts@@ -8,7 +8,9 @@ import { NativeEvidenceBudget } from '../engines/kit/nativeEvidence.js'import { controlDescriptors, nativeProcessControl, parseNativeProcessControl } from './nativeProcessControl.js'import { env } from '../config/env.js'-const host = vi.hoisted(() => ({ root: '', exec: vi.fn(), helper: vi.fn(), before: undefined as undefined | ((operation: string, path: string) => void) }))+const host = vi.hoisted(() => ({ root: '', exec: vi.fn(), helper: vi.fn(), before: undefined as undefined | ((operation: string, path: string) => void),+ // A filesystem lists a directory in its own order (a tmpfs, ext4's hash): CI listed task 99 before 42.+ listDescending: false }))vi.mock('node:os', async original => ({ ...await original<object>(), platform: () => process.platform }))vi.mock('node:child_process', async original => ({ ...await original<object>(), execFile: host.exec }))vi.mock('./nativeProcessImages.js', async original => ({ ...await original<object>(), bundledProcessImageHelper: host.helper }))@@ -22,6 +24,16 @@ vi.mock('node:fs', async original => {return Reflect.apply(fs[method], fs, [mapped(path), ...args])}) as never}+ const opendir = result.opendirSync+ result.opendirSync = ((path: unknown, ...args: unknown[]) => {+ const directory = Reflect.apply(opendir, fs, [path, ...args]) as import('node:fs').Dir+ if (!host.listDescending) return directory+ const listed: import('node:fs').Dirent[] = []+ for (let next = directory.readSync(); next; next = directory.readSync()) listed.push(next)+ directory.closeSync()+ listed.sort((a, b) => Number(b.name) - Number(a.name))+ return { readSync: () => listed.shift() ?? null, closeSync: () => {} }+ }) as neverreturn result})const platform = Object.getOwnPropertyDescriptor(process, 'platform')!@@ -175,6 +187,15 @@ describe('joined Linux control proof', () => {file(join(host.root, 'proc/45/stat'), `45 (exited) Z 42 ${Array(17).fill(0).join(' ')} 100\n`)expect((await nativeProcessControl([42, 43, 44], 42, budget())).children).toEqual([43, 44])})+ it('gives the children in thread order, however the directory lists its threads', async () => {+ processRow(43, 42); processRow(44, 42)+ file(join(host.root, 'proc/42/task/42/children'), '43 ')+ file(join(host.root, 'proc/42/task/99/children'), '44 ')+ host.listDescending = true+ try {+ expect((await nativeProcessControl([42, 43, 44], 42, budget())).children).toEqual([43, 44])+ } finally { host.listDescending = false }+ })it.each(['descriptor', 'generation', 'image', 'child', 'task'])('holds a changing %s in the joined observation', async mode => {let calls = 0host.before = (operation, path) => {diff --git a/cli/src/lib/nativeProcessControl.ts b/cli/src/lib/nativeProcessControl.tsindex dda25e527..c88fa66e7 100644--- a/cli/src/lib/nativeProcessControl.ts+++ b/cli/src/lib/nativeProcessControl.ts@@ -141,7 +141,9 @@ function linuxControl(pids: readonly number[], parent: number, budget: NativeEviif (!parent) return []const tasks = entries(`/proc/${parent}/task`, 128), ids: number[] = []if (!tasks.length) return nativeUnavailable('the native task pool is empty')- for (const task of tasks) {+ // In thread order, as /proc lists them: a directory read elsewhere (a test's tmpfs, a hashed ext4) lists in+ // its own order, and the children came back in it (CI, 2026-10-20: [44, 43] for [43, 44]).+ for (const task of [...tasks].sort((a, b) => Number(a) - Number(b))) {const text = bytes(`/proc/${parent}/task/${task}/children`, 1024)if (!/^(?:\d+\s+)*$/.test(text)) return nativeUnavailable('the native child pool is malformed')for (const value of text.trim().split(/\s+/).filter(Boolean)) {diff --git a/docs/e2e-findings-2026-09-22.md b/docs/e2e-findings-2026-09-22.mdnew file mode 100644index 000000000..dc568036c--- /dev/null+++ b/docs/e2e-findings-2026-09-22.md@@ -0,0 +1,39 @@+# E2E grid-switch — finding 2026-09-22: codex resume onto a grid fails after an MCP call++**Symptom.** A codex 0.155.x session that has called an MCP tool on its own login, moved onto a grid+model (any node: `Qwen/Qwen3.8-27B`, `DeepSeek-V4-Flash-0731`) and resumed, never answers again:+`■ stream disconnected before completion: transient provider recovery exhausted`. A FRESH session+on the same grid model answers fine — which is why "switching in the app works" and this only shows+up in the real scenario (use the person's tool + MCP, then switch, keep talking).++**Root cause (isolated with a curl replay, no harness involved).** The relay answers the resumed+request with `event: response.failed` / `error.code: server_error "transient provider recovery+exhausted"`. The replayed history contains two Responses item types codex emits when it loads a deferred+MCP tool — `tool_search_call` / `tool_search_output`. Not new in 0.155: on by default since+`rust-v0.150.0` (upstream #29486, 2026-06-22; features `tool_search` / `tool_search_always_defer_mcp_tools`+are "Removed" = baked in). It surfaced today because the scenario (use an MCP tool on the vendor, THEN+switch) had never been run — the e2e must run on a schedule, not only when a version changes. Removing ONLY those two+items from `input` makes the same request complete (`RESUME_40+2` — the model even recalls the+conversation). `reasoning` items (with `encrypted_content`) and `function_call` items are harmless.++| replayed request | relay result |+|-------------------------------------------|-----------------------------------|+| as recorded (resume history) | `response.failed` server_error |+| minus `tool_search_call/_output` | `completed`, answers + recalls |+| minus `reasoning` only | `response.failed` |+| minus `reasoning` + `tool_search_*` | `completed` |++**Where to fix.** The grid relay, when forwarding a Responses request to a non-OpenAI provider:+drop (or translate) `tool_search_call` / `tool_search_output` input items — they are discovery+bookkeeping; the real call is the `function_call` that follows. Codex cannot be told to stop+emitting them.++**Repro.** `cli/src/e2e/runGridSwitchTrace.ts` (live, creates its own agent) reproduces it on every+run at `grid / recall`; the isolated replay is a `POST <grid>/relay/v1/responses` with the recorded+`input` minus the two item types (script kept in the run's scratch folder during the session).++**Side notes from the same runs.**+- codex 0.155.0 → 0.155.1 got upgraded on this Mac by run 3 (the probe's Enter landed on the+ "Update now" dialog; now answered with Skip before any check).+- The codex subscription hit its usage limit during these runs (`You've hit your usage limit`);+ the subscription leg cannot run again until it resets.diff --git a/docs/e2e-grid-matrix-plan.md b/docs/e2e-grid-matrix-plan.mdnew file mode 100644index 000000000..96229f9d8--- /dev/null+++ b/docs/e2e-grid-matrix-plan.md@@ -0,0 +1,183 @@+# E2E Grid-Switch Matrix — Plan++Goal: when a NEW version of a coding-agent tool (codex / claude) is acknowledged,+verify it against the **real** harness grid-switch journey (not a mock): an agent on its own+subscription is moved onto a grid model, resumed, checked that it still answers / uses tools /+has its MCP servers / spawns a subagent, then moved back to its own login and checked again — with every step/value logged under a+namespaced session so we can audit what each agent read and did, catch hallucination / wrong+decisions / wrong actions, and produce a review from a free-model watchdog.++---++## 1. Two layers of environment (decide A/B/C)++| Layer | What runs | Needs | Sandboxable? |+|---|---|---|---|+| **Controller** (drive + watchdog) | `gridSwitchDriver` + `paneProbe` (Node) + the opencode `e2e-watchdog` agent (free model) | Node, tmux, opencode | ⚠️ needs the pane: runs on the machine under test |+| **Under-test** (the machine being tested) | harness daemon + tmux + real codex/claude + grid sign-in | tmux, interactive login shell, **codex auth**, **claude login**, **grid sign-in** | ⚠️ only if you provision all login/state inside |++> **DECIDED: Option A — code-managed, ZERO-CONFIG, deterministic scripts on the real machine.**+> All logic lives in repo code (argv-package, matrix script, watchdog agent, session naming).+> Every parameter is auto-derived in code (self-discovering) — daemon port from `env.PORT`, machineId+> from the daemon, engine version from `--version`/`cli_version`, grid model picked automatically from+> available, grid endpoint/credential from the machine's grid login, session name from timestamp.+> No config file, no manual flags, no drift → consistent wherever the same prerequisites are met.+> Under-test reuses the already-running harness machine (codex auth/tmux/grid sign-in). No workflow+> container is required for consistency; a container is OPTIONAL only for the watchdog (read-only+> trace, no credentials) if blast-radius isolation is wanted later.++- **A ✔ (chosen) — code-managed scripts; under-test on the real machine.**+ Fastest, least manual deploy. Real machine already has `~/.codex/auth.json`, tmux, zsh.+ Watchdog runs on the machine (or optionally in an isolated container) and only reads the trace.+- **B — everything in a container sandbox.**+ More setup: must mount codex/claude/grid auth + tmux + harness and keep login state fresh+ for every matrix run. Rejected: too much manual ops.+- **C — split relay: loopback local relay for the relay/convert part ($0), separate real+ subscription→grid run on the real machine.**++---++## 2. Session naming — one name per run, auditable end-to-end++Convention: `session = <engine>@<version>-><testcase>@<grid-model>--<timestamp>`++Examples:+```[email protected]>grid-switch@grid:gpt-5-mini--20260921T1805Z[email protected]>back-home@opus--20260921T1810Z+claude@UNKNOWN->grid-switch@none--20260921T1820Z # engine not logged in -> still nameable + notified+```++The SAME name is stamped on everything from one run:+```[email protected]>[email protected] # driver step/value log+report-…json / [email protected]>…json # what the watchdog reads; its review is written back into it[email protected]>...md # watchdog review output+```+`<engine>@UNKNOWN` is used when a tool is present but not logged in, so the run still has an+audit name and a NOTIFY instead of a hard failure.++---++## 3. argv-package — the real command the harness enters++Spawn is `tmux respawn-pane -k` with the engine argv from `buildEngineCommandArgv`+(`cli/src/lib/engineLaunch.ts`) + env/clearEnv from `launchOverrides.ts` → `gridLaunch.ts`.++- **codex → grid:** `codex [resume <session>] -c model_provider="grid" -c model_providers.grid.name=…+ -c model_providers.grid.base_url=<grid>/relay/v1 -c model_providers.grid.env_key=…+ -c model_providers.grid.wire_api="responses" -c model_providers.grid.supports_websockets="true"+ [-c web_search="disabled"]`; env `OPENAI_API_KEY=<grid token>`; clear vendor creds.+- **claude → grid:** `claude --resume <session> [mode]`; env `ANTHROPIC_BASE_URL=<grid anthropic relay>`,+ `ANTHROPIC_MODEL=<model>`; clear `ANTHROPIC_API_KEY`.+- **back to own login:** clearEnv of grid vars + respawn with `ownLoginProviderArgs` (codex re-reads+ `config.toml`; claude returns to its own `ANTHROPIC_*`).++The E2E matrix builds these argv itself (same builder) so it spawns exactly like the harness.++---++## 3b. Real-test models (fixed per user decision)++- **Subscription leg** (codex/claude answering on their OWN login) uses a fixed cheap model,+ enabled in the tmux launch (`cli/src/e2e/fixedModels.ts`):+ - codex → `gpt-5.5`+ - claude → `claude-sonnet-5`+ (codex/claude expose no model-list CLI, so these are explicit and tuneable in one place.)+- **Grid leg** picks the smallest available model automatically from `grid_models_list`.++---++## 4. Real-matrix flow (per engine × version)++```+1. prep: daemon + real agent in a tmux pane + engine login present+2. for each engine (codex/claude) × version:+ a. agent answers on its OWN subscription (real round-trip)+ b. driver -> agent_retarget grid (payload gridModel/gridName)+ c. harness respawns with grid argv; driver PINGS the engine; wait for answer from the GRID+ d. verify the answer came from the grid, not the old subscription+ e. driver -> agent_retarget clearGrid; verify back on own login+ f. write trace-<session>.json + per-engine log+3. engine not logged in -> NOTIFY (no hard fail); still name the session @UNKNOWN+ grid not sign-in -> NOTIFY (per decision)+```++Handles: refusal codes (AGENT_BUSY, RETARGET_UNSUPPORTED_BACKEND, GRID_UNAVAILABLE, …) as+surfaced results, and surfaces every step in the trace so a reviewer can audit what happened.++---++## 5. Where each piece lives (decided 2026-09-22)++Two repos, split by who owns the code under test:++- **autonomous-harness** (this repo) — the **driver**: `harness e2e plan` (trigger: installed vs+ latest) and `harness e2e grid-switch` (`cli/src/e2e/`): creates a fresh agent through the daemon,+ lays out the person's project (`tools/calc.sh` + MCP `e2e_calc`, `workspace.ts`), runs the+ conversation subscription → grid → back-home (`smokeChecks.ts` scenario: tool / mcp / recall,+ proven by the tool/MCP logs, `paneProbe.ts`), moves it with `agent_retarget` (own grid, or a+ relay under test via the `grid` override), and leaves a **review bundle** in+ `~/.harness/cli/data/e2e/runs/<session>/` (trace, daemon log slice, workspace logs, manifest).+- **autonomous-grid-cli** (`e2e/`) — the **reviewer and the fix loop**, next to the relay it fixes:+ `agents/e2e-watchdog.md` (system prompt) + `skills/e2e-watchdog/SKILL.md` (quick view, report+ fields, clean-run shape, find/reproduce, relay bisect, fix + PR rules, report shape),+ `install.sh` (sync into `~/.config/opencode`), `run.sh` (the cron entry: plan → grid-switch →+ `opencode run --agent e2e-watchdog … "Run report: <bundle>/trace.json. What got stuck?"` → bundle+ gains `watchdog.jsonl` + `watchdog.md` + the opencode session id to reopen), and `stack/` (a+ relay checkout under test: strict node, logging proxy, `matrix.sh`, the recorded requests).+- **The grid server** runs it: harness daemon + tmux + codex/claude logins + opencode (free model),+ cron `e2e/run.sh`. The e2e-watchdog reads the relay code in place, tests it, opens the PR there.++Verified 2026-09-22 on this Mac: the relay-commit matrix (`matrix.sh f5bbb4e 00fa887 22519fa+d6c5489 main` × recorded codex requests R/C/T) fails and passes exactly at the commit that fixed+each shape; `main` still fails T (`tool_search_*`, codex 0.155.1) — the open ticket for the agent.++---++## 6. Cost / accounts (honest)++- **Watchdog:** $0 (opencode free model, verified).+- **Real matrix** (`subscription -> grid -> back`): each grid ping costs real tokens against the+ codex/claude subscription + grid model. Small per run, but real. **Accepted** per earlier+ direction; the offline driver spec (`gridSwitchDriver.spec.ts`) remains the $0 CI path.+- **Accounts needed:** codex (present ✅), claude (login pending ❌ -> notify), grid sign-in (pending ❌ -> notify).++---++## 7. Machine state today (verified)++| Item | Status |+|---|---|+| codex binary + auth | ✅ `~/.codex/auth.json` present |+| claude binary | ✅ present; login ❌ (no `~/.claude/.credentials.json`) -> NOTIFY |+| tmux / sqlite3 / zsh | ✅ present |+| opencode binary + free models | ✅ `opencode/big-pickle` verified cost 0 (reviewer only) |+| grid sign-in | ❓ not confirmed -> check+notify |++---++## 8. Operational loop (manual once, then fully automated)++**Manual once (user):** install harness + codex + claude on the server; sign in+codex / claude / grid. This is the ONLY manual step.++**Then automated (no manual action):**+1. boot harness daemon if not running+2. detect current versions (codex/claude/grid) from `--version`/`cli_version`+3. update to latest when a new version is acknowledged+4. run the matrix E2E per version: subscription -> switch grid -> real ping -> verify -> back home+ (session naming for auditability)+5. missing login / not signed in -> NOTIFY, no hard fail+6. watchdog (opencode `e2e-watchdog` agent + skill, free model) reads the trace -> what stuck, reproduced, PASS/FAIL + notice++---++## 9. Build order (pending env choice A/B/C)++1. **argv-package** — expose the exact grid/back-home command builder the matrix reuses.+2. **real-matrix script** — driver + switch + ping + verify + notify, with session naming.+3. **watchdog** — opencode `e2e-watchdog` agent + skill, smoke checks per leg, pane probe. ✔+4. CI/scheduling wrapper for when a new version is acknowledged (per-version, not blanket).++**BLOCKED ON:** env choice (A/B/C) + confirming grid sign-in + claude login (or notify-only).