Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
test(openai-codex): add a single-call delta scenario to the bench
Jack Rudenko8c46f034b369
diff --git a/ai-docs/benches/codex-code-mode/README.md b/ai-docs/benches/codex-code-mode/README.mdindex 44381e7e..2cb86785 100644--- a/ai-docs/benches/codex-code-mode/README.md+++ b/ai-docs/benches/codex-code-mode/README.md@@ -32,6 +32,7 @@ credential.| `classic` | off (`CLAUDISH_CODE_MODE=off`) | Phase 7's parallel-Read prompt; the answer is 6 | answer 6; three Reads reached Claude Code with `file_path` ending in `a.ts`, `b.ts`, `c.ts`; no tool_result carries `InputValidationError` or a missing `file_path`; the log says `[CodeMode] off (` || `auto` | default (`auto`) | the same | answer 6; the same argument Checks; the log says `[CodeMode] on (<shape>, discovery: …)`; **every relayed exec resumed** and none took the lost-state path || `auto-edit` | default (`auto`) | Phase 7 V4: create `notes.txt` containing `alpha`, run `wc -c notes.txt` through Bash, report the byte count | `notes.txt` contains `alpha`; the number `wc` printed is in the answer; Bash was used with no input error; code mode on; every relayed exec resumed |+| `single-call` | off (`CLAUDISH_CODE_MODE=off`) | run `cat token.txt` as one Bash call on its own, report the line | the answer carries `kestrel-4182` (only in `token.txt`); Bash ran with `command: "cat token.txt"` and its tool_result carries the token; no tool_result carries an input error; in the log that call was **alone in its response, streamed >0 argument deltas that add up to the `.done` arguments, and claudish emitted nothing from `.done` for it** (`single_call_streamed`); the log says `[CodeMode] off (` |Every Scenario also carries:@@ -49,7 +50,34 @@ a failure.The testdata (`a.ts`, `b.ts`, `c.ts`) is copied from the Phase 7 run directories(`it2/V2a/p7i2-V2a-304-1`, `it2/V4/p7i2-V4-304-1`). `notes.txt` is absent, so a session that does-nothing fails `auto-edit`.+nothing fails `auto-edit`. `testdata/single-call/token.txt` holds a token the prompt never names.++## `single-call`: the streamed case (replaces live check V3)++`classic` covers a call Codex sends with NO argument deltas. `single-call` covers the other half+of `emitUnforwardedArguments` (`handlers/shared/stream-parsers/responses/responses-leg.ts`): a+lone function call, whose arguments Codex DOES stream as `response.function_call_arguments.delta`+events. claudish must forward those deltas and send nothing more; sending the `.done` arguments+as well would hand Claude Code `{…}{…}`, which is not valid JSON. On the subscription this was+measured before the Scenario existed: in the first full run a lone `Bash pwd` streamed 30 deltas+and the log has no `zero-delta function_call Bash` line.++`checks/single-call-streamed.ts` (a `ts` Check) reads `logs/claudish_*.log` from the Session's+`work_dir`, parses the `[SSE:responses]` lines, and finds every `Bash` function_call whose+`.done` arguments carry `command: "cat token.txt"`. Each must be alone in its+`response.completed`, carry more than zero deltas, and its deltas concatenated must equal the+`.done` arguments. The log must also carry no `function_call Bash: emitted N-byte suffix`, no+`done arguments diverge … for Bash`, and no more `zero-delta function_call Bash` lines than there+are Bash calls that really streamed zero deltas. The Session half (`args_intact`) proves Claude+Code ran the call with those arguments and got the token back with no input error.++**How it was proven able to fail.** `madbench check` fails all six of its Checks. The Check's+pure `analyse` was also run against real logs from the first full run+(`ai-docs/sessions/codex-code-mode-bench-20261020-172800-a1c3/single-call/prove-check.ts`,+gitignored): it passes the lone streamed `Bash pwd` (30 deltas), and fails the same log's three+parallel zero-delta Reads (`streamed 0 argument deltas`, `shared its response with 3+function_call(s)`), a command that never ran, the log with its delta events removed, and the log+with claudish's zero-delta line attributed to the streamed Bash call.## Running it@@ -65,7 +93,7 @@ The free gates, from this folder:```bashmadbench list madbench.yamlmadbench preflight madbench.yaml-madbench check madbench.yaml # negative control: all 19 Checks must fail+madbench check madbench.yaml # negative control: all 25 Checks must failmadbench madbench.yaml --harness mock # offline dry run```@@ -95,10 +123,11 @@ Why each setting:own path, writes `claudish-provenance.txt`, and adds `--debug-claudish --log-level debug -q`.The debug log in `logs/claudish_*.log` is what the `exec` Checks read; `-q` returns before theupdate prompt, which would otherwise wait on a TTY nobody answers.-- **`CLAUDISH_CODE_MODE=off`** on `classic` is claudish's documented equivalent of- `--code-mode off` (`code-mode/policy.ts`: flag > env > project > global > `auto`). Nothing- passes the flag, and `harness.args` belongs to the whole Bench, so the environment is how one- Scenario turns code mode off.+- **`CLAUDISH_CODE_MODE=off`** on `classic` and `single-call` is claudish's documented+ equivalent of `--code-mode off` (`code-mode/policy.ts`: flag > env > project > global >+ `auto`). Nothing passes the flag, and `harness.args` belongs to the whole Bench (madbench+ 0.47.0 refuses `harness:` on a Scenario), so the environment is how one Scenario turns code+ mode off. A madbench feature request for per-Scenario harness args is drafted, not yet filed.- **`driver: false`**: every prompt is an order, so a question from the agent fails the Scenarioby name instead of being answered by a model and becoming a variable.@@ -122,7 +151,9 @@ jq -r '.results[] | "\(.scenario_id) \(.status)", (.checks[] | select(.result.pa| `subscription` | a call went somewhere other than `chatgpt.com/backend-api/codex`, or another handler served the request. **That is metered spend or a fallback, never a pass** | the `uniq -c` table in the reason || `provenance` | the shim did not run, so the build under test is unknown | `harness.binary`; `bun run build` || `bash_output_reported` | `wc` ran but its number is not in the answer, or Bash never ran | the Session's Bash `tool_result` |-| `answer` | the final answer has no 6 | read the other Checks first: on `auto` a right answer with a failed `resumed` is D1 |+| `single_call_streamed` with `streamed 0 argument deltas` or `shared its response` | the model put the call in a multi-call turn, or Codex stopped streaming deltas for lone calls: this sample did not exercise the streamed case | the Check's `calls_of_tool` evidence; re-run |+| `single_call_streamed` with a `zero-delta`, `suffix` or `diverge` line | **claudish emitted arguments from `.done` on top of streamed deltas**: the duplicate-arguments defect | `emitUnforwardedArguments`; `args_intact` should show the input error |+| `answer` | the final answer has no 6 (`kestrel-4182` on `single-call`) | read the other Checks first: on `auto` a right answer with a failed `resumed` is D1 |An exec Check's reason ends with the tail of its own output on failure (the matching log lines andthe `relayed= resumed= lost=` tally), so most failures can be read straight from the report.diff --git a/ai-docs/benches/codex-code-mode/checks/single-call-streamed.ts b/ai-docs/benches/codex-code-mode/checks/single-call-streamed.tsnew file mode 100644index 00000000..8794c78d--- /dev/null+++ b/ai-docs/benches/codex-code-mode/checks/single-call-streamed.ts@@ -0,0 +1,169 @@+// single-call: when Codex DOES stream argument deltas for a function call,+// claudish forwards those deltas and nothing else. The zero-delta fix+// (responses-leg.ts, emitUnforwardedArguments) sends the `.done` arguments+// only when no delta was forwarded, or the missing suffix when the deltas fell+// short. Sending them on top of streamed deltas would hand Claude Code the+// arguments twice (`{…}{…}`), which is not valid JSON.+//+// Reads claudish's debug log (logs/claudish_*.log in the workspace) and finds+// every function_call named `tool` whose final arguments carry every key of+// `args` with exactly that value. Each must be:+// - alone: its response.completed lists exactly one function_call;+// - streamed: more than zero response.function_call_arguments.delta events;+// - complete: the deltas concatenated equal the `.done` arguments, so there+// was no suffix left for claudish to add.+// And the log must carry no claudish line saying it emitted arguments for a+// streamed call of that tool: no `function_call <tool>: emitted N-byte+// suffix`, no `done arguments diverge … for <tool>`, and no more+// `zero-delta function_call <tool>` lines than there are calls of that tool+// that really streamed zero deltas.+//+// The pure part is `analyse`, exported so it can be run against stored real+// logs (README.md, "How single-call was proven").+import { readdirSync, readFileSync } from "node:fs";+import { join } from "node:path";++type Item = { id: string; name: string; deltas: number; streamed: string; done?: string; fnInResponse?: number };++export type Analysis = { pass: boolean; reason: string; evidence: Record<string, unknown> };++const SSE = /^\[[^\]]*\] \[SSE:responses\] (\{.*\})\s*$/;++function parsed(args: string | undefined): Record<string, unknown> | undefined {+ if (!args) return undefined;+ try {+ const v = JSON.parse(args);+ return v && typeof v === "object" ? (v as Record<string, unknown>) : undefined;+ } catch {+ return undefined;+ }+}++/** Every key of `want` is present in the call's final arguments with exactly that value. */+function matches(args: string | undefined, want: Record<string, unknown>): boolean {+ const got = parsed(args);+ return !!got && Object.entries(want).every(([k, v]) => got[k] === v);+}++export function analyse(log: string, tool: string, want: Record<string, unknown>): Analysis {+ const items = new Map<string, Item>();+ let zeroDeltaLines = 0;+ let suffixLines = 0;+ let divergeLines = 0;+ const t = tool.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");+ const zeroRe = new RegExp(`\\[ResponsesSSE\\] zero-delta function_call ${t}:`);+ const suffixRe = new RegExp(`\\[ResponsesSSE\\] function_call ${t}: emitted \\d+-byte suffix`);+ const divergeRe = new RegExp(`\\[ResponsesSSE\\] done arguments diverge from streamed deltas for ${t}\\b`);++ for (const line of log.split("\n")) {+ if (zeroRe.test(line)) zeroDeltaLines++;+ if (suffixRe.test(line)) suffixLines++;+ if (divergeRe.test(line)) divergeLines++;+ const m = SSE.exec(line);+ if (!m) continue;+ let ev: any;+ try {+ ev = JSON.parse(m[1]);+ } catch {+ continue; // a line cut by the logger carries nothing we can trust+ }+ switch (ev.type) {+ case "response.output_item.added":+ if (ev.item?.type === "function_call" && typeof ev.item.id === "string") {+ items.set(ev.item.id, { id: ev.item.id, name: ev.item.name, deltas: 0, streamed: "" });+ }+ break;+ case "response.function_call_arguments.delta": {+ const it = items.get(ev.item_id);+ if (it) {+ it.deltas++;+ it.streamed += ev.delta ?? "";+ }+ break;+ }+ case "response.function_call_arguments.done": {+ const it = items.get(ev.item_id);+ if (it) it.done = ev.arguments ?? "";+ break;+ }+ case "response.completed": {+ const out: any[] = Array.isArray(ev.response?.output) ? ev.response.output : [];+ const fns = out.filter((o) => o?.type === "function_call");+ for (const f of fns) {+ const it = items.get(f.id);+ if (!it) continue;+ it.fnInResponse = fns.length;+ if (it.done === undefined && typeof f.arguments === "string") it.done = f.arguments;+ }+ break;+ }+ }+ }++ const ofTool = [...items.values()].filter((i) => i.name === tool);+ const targets = ofTool.filter((i) => matches(i.done, want));+ const zeroDeltaCalls = ofTool.filter((i) => i.deltas === 0 && (i.done ?? "") !== "").length;+ const rows = targets.map((x) => ({+ id: x.id,+ deltas: x.deltas,+ alone: x.fnInResponse === 1,+ function_calls_in_response: x.fnInResponse ?? null,+ streamed_equals_done: x.streamed === x.done,+ done_bytes: (x.done ?? "").length,+ }));+ const evidence = {+ tool,+ args: want,+ target_calls: rows,+ calls_of_tool: ofTool.map((i) => ({ id: i.id, deltas: i.deltas, args: parsed(i.done) ?? i.done ?? null })),+ zero_delta_lines: zeroDeltaLines,+ zero_delta_calls: zeroDeltaCalls,+ suffix_lines: suffixLines,+ diverge_lines: divergeLines,+ };++ const problems: string[] = [];+ if (targets.length === 0) {+ problems.push(`no ${tool} function_call with arguments ${JSON.stringify(want)} in the log (${ofTool.length} ${tool} call(s) seen)`);+ }+ for (const r of rows) {+ if (!r.alone) problems.push(`${r.id} shared its response with ${r.function_calls_in_response ?? "an unknown number of"} function_call(s)`);+ if (r.deltas === 0) problems.push(`${r.id} streamed 0 argument deltas (a zero-delta call, not the streamed case)`);+ else if (!r.streamed_equals_done) problems.push(`${r.id}: its ${r.deltas} deltas do not add up to the .done arguments`);+ }+ if (zeroDeltaLines > zeroDeltaCalls) {+ problems.push(+ `${zeroDeltaLines} 'zero-delta function_call ${tool}' line(s) for ${zeroDeltaCalls} zero-delta call(s): claudish emitted .done arguments for a streamed call`,+ );+ }+ if (suffixLines > 0) problems.push(`${suffixLines} 'emitted N-byte suffix' line(s) for ${tool}`);+ if (divergeLines > 0) problems.push(`${divergeLines} 'done arguments diverge' line(s) for ${tool}`);++ if (problems.length > 0) return { pass: false, reason: problems.join("; "), evidence };+ return {+ pass: true,+ reason: `${rows.length} lone ${tool} call(s) with ${JSON.stringify(want)} streamed (${rows.map((r) => `${r.deltas} deltas`).join(", ")}); deltas == .done; no claudish emission from .done`,+ evidence,+ };+}++export default function evaluate(+ _output: string,+ context: { config?: { tool?: string; args?: Record<string, unknown> }; session?: { work_dir?: string } },+) {+ const tool = context.config?.tool ?? "Bash";+ const want = context.config?.args;+ if (!want || Object.keys(want).length === 0) throw new Error("single-call-streamed: config.args is required");+ const wd = context.session?.work_dir;+ if (!wd) return { pass: false, score: 0, reason: "the Session carries no work_dir, so the claudish log cannot be read" };+ let names: string[] = [];+ try {+ names = readdirSync(join(wd, "logs")).filter((n) => /^claudish_.*\.log$/.test(n));+ } catch {+ // no logs directory: claudish never ran in this workspace+ }+ if (names.length === 0) return { pass: false, score: 0, reason: `no logs/claudish_*.log under ${wd}` };+ const log = names.map((n) => readFileSync(join(wd, "logs", n), "utf8")).join("\n");+ const a = analyse(log, tool, want);+ return { pass: a.pass, score: a.pass ? 1 : 0, reason: a.reason, evidence: { ...a.evidence, logs: names } };+}diff --git a/ai-docs/benches/codex-code-mode/madbench.yaml b/ai-docs/benches/codex-code-mode/madbench.yamlindex 831da5be..5b4b2421 100644--- a/ai-docs/benches/codex-code-mode/madbench.yaml+++ b/ai-docs/benches/codex-code-mode/madbench.yaml@@ -65,7 +65,8 @@ scenarios:regex: trueevent_type: tool_resulteq: 0- - type: exec+ - &code_mode_off+ type: execmetric: code_mode_offcmd: 'L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" && grep -h "\[CodeMode\] off (" $L && ! grep -q "\[CodeMode\] on (" $L'# Measures whether this session hit the defect's trigger at all: Codex@@ -171,3 +172,53 @@ scenarios:- *resumed- *subscription- *provenance++ # ── single-call: one classic tool call whose arguments Codex DOES stream ───+ # Replaces the hand-run live check V3 on the subscription. A lone function+ # call streams its arguments as function_call_arguments.delta events+ # (measured: the lone `Bash pwd` in the first full run streamed 30 deltas,+ # and claudish logged no zero-delta line for it). claudish must forward+ # those deltas and NOT also emit the .done arguments, or Claude Code gets+ # the arguments twice and the tool input is not valid JSON.+ - name: single-call+ description: "code mode off: one lone Bash call streams its argument deltas; claudish forwards them once"+ testdata: ./testdata/single-call+ # Same mechanism as classic: harness.args is per Bench.+ sandbox:+ level: workspace+ env:+ CLAUDISH_CODE_MODE: "off"+ prompt: "Use the Bash tool to run exactly this command: cat token.txt — make it a single Bash call on its own, with no other tool call before or after it. Then reply with the line it printed."+ checks:+ # The token is only in token.txt, never in the prompt, so an answer that+ # carries it came from the tool's output.+ - type: regex+ metric: answer+ value: '\bkestrel-4182\b'+ # Claude Code ran the call with the intended arguments, its output came+ # back, and no tool_result carries an input error (arguments sent twice+ # are not valid JSON, and come back as one). The absence half sits+ # inside the set so the set as a whole is a positive Check.+ - type: assert-set+ metric: args_intact+ checks:+ - { type: session:tool-args-match, value: Bash, tool_args: { command: "cat token.txt" } }+ - { type: session:match, value: "kestrel-4182", event_type: tool_result }+ - type: session:match+ value: "InputValidationError|not valid JSON|invalid tool input"+ regex: true+ ignore_case: true+ event_type: tool_result+ eq: 0+ # In claudish's log: that call was alone in its response, streamed >0+ # argument deltas that add up to the .done arguments, and claudish wrote+ # no zero-delta / suffix / divergence line for it.+ - type: ts+ metric: single_call_streamed+ value: "file://./checks/single-call-streamed.ts"+ config:+ tool: Bash+ args: { command: "cat token.txt" }+ - *code_mode_off+ - *subscription+ - *provenancediff --git a/ai-docs/benches/codex-code-mode/testdata/single-call/token.txt b/ai-docs/benches/codex-code-mode/testdata/single-call/token.txtnew file mode 100644index 00000000..f73add09--- /dev/null+++ b/ai-docs/benches/codex-code-mode/testdata/single-call/token.txt@@ -0,0 +1 @@+kestrel-4182