Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

test(openai-codex): add a single-call delta scenario to the bench

Jack Rudenko8c46f034b369
diff --git a/ai-docs/benches/codex-code-mode/README.md b/ai-docs/benches/codex-code-mode/README.md
index 44381e7e..2cb86785 100644
--- a/ai-docs/benches/codex-code-mode/README.md
+++ b/ai-docs/benches/codex-code-mode/README.md
@@ -32,6 +32,7 @@ credential.
| `classic` | off (`CLAUDISH_CODE_MODE=off`) | Phase 7's parallel-Read prompt; the answer is 6 | answer 6; three Reads reached Claude Code with `file_path` ending in `a.ts`, `b.ts`, `c.ts`; no tool_result carries `InputValidationError` or a missing `file_path`; the log says `[CodeMode] off (` |
| `auto` | default (`auto`) | the same | answer 6; the same argument Checks; the log says `[CodeMode] on (<shape>, discovery: …)`; **every relayed exec resumed** and none took the lost-state path |
| `auto-edit` | default (`auto`) | Phase 7 V4: create `notes.txt` containing `alpha`, run `wc -c notes.txt` through Bash, report the byte count | `notes.txt` contains `alpha`; the number `wc` printed is in the answer; Bash was used with no input error; code mode on; every relayed exec resumed |
+| `single-call` | off (`CLAUDISH_CODE_MODE=off`) | run `cat token.txt` as one Bash call on its own, report the line | the answer carries `kestrel-4182` (only in `token.txt`); Bash ran with `command: "cat token.txt"` and its tool_result carries the token; no tool_result carries an input error; in the log that call was **alone in its response, streamed >0 argument deltas that add up to the `.done` arguments, and claudish emitted nothing from `.done` for it** (`single_call_streamed`); the log says `[CodeMode] off (` |
Every Scenario also carries:
@@ -49,7 +50,34 @@ a failure.
The testdata (`a.ts`, `b.ts`, `c.ts`) is copied from the Phase 7 run directories
(`it2/V2a/p7i2-V2a-304-1`, `it2/V4/p7i2-V4-304-1`). `notes.txt` is absent, so a session that does
-nothing fails `auto-edit`.
+nothing fails `auto-edit`. `testdata/single-call/token.txt` holds a token the prompt never names.
+
+## `single-call`: the streamed case (replaces live check V3)
+
+`classic` covers a call Codex sends with NO argument deltas. `single-call` covers the other half
+of `emitUnforwardedArguments` (`handlers/shared/stream-parsers/responses/responses-leg.ts`): a
+lone function call, whose arguments Codex DOES stream as `response.function_call_arguments.delta`
+events. claudish must forward those deltas and send nothing more; sending the `.done` arguments
+as well would hand Claude Code `{…}{…}`, which is not valid JSON. On the subscription this was
+measured before the Scenario existed: in the first full run a lone `Bash pwd` streamed 30 deltas
+and the log has no `zero-delta function_call Bash` line.
+
+`checks/single-call-streamed.ts` (a `ts` Check) reads `logs/claudish_*.log` from the Session's
+`work_dir`, parses the `[SSE:responses]` lines, and finds every `Bash` function_call whose
+`.done` arguments carry `command: "cat token.txt"`. Each must be alone in its
+`response.completed`, carry more than zero deltas, and its deltas concatenated must equal the
+`.done` arguments. The log must also carry no `function_call Bash: emitted N-byte suffix`, no
+`done arguments diverge … for Bash`, and no more `zero-delta function_call Bash` lines than there
+are Bash calls that really streamed zero deltas. The Session half (`args_intact`) proves Claude
+Code ran the call with those arguments and got the token back with no input error.
+
+**How it was proven able to fail.** `madbench check` fails all six of its Checks. The Check's
+pure `analyse` was also run against real logs from the first full run
+(`ai-docs/sessions/codex-code-mode-bench-20261020-172800-a1c3/single-call/prove-check.ts`,
+gitignored): it passes the lone streamed `Bash pwd` (30 deltas), and fails the same log's three
+parallel zero-delta Reads (`streamed 0 argument deltas`, `shared its response with 3
+function_call(s)`), a command that never ran, the log with its delta events removed, and the log
+with claudish's zero-delta line attributed to the streamed Bash call.
## Running it
@@ -65,7 +93,7 @@ The free gates, from this folder:
```bash
madbench list madbench.yaml
madbench preflight madbench.yaml
-madbench check madbench.yaml # negative control: all 19 Checks must fail
+madbench check madbench.yaml # negative control: all 25 Checks must fail
madbench madbench.yaml --harness mock # offline dry run
```
@@ -95,10 +123,11 @@ Why each setting:
own path, writes `claudish-provenance.txt`, and adds `--debug-claudish --log-level debug -q`.
The debug log in `logs/claudish_*.log` is what the `exec` Checks read; `-q` returns before the
update prompt, which would otherwise wait on a TTY nobody answers.
-- **`CLAUDISH_CODE_MODE=off`** on `classic` is claudish's documented equivalent of
- `--code-mode off` (`code-mode/policy.ts`: flag > env > project > global > `auto`). Nothing
- passes the flag, and `harness.args` belongs to the whole Bench, so the environment is how one
- Scenario turns code mode off.
+- **`CLAUDISH_CODE_MODE=off`** on `classic` and `single-call` is claudish's documented
+ equivalent of `--code-mode off` (`code-mode/policy.ts`: flag > env > project > global >
+ `auto`). Nothing passes the flag, and `harness.args` belongs to the whole Bench (madbench
+ 0.47.0 refuses `harness:` on a Scenario), so the environment is how one Scenario turns code
+ mode off. A madbench feature request for per-Scenario harness args is drafted, not yet filed.
- **`driver: false`**: every prompt is an order, so a question from the agent fails the Scenario
by name instead of being answered by a model and becoming a variable.
@@ -122,7 +151,9 @@ jq -r '.results[] | "\(.scenario_id) \(.status)", (.checks[] | select(.result.pa
| `subscription` | a call went somewhere other than `chatgpt.com/backend-api/codex`, or another handler served the request. **That is metered spend or a fallback, never a pass** | the `uniq -c` table in the reason |
| `provenance` | the shim did not run, so the build under test is unknown | `harness.binary`; `bun run build` |
| `bash_output_reported` | `wc` ran but its number is not in the answer, or Bash never ran | the Session's Bash `tool_result` |
-| `answer` | the final answer has no 6 | read the other Checks first: on `auto` a right answer with a failed `resumed` is D1 |
+| `single_call_streamed` with `streamed 0 argument deltas` or `shared its response` | the model put the call in a multi-call turn, or Codex stopped streaming deltas for lone calls: this sample did not exercise the streamed case | the Check's `calls_of_tool` evidence; re-run |
+| `single_call_streamed` with a `zero-delta`, `suffix` or `diverge` line | **claudish emitted arguments from `.done` on top of streamed deltas**: the duplicate-arguments defect | `emitUnforwardedArguments`; `args_intact` should show the input error |
+| `answer` | the final answer has no 6 (`kestrel-4182` on `single-call`) | read the other Checks first: on `auto` a right answer with a failed `resumed` is D1 |
An exec Check's reason ends with the tail of its own output on failure (the matching log lines and
the `relayed= resumed= lost=` tally), so most failures can be read straight from the report.
diff --git a/ai-docs/benches/codex-code-mode/checks/single-call-streamed.ts b/ai-docs/benches/codex-code-mode/checks/single-call-streamed.ts
new file mode 100644
index 00000000..8794c78d
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/checks/single-call-streamed.ts
@@ -0,0 +1,169 @@
+// single-call: when Codex DOES stream argument deltas for a function call,
+// claudish forwards those deltas and nothing else. The zero-delta fix
+// (responses-leg.ts, emitUnforwardedArguments) sends the `.done` arguments
+// only when no delta was forwarded, or the missing suffix when the deltas fell
+// short. Sending them on top of streamed deltas would hand Claude Code the
+// arguments twice (`{…}{…}`), which is not valid JSON.
+//
+// Reads claudish's debug log (logs/claudish_*.log in the workspace) and finds
+// every function_call named `tool` whose final arguments carry every key of
+// `args` with exactly that value. Each must be:
+// - alone: its response.completed lists exactly one function_call;
+// - streamed: more than zero response.function_call_arguments.delta events;
+// - complete: the deltas concatenated equal the `.done` arguments, so there
+// was no suffix left for claudish to add.
+// And the log must carry no claudish line saying it emitted arguments for a
+// streamed call of that tool: no `function_call <tool>: emitted N-byte
+// suffix`, no `done arguments diverge … for <tool>`, and no more
+// `zero-delta function_call <tool>` lines than there are calls of that tool
+// that really streamed zero deltas.
+//
+// The pure part is `analyse`, exported so it can be run against stored real
+// logs (README.md, "How single-call was proven").
+import { readdirSync, readFileSync } from "node:fs";
+import { join } from "node:path";
+
+type Item = { id: string; name: string; deltas: number; streamed: string; done?: string; fnInResponse?: number };
+
+export type Analysis = { pass: boolean; reason: string; evidence: Record<string, unknown> };
+
+const SSE = /^\[[^\]]*\] \[SSE:responses\] (\{.*\})\s*$/;
+
+function parsed(args: string | undefined): Record<string, unknown> | undefined {
+ if (!args) return undefined;
+ try {
+ const v = JSON.parse(args);
+ return v && typeof v === "object" ? (v as Record<string, unknown>) : undefined;
+ } catch {
+ return undefined;
+ }
+}
+
+/** Every key of `want` is present in the call's final arguments with exactly that value. */
+function matches(args: string | undefined, want: Record<string, unknown>): boolean {
+ const got = parsed(args);
+ return !!got && Object.entries(want).every(([k, v]) => got[k] === v);
+}
+
+export function analyse(log: string, tool: string, want: Record<string, unknown>): Analysis {
+ const items = new Map<string, Item>();
+ let zeroDeltaLines = 0;
+ let suffixLines = 0;
+ let divergeLines = 0;
+ const t = tool.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+ const zeroRe = new RegExp(`\\[ResponsesSSE\\] zero-delta function_call ${t}:`);
+ const suffixRe = new RegExp(`\\[ResponsesSSE\\] function_call ${t}: emitted \\d+-byte suffix`);
+ const divergeRe = new RegExp(`\\[ResponsesSSE\\] done arguments diverge from streamed deltas for ${t}\\b`);
+
+ for (const line of log.split("\n")) {
+ if (zeroRe.test(line)) zeroDeltaLines++;
+ if (suffixRe.test(line)) suffixLines++;
+ if (divergeRe.test(line)) divergeLines++;
+ const m = SSE.exec(line);
+ if (!m) continue;
+ let ev: any;
+ try {
+ ev = JSON.parse(m[1]);
+ } catch {
+ continue; // a line cut by the logger carries nothing we can trust
+ }
+ switch (ev.type) {
+ case "response.output_item.added":
+ if (ev.item?.type === "function_call" && typeof ev.item.id === "string") {
+ items.set(ev.item.id, { id: ev.item.id, name: ev.item.name, deltas: 0, streamed: "" });
+ }
+ break;
+ case "response.function_call_arguments.delta": {
+ const it = items.get(ev.item_id);
+ if (it) {
+ it.deltas++;
+ it.streamed += ev.delta ?? "";
+ }
+ break;
+ }
+ case "response.function_call_arguments.done": {
+ const it = items.get(ev.item_id);
+ if (it) it.done = ev.arguments ?? "";
+ break;
+ }
+ case "response.completed": {
+ const out: any[] = Array.isArray(ev.response?.output) ? ev.response.output : [];
+ const fns = out.filter((o) => o?.type === "function_call");
+ for (const f of fns) {
+ const it = items.get(f.id);
+ if (!it) continue;
+ it.fnInResponse = fns.length;
+ if (it.done === undefined && typeof f.arguments === "string") it.done = f.arguments;
+ }
+ break;
+ }
+ }
+ }
+
+ const ofTool = [...items.values()].filter((i) => i.name === tool);
+ const targets = ofTool.filter((i) => matches(i.done, want));
+ const zeroDeltaCalls = ofTool.filter((i) => i.deltas === 0 && (i.done ?? "") !== "").length;
+ const rows = targets.map((x) => ({
+ id: x.id,
+ deltas: x.deltas,
+ alone: x.fnInResponse === 1,
+ function_calls_in_response: x.fnInResponse ?? null,
+ streamed_equals_done: x.streamed === x.done,
+ done_bytes: (x.done ?? "").length,
+ }));
+ const evidence = {
+ tool,
+ args: want,
+ target_calls: rows,
+ calls_of_tool: ofTool.map((i) => ({ id: i.id, deltas: i.deltas, args: parsed(i.done) ?? i.done ?? null })),
+ zero_delta_lines: zeroDeltaLines,
+ zero_delta_calls: zeroDeltaCalls,
+ suffix_lines: suffixLines,
+ diverge_lines: divergeLines,
+ };
+
+ const problems: string[] = [];
+ if (targets.length === 0) {
+ problems.push(`no ${tool} function_call with arguments ${JSON.stringify(want)} in the log (${ofTool.length} ${tool} call(s) seen)`);
+ }
+ for (const r of rows) {
+ if (!r.alone) problems.push(`${r.id} shared its response with ${r.function_calls_in_response ?? "an unknown number of"} function_call(s)`);
+ if (r.deltas === 0) problems.push(`${r.id} streamed 0 argument deltas (a zero-delta call, not the streamed case)`);
+ else if (!r.streamed_equals_done) problems.push(`${r.id}: its ${r.deltas} deltas do not add up to the .done arguments`);
+ }
+ if (zeroDeltaLines > zeroDeltaCalls) {
+ problems.push(
+ `${zeroDeltaLines} 'zero-delta function_call ${tool}' line(s) for ${zeroDeltaCalls} zero-delta call(s): claudish emitted .done arguments for a streamed call`,
+ );
+ }
+ if (suffixLines > 0) problems.push(`${suffixLines} 'emitted N-byte suffix' line(s) for ${tool}`);
+ if (divergeLines > 0) problems.push(`${divergeLines} 'done arguments diverge' line(s) for ${tool}`);
+
+ if (problems.length > 0) return { pass: false, reason: problems.join("; "), evidence };
+ return {
+ pass: true,
+ reason: `${rows.length} lone ${tool} call(s) with ${JSON.stringify(want)} streamed (${rows.map((r) => `${r.deltas} deltas`).join(", ")}); deltas == .done; no claudish emission from .done`,
+ evidence,
+ };
+}
+
+export default function evaluate(
+ _output: string,
+ context: { config?: { tool?: string; args?: Record<string, unknown> }; session?: { work_dir?: string } },
+) {
+ const tool = context.config?.tool ?? "Bash";
+ const want = context.config?.args;
+ if (!want || Object.keys(want).length === 0) throw new Error("single-call-streamed: config.args is required");
+ const wd = context.session?.work_dir;
+ if (!wd) return { pass: false, score: 0, reason: "the Session carries no work_dir, so the claudish log cannot be read" };
+ let names: string[] = [];
+ try {
+ names = readdirSync(join(wd, "logs")).filter((n) => /^claudish_.*\.log$/.test(n));
+ } catch {
+ // no logs directory: claudish never ran in this workspace
+ }
+ if (names.length === 0) return { pass: false, score: 0, reason: `no logs/claudish_*.log under ${wd}` };
+ const log = names.map((n) => readFileSync(join(wd, "logs", n), "utf8")).join("\n");
+ const a = analyse(log, tool, want);
+ return { pass: a.pass, score: a.pass ? 1 : 0, reason: a.reason, evidence: { ...a.evidence, logs: names } };
+}
diff --git a/ai-docs/benches/codex-code-mode/madbench.yaml b/ai-docs/benches/codex-code-mode/madbench.yaml
index 831da5be..5b4b2421 100644
--- a/ai-docs/benches/codex-code-mode/madbench.yaml
+++ b/ai-docs/benches/codex-code-mode/madbench.yaml
@@ -65,7 +65,8 @@ scenarios:
regex: true
event_type: tool_result
eq: 0
- - type: exec
+ - &code_mode_off
+ type: exec
metric: code_mode_off
cmd: 'L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" && grep -h "\[CodeMode\] off (" $L && ! grep -q "\[CodeMode\] on (" $L'
# Measures whether this session hit the defect's trigger at all: Codex
@@ -171,3 +172,53 @@ scenarios:
- *resumed
- *subscription
- *provenance
+
+ # ── single-call: one classic tool call whose arguments Codex DOES stream ───
+ # Replaces the hand-run live check V3 on the subscription. A lone function
+ # call streams its arguments as function_call_arguments.delta events
+ # (measured: the lone `Bash pwd` in the first full run streamed 30 deltas,
+ # and claudish logged no zero-delta line for it). claudish must forward
+ # those deltas and NOT also emit the .done arguments, or Claude Code gets
+ # the arguments twice and the tool input is not valid JSON.
+ - name: single-call
+ description: "code mode off: one lone Bash call streams its argument deltas; claudish forwards them once"
+ testdata: ./testdata/single-call
+ # Same mechanism as classic: harness.args is per Bench.
+ sandbox:
+ level: workspace
+ env:
+ CLAUDISH_CODE_MODE: "off"
+ prompt: "Use the Bash tool to run exactly this command: cat token.txt — make it a single Bash call on its own, with no other tool call before or after it. Then reply with the line it printed."
+ checks:
+ # The token is only in token.txt, never in the prompt, so an answer that
+ # carries it came from the tool's output.
+ - type: regex
+ metric: answer
+ value: '\bkestrel-4182\b'
+ # Claude Code ran the call with the intended arguments, its output came
+ # back, and no tool_result carries an input error (arguments sent twice
+ # are not valid JSON, and come back as one). The absence half sits
+ # inside the set so the set as a whole is a positive Check.
+ - type: assert-set
+ metric: args_intact
+ checks:
+ - { type: session:tool-args-match, value: Bash, tool_args: { command: "cat token.txt" } }
+ - { type: session:match, value: "kestrel-4182", event_type: tool_result }
+ - type: session:match
+ value: "InputValidationError|not valid JSON|invalid tool input"
+ regex: true
+ ignore_case: true
+ event_type: tool_result
+ eq: 0
+ # In claudish's log: that call was alone in its response, streamed >0
+ # argument deltas that add up to the .done arguments, and claudish wrote
+ # no zero-delta / suffix / divergence line for it.
+ - type: ts
+ metric: single_call_streamed
+ value: "file://./checks/single-call-streamed.ts"
+ config:
+ tool: Bash
+ args: { command: "cat token.txt" }
+ - *code_mode_off
+ - *subscription
+ - *provenance
diff --git a/ai-docs/benches/codex-code-mode/testdata/single-call/token.txt b/ai-docs/benches/codex-code-mode/testdata/single-call/token.txt
new file mode 100644
index 00000000..f73add09
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/testdata/single-call/token.txt
@@ -0,0 +1 @@
+kestrel-4182