Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
codex-code-mode: zero-delta tool calls and code-mode resume on cx@
A re-runnable regression check for two shipped changes on the claudish provider
openai-codex (cx@, the ChatGPT Codex subscription, signed with the OAuth credential in
~/.claudish/codex-oauth.json):
- Zero-delta function calls. In a multi-call turn Codex sends no argument deltas, only the
finished arguments in
.done. claudish must still hand Claude Code the full tool input. Before the fix, parallel Reads came back asInputValidationError: ... The required parameter 'file_path' is missing. - Code mode (
--code-mode auto|on|off, defaultauto). The model gets one freeformexectool; claudish runs its JavaScript in a QuickJS sandbox, relays eachtools.X()call to Claude Code as atool_useblock, and resumes the script from thetool_results. Defect D1 (fixed in4cb34e54): Claude Code 2.1.303 appended a trailingrole: "system"message, so no script ever resumed, while the final answers still looked right because the nested results followed claudish's lost-state note. A correct answer proves nothing here; only claudish's debug log shows whether the script resumed.
Design rationale: ai-docs/architecture/code-mode.md and ai-docs/architecture/adapters.md.
The hand-run validation this Bench replaces is in
ai-docs/sessions/dev-feature-codex-tool-calls-20261016-143706-dc2c/validation/ (gitignored; it
does not survive a fresh clone).
Real Claude Code drives this worktree's claudish build against the real Codex backend. There is no mock upstream, and nothing is billed per token: every request must sign with the Codex OAuth credential.
The Scenarios
| Scenario | Code mode | Prompt | Passes when |
|---|---|---|---|
classic |
off (CLAUDISH_CODE_MODE=off) |
Phase 7's parallel-Read prompt; the answer is 6 | answer 6; three Reads reached Claude Code with file_path ending in a.ts, b.ts, c.ts; no tool_result carries InputValidationError or a missing file_path; the log says [CodeMode] off ( |
auto |
default (auto) |
the same | answer 6; the same argument Checks; the log says [CodeMode] on (<shape>, discovery: …); every relayed exec resumed and none took the lost-state path |
auto-edit |
default (auto) |
Phase 7 V4: create notes.txt containing alpha, run wc -c notes.txt through Bash, report the byte count |
notes.txt contains alpha; the number wc printed is in the answer; Bash was used with no input error; code mode on; every relayed exec resumed |
single-call |
off (CLAUDISH_CODE_MODE=off) |
run cat token.txt as one Bash call on its own, report the line |
the answer carries kestrel-4182 (only in token.txt); Bash ran with command: "cat token.txt" and its tool_result carries the token; no tool_result carries an input error; in the log that call was alone in its response, streamed >0 argument deltas that add up to the .done arguments, and claudish emitted nothing from .done for it (single_call_streamed); the log says [CodeMode] off ( |
Every Scenario also carries:
subscription: everyCalling API:line in the log ishttps://chatgpt.com/backend-api/codex/…(only the OAuth credential signs there; the API-key path goes toapi.openai.com), at least one exists, and everyHandler: provider=line isopenai-codex. A metered hop or a fallback to another provider fails it.provenance:claudish-under-testwroteclaudish-provenance.txtnaming thedist/index.jsit ran and itsclaudish version, so the report records which build ran.
classic also has the readout zero_delta_seen: the log has
[ResponsesSSE] zero-delta function_call … emitted N bytes from done with no emitted 0 bytes.
It is a readout because a session whose model happened to stream deltas is a weaker sample, not
a failure.
The testdata (a.ts, b.ts, c.ts) is copied from the Phase 7 run directories
(it2/V2a/p7i2-V2a-304-1, it2/V4/p7i2-V4-304-1). notes.txt is absent, so a session that does
nothing fails auto-edit. testdata/single-call/token.txt holds a token the prompt never names.
single-call: the streamed case (replaces live check V3)
classic covers a call Codex sends with NO argument deltas. single-call covers the other half
of emitUnforwardedArguments (handlers/shared/stream-parsers/responses/responses-leg.ts): a
lone function call, whose arguments Codex DOES stream as response.function_call_arguments.delta
events. claudish must forward those deltas and send nothing more; sending the .done arguments
as well would hand Claude Code {…}{…}, which is not valid JSON. On the subscription this was
measured before the Scenario existed: in the first full run a lone Bash pwd streamed 30 deltas
and the log has no zero-delta function_call Bash line.
checks/single-call-streamed.ts (a ts Check) reads logs/claudish_*.log from the Session's
work_dir, parses the [SSE:responses] lines, and finds every Bash function_call whose
.done arguments carry command: "cat token.txt". Each must be alone in its
response.completed, carry more than zero deltas, and its deltas concatenated must equal the
.done arguments. The log must also carry no function_call Bash: emitted N-byte suffix, no
done arguments diverge … for Bash, and no more zero-delta function_call Bash lines than there
are Bash calls that really streamed zero deltas. The Session half (args_intact) proves Claude
Code ran the call with those arguments and got the token back with no input error.
How it was proven able to fail. madbench check fails all six of its Checks. The Check's
pure analyse was also run against real logs from the first full run
(ai-docs/sessions/codex-code-mode-bench-20261020-172800-a1c3/single-call/prove-check.ts,
gitignored): it passes the lone streamed Bash pwd (30 deltas), and fails the same log's three
parallel zero-delta Reads (streamed 0 argument deltas, shared its response with 3 function_call(s)), a command that never ran, the log with its delta events removed, and the log
with claudish's zero-delta line attributed to the streamed Bash call.
Running it
Build first. The Bench runs packages/cli/dist/index.js from this worktree, never the claudish
on PATH:
bun run build # from the repo root
The free gates, from this folder:
madbench list madbench.yaml
madbench preflight madbench.yaml
madbench check madbench.yaml # negative control: all 25 Checks must fail
madbench madbench.yaml --harness mock # offline dry run
The real run, in a visible terminal (not piped, not backgrounded):
madbench madbench.yaml --concurrency 4 --trials 5 --report-json <out.json>
- Concurrency needs a claudish build with
1066489b(see "Concurrency" below). An older build fails Scenarios that start together withagent never started. --trials 5gives 20 sessions, all on the subscription. Phase 7 used 10/10/3 sessions by hand;--scenario auto --trials 10adds depth where D1 lived.- A graded miss exits 0. Add
--fail-on-failurewhere a miss must turn the job red. madbench grade <out.json> --bench madbench.yamlre-grades a stored report offline. TheexecandtsChecks show their recorded verdicts, because the workspace is gone.
Why each setting:
sandbox: workspace, nothome.homereplacesHOME, which hides~/.claudish/codex-oauth.json;cx@could then only sign with an API key (metered) or not at all.workspacekeeps the realHOME, and that is the subscription.binary: ./claudish-under-test: a shim that derivespackages/cli/dist/index.jsfrom its own path, writesclaudish-provenance.txt, and adds--debug-claudish --log-level debug -q. The debug log inlogs/claudish_*.logis what theexecChecks read;-qreturns before the update prompt, which would otherwise wait on a TTY nobody answers.CLAUDISH_CODE_MODE=offonclassicandsingle-callis claudish's documented equivalent of--code-mode off(code-mode/policy.ts: flag > env > project > global >auto). Nothing passes the flag, andharness.argsbelongs to the whole Bench (madbench 0.47.0 refusesharness:on a Scenario), so the environment is how one Scenario turns code mode off. Per-Scenario harness args are requested upstream as MadAppGang/madbench#141.driver: false: every prompt is an order, so a question from the agent fails the Scenario by name instead of being answered by a model and becoming a variable.
Concurrency
The first smoke run, three Scenarios at once, reported two agent never started: agent did not start within 1m0s with Claude Code ready in both panes. The cause is claudish's, not madbench's:
createTempSettingsFile (packages/cli/src/claude-runner.ts) named its --settings overlay
~/.claudish/settings-<Date.now()>.json before 1066489b. madbench starts concurrent Scenarios in lock-step (two
claudish debug logs opened in the same millisecond), so two or three processes write ONE file and
every Claude Code loads the last writer's statusLine and hooks. madbench waits for a sample in
each run's own statusline.jsonl; the losers' stay empty, and all the samples land in the
winner's. At --concurrency 3 --trials 3, 2 of 3 trials collided this way; one collided winner
then also hung for 300 s, because its sink carried other sessions' samples. Full evidence:
ai-docs/sessions/codex-code-mode-bench-20261020-172800-a1c3/conc/DIAGNOSIS.md (gitignored).
claudish 1066489b fixed it: the overlay is now settings-<pid>-<Date.now()>.json (and the
Windows status script status-<pid>-<Date.now()>.js), so two processes started in the same
millisecond still write two files. The Bench carries no workaround; it needs a build with that
commit.
Reading a failure
Read .summary.errors before .summary.failed: an errored Scenario never graded.
jq -c '.summary' out.json
jq -r '.results[] | "\(.scenario_id) \(.status)", (.checks[] | select(.result.pass==false) | " \(.spec.metric): \(.result.reason)")' out.json
| Failed Check | Means | Look at |
|---|---|---|
resumed with relayed=N resumed=0 |
D1 is back: scripts were relayed and never resumed. The answers may still be right | the inbound request's last message (code-mode/resume.ts, lastMessageResults); Claude Code's version |
resumed with lost>0 |
a script took the lost-state path (script lost, answered as Script failed) |
the log line the reason prints |
resumed with relayed=0 |
code mode was on but the model never called a nested tool through exec | the transcript; code_mode_on |
args_intact |
a Read lost its file_path, or a tool_result carries InputValidationError: the zero-delta defect |
zero_delta_seen, and the [ResponsesSSE] zero-delta lines |
code_mode_on |
auto did not choose code mode from discovery | the [CodeMode] off (<reason>) line; Codex /models ([CodexDiscovery]) |
code_mode_off |
classic ran with code mode on: the environment did not reach claudish |
the Scenario's sandbox.env |
subscription |
a call went somewhere other than chatgpt.com/backend-api/codex, or another handler served the request. That is metered spend or a fallback, never a pass |
the uniq -c table in the reason |
provenance |
the shim did not run, so the build under test is unknown | harness.binary; bun run build |
bash_output_reported |
wc ran but its number is not in the answer, or Bash never ran |
the Session's Bash tool_result |
single_call_streamed with streamed 0 argument deltas or shared its response |
the model put the call in a multi-call turn, or Codex stopped streaming deltas for lone calls: this sample did not exercise the streamed case | the Check's calls_of_tool evidence; re-run |
single_call_streamed with a zero-delta, suffix or diverge line |
claudish emitted arguments from .done on top of streamed deltas: the duplicate-arguments defect |
emitUnforwardedArguments; args_intact should show the input error |
answer |
the final answer has no 6 (kestrel-4182 on single-call) |
read the other Checks first: on auto a right answer with a failed resumed is D1 |
An exec Check's reason ends with the tail of its own output on failure (the matching log lines and
the relayed= resumed= lost= tally), so most failures can be read straight from the report.
Keeping the logs
madbench deletes the workspace after the run, but an exec Check keeps its command's output in the
report's Evidence whether it passes or fails (MadAppGang/madbench#142). The log lines a Check passed
on are therefore in the report itself; read them there.