Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

test(openai-codex): add a madbench bench for zero-delta and code mode

Jack Rudenko5a4f1a519438
diff --git a/ai-docs/benches/codex-code-mode/README.md b/ai-docs/benches/codex-code-mode/README.md
new file mode 100644
index 00000000..44381e7e
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/README.md
@@ -0,0 +1,137 @@
+# codex-code-mode: zero-delta tool calls and code-mode resume on `cx@`
+
+A re-runnable regression check for two shipped changes on the claudish provider
+`openai-codex` (`cx@`, the ChatGPT Codex subscription, signed with the OAuth credential in
+`~/.claudish/codex-oauth.json`):
+
+1. **Zero-delta function calls.** In a multi-call turn Codex sends no argument deltas, only the
+ finished arguments in `.done`. claudish must still hand Claude Code the full tool input.
+ Before the fix, parallel Reads came back as `InputValidationError: ... The required
+ parameter 'file_path' is missing`.
+2. **Code mode** (`--code-mode auto|on|off`, default `auto`). The model gets one freeform `exec`
+ tool; claudish runs its JavaScript in a QuickJS sandbox, relays each `tools.X()` call to
+ Claude Code as a `tool_use` block, and resumes the script from the `tool_result`s. Defect D1
+ (fixed in `4cb34e54`): Claude Code 2.1.303 appended a trailing `role: "system"` message, so
+ no script ever resumed, while the final answers still looked right because the nested results
+ followed claudish's lost-state note. **A correct answer proves nothing here; only claudish's
+ debug log shows whether the script resumed.**
+
+Design rationale: `ai-docs/architecture/code-mode.md` and `ai-docs/architecture/adapters.md`.
+The hand-run validation this Bench replaces is in
+`ai-docs/sessions/dev-feature-codex-tool-calls-20261016-143706-dc2c/validation/` (gitignored; it
+does not survive a fresh clone).
+
+Real Claude Code drives this worktree's claudish build against the real Codex backend. There is
+no mock upstream, and nothing is billed per token: every request must sign with the Codex OAuth
+credential.
+
+## The Scenarios
+
+| Scenario | Code mode | Prompt | Passes when |
+|---|---|---|---|
+| `classic` | off (`CLAUDISH_CODE_MODE=off`) | Phase 7's parallel-Read prompt; the answer is 6 | answer 6; three Reads reached Claude Code with `file_path` ending in `a.ts`, `b.ts`, `c.ts`; no tool_result carries `InputValidationError` or a missing `file_path`; the log says `[CodeMode] off (` |
+| `auto` | default (`auto`) | the same | answer 6; the same argument Checks; the log says `[CodeMode] on (<shape>, discovery: …)`; **every relayed exec resumed** and none took the lost-state path |
+| `auto-edit` | default (`auto`) | Phase 7 V4: create `notes.txt` containing `alpha`, run `wc -c notes.txt` through Bash, report the byte count | `notes.txt` contains `alpha`; the number `wc` printed is in the answer; Bash was used with no input error; code mode on; every relayed exec resumed |
+
+Every Scenario also carries:
+
+- **`subscription`**: every `Calling API:` line in the log is
+ `https://chatgpt.com/backend-api/codex/…` (only the OAuth credential signs there; the API-key
+ path goes to `api.openai.com`), at least one exists, and every `Handler: provider=` line is
+ `openai-codex`. A metered hop or a fallback to another provider fails it.
+- **`provenance`**: `claudish-under-test` wrote `claudish-provenance.txt` naming the
+ `dist/index.js` it ran and its `claudish version`, so the report records which build ran.
+
+`classic` also has the readout `zero_delta_seen`: the log has
+`[ResponsesSSE] zero-delta function_call … emitted N bytes from done` with no `emitted 0 bytes`.
+It is a readout because a session whose model happened to stream deltas is a weaker sample, not
+a failure.
+
+The testdata (`a.ts`, `b.ts`, `c.ts`) is copied from the Phase 7 run directories
+(`it2/V2a/p7i2-V2a-304-1`, `it2/V4/p7i2-V4-304-1`). `notes.txt` is absent, so a session that does
+nothing fails `auto-edit`.
+
+## Running it
+
+Build first. The Bench runs `packages/cli/dist/index.js` from this worktree, never the claudish
+on `PATH`:
+
+```bash
+bun run build # from the repo root
+```
+
+The free gates, from this folder:
+
+```bash
+madbench list madbench.yaml
+madbench preflight madbench.yaml
+madbench check madbench.yaml # negative control: all 19 Checks must fail
+madbench madbench.yaml --harness mock # offline dry run
+```
+
+The real run, in a visible terminal (not piped, not backgrounded):
+
+```bash
+CODEX_CODE_MODE_KEEP_LOGS=<dir> \
+ madbench madbench.yaml --concurrency 1 --trials 5 --report-json <out.json>
+```
+
+- **`--concurrency 1` is required.** On the first real run, with three claudish sessions starting
+ at once, two were reported `agent never started: agent did not start within 1m0s`, although
+ their panes showed Claude Code ready at the prompt. One at a time, all three started. The cause
+ has not been isolated.
+- `--trials 5` gives 15 sessions (about 15 minutes), all on the subscription. Phase 7 used
+ 10/10/3 sessions by hand; `--scenario auto --trials 10` adds depth where D1 lived.
+- A graded miss exits 0. Add `--fail-on-failure` where a miss must turn the job red.
+- `madbench grade <out.json> --bench madbench.yaml` re-grades a stored report offline. The `exec`
+ and `ts` Checks show their recorded verdicts, because the workspace is gone.
+
+Why each setting:
+
+- **`sandbox: workspace`**, not `home`. `home` replaces `HOME`, which hides
+ `~/.claudish/codex-oauth.json`; `cx@` could then only sign with an API key (metered) or not at
+ all. `workspace` keeps the real `HOME`, and that is the subscription.
+- **`binary: ./claudish-under-test`**: a shim that derives `packages/cli/dist/index.js` from its
+ own path, writes `claudish-provenance.txt`, and adds `--debug-claudish --log-level debug -q`.
+ The debug log in `logs/claudish_*.log` is what the `exec` Checks read; `-q` returns before the
+ update prompt, which would otherwise wait on a TTY nobody answers.
+- **`CLAUDISH_CODE_MODE=off`** on `classic` is claudish's documented equivalent of
+ `--code-mode off` (`code-mode/policy.ts`: flag > env > project > global > `auto`). Nothing
+ passes the flag, and `harness.args` belongs to the whole Bench, so the environment is how one
+ Scenario turns code mode off.
+- **`driver: false`**: every prompt is an order, so a question from the agent fails the Scenario
+ by name instead of being answered by a model and becoming a variable.
+
+## Reading a failure
+
+Read `.summary.errors` before `.summary.failed`: an errored Scenario never graded.
+
+```bash
+jq -c '.summary' out.json
+jq -r '.results[] | "\(.scenario_id) \(.status)", (.checks[] | select(.result.pass==false) | " \(.spec.metric): \(.result.reason)")' out.json
+```
+
+| Failed Check | Means | Look at |
+|---|---|---|
+| `resumed` with `relayed=N resumed=0` | **D1 is back**: scripts were relayed and never resumed. The answers may still be right | the inbound request's last message (`code-mode/resume.ts`, `lastMessageResults`); Claude Code's version |
+| `resumed` with `lost>0` | a script took the lost-state path (`script lost`, `answered as Script failed`) | the log line the reason prints |
+| `resumed` with `relayed=0` | code mode was on but the model never called a nested tool through exec | the transcript; `code_mode_on` |
+| `args_intact` | a Read lost its `file_path`, or a tool_result carries `InputValidationError`: the zero-delta defect | `zero_delta_seen`, and the `[ResponsesSSE] zero-delta` lines |
+| `code_mode_on` | auto did not choose code mode from discovery | the `[CodeMode] off (<reason>)` line; Codex `/models` (`[CodexDiscovery]`) |
+| `code_mode_off` | `classic` ran with code mode on: the environment did not reach claudish | the Scenario's `sandbox.env` |
+| `subscription` | a call went somewhere other than `chatgpt.com/backend-api/codex`, or another handler served the request. **That is metered spend or a fallback, never a pass** | the `uniq -c` table in the reason |
+| `provenance` | the shim did not run, so the build under test is unknown | `harness.binary`; `bun run build` |
+| `bash_output_reported` | `wc` ran but its number is not in the answer, or Bash never ran | the Session's Bash `tool_result` |
+| `answer` | the final answer has no 6 | read the other Checks first: on `auto` a right answer with a failed `resumed` is D1 |
+
+An exec Check's reason ends with the tail of its own output on failure (the matching log lines and
+the `relayed= resumed= lost=` tally), so most failures can be read straight from the report.
+
+## Keeping the logs
+
+An `exec` Check keeps its output only on failure, and madbench deletes the workspace after the run,
+so a passing Check keeps none of the log lines it passed on. Until madbench keeps that output in the
+report (feature request drafted, not yet filed), the `provenance` Check carries a TEMPORARY first
+line: with `CODEX_CODE_MODE_KEEP_LOGS=<dir>` set it copies `logs/claudish_*.log` and
+`claudish-provenance.txt` to `<dir>/<workspace name>/`. Match a folder to its Scenario with the
+report's `.results[].session.work_dir`. Delete that line when madbench ships the feature.
diff --git a/ai-docs/benches/codex-code-mode/checks/bash-count-reported.ts b/ai-docs/benches/codex-code-mode/checks/bash-count-reported.ts
new file mode 100644
index 00000000..58bcaa20
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/checks/bash-count-reported.ts
@@ -0,0 +1,43 @@
+// Did the Bash output reach the model? `wc -c notes.txt` prints
+// " 5 notes.txt"; the model can only state that number if claudish
+// relayed the Bash tool_result back into the exec script and on to Codex.
+// Pass when some Bash call's result printed "<n> notes.txt" AND the final
+// answer contains that same <n> as a whole number.
+//
+// Reads every Bash call's output wherever the Session carries it: a call's
+// `result` / `result_tag`, and the matching tool_result Event's `tool_out`.
+// An Event keeps tool output in `tool_out`, not `text`, so a text-only read
+// sees no Bash output at all (measured on the first real run).
+type Call = { name?: string; tool_call_id?: string; result?: unknown; result_tag?: unknown };
+type Event = { type?: string; tool_call_id?: string; tool_out?: unknown };
+
+export default function evaluate(
+ output: string,
+ context: { session?: { calls?: Call[]; events?: Event[] } },
+) {
+ const calls = (context.session?.calls ?? []).filter((c) => c.name === "Bash");
+ const bashIds = new Set(calls.map((c) => c.tool_call_id).filter(Boolean));
+ const outputs: unknown[] = calls.flatMap((c) => [c.result, c.result_tag]);
+ for (const e of context.session?.events ?? []) {
+ if (e.type === "tool_result" && e.tool_call_id && bashIds.has(e.tool_call_id)) outputs.push(e.tool_out);
+ }
+ const counts: string[] = [];
+ for (const o of outputs) {
+ if (typeof o !== "string") continue;
+ const m = /(?:^|\s)(\d+)\s+notes\.txt\b/.exec(o);
+ if (m && !counts.includes(m[1])) counts.push(m[1]);
+ }
+ if (counts.length === 0) {
+ return { pass: false, score: 0, reason: "no Bash result printed '<n> notes.txt' (wc -c never ran or its output was not captured)" };
+ }
+ const reported = counts.find((n) => new RegExp(`\\b${n}\\b`).test(output));
+ const pass = reported !== undefined;
+ return {
+ pass,
+ score: pass ? 1 : 0,
+ reason: pass
+ ? `wc printed ${reported} and the answer states ${reported}`
+ : `wc printed ${counts.join(", ")}; the answer states none of them`,
+ evidence: { wc_counts: counts, final_output: output.slice(0, 300) },
+ };
+}
diff --git a/ai-docs/benches/codex-code-mode/claudish-under-test b/ai-docs/benches/codex-code-mode/claudish-under-test
new file mode 100755
index 00000000..63471b6d
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/claudish-under-test
@@ -0,0 +1,23 @@
+#!/bin/sh
+# harness.binary for this Bench: THIS worktree's built claudish
+# (packages/cli/dist/index.js, the output of `bun run build`), never the
+# claudish on PATH. The path is derived from this file's own location, so a
+# run cannot pick up a global install by accident.
+#
+# Before handing over it writes claudish-provenance.txt into the workspace
+# (the agent's cwd); the `provenance` Check reads it back, so the report
+# records which build ran. It then adds the debug-log flags every Check that
+# reads logs/claudish_*.log depends on, and `-q`, which returns before
+# claudish's update prompt (its "nobody can answer" test is
+# process.stdin.isTTY, which is true under magmux).
+set -eu
+here=$(cd "$(dirname "$0")" && pwd)
+root=$(cd "$here/../../.." && pwd)
+dist="$root/packages/cli/dist/index.js"
+{
+ echo "dist=$dist"
+ echo "version=$(bun "$dist" --version 2>/dev/null | head -n 1)"
+ echo "head=$(git -C "$root" rev-parse HEAD 2>/dev/null || echo unknown)"
+ echo "dist_mtime=$(stat -f %Sm -t %Y-%m-%dT%H:%M:%S "$dist")"
+} > claudish-provenance.txt
+exec bun "$dist" --debug-claudish --log-level debug -q "$@"
diff --git a/ai-docs/benches/codex-code-mode/madbench.yaml b/ai-docs/benches/codex-code-mode/madbench.yaml
new file mode 100644
index 00000000..831da5be
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/madbench.yaml
@@ -0,0 +1,173 @@
+description: "codex-code-mode: cx@ zero-delta tool calls keep their arguments, and code-mode exec scripts resume"
+
+# Real Claude Code -> this worktree's claudish build -> the REAL Codex backend
+# on the ChatGPT subscription (cx@, OAuth credential ~/.claudish/codex-oauth.json).
+# No mock upstream. README.md explains each Check and how to read a failure.
+#
+# Every claim that matters here is in claudish's debug log, not in the answer:
+# D1 shipped with right answers and zero resumed scripts. The `exec` Checks
+# read logs/claudish_*.log, which claudish writes into the workspace because
+# claudish-under-test passes --debug-claudish.
+
+harness:
+ type: claude-code
+ # This worktree's packages/cli/dist/index.js, via a shim that records the
+ # build it ran (claudish-provenance.txt). A path with a `/` resolves against
+ # this file, so the claudish on PATH is never used.
+ binary: ./claudish-under-test
+ # Interactive (the default): Bash parks at an approval menu nobody answers
+ # unless the mode is bypassPermissions. claudish forwards it to claude.
+ args: ["--permission-mode", "bypassPermissions"]
+
+defaults:
+ # workspace, not home: home replaces HOME and hides ~/.claudish, so the
+ # Codex OAuth credential would be gone and cx@ could only sign metered (or
+ # not at all). workspace keeps the real HOME; that is the subscription.
+ sandbox: workspace
+ # Every prompt is an order, so a question is a defect: fail it by name
+ # instead of letting a Driver answer it and become a variable.
+ driver: false
+ # Phase 7 measured 8-20 s per -p session; an interactive cold start adds
+ # Claude Code's boot. Tighten after more runs.
+ timeout: 300s
+
+scenarios:
+ # ── classic: --code-mode off, the Part A regression ─────────────────────────
+ - name: classic
+ description: "code mode off: three parallel Reads arrive with their file_path (zero-delta function calls)"
+ testdata: ./testdata/parallel-read
+ # claudish's documented equivalent of `--code-mode off`
+ # (code-mode/policy.ts: flag > CLAUDISH_CODE_MODE > project > global >
+ # auto). Nothing passes the flag, so this decides it. An env var because
+ # harness.args is per Bench and this Scenario alone runs classic.
+ sandbox:
+ level: workspace
+ env:
+ CLAUDISH_CODE_MODE: "off"
+ prompt: "Use the Read tool to read a.ts, b.ts and c.ts in this directory. Issue all three Read calls in parallel in ONE turn. Then reply with the sum of the three exported numbers."
+ checks:
+ - type: regex
+ metric: answer
+ value: '\b6\b'
+ # The Reads reached Claude Code with their arguments, and none came back
+ # as `InputValidationError: ... The required parameter 'file_path' is
+ # missing` (the pre-fix symptom). The absence half sits inside the set
+ # so the set as a whole is a positive Check.
+ - type: assert-set
+ metric: args_intact
+ checks:
+ - { type: session:tool-args-match, value: Read, tool_args: { file_path: "suffix:a.ts" } }
+ - { type: session:tool-args-match, value: Read, tool_args: { file_path: "suffix:b.ts" } }
+ - { type: session:tool-args-match, value: Read, tool_args: { file_path: "suffix:c.ts" } }
+ - type: session:match
+ value: "InputValidationError|required parameter .file_path. is missing"
+ regex: true
+ event_type: tool_result
+ eq: 0
+ - type: exec
+ metric: code_mode_off
+ cmd: 'L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" && grep -h "\[CodeMode\] off (" $L && ! grep -q "\[CodeMode\] on (" $L'
+ # Measures whether this session hit the defect's trigger at all: Codex
+ # sent a function_call with no argument deltas and claudish emitted the
+ # arguments from `.done`. A readout: a session whose model happened to
+ # stream deltas is not a failure, just a weaker sample.
+ - type: exec
+ metric: zero_delta_seen
+ readout: true
+ cmd: 'L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" && grep -h "zero-delta function_call" $L && ! grep -q "zero-delta function_call.*emitted 0 bytes" $L'
+ - &subscription
+ type: exec
+ metric: subscription
+ # Every upstream call went to chatgpt.com/backend-api/codex (only the
+ # OAuth credential signs there; an API key goes to api.openai.com), and
+ # every handler was openai-codex: no metered hop, no fallback.
+ cmd: 'L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" && grep -h -e "Calling API:" -e "Handler: provider=" -e "\[Wire\]" $L | sed "s/^\[[^]]*\] //" | sort | uniq -c && grep -q "Calling API: https://chatgpt.com/backend-api/codex/" $L && ! grep -h "Calling API:" $L | grep -vq "Calling API: https://chatgpt.com/backend-api/codex/" && ! grep -h "Handler: provider=" $L | grep -vq "Handler: provider=openai-codex,"'
+ - &provenance
+ type: exec
+ metric: provenance
+ # The shim ran, so this worktree's dist/index.js is what claudish was.
+ #
+ # TEMPORARY — the first line of this cmd stands in for the madbench
+ # feature request "exec: keep the command's output on a passing Check"
+ # (draft unsent; README.md, "Keeping the logs"). Delete it when madbench
+ # ships it. An exec Check records its output only on failure and the
+ # workspace is deleted after the run, so a PASS keeps none of the log
+ # lines that proved it. With CODEX_CODE_MODE_KEEP_LOGS=<dir> set, the
+ # logs are copied to <dir>/<workspace name>/ (map it to a Scenario by
+ # the report's session.work_dir). Unset, it does nothing.
+ cmd: |
+ if [ -n "${CODEX_CODE_MODE_KEEP_LOGS:-}" ]; then d="$CODEX_CODE_MODE_KEEP_LOGS/$(basename "$PWD")"; mkdir -p "$d" && cp claudish-provenance.txt logs/claudish_*.log "$d/" 2>/dev/null; fi
+ cat claudish-provenance.txt && grep -q "^dist=/.*/packages/cli/dist/index.js$" claudish-provenance.txt && grep -q "^version=claudish version " claudish-provenance.txt
+
+ # ── auto: code mode by discovery, the D1 regression ─────────────────────────
+ - name: auto
+ description: "code mode auto: the same Reads go through one exec script, and every relayed script resumes"
+ testdata: ./testdata/parallel-read
+ prompt: "Use the Read tool to read a.ts, b.ts and c.ts in this directory. Issue all three Read calls in parallel in ONE turn. Then reply with the sum of the three exported numbers."
+ checks:
+ - type: regex
+ metric: answer
+ value: '\b6\b'
+ - type: assert-set
+ metric: args_intact
+ checks:
+ - { type: session:tool-args-match, value: Read, tool_args: { file_path: "suffix:a.ts" } }
+ - { type: session:tool-args-match, value: Read, tool_args: { file_path: "suffix:b.ts" } }
+ - { type: session:tool-args-match, value: Read, tool_args: { file_path: "suffix:c.ts" } }
+ - type: session:match
+ value: "InputValidationError|required parameter .file_path. is missing"
+ regex: true
+ event_type: tool_result
+ eq: 0
+ - &code_mode_on
+ type: exec
+ metric: code_mode_on
+ # auto chose code mode from Codex /models discovery, not a forced flag.
+ cmd: 'L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" && grep -h "\[CodeMode\] on (" $L && grep -q "\[CodeMode\] on ([a-z]*, discovery: " $L'
+ - &resumed
+ type: exec
+ metric: resumed
+ # THE D1 detector. Every exec relayed to Claude Code as tool_use blocks
+ # must later log `exec <id>: resumed;` (completed, blocked again or a
+ # genuine script failure), and no script may take the lost-state path.
+ # D1 had right answers and relayed=N resumed=0.
+ cmd: |
+ L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" || exit 1
+ grep -h -e "relayed as" -e ": resumed;" -e "script lost" -e "answered as Script failed" $L | sed "s/^\[[^]]*\] //"
+ awk '
+ /\[ResponsesTurn\] exec call_[A-Za-z0-9]+ relayed as/ { id=$0; sub(/.*\] exec /,"",id); sub(/ .*/,"",id); r[id]=1 }
+ /\[CodeMode\] exec call_[A-Za-z0-9]+: resumed;/ { id=$0; sub(/.*\] exec /,"",id); sub(/:.*/,"",id); if (id in r) s[id]=1 }
+ /script lost|answered as Script failed/ { lost++ }
+ END { n=0; m=0; for (i in r) { n++; if (i in s) m++ }
+ printf "relayed=%d resumed=%d lost=%d\n", n, m, lost+0
+ exit (n > 0 && m == n && lost == 0) ? 0 : 1 }' $L
+ - *subscription
+ - *provenance
+
+ # ── auto-edit: a script that writes, then runs Bash, then resumes ───────────
+ - name: auto-edit
+ description: "code mode auto: create a file and run Bash through exec; the Bash output reaches the model"
+ testdata: ./testdata/edit-bash
+ prompt: "Create notes.txt containing the word alpha, then use the Bash tool to run `wc -c notes.txt` and tell me the byte count."
+ checks:
+ # The file changed as asked (absent in the testdata, so a no-op fails).
+ - type: exec
+ metric: file_written
+ cmd: 'test -f notes.txt && grep -q alpha notes.txt && cat notes.txt'
+ # wc ran through Bash, and the number it printed is in the answer.
+ - type: ts
+ metric: bash_output_reported
+ value: "file://./checks/bash-count-reported.ts"
+ - type: assert-set
+ metric: args_intact
+ checks:
+ - { type: session:tool-used, value: Bash }
+ - type: session:match
+ value: "InputValidationError"
+ event_type: tool_result
+ eq: 0
+ - *code_mode_on
+ - *resumed
+ - *subscription
+ - *provenance
diff --git a/ai-docs/benches/codex-code-mode/testdata/edit-bash/a.ts b/ai-docs/benches/codex-code-mode/testdata/edit-bash/a.ts
new file mode 100644
index 00000000..cc798ff5
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/testdata/edit-bash/a.ts
@@ -0,0 +1 @@
+export const a = 1;
diff --git a/ai-docs/benches/codex-code-mode/testdata/edit-bash/b.ts b/ai-docs/benches/codex-code-mode/testdata/edit-bash/b.ts
new file mode 100644
index 00000000..20210308
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/testdata/edit-bash/b.ts
@@ -0,0 +1 @@
+export const b = 2;
diff --git a/ai-docs/benches/codex-code-mode/testdata/edit-bash/c.ts b/ai-docs/benches/codex-code-mode/testdata/edit-bash/c.ts
new file mode 100644
index 00000000..5f0cabef
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/testdata/edit-bash/c.ts
@@ -0,0 +1 @@
+export const c = 3;
diff --git a/ai-docs/benches/codex-code-mode/testdata/parallel-read/a.ts b/ai-docs/benches/codex-code-mode/testdata/parallel-read/a.ts
new file mode 100644
index 00000000..cc798ff5
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/testdata/parallel-read/a.ts
@@ -0,0 +1 @@
+export const a = 1;
diff --git a/ai-docs/benches/codex-code-mode/testdata/parallel-read/b.ts b/ai-docs/benches/codex-code-mode/testdata/parallel-read/b.ts
new file mode 100644
index 00000000..20210308
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/testdata/parallel-read/b.ts
@@ -0,0 +1 @@
+export const b = 2;
diff --git a/ai-docs/benches/codex-code-mode/testdata/parallel-read/c.ts b/ai-docs/benches/codex-code-mode/testdata/parallel-read/c.ts
new file mode 100644
index 00000000..5f0cabef
--- /dev/null
+++ b/ai-docs/benches/codex-code-mode/testdata/parallel-read/c.ts
@@ -0,0 +1 @@
+export const c = 3;