Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
test(openai-codex): space bench claudish launches 100 ms apart
Jack Rudenkoe7f4c2ead0e2
diff --git a/ai-docs/benches/codex-code-mode/README.md b/ai-docs/benches/codex-code-mode/README.mdindex 2cb86785..57f37b94 100644--- a/ai-docs/benches/codex-code-mode/README.md+++ b/ai-docs/benches/codex-code-mode/README.md@@ -101,15 +101,13 @@ The real run, in a visible terminal (not piped, not backgrounded):```bashCODEX_CODE_MODE_KEEP_LOGS=<dir> \- madbench madbench.yaml --concurrency 1 --trials 5 --report-json <out.json>+ madbench madbench.yaml --concurrency 4 --trials 5 --report-json <out.json>```-- **`--concurrency 1` is required.** On the first real run, with three claudish sessions starting- at once, two were reported `agent never started: agent did not start within 1m0s`, although- their panes showed Claude Code ready at the prompt. One at a time, all three started. The cause- has not been isolated.-- `--trials 5` gives 15 sessions (about 15 minutes), all on the subscription. Phase 7 used- 10/10/3 sessions by hand; `--scenario auto --trials 10` adds depth where D1 lived.+- **Concurrency works through the shim's launch lock** (see "Concurrency" below). Without it,+ Scenarios that start together fail `agent never started` because of a claudish defect.+- `--trials 5` gives 20 sessions, all on the subscription. Phase 7 used 10/10/3 sessions by+ hand; `--scenario auto --trials 10` adds depth where D1 lived.- A graded miss exits 0. Add `--fail-on-failure` where a miss must turn the job red.- `madbench grade <out.json> --bench madbench.yaml` re-grades a stored report offline. The `exec`and `ts` Checks show their recorded verdicts, because the workspace is gone.@@ -131,6 +129,23 @@ Why each setting:- **`driver: false`**: every prompt is an order, so a question from the agent fails the Scenarioby name instead of being answered by a model and becoming a variable.+## Concurrency++The first smoke run, three Scenarios at once, reported two `agent never started: agent did not+start within 1m0s` with Claude Code ready in both panes. The cause is claudish's, not madbench's:+`createTempSettingsFile` (`packages/cli/src/claude-runner.ts`) names its `--settings` overlay+`~/.claudish/settings-<Date.now()>.json`. madbench starts concurrent Scenarios in lock-step (two+claudish debug logs opened in the same millisecond), so two or three processes write ONE file and+every Claude Code loads the last writer's statusLine and hooks. madbench waits for a sample in+each run's own `statusline.jsonl`; the losers' stay empty, and all the samples land in the+winner's. At `--concurrency 3 --trials 3`, 2 of 3 trials collided this way; one collided winner+then also hung for 300 s, because its sink carried other sessions' samples. Full evidence:+`ai-docs/sessions/codex-code-mode-bench-20261020-172800-a1c3/conc/DIAGNOSIS.md` (gitignored).++`claudish-under-test` carries a TEMPORARY launch lock (`/tmp/claudish-under-test.launch.lock`,+held 100 ms per launch) so every process reads a different millisecond. Delete it, and this+section's workaround, when claudish names the settings file uniquely per process.+## Reading a failureRead `.summary.errors` before `.summary.failed`: an errored Scenario never graded.diff --git a/ai-docs/benches/codex-code-mode/claudish-under-test b/ai-docs/benches/codex-code-mode/claudish-under-testindex 63471b6d..ad285ffa 100755--- a/ai-docs/benches/codex-code-mode/claudish-under-test+++ b/ai-docs/benches/codex-code-mode/claudish-under-test@@ -20,4 +20,25 @@ dist="$root/packages/cli/dist/index.js"echo "head=$(git -C "$root" rev-parse HEAD 2>/dev/null || echo unknown)"echo "dist_mtime=$(stat -f %Sm -t %Y-%m-%dT%H:%M:%S "$dist")"} > claudish-provenance.txt++# TEMPORARY — stands in for a claudish fix: createTempSettingsFile+# (packages/cli/src/claude-runner.ts) names its file+# ~/.claudish/settings-<Date.now()>.json, so two claudish processes started in+# the same millisecond write ONE file, and both Claude Codes load the last+# writer's statusLine and hooks. madbench starts concurrent Scenarios within a+# millisecond of each other, so the losers' statusLine sinks stay empty and+# madbench reports "agent never started" (diagnosis: README.md, "Concurrency").+# This spaces launches at least 100 ms apart with a lock directory, so each+# process reads a different Date.now(). Delete it when claudish names the+# file uniquely per process.+lock=/tmp/claudish-under-test.launch.lock+waited=0+until mkdir "$lock" 2>/dev/null; do+ waited=$((waited + 1))+ # a lock left by a killed launch is never older than a few seconds+ if [ "$waited" -gt 200 ]; then rmdir "$lock" 2>/dev/null || true; waited=0; fi+ sleep 0.02+done+( sleep 0.1; rmdir "$lock" 2>/dev/null ) &+exec bun "$dist" --debug-claudish --log-level debug -q "$@"