Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

test(madbench): migrate cli.madbench.yaml Checks to madbench 0.46.0

Jack Rudenko87391475d412
diff --git a/ai-docs/benches/recommended-models/README.md b/ai-docs/benches/recommended-models/README.md
index bde105e6..4b31a36e 100644
--- a/ai-docs/benches/recommended-models/README.md
+++ b/ai-docs/benches/recommended-models/README.md
@@ -43,6 +43,8 @@ tier that left the list.
## Running
+The Benches need madbench 0.46.0 or later: a Check's keys sit beside `type:`.
+
The free gates (no spend):
```
@@ -71,9 +73,12 @@ A graded miss exits 0. Add `--fail-on-failure` where a miss must go red.
(config, 1Password and Keychain references, catalog cache). madbench's own example
`local-agent-orchestration` measured 20/20 Scenarios failing as "agent never started"
under `home`. The cost: the run shares this machine's `HOME`.
-- **The CLI anti-cheat is inside the outcome Check.** "The four cases are still in the test
- file" passes against an agent that did nothing, so on its own it fails `madbench check`
- as WRONGLY PASSED. `bun test && grep ...` passes only for a real fix. It counts lines, so it is weak; MadAppGang/madbench#131 asks for a `file-changes` Check to replace it.
+- **The CLI anti-cheat is a `file-changes` Check.** `exec: bun test` proves the work;
+ `file-changes` with `must: {update: slugify.ts}` and `must-not: {touch: slugify.test.ts}`
+ proves it stayed inside its boundary. A ban alone passes against an agent that did
+ nothing, and `madbench check` reports it WRONGLY PASSED, so it sits beside the exec Check.
+ It replaces a `grep -c` line count (MadAppGang/madbench#131). Needs madbench 0.46.0 or
+ later: a Check's keys sit beside `type:`.
- **The MCP detector reads the tool result.** The host Claude can compute 7723 itself; it
cannot write an MCP `tool_result`. `session:match` with `type: tool_result` reads only
what claudish returned, and a failed route returns `Error: ...` with no number.
diff --git a/ai-docs/benches/recommended-models/cli.madbench.yaml b/ai-docs/benches/recommended-models/cli.madbench.yaml
index be16a42f..fd7abf63 100644
--- a/ai-docs/benches/recommended-models/cli.madbench.yaml
+++ b/ai-docs/benches/recommended-models/cli.madbench.yaml
@@ -35,14 +35,16 @@ scenarios:
The test in slugify.test.ts fails. Fix slugify.ts so that `bun test`
passes. Do not change slugify.test.ts. Run `bun test` to confirm.
checks:
- # The outcome, with its anti-cheat in the same Check: `bun test` passes AND
- # the four cases are still there. Apart, the anti-cheat half passes under
- # a no-op agent and `madbench check` reports it WRONGLY PASSED.
+ # The work: `bun test` passes in the agent's tree.
- type: exec
- args:
- cmd: [sh, -c, 'bun test && test "$(grep -c "expect(slugify" slugify.test.ts)" -ge 4']
- expected_exit: 0
- timeout: 60s
+ cmd: bun test
+ expected_exit: 0
+ timeout: 60s
+ # The boundary: the fix is in slugify.ts and the test file is untouched. A ban alone
+ # passes under an agent that did nothing; the exec Check above proves the work.
+ - type: file-changes
+ must: { update: slugify.ts }
+ must-not: { touch: slugify.test.ts }
# The model drove tools through the proxy: it read the code...
- type: any-of
checks: