Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
test(madbench): migrate cli.madbench.yaml Checks to madbench 0.46.0
Jack Rudenko87391475d412
diff --git a/ai-docs/benches/recommended-models/README.md b/ai-docs/benches/recommended-models/README.mdindex bde105e6..4b31a36e 100644--- a/ai-docs/benches/recommended-models/README.md+++ b/ai-docs/benches/recommended-models/README.md@@ -43,6 +43,8 @@ tier that left the list.## Running+The Benches need madbench 0.46.0 or later: a Check's keys sit beside `type:`.+The free gates (no spend):```@@ -71,9 +73,12 @@ A graded miss exits 0. Add `--fail-on-failure` where a miss must go red.(config, 1Password and Keychain references, catalog cache). madbench's own example`local-agent-orchestration` measured 20/20 Scenarios failing as "agent never started"under `home`. The cost: the run shares this machine's `HOME`.-- **The CLI anti-cheat is inside the outcome Check.** "The four cases are still in the test- file" passes against an agent that did nothing, so on its own it fails `madbench check`- as WRONGLY PASSED. `bun test && grep ...` passes only for a real fix. It counts lines, so it is weak; MadAppGang/madbench#131 asks for a `file-changes` Check to replace it.+- **The CLI anti-cheat is a `file-changes` Check.** `exec: bun test` proves the work;+ `file-changes` with `must: {update: slugify.ts}` and `must-not: {touch: slugify.test.ts}`+ proves it stayed inside its boundary. A ban alone passes against an agent that did+ nothing, and `madbench check` reports it WRONGLY PASSED, so it sits beside the exec Check.+ It replaces a `grep -c` line count (MadAppGang/madbench#131). Needs madbench 0.46.0 or+ later: a Check's keys sit beside `type:`.- **The MCP detector reads the tool result.** The host Claude can compute 7723 itself; itcannot write an MCP `tool_result`. `session:match` with `type: tool_result` reads onlywhat claudish returned, and a failed route returns `Error: ...` with no number.diff --git a/ai-docs/benches/recommended-models/cli.madbench.yaml b/ai-docs/benches/recommended-models/cli.madbench.yamlindex be16a42f..fd7abf63 100644--- a/ai-docs/benches/recommended-models/cli.madbench.yaml+++ b/ai-docs/benches/recommended-models/cli.madbench.yaml@@ -35,14 +35,16 @@ scenarios:The test in slugify.test.ts fails. Fix slugify.ts so that `bun test`passes. Do not change slugify.test.ts. Run `bun test` to confirm.checks:- # The outcome, with its anti-cheat in the same Check: `bun test` passes AND- # the four cases are still there. Apart, the anti-cheat half passes under- # a no-op agent and `madbench check` reports it WRONGLY PASSED.+ # The work: `bun test` passes in the agent's tree.- type: exec- args:- cmd: [sh, -c, 'bun test && test "$(grep -c "expect(slugify" slugify.test.ts)" -ge 4']- expected_exit: 0- timeout: 60s+ cmd: bun test+ expected_exit: 0+ timeout: 60s+ # The boundary: the fix is in slugify.ts and the test file is untouched. A ban alone+ # passes under an agent that did nothing; the exec Check above proves the work.+ - type: file-changes+ must: { update: slugify.ts }+ must-not: { touch: slugify.test.ts }# The model drove tools through the proxy: it read the code...- type: any-ofchecks: