Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
test(madbench): add create-session and team Scenarios to the MCP Bench
Jack Rudenko1ad50e77d6ac
diff --git a/ai-docs/benches/recommended-models/README.md b/ai-docs/benches/recommended-models/README.mdindex 0e52f31d..37b056d0 100644--- a/ai-docs/benches/recommended-models/README.md+++ b/ai-docs/benches/recommended-models/README.md@@ -6,11 +6,25 @@ list category and Bench.| Bench | Path under test | The model must ||---|---|---|| `cli.madbench.yaml` | `claudish --model <spec>`: Claude Code through the proxy, interactive | fix `slugify.ts` with real tool calls (Read or Bash, then Edit or Write) until `bun test` passes |-| `mcp.madbench.yaml` | native Claude Code → claudish MCP `run_prompt` → `runPromptViaProxy` | answer `4817 + 2906` inside the MCP tool result |+| `mcp.madbench.yaml` | native Claude Code → the claudish MCP server, three Scenarios (below) | answer `4817 + 2906` inside an MCP or Read tool result |The CLI Bench is the stronger test: it exercises the wire, streaming tool calls and the-Claude Code TUI. `run_prompt` sends one text request with no tools, so the MCP Bench proves-that a route answers, not that the model can use tools.+Claude Code TUI. The MCP Bench sends one text task with no tools, so it proves that each+MCP path reaches the model and returns its answer, not that the model can use tools.++The MCP Bench's three Scenarios each send the same task (`What is 4817 + 2906?`) by a+different tool, and each grades the same three things: the server came up, the call named+the model under test letter for letter (`session:tool-args-match` on `{{model}}`), and+`7723` appears in a `tool_result` the host did not write.++| Scenario | The host | Where `7723` must appear |+|---|---|---|+| `run-prompt` | calls `run_prompt` once | `run_prompt`'s own result |+| `create-session` | calls `create_session` once, polls `list_sessions` with `sleep 10` between polls, then calls `get_output` | `get_output`'s result |+| `team` | calls `team` mode `run` with `models: ["{{model}}"]`, polls mode `status`, then reads `team-sum/response-<slot>.md` | the Read of the slot's response file |++`run-prompt` and `create-session` also require exactly one call (`eq: 1`), and `team`'s+`models` list is matched exactly, so a second model cannot supply the answer.## The variants come from the catalog@@ -72,8 +86,16 @@ A graded miss exits 0. Add `--fail-on-failure` where a miss must go red.- **The MCP detector reads the tool result.** The host Claude can compute 7723 itself; itcannot write an MCP `tool_result`. `session:match` with `type: tool_result` reads onlywhat claudish returned, and a failed route returns `Error: ...` with no number.-- **Exactly one `run_prompt` call.** A retry with a second model could supply the answer- the first model failed to give.+- **Exactly one `run_prompt` or `create_session` call.** A retry with a second model could+ supply the answer the first model failed to give.+- **`--allowedTools "Bash(sleep:*)"`.** The server pushes no session events, so the host+ polls `create_session` and `team`. Under `--print` with no permission mode an un-allowed+ Bash call is denied, and polling with no wait burns the host's turns. madbench appends+ `args:` last and pipes the prompt on stdin, so the variadic flag swallows nothing.+ `run-prompt` makes no Bash call and is unaffected.+- **Check keys are written flat, beside `type:`** (`eq:`, `tool_args:`, `event_type:`,+ `regex:`). madbench 0.46.0 refuses a nested `args:` map on these Checks at preflight+ (`unknown key "args"`).- **The MCP server is declared inline in `args:`, not with `config.dirs`.** `config.dirs`needs sandbox level `home`, and claudish needs `workspace`. madbench 0.43.2 refuses thatpair only at run time ("harness.config.dirs cannot apply at sandbox level workspace"),diff --git a/ai-docs/benches/recommended-models/mcp.madbench.yaml b/ai-docs/benches/recommended-models/mcp.madbench.yamlindex 049cda45..1e54dce2 100644--- a/ai-docs/benches/recommended-models/mcp.madbench.yaml+++ b/ai-docs/benches/recommended-models/mcp.madbench.yaml@@ -1,4 +1,4 @@-description: "claudish MCP: does run_prompt get an answer from a recommended model?"+description: "claudish MCP: do run_prompt, create_session and team get an answer from a recommended model?"# One model spec per run. `mcp.experiment.yaml` sweeps it over the recommended# list; `gen-experiments.ts` writes that file from `claudish --models-top --json`.@@ -16,17 +16,25 @@ harness:# pair at run time, after preflight and check both pass). Inline JSON, not a# file path, because a path would resolve in the workspace.# --strict-mcp-config keeps the user's installed claudish plugin server out.+ # --allowedTools "Bash(sleep:*)": create_session and team are polled (the+ # server pushes no session events), and in --print with no permission mode+ # an un-allowed Bash call is denied, so the host could not wait between+ # polls. madbench appends args last and pipes the prompt on stdin, so the+ # variadic --allowedTools swallows nothing.args:- --mcp-config- '{"mcpServers":{"claudish":{"command":"claudish","args":["--mcp"]}}}'- --strict-mcp-config+ - --allowedTools+ - "Bash(sleep:*)"defaults:# workspace, not home: the MCP server is `claudish --mcp`, and it needs the# real ~/.claudish for its config, credentials and catalog cache.sandbox: workspace- # One request through the proxy, no tools: --print is enough and needs no- # permission mode.+ # The host only calls MCP tools and `sleep`, so --print is enough. The+ # interactive Claude Code a session or team slot runs in is claudish's own+ # pane, not this Harness.interactive: falsetimeout: 360scapture: log@@ -56,15 +64,87 @@ scenarios:# the answer the first model failed to give.- type: session:tool-usedvalue: mcp__claudish__run_prompt- args: { eq: 1 }+ eq: 1# The call named the model under test, letter for letter.- type: session:tool-args-matchvalue: mcp__claudish__run_prompt- args:- args: { model: "{{model}}" }+ tool_args: { model: "{{model}}" }# THE detector: the answer is in the MCP tool's own result. The host can# compute 7723 itself, but it cannot write a tool_result. A failed route# returns "Error: ..." and no number.- type: session:matchvalue: '\b7723\b'- args: { type: tool_result, regex: true }+ event_type: tool_result+ regex: true++ - name: create-session+ description: "The host runs the model under test as a one-shot session and relays its answer"+ timeout: 600s+ prompt: |+ Use the claudish MCP tools to run one session with the model under test.++ 1. Call `create_session` exactly once, with these arguments:+ - model: {{model}}+ - prompt: What is 4817 + 2906? Reply with the number only.+ 2. Poll `list_sessions` with include_completed: true. Between polls run+ `sleep 10` with Bash. Stop when that session's state is COMPLETED,+ FAILED, TIMEOUT, CANCELLED or EMPTY.+ 3. Call `get_output` once with that session_id.++ Use the model string exactly as written. Then reply with the text+ get_output returned, verbatim. Do not compute the sum yourself and do+ not call run_prompt, team or compare_models.+ checks:+ - type: environment:mcp-reachable+ value: claudish+ - type: environment:mcp-connected+ value: claudish+ # One session: a second session on another model could supply the answer.+ - type: session:tool-used+ value: mcp__claudish__create_session+ eq: 1+ - type: session:tool-args-match+ value: mcp__claudish__create_session+ tool_args: { model: "{{model}}" }+ # The answer is in get_output's tool_result, which only the session wrote.+ - type: session:match+ value: '\b7723\b'+ event_type: tool_result+ regex: true++ - name: team+ description: "The host runs a one-slot team on the model under test and relays the slot's answer"+ timeout: 600s+ prompt: |+ Use the claudish MCP tool `team` to run the model under test on one task.++ 1. Call `team` with mode "run" exactly once, with these arguments:+ - mode: run+ - path: team-sum+ - models: ["{{model}}"]+ - input: What is 4817 + 2906? Reply with the number only.+ 2. Poll `team` with mode "status", the same path and the run_id the run+ returned. Between polls run `sleep 10` with Bash. Stop when the slot's+ state is COMPLETED, FAILED, CANCELLED or EMPTY.+ 3. Read the file team-sum/response-<slot id>.md, where the slot id is the+ one key of the `slots` map the run returned.++ Use the model string exactly as written. Then reply with the file's+ content, verbatim. Do not compute the sum yourself and do not call+ run_prompt, create_session or compare_models.+ checks:+ - type: environment:mcp-reachable+ value: claudish+ - type: environment:mcp-connected+ value: claudish+ # The run named the model under test, and only it: a list expectation+ # is matched exactly, so a second model in the run fails this Check.+ - type: session:tool-args-match+ value: mcp__claudish__team+ tool_args: { mode: run, models: ["{{model}}"] }+ # The slot's answer reaches the host only through the response file the+ # slot wrote; the host cannot author a Read tool_result.+ - type: session:match+ value: '\b7723\b'+ event_type: tool_result+ regex: true