Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

test(madbench): add create-session and team Scenarios to the MCP Bench

Jack Rudenko1ad50e77d6ac
diff --git a/ai-docs/benches/recommended-models/README.md b/ai-docs/benches/recommended-models/README.md
index 0e52f31d..37b056d0 100644
--- a/ai-docs/benches/recommended-models/README.md
+++ b/ai-docs/benches/recommended-models/README.md
@@ -6,11 +6,25 @@ list category and Bench.
| Bench | Path under test | The model must |
|---|---|---|
| `cli.madbench.yaml` | `claudish --model <spec>`: Claude Code through the proxy, interactive | fix `slugify.ts` with real tool calls (Read or Bash, then Edit or Write) until `bun test` passes |
-| `mcp.madbench.yaml` | native Claude Code → claudish MCP `run_prompt` → `runPromptViaProxy` | answer `4817 + 2906` inside the MCP tool result |
+| `mcp.madbench.yaml` | native Claude Code → the claudish MCP server, three Scenarios (below) | answer `4817 + 2906` inside an MCP or Read tool result |
The CLI Bench is the stronger test: it exercises the wire, streaming tool calls and the
-Claude Code TUI. `run_prompt` sends one text request with no tools, so the MCP Bench proves
-that a route answers, not that the model can use tools.
+Claude Code TUI. The MCP Bench sends one text task with no tools, so it proves that each
+MCP path reaches the model and returns its answer, not that the model can use tools.
+
+The MCP Bench's three Scenarios each send the same task (`What is 4817 + 2906?`) by a
+different tool, and each grades the same three things: the server came up, the call named
+the model under test letter for letter (`session:tool-args-match` on `{{model}}`), and
+`7723` appears in a `tool_result` the host did not write.
+
+| Scenario | The host | Where `7723` must appear |
+|---|---|---|
+| `run-prompt` | calls `run_prompt` once | `run_prompt`'s own result |
+| `create-session` | calls `create_session` once, polls `list_sessions` with `sleep 10` between polls, then calls `get_output` | `get_output`'s result |
+| `team` | calls `team` mode `run` with `models: ["{{model}}"]`, polls mode `status`, then reads `team-sum/response-<slot>.md` | the Read of the slot's response file |
+
+`run-prompt` and `create-session` also require exactly one call (`eq: 1`), and `team`'s
+`models` list is matched exactly, so a second model cannot supply the answer.
## The variants come from the catalog
@@ -72,8 +86,16 @@ A graded miss exits 0. Add `--fail-on-failure` where a miss must go red.
- **The MCP detector reads the tool result.** The host Claude can compute 7723 itself; it
cannot write an MCP `tool_result`. `session:match` with `type: tool_result` reads only
what claudish returned, and a failed route returns `Error: ...` with no number.
-- **Exactly one `run_prompt` call.** A retry with a second model could supply the answer
- the first model failed to give.
+- **Exactly one `run_prompt` or `create_session` call.** A retry with a second model could
+ supply the answer the first model failed to give.
+- **`--allowedTools "Bash(sleep:*)"`.** The server pushes no session events, so the host
+ polls `create_session` and `team`. Under `--print` with no permission mode an un-allowed
+ Bash call is denied, and polling with no wait burns the host's turns. madbench appends
+ `args:` last and pipes the prompt on stdin, so the variadic flag swallows nothing.
+ `run-prompt` makes no Bash call and is unaffected.
+- **Check keys are written flat, beside `type:`** (`eq:`, `tool_args:`, `event_type:`,
+ `regex:`). madbench 0.46.0 refuses a nested `args:` map on these Checks at preflight
+ (`unknown key "args"`).
- **The MCP server is declared inline in `args:`, not with `config.dirs`.** `config.dirs`
needs sandbox level `home`, and claudish needs `workspace`. madbench 0.43.2 refuses that
pair only at run time ("harness.config.dirs cannot apply at sandbox level workspace"),
diff --git a/ai-docs/benches/recommended-models/mcp.madbench.yaml b/ai-docs/benches/recommended-models/mcp.madbench.yaml
index 049cda45..1e54dce2 100644
--- a/ai-docs/benches/recommended-models/mcp.madbench.yaml
+++ b/ai-docs/benches/recommended-models/mcp.madbench.yaml
@@ -1,4 +1,4 @@
-description: "claudish MCP: does run_prompt get an answer from a recommended model?"
+description: "claudish MCP: do run_prompt, create_session and team get an answer from a recommended model?"
# One model spec per run. `mcp.experiment.yaml` sweeps it over the recommended
# list; `gen-experiments.ts` writes that file from `claudish --models-top --json`.
@@ -16,17 +16,25 @@ harness:
# pair at run time, after preflight and check both pass). Inline JSON, not a
# file path, because a path would resolve in the workspace.
# --strict-mcp-config keeps the user's installed claudish plugin server out.
+ # --allowedTools "Bash(sleep:*)": create_session and team are polled (the
+ # server pushes no session events), and in --print with no permission mode
+ # an un-allowed Bash call is denied, so the host could not wait between
+ # polls. madbench appends args last and pipes the prompt on stdin, so the
+ # variadic --allowedTools swallows nothing.
args:
- --mcp-config
- '{"mcpServers":{"claudish":{"command":"claudish","args":["--mcp"]}}}'
- --strict-mcp-config
+ - --allowedTools
+ - "Bash(sleep:*)"
defaults:
# workspace, not home: the MCP server is `claudish --mcp`, and it needs the
# real ~/.claudish for its config, credentials and catalog cache.
sandbox: workspace
- # One request through the proxy, no tools: --print is enough and needs no
- # permission mode.
+ # The host only calls MCP tools and `sleep`, so --print is enough. The
+ # interactive Claude Code a session or team slot runs in is claudish's own
+ # pane, not this Harness.
interactive: false
timeout: 360s
capture: log
@@ -56,15 +64,87 @@ scenarios:
# the answer the first model failed to give.
- type: session:tool-used
value: mcp__claudish__run_prompt
- args: { eq: 1 }
+ eq: 1
# The call named the model under test, letter for letter.
- type: session:tool-args-match
value: mcp__claudish__run_prompt
- args:
- args: { model: "{{model}}" }
+ tool_args: { model: "{{model}}" }
# THE detector: the answer is in the MCP tool's own result. The host can
# compute 7723 itself, but it cannot write a tool_result. A failed route
# returns "Error: ..." and no number.
- type: session:match
value: '\b7723\b'
- args: { type: tool_result, regex: true }
+ event_type: tool_result
+ regex: true
+
+ - name: create-session
+ description: "The host runs the model under test as a one-shot session and relays its answer"
+ timeout: 600s
+ prompt: |
+ Use the claudish MCP tools to run one session with the model under test.
+
+ 1. Call `create_session` exactly once, with these arguments:
+ - model: {{model}}
+ - prompt: What is 4817 + 2906? Reply with the number only.
+ 2. Poll `list_sessions` with include_completed: true. Between polls run
+ `sleep 10` with Bash. Stop when that session's state is COMPLETED,
+ FAILED, TIMEOUT, CANCELLED or EMPTY.
+ 3. Call `get_output` once with that session_id.
+
+ Use the model string exactly as written. Then reply with the text
+ get_output returned, verbatim. Do not compute the sum yourself and do
+ not call run_prompt, team or compare_models.
+ checks:
+ - type: environment:mcp-reachable
+ value: claudish
+ - type: environment:mcp-connected
+ value: claudish
+ # One session: a second session on another model could supply the answer.
+ - type: session:tool-used
+ value: mcp__claudish__create_session
+ eq: 1
+ - type: session:tool-args-match
+ value: mcp__claudish__create_session
+ tool_args: { model: "{{model}}" }
+ # The answer is in get_output's tool_result, which only the session wrote.
+ - type: session:match
+ value: '\b7723\b'
+ event_type: tool_result
+ regex: true
+
+ - name: team
+ description: "The host runs a one-slot team on the model under test and relays the slot's answer"
+ timeout: 600s
+ prompt: |
+ Use the claudish MCP tool `team` to run the model under test on one task.
+
+ 1. Call `team` with mode "run" exactly once, with these arguments:
+ - mode: run
+ - path: team-sum
+ - models: ["{{model}}"]
+ - input: What is 4817 + 2906? Reply with the number only.
+ 2. Poll `team` with mode "status", the same path and the run_id the run
+ returned. Between polls run `sleep 10` with Bash. Stop when the slot's
+ state is COMPLETED, FAILED, CANCELLED or EMPTY.
+ 3. Read the file team-sum/response-<slot id>.md, where the slot id is the
+ one key of the `slots` map the run returned.
+
+ Use the model string exactly as written. Then reply with the file's
+ content, verbatim. Do not compute the sum yourself and do not call
+ run_prompt, create_session or compare_models.
+ checks:
+ - type: environment:mcp-reachable
+ value: claudish
+ - type: environment:mcp-connected
+ value: claudish
+ # The run named the model under test, and only it: a list expectation
+ # is matched exactly, so a second model in the run fails this Check.
+ - type: session:tool-args-match
+ value: mcp__claudish__team
+ tool_args: { mode: run, models: ["{{model}}"] }
+ # The slot's answer reaches the host only through the response file the
+ # slot wrote; the host cannot author a Read tool_result.
+ - type: session:match
+ value: '\b7723\b'
+ event_type: tool_result
+ regex: true