Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

test(madbench): add CLI and MCP benches for the recommended models

Jack Rudenkof96107326f82
diff --git a/ai-docs/benches/recommended-models/README.md b/ai-docs/benches/recommended-models/README.md
new file mode 100644
index 00000000..f7a5b478
--- /dev/null
+++ b/ai-docs/benches/recommended-models/README.md
@@ -0,0 +1,83 @@
+# recommended-models: does every recommended model work through claudish?
+
+Two Benches, one question each, swept over the recommended list by two Experiments.
+
+| Bench | Path under test | The model must |
+|---|---|---|
+| `cli.madbench.yaml` | `claudish --model <spec>`: Claude Code through the proxy, interactive | fix `slugify.ts` with real tool calls (Read or Bash, then Edit or Write) until `bun test` passes |
+| `mcp.madbench.yaml` | native Claude Code → claudish MCP `run_prompt` → `runPromptViaProxy` | answer `4817 + 2906` inside the MCP tool result |
+
+The CLI Bench is the stronger test: it exercises the wire, streaming tool calls and the
+Claude Code TUI. `run_prompt` sends one text request with no tools, so the MCP Bench proves
+that a route answers, not that the model can use tools.
+
+## The variants come from the catalog
+
+```
+bun scripts/gen-experiments.ts
+```
+
+Every command in this README runs from this folder
+(`ai-docs/benches/recommended-models/`). The Benches resolve their own paths against the
+Bench file, and the script writes next to itself, so the folder is the only working
+directory to remember.
+
+writes `cli.experiment.yaml` and `mcp.experiment.yaml` from `claudish --models-top --json`:
+first every recommended model by bare name (routing chooses the hop), then every distinct
+subscription command the list publishes (`[email protected]`, `kc@k3`, ...). The list changes
+with each catalog generation, so re-run the script before a real run. Each file names the
+generation it was written from.
+
+## Running
+
+The free gates (no spend):
+
+```
+madbench preflight cli.madbench.yaml
+madbench check cli.madbench.yaml
+madbench preflight mcp.madbench.yaml
+madbench check mcp.madbench.yaml
+```
+
+A real run, one variant first:
+
+```
+madbench cli.experiment.yaml --variant kimi-k3 --report-json <path>
+madbench mcp.experiment.yaml --report-json <path>
+```
+
+Both Benches run the `claudish` on `PATH`. Check `claudish --version` first: a stale global
+install tests the old release, not the one under review.
+
+A graded miss exits 0. Add `--fail-on-failure` where a miss must go red.
+
+## Why each choice
+
+- **`sandbox: workspace`, not `home`.** `home` replaces `HOME`, which hides `~/.claudish`
+ (config, 1Password and Keychain references, catalog cache). madbench's own example
+ `local-agent-orchestration` measured 20/20 Scenarios failing as "agent never started"
+ under `home`. The cost: the run shares this machine's `HOME`.
+- **The CLI anti-cheat is inside the outcome Check.** "The four cases are still in the test
+ file" passes against an agent that did nothing, so on its own it fails `madbench check`
+ as WRONGLY PASSED. `bun test && grep ...` passes only for a real fix. It counts lines, so it is weak; MadAppGang/madbench#131 asks for a `file-changes` Check to replace it.
+- **The MCP detector reads the tool result.** The host Claude can compute 7723 itself; it
+ cannot write an MCP `tool_result`. `session:match` with `type: tool_result` reads only
+ what claudish returned, and a failed route returns `Error: ...` with no number.
+- **Exactly one `run_prompt` call.** A retry with a second model could supply the answer
+ the first model failed to give.
+- **The MCP server is declared inline in `args:`, not with `config.dirs`.** `config.dirs`
+ needs sandbox level `home`, and claudish needs `workspace`. madbench 0.43.2 refuses that
+ pair only at run time ("harness.config.dirs cannot apply at sandbox level workspace"),
+ after `preflight` and `check` both passed (MadAppGang/madbench#130). The value is JSON, not a file path, because a
+ path would resolve in the workspace. `--strict-mcp-config` keeps the user's installed
+ claudish plugin server out of the run.
+- **`environment:mcp-connected`, not `environment:tool-available`.** `tool-available`
+ always errors under the mock, so the negative control could never hold
+ (madbench `docs/checks.md`, the environment table).
+- **The MCP Bench errors under `--harness mock`.** `environment:mcp-reachable` reads the
+ run's MCP preflight, and the mock starts no server. `madbench check` lists it as not
+ applicable; the negative control is the gate, not the mock run's exit code.
+- **The MCP host model is fixed** (`claude-haiku-5-5`, billed to the subscription), so the
+ variants differ only in the prompt and the Checks. That Experiment declares no
+ `guard_changes:` because its staged inputs never differ; the CLI Experiment allows
+ `harness/model` and nothing else.
diff --git a/ai-docs/benches/recommended-models/cli.experiment.yaml b/ai-docs/benches/recommended-models/cli.experiment.yaml
new file mode 100644
index 00000000..2300b1ea
--- /dev/null
+++ b/ai-docs/benches/recommended-models/cli.experiment.yaml
@@ -0,0 +1,80 @@
+# GENERATED by scripts/gen-experiments.ts from `claudish --models-top --json`.
+# Catalog generation g-20261014030532231-748cc4dd, written 2026-10-14T03:36:04.684Z.
+# Do not edit by hand: re-run the script. 36 variants.
+description: "cli: every recommended model and subscription command"
+bench: ./cli.madbench.yaml
+guard_changes:
+ allow: [harness/model]
+variants:
+ - name: "claude-opus-5-5"
+ params: { model: "claude-opus-5-5" }
+ - name: "gpt-6.1-sol"
+ params: { model: "gpt-6.1-sol" }
+ - name: "kimi-k3"
+ params: { model: "kimi-k3" }
+ - name: "grok-4.7"
+ params: { model: "grok-4.7" }
+ - name: "gemini-4-argon"
+ params: { model: "gemini-4-argon" }
+ - name: "qwen3.8-max-0902"
+ params: { model: "qwen3.8-max-0902" }
+ - name: "glm-5.3"
+ params: { model: "glm-5.3" }
+ - name: "minimax-m3"
+ params: { model: "minimax-m3" }
+ - name: "deepseek-v3.2-exp"
+ params: { model: "deepseek-v3.2-exp" }
+ - name: "claude-haiku-5-5"
+ params: { model: "claude-haiku-5-5" }
+ - name: "deepseek-v4.1-flash"
+ params: { model: "deepseek-v4.1-flash" }
+ - name: "gemini-3.5-flash-lite"
+ params: { model: "gemini-3.5-flash-lite" }
+ - name: "mistral-small-2603"
+ params: { model: "mistral-small-2603" }
+ - name: "gpt-5.4-nano-2026-03-17"
+ params: { model: "gpt-5.4-nano-2026-03-17" }
+ - name: "qwen3.8-omni-flash"
+ params: { model: "qwen3.8-omni-flash" }
+ - name: "grok-build-0.1"
+ params: { model: "grok-build-0.1" }
+ - name: "glm-5.3-flash"
+ params: { model: "glm-5.3-flash" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "kc@k3"
+ params: { model: "kc@k3" }
+ - name: "oc@kimi-k3"
+ params: { model: "oc@kimi-k3" }
+ - name: "zengo@kimi-k3"
+ params: { model: "zengo@kimi-k3" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "mmc@MiniMax-M3"
+ params: { model: "mmc@MiniMax-M3" }
+ - name: "oc@minimax-m3"
+ params: { model: "oc@minimax-m3" }
+ - name: "zengo@minimax-m3"
+ params: { model: "zengo@minimax-m3" }
+ - name: "zengo@claude-haiku-5-5"
+ params: { model: "zengo@claude-haiku-5-5" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "zengo@deepseek-flash"
+ params: { model: "zengo@deepseek-flash" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
diff --git a/ai-docs/benches/recommended-models/cli.madbench.yaml b/ai-docs/benches/recommended-models/cli.madbench.yaml
new file mode 100644
index 00000000..be16a42f
--- /dev/null
+++ b/ai-docs/benches/recommended-models/cli.madbench.yaml
@@ -0,0 +1,55 @@
+description: "claudish CLI: can a recommended model fix a failing test through Claude Code?"
+
+# One model spec per run. `cli.experiment.yaml` sweeps it over the recommended
+# list; `gen-experiments.ts` writes that file from `claudish --models-top --json`.
+params:
+ model: kimi-k3
+
+harness:
+ type: claude-code
+ # The program under test. madbench refuses a non-Anthropic model unless the
+ # binary is claudish, and `claudish --model <spec>` is the user's own path:
+ # routing, the wire built from the catalog, the stream parser, tool calls.
+ binary: claudish
+ model: "{{model}}"
+ # Interactive (the default): Bash parks at an approval menu nobody answers
+ # unless the mode is bypassPermissions. claudish forwards the flag to claude.
+ args: ["--permission-mode", "bypassPermissions"]
+
+defaults:
+ # workspace, not home. Level `home` replaces HOME, which hides ~/.claudish
+ # (config, 1Password and Keychain references, catalog cache). The madbench
+ # example local-agent-orchestration measured 20/20 Scenarios failing as
+ # "agent never started" under `home`.
+ sandbox: workspace
+ # A reasoning model on a cold catalog can need the 30 s catalog wait plus
+ # several turns. Tighten after 2-3 real runs.
+ timeout: 480s
+ capture: log
+
+scenarios:
+ - name: fix-slugify
+ description: "Read a failing test, edit the source, run the test"
+ testdata: ./testdata/cli
+ prompt: |
+ The test in slugify.test.ts fails. Fix slugify.ts so that `bun test`
+ passes. Do not change slugify.test.ts. Run `bun test` to confirm.
+ checks:
+ # The outcome, with its anti-cheat in the same Check: `bun test` passes AND
+ # the four cases are still there. Apart, the anti-cheat half passes under
+ # a no-op agent and `madbench check` reports it WRONGLY PASSED.
+ - type: exec
+ args:
+ cmd: [sh, -c, 'bun test && test "$(grep -c "expect(slugify" slugify.test.ts)" -ge 4']
+ expected_exit: 0
+ timeout: 60s
+ # The model drove tools through the proxy: it read the code...
+ - type: any-of
+ checks:
+ - { type: session:tool-used, value: Read }
+ - { type: session:tool-used, value: Bash }
+ # ...and changed it with a file tool.
+ - type: any-of
+ checks:
+ - { type: session:tool-used, value: Edit }
+ - { type: session:tool-used, value: Write }
diff --git a/ai-docs/benches/recommended-models/mcp.experiment.yaml b/ai-docs/benches/recommended-models/mcp.experiment.yaml
new file mode 100644
index 00000000..4510ea1b
--- /dev/null
+++ b/ai-docs/benches/recommended-models/mcp.experiment.yaml
@@ -0,0 +1,78 @@
+# GENERATED by scripts/gen-experiments.ts from `claudish --models-top --json`.
+# Catalog generation g-20261014030532231-748cc4dd, written 2026-10-14T03:36:04.684Z.
+# Do not edit by hand: re-run the script. 36 variants.
+description: "mcp: every recommended model and subscription command"
+bench: ./mcp.madbench.yaml
+variants:
+ - name: "claude-opus-5-5"
+ params: { model: "claude-opus-5-5" }
+ - name: "gpt-6.1-sol"
+ params: { model: "gpt-6.1-sol" }
+ - name: "kimi-k3"
+ params: { model: "kimi-k3" }
+ - name: "grok-4.7"
+ params: { model: "grok-4.7" }
+ - name: "gemini-4-argon"
+ params: { model: "gemini-4-argon" }
+ - name: "qwen3.8-max-0902"
+ params: { model: "qwen3.8-max-0902" }
+ - name: "glm-5.3"
+ params: { model: "glm-5.3" }
+ - name: "minimax-m3"
+ params: { model: "minimax-m3" }
+ - name: "deepseek-v3.2-exp"
+ params: { model: "deepseek-v3.2-exp" }
+ - name: "claude-haiku-5-5"
+ params: { model: "claude-haiku-5-5" }
+ - name: "deepseek-v4.1-flash"
+ params: { model: "deepseek-v4.1-flash" }
+ - name: "gemini-3.5-flash-lite"
+ params: { model: "gemini-3.5-flash-lite" }
+ - name: "mistral-small-2603"
+ params: { model: "mistral-small-2603" }
+ - name: "gpt-5.4-nano-2026-03-17"
+ params: { model: "gpt-5.4-nano-2026-03-17" }
+ - name: "qwen3.8-omni-flash"
+ params: { model: "qwen3.8-omni-flash" }
+ - name: "grok-build-0.1"
+ params: { model: "grok-build-0.1" }
+ - name: "glm-5.3-flash"
+ params: { model: "glm-5.3-flash" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "kc@k3"
+ params: { model: "kc@k3" }
+ - name: "oc@kimi-k3"
+ params: { model: "oc@kimi-k3" }
+ - name: "zengo@kimi-k3"
+ params: { model: "zengo@kimi-k3" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "mmc@MiniMax-M3"
+ params: { model: "mmc@MiniMax-M3" }
+ - name: "oc@minimax-m3"
+ params: { model: "oc@minimax-m3" }
+ - name: "zengo@minimax-m3"
+ params: { model: "zengo@minimax-m3" }
+ - name: "zengo@claude-haiku-5-5"
+ params: { model: "zengo@claude-haiku-5-5" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "zengo@deepseek-flash"
+ params: { model: "zengo@deepseek-flash" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
+ - name: "[email protected]"
+ params: { model: "[email protected]" }
diff --git a/ai-docs/benches/recommended-models/mcp.madbench.yaml b/ai-docs/benches/recommended-models/mcp.madbench.yaml
new file mode 100644
index 00000000..049cda45
--- /dev/null
+++ b/ai-docs/benches/recommended-models/mcp.madbench.yaml
@@ -0,0 +1,70 @@
+description: "claudish MCP: does run_prompt get an answer from a recommended model?"
+
+# One model spec per run. `mcp.experiment.yaml` sweeps it over the recommended
+# list; `gen-experiments.ts` writes that file from `claudish --models-top --json`.
+params:
+ model: kimi-k3
+
+harness:
+ type: claude-code
+ # The host is native Claude Code. The model under test is reached ONLY
+ # through the claudish MCP server, so the Harness model stays fixed across
+ # every variant and the guard has nothing to audit.
+ model: claude-haiku-5-5
+ # The server is declared inline, not through config.dirs: config.dirs needs
+ # sandbox level `home`, and claudish needs `workspace` (madbench refuses the
+ # pair at run time, after preflight and check both pass). Inline JSON, not a
+ # file path, because a path would resolve in the workspace.
+ # --strict-mcp-config keeps the user's installed claudish plugin server out.
+ args:
+ - --mcp-config
+ - '{"mcpServers":{"claudish":{"command":"claudish","args":["--mcp"]}}}'
+ - --strict-mcp-config
+
+defaults:
+ # workspace, not home: the MCP server is `claudish --mcp`, and it needs the
+ # real ~/.claudish for its config, credentials and catalog cache.
+ sandbox: workspace
+ # One request through the proxy, no tools: --print is enough and needs no
+ # permission mode.
+ interactive: false
+ timeout: 360s
+ capture: log
+
+scenarios:
+ - name: run-prompt
+ description: "The host calls run_prompt once with the model under test and relays the answer"
+ prompt: |
+ Call the claudish MCP tool `run_prompt` exactly once, with these arguments:
+
+ - model: {{model}}
+ - prompt: What is 4817 + 2906? Reply with the number only.
+
+ Use the model string exactly as written. Then reply with the text the
+ tool returned, verbatim. Do not compute the sum yourself and do not
+ call any other tool.
+ checks:
+ # Before any spend: the server started and answered tools/list.
+ - type: environment:mcp-reachable
+ value: claudish
+ # The agent's own startup record lists the server as connected.
+ # (environment:tool-available would say more, but it always ERRORS under
+ # the mock, so the negative control could never hold: docs/checks.md.)
+ - type: environment:mcp-connected
+ value: claudish
+ # Exactly one call: a retry on a second model could otherwise supply
+ # the answer the first model failed to give.
+ - type: session:tool-used
+ value: mcp__claudish__run_prompt
+ args: { eq: 1 }
+ # The call named the model under test, letter for letter.
+ - type: session:tool-args-match
+ value: mcp__claudish__run_prompt
+ args:
+ args: { model: "{{model}}" }
+ # THE detector: the answer is in the MCP tool's own result. The host can
+ # compute 7723 itself, but it cannot write a tool_result. A failed route
+ # returns "Error: ..." and no number.
+ - type: session:match
+ value: '\b7723\b'
+ args: { type: tool_result, regex: true }
diff --git a/ai-docs/benches/recommended-models/scripts/gen-experiments.ts b/ai-docs/benches/recommended-models/scripts/gen-experiments.ts
new file mode 100644
index 00000000..4939b84f
--- /dev/null
+++ b/ai-docs/benches/recommended-models/scripts/gen-experiments.ts
@@ -0,0 +1,91 @@
+#!/usr/bin/env bun
+/**
+ * Write cli.experiment.yaml and mcp.experiment.yaml from the recommended list.
+ *
+ * bun ai-docs/benches/recommended-models/scripts/gen-experiments.ts
+ *
+ * The list is read from `claudish --models-top --json`, never typed here: it
+ * changes with every catalog generation, and a hand-kept copy is the hardcoded
+ * model data CLAUDE.md forbids. Re-run this before a real run.
+ *
+ * Two kinds of variant, in this order:
+ * 1. every recommended model by its bare name, so routing chooses the hop the
+ * user would get (the first one is the Experiment's baseline);
+ * 2. every distinct subscription command the list publishes (`[email protected]`,
+ * `kc@k3`), so each subscription connection is exercised on its own. A row
+ * with no provider shortcut is the native route and is already in (1).
+ */
+import { writeFileSync } from "node:fs";
+import { join, resolve } from "node:path";
+
+interface Subscription {
+ prefix?: string;
+ command: string;
+}
+interface RecommendedModel {
+ id: string;
+ category: string;
+ subscriptions?: Subscription[];
+}
+interface RecommendedDoc {
+ generationId: string;
+ models: RecommendedModel[];
+}
+
+const BENCH_DIR = resolve(import.meta.dir, "..");
+
+const proc = Bun.spawnSync(["claudish", "--models-top", "--json"], { stderr: "inherit" });
+if (proc.exitCode !== 0) {
+ console.error(`claudish --models-top --json exited ${proc.exitCode}`);
+ process.exit(1);
+}
+const doc = JSON.parse(proc.stdout.toString()) as RecommendedDoc;
+
+const variants: Array<{ name: string; model: string }> = [];
+const seenModels = new Set<string>();
+const add = (model: string) => {
+ if (seenModels.has(model)) return;
+ seenModels.add(model);
+ // The variant is named by its model spec, `@` and all: madbench accepts it in
+ // the file and in `--variant '[email protected]'`, and the report then names the
+ // exact spec that was sent.
+ variants.push({ name: model, model });
+};
+
+for (const m of doc.models) if (m.category !== "subscription") add(m.id);
+for (const m of doc.models) {
+ if (m.category !== "subscription") continue;
+ for (const s of m.subscriptions ?? []) if (s.prefix) add(s.command);
+}
+
+const header = (bench: string) =>
+ [
+ `# GENERATED by scripts/gen-experiments.ts from \`claudish --models-top --json\`.`,
+ `# Catalog generation ${doc.generationId}, written ${new Date().toISOString()}.`,
+ `# Do not edit by hand: re-run the script. ${variants.length} variants.`,
+ `description: ${JSON.stringify(`${bench}: every recommended model and subscription command`)}`,
+ `bench: ./${bench}.madbench.yaml`,
+ ].join("\n");
+
+const variantLines = variants
+ .map((v) => ` - name: ${JSON.stringify(v.name)}\n params: { model: ${JSON.stringify(v.model)} }`)
+ .join("\n");
+
+// The CLI Bench passes the model as harness.model, so the variants differ there
+// and only there. The MCP Bench keeps harness.model fixed (the model under test
+// is in the prompt and the Checks), so its staged inputs never differ.
+const cli = `${header("cli")}
+guard_changes:
+ allow: [harness/model]
+variants:
+${variantLines}
+`;
+const mcp = `${header("mcp")}
+variants:
+${variantLines}
+`;
+
+writeFileSync(join(BENCH_DIR, "cli.experiment.yaml"), cli);
+writeFileSync(join(BENCH_DIR, "mcp.experiment.yaml"), mcp);
+console.log(`generation ${doc.generationId}: ${variants.length} variants`);
+for (const v of variants) console.log(` ${v.name}`);
diff --git a/ai-docs/benches/recommended-models/testdata/cli/slugify.test.ts b/ai-docs/benches/recommended-models/testdata/cli/slugify.test.ts
new file mode 100644
index 00000000..821ef134
--- /dev/null
+++ b/ai-docs/benches/recommended-models/testdata/cli/slugify.test.ts
@@ -0,0 +1,9 @@
+import { expect, test } from "bun:test";
+import { slugify } from "./slugify";
+
+test("slugify", () => {
+ expect(slugify("Hello World")).toBe("hello-world");
+ expect(slugify("Hello, World!")).toBe("hello-world");
+ expect(slugify(" Many spaces here ")).toBe("many-spaces-here");
+ expect(slugify("Version 2.0 -- Release")).toBe("version-2-0-release");
+});
diff --git a/ai-docs/benches/recommended-models/testdata/cli/slugify.ts b/ai-docs/benches/recommended-models/testdata/cli/slugify.ts
new file mode 100644
index 00000000..a94db143
--- /dev/null
+++ b/ai-docs/benches/recommended-models/testdata/cli/slugify.ts
@@ -0,0 +1,5 @@
+// Turn a title into a URL slug: lower case, ASCII letters and digits only,
+// words joined by single hyphens, no hyphen at either end.
+export function slugify(title: string): string {
+ return title.toLowerCase().replace(/ /g, "-");
+}