Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
test(madbench): add CLI and MCP benches for the recommended models
Jack Rudenkof96107326f82
diff --git a/ai-docs/benches/recommended-models/README.md b/ai-docs/benches/recommended-models/README.mdnew file mode 100644index 00000000..f7a5b478--- /dev/null+++ b/ai-docs/benches/recommended-models/README.md@@ -0,0 +1,83 @@+# recommended-models: does every recommended model work through claudish?++Two Benches, one question each, swept over the recommended list by two Experiments.++| Bench | Path under test | The model must |+|---|---|---|+| `cli.madbench.yaml` | `claudish --model <spec>`: Claude Code through the proxy, interactive | fix `slugify.ts` with real tool calls (Read or Bash, then Edit or Write) until `bun test` passes |+| `mcp.madbench.yaml` | native Claude Code → claudish MCP `run_prompt` → `runPromptViaProxy` | answer `4817 + 2906` inside the MCP tool result |++The CLI Bench is the stronger test: it exercises the wire, streaming tool calls and the+Claude Code TUI. `run_prompt` sends one text request with no tools, so the MCP Bench proves+that a route answers, not that the model can use tools.++## The variants come from the catalog++```+bun scripts/gen-experiments.ts+```++Every command in this README runs from this folder+(`ai-docs/benches/recommended-models/`). The Benches resolve their own paths against the+Bench file, and the script writes next to itself, so the folder is the only working+directory to remember.++writes `cli.experiment.yaml` and `mcp.experiment.yaml` from `claudish --models-top --json`:+first every recommended model by bare name (routing chooses the hop), then every distinct+subscription command the list publishes (`[email protected]`, `kc@k3`, ...). The list changes+with each catalog generation, so re-run the script before a real run. Each file names the+generation it was written from.++## Running++The free gates (no spend):++```+madbench preflight cli.madbench.yaml+madbench check cli.madbench.yaml+madbench preflight mcp.madbench.yaml+madbench check mcp.madbench.yaml+```++A real run, one variant first:++```+madbench cli.experiment.yaml --variant kimi-k3 --report-json <path>+madbench mcp.experiment.yaml --report-json <path>+```++Both Benches run the `claudish` on `PATH`. Check `claudish --version` first: a stale global+install tests the old release, not the one under review.++A graded miss exits 0. Add `--fail-on-failure` where a miss must go red.++## Why each choice++- **`sandbox: workspace`, not `home`.** `home` replaces `HOME`, which hides `~/.claudish`+ (config, 1Password and Keychain references, catalog cache). madbench's own example+ `local-agent-orchestration` measured 20/20 Scenarios failing as "agent never started"+ under `home`. The cost: the run shares this machine's `HOME`.+- **The CLI anti-cheat is inside the outcome Check.** "The four cases are still in the test+ file" passes against an agent that did nothing, so on its own it fails `madbench check`+ as WRONGLY PASSED. `bun test && grep ...` passes only for a real fix. It counts lines, so it is weak; MadAppGang/madbench#131 asks for a `file-changes` Check to replace it.+- **The MCP detector reads the tool result.** The host Claude can compute 7723 itself; it+ cannot write an MCP `tool_result`. `session:match` with `type: tool_result` reads only+ what claudish returned, and a failed route returns `Error: ...` with no number.+- **Exactly one `run_prompt` call.** A retry with a second model could supply the answer+ the first model failed to give.+- **The MCP server is declared inline in `args:`, not with `config.dirs`.** `config.dirs`+ needs sandbox level `home`, and claudish needs `workspace`. madbench 0.43.2 refuses that+ pair only at run time ("harness.config.dirs cannot apply at sandbox level workspace"),+ after `preflight` and `check` both passed (MadAppGang/madbench#130). The value is JSON, not a file path, because a+ path would resolve in the workspace. `--strict-mcp-config` keeps the user's installed+ claudish plugin server out of the run.+- **`environment:mcp-connected`, not `environment:tool-available`.** `tool-available`+ always errors under the mock, so the negative control could never hold+ (madbench `docs/checks.md`, the environment table).+- **The MCP Bench errors under `--harness mock`.** `environment:mcp-reachable` reads the+ run's MCP preflight, and the mock starts no server. `madbench check` lists it as not+ applicable; the negative control is the gate, not the mock run's exit code.+- **The MCP host model is fixed** (`claude-haiku-5-5`, billed to the subscription), so the+ variants differ only in the prompt and the Checks. That Experiment declares no+ `guard_changes:` because its staged inputs never differ; the CLI Experiment allows+ `harness/model` and nothing else.diff --git a/ai-docs/benches/recommended-models/cli.experiment.yaml b/ai-docs/benches/recommended-models/cli.experiment.yamlnew file mode 100644index 00000000..2300b1ea--- /dev/null+++ b/ai-docs/benches/recommended-models/cli.experiment.yaml@@ -0,0 +1,80 @@+# GENERATED by scripts/gen-experiments.ts from `claudish --models-top --json`.+# Catalog generation g-20261014030532231-748cc4dd, written 2026-10-14T03:36:04.684Z.+# Do not edit by hand: re-run the script. 36 variants.+description: "cli: every recommended model and subscription command"+bench: ./cli.madbench.yaml+guard_changes:+ allow: [harness/model]+variants:+ - name: "claude-opus-5-5"+ params: { model: "claude-opus-5-5" }+ - name: "gpt-6.1-sol"+ params: { model: "gpt-6.1-sol" }+ - name: "kimi-k3"+ params: { model: "kimi-k3" }+ - name: "grok-4.7"+ params: { model: "grok-4.7" }+ - name: "gemini-4-argon"+ params: { model: "gemini-4-argon" }+ - name: "qwen3.8-max-0902"+ params: { model: "qwen3.8-max-0902" }+ - name: "glm-5.3"+ params: { model: "glm-5.3" }+ - name: "minimax-m3"+ params: { model: "minimax-m3" }+ - name: "deepseek-v3.2-exp"+ params: { model: "deepseek-v3.2-exp" }+ - name: "claude-haiku-5-5"+ params: { model: "claude-haiku-5-5" }+ - name: "deepseek-v4.1-flash"+ params: { model: "deepseek-v4.1-flash" }+ - name: "gemini-3.5-flash-lite"+ params: { model: "gemini-3.5-flash-lite" }+ - name: "mistral-small-2603"+ params: { model: "mistral-small-2603" }+ - name: "gpt-5.4-nano-2026-03-17"+ params: { model: "gpt-5.4-nano-2026-03-17" }+ - name: "qwen3.8-omni-flash"+ params: { model: "qwen3.8-omni-flash" }+ - name: "grok-build-0.1"+ params: { model: "grok-build-0.1" }+ - name: "glm-5.3-flash"+ params: { model: "glm-5.3-flash" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "kc@k3"+ params: { model: "kc@k3" }+ - name: "oc@kimi-k3"+ params: { model: "oc@kimi-k3" }+ - name: "zengo@kimi-k3"+ params: { model: "zengo@kimi-k3" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "mmc@MiniMax-M3"+ params: { model: "mmc@MiniMax-M3" }+ - name: "oc@minimax-m3"+ params: { model: "oc@minimax-m3" }+ - name: "zengo@minimax-m3"+ params: { model: "zengo@minimax-m3" }+ - name: "zengo@claude-haiku-5-5"+ params: { model: "zengo@claude-haiku-5-5" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "zengo@deepseek-flash"+ params: { model: "zengo@deepseek-flash" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }diff --git a/ai-docs/benches/recommended-models/cli.madbench.yaml b/ai-docs/benches/recommended-models/cli.madbench.yamlnew file mode 100644index 00000000..be16a42f--- /dev/null+++ b/ai-docs/benches/recommended-models/cli.madbench.yaml@@ -0,0 +1,55 @@+description: "claudish CLI: can a recommended model fix a failing test through Claude Code?"++# One model spec per run. `cli.experiment.yaml` sweeps it over the recommended+# list; `gen-experiments.ts` writes that file from `claudish --models-top --json`.+params:+ model: kimi-k3++harness:+ type: claude-code+ # The program under test. madbench refuses a non-Anthropic model unless the+ # binary is claudish, and `claudish --model <spec>` is the user's own path:+ # routing, the wire built from the catalog, the stream parser, tool calls.+ binary: claudish+ model: "{{model}}"+ # Interactive (the default): Bash parks at an approval menu nobody answers+ # unless the mode is bypassPermissions. claudish forwards the flag to claude.+ args: ["--permission-mode", "bypassPermissions"]++defaults:+ # workspace, not home. Level `home` replaces HOME, which hides ~/.claudish+ # (config, 1Password and Keychain references, catalog cache). The madbench+ # example local-agent-orchestration measured 20/20 Scenarios failing as+ # "agent never started" under `home`.+ sandbox: workspace+ # A reasoning model on a cold catalog can need the 30 s catalog wait plus+ # several turns. Tighten after 2-3 real runs.+ timeout: 480s+ capture: log++scenarios:+ - name: fix-slugify+ description: "Read a failing test, edit the source, run the test"+ testdata: ./testdata/cli+ prompt: |+ The test in slugify.test.ts fails. Fix slugify.ts so that `bun test`+ passes. Do not change slugify.test.ts. Run `bun test` to confirm.+ checks:+ # The outcome, with its anti-cheat in the same Check: `bun test` passes AND+ # the four cases are still there. Apart, the anti-cheat half passes under+ # a no-op agent and `madbench check` reports it WRONGLY PASSED.+ - type: exec+ args:+ cmd: [sh, -c, 'bun test && test "$(grep -c "expect(slugify" slugify.test.ts)" -ge 4']+ expected_exit: 0+ timeout: 60s+ # The model drove tools through the proxy: it read the code...+ - type: any-of+ checks:+ - { type: session:tool-used, value: Read }+ - { type: session:tool-used, value: Bash }+ # ...and changed it with a file tool.+ - type: any-of+ checks:+ - { type: session:tool-used, value: Edit }+ - { type: session:tool-used, value: Write }diff --git a/ai-docs/benches/recommended-models/mcp.experiment.yaml b/ai-docs/benches/recommended-models/mcp.experiment.yamlnew file mode 100644index 00000000..4510ea1b--- /dev/null+++ b/ai-docs/benches/recommended-models/mcp.experiment.yaml@@ -0,0 +1,78 @@+# GENERATED by scripts/gen-experiments.ts from `claudish --models-top --json`.+# Catalog generation g-20261014030532231-748cc4dd, written 2026-10-14T03:36:04.684Z.+# Do not edit by hand: re-run the script. 36 variants.+description: "mcp: every recommended model and subscription command"+bench: ./mcp.madbench.yaml+variants:+ - name: "claude-opus-5-5"+ params: { model: "claude-opus-5-5" }+ - name: "gpt-6.1-sol"+ params: { model: "gpt-6.1-sol" }+ - name: "kimi-k3"+ params: { model: "kimi-k3" }+ - name: "grok-4.7"+ params: { model: "grok-4.7" }+ - name: "gemini-4-argon"+ params: { model: "gemini-4-argon" }+ - name: "qwen3.8-max-0902"+ params: { model: "qwen3.8-max-0902" }+ - name: "glm-5.3"+ params: { model: "glm-5.3" }+ - name: "minimax-m3"+ params: { model: "minimax-m3" }+ - name: "deepseek-v3.2-exp"+ params: { model: "deepseek-v3.2-exp" }+ - name: "claude-haiku-5-5"+ params: { model: "claude-haiku-5-5" }+ - name: "deepseek-v4.1-flash"+ params: { model: "deepseek-v4.1-flash" }+ - name: "gemini-3.5-flash-lite"+ params: { model: "gemini-3.5-flash-lite" }+ - name: "mistral-small-2603"+ params: { model: "mistral-small-2603" }+ - name: "gpt-5.4-nano-2026-03-17"+ params: { model: "gpt-5.4-nano-2026-03-17" }+ - name: "qwen3.8-omni-flash"+ params: { model: "qwen3.8-omni-flash" }+ - name: "grok-build-0.1"+ params: { model: "grok-build-0.1" }+ - name: "glm-5.3-flash"+ params: { model: "glm-5.3-flash" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "kc@k3"+ params: { model: "kc@k3" }+ - name: "oc@kimi-k3"+ params: { model: "oc@kimi-k3" }+ - name: "zengo@kimi-k3"+ params: { model: "zengo@kimi-k3" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "mmc@MiniMax-M3"+ params: { model: "mmc@MiniMax-M3" }+ - name: "oc@minimax-m3"+ params: { model: "oc@minimax-m3" }+ - name: "zengo@minimax-m3"+ params: { model: "zengo@minimax-m3" }+ - name: "zengo@claude-haiku-5-5"+ params: { model: "zengo@claude-haiku-5-5" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "zengo@deepseek-flash"+ params: { model: "zengo@deepseek-flash" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }+ - name: "[email protected]"+ params: { model: "[email protected]" }diff --git a/ai-docs/benches/recommended-models/mcp.madbench.yaml b/ai-docs/benches/recommended-models/mcp.madbench.yamlnew file mode 100644index 00000000..049cda45--- /dev/null+++ b/ai-docs/benches/recommended-models/mcp.madbench.yaml@@ -0,0 +1,70 @@+description: "claudish MCP: does run_prompt get an answer from a recommended model?"++# One model spec per run. `mcp.experiment.yaml` sweeps it over the recommended+# list; `gen-experiments.ts` writes that file from `claudish --models-top --json`.+params:+ model: kimi-k3++harness:+ type: claude-code+ # The host is native Claude Code. The model under test is reached ONLY+ # through the claudish MCP server, so the Harness model stays fixed across+ # every variant and the guard has nothing to audit.+ model: claude-haiku-5-5+ # The server is declared inline, not through config.dirs: config.dirs needs+ # sandbox level `home`, and claudish needs `workspace` (madbench refuses the+ # pair at run time, after preflight and check both pass). Inline JSON, not a+ # file path, because a path would resolve in the workspace.+ # --strict-mcp-config keeps the user's installed claudish plugin server out.+ args:+ - --mcp-config+ - '{"mcpServers":{"claudish":{"command":"claudish","args":["--mcp"]}}}'+ - --strict-mcp-config++defaults:+ # workspace, not home: the MCP server is `claudish --mcp`, and it needs the+ # real ~/.claudish for its config, credentials and catalog cache.+ sandbox: workspace+ # One request through the proxy, no tools: --print is enough and needs no+ # permission mode.+ interactive: false+ timeout: 360s+ capture: log++scenarios:+ - name: run-prompt+ description: "The host calls run_prompt once with the model under test and relays the answer"+ prompt: |+ Call the claudish MCP tool `run_prompt` exactly once, with these arguments:++ - model: {{model}}+ - prompt: What is 4817 + 2906? Reply with the number only.++ Use the model string exactly as written. Then reply with the text the+ tool returned, verbatim. Do not compute the sum yourself and do not+ call any other tool.+ checks:+ # Before any spend: the server started and answered tools/list.+ - type: environment:mcp-reachable+ value: claudish+ # The agent's own startup record lists the server as connected.+ # (environment:tool-available would say more, but it always ERRORS under+ # the mock, so the negative control could never hold: docs/checks.md.)+ - type: environment:mcp-connected+ value: claudish+ # Exactly one call: a retry on a second model could otherwise supply+ # the answer the first model failed to give.+ - type: session:tool-used+ value: mcp__claudish__run_prompt+ args: { eq: 1 }+ # The call named the model under test, letter for letter.+ - type: session:tool-args-match+ value: mcp__claudish__run_prompt+ args:+ args: { model: "{{model}}" }+ # THE detector: the answer is in the MCP tool's own result. The host can+ # compute 7723 itself, but it cannot write a tool_result. A failed route+ # returns "Error: ..." and no number.+ - type: session:match+ value: '\b7723\b'+ args: { type: tool_result, regex: true }diff --git a/ai-docs/benches/recommended-models/scripts/gen-experiments.ts b/ai-docs/benches/recommended-models/scripts/gen-experiments.tsnew file mode 100644index 00000000..4939b84f--- /dev/null+++ b/ai-docs/benches/recommended-models/scripts/gen-experiments.ts@@ -0,0 +1,91 @@+#!/usr/bin/env bun+/**+ * Write cli.experiment.yaml and mcp.experiment.yaml from the recommended list.+ *+ * bun ai-docs/benches/recommended-models/scripts/gen-experiments.ts+ *+ * The list is read from `claudish --models-top --json`, never typed here: it+ * changes with every catalog generation, and a hand-kept copy is the hardcoded+ * model data CLAUDE.md forbids. Re-run this before a real run.+ *+ * Two kinds of variant, in this order:+ * 1. every recommended model by its bare name, so routing chooses the hop the+ * user would get (the first one is the Experiment's baseline);+ * 2. every distinct subscription command the list publishes (`[email protected]`,+ * `kc@k3`), so each subscription connection is exercised on its own. A row+ * with no provider shortcut is the native route and is already in (1).+ */+import { writeFileSync } from "node:fs";+import { join, resolve } from "node:path";++interface Subscription {+ prefix?: string;+ command: string;+}+interface RecommendedModel {+ id: string;+ category: string;+ subscriptions?: Subscription[];+}+interface RecommendedDoc {+ generationId: string;+ models: RecommendedModel[];+}++const BENCH_DIR = resolve(import.meta.dir, "..");++const proc = Bun.spawnSync(["claudish", "--models-top", "--json"], { stderr: "inherit" });+if (proc.exitCode !== 0) {+ console.error(`claudish --models-top --json exited ${proc.exitCode}`);+ process.exit(1);+}+const doc = JSON.parse(proc.stdout.toString()) as RecommendedDoc;++const variants: Array<{ name: string; model: string }> = [];+const seenModels = new Set<string>();+const add = (model: string) => {+ if (seenModels.has(model)) return;+ seenModels.add(model);+ // The variant is named by its model spec, `@` and all: madbench accepts it in+ // the file and in `--variant '[email protected]'`, and the report then names the+ // exact spec that was sent.+ variants.push({ name: model, model });+};++for (const m of doc.models) if (m.category !== "subscription") add(m.id);+for (const m of doc.models) {+ if (m.category !== "subscription") continue;+ for (const s of m.subscriptions ?? []) if (s.prefix) add(s.command);+}++const header = (bench: string) =>+ [+ `# GENERATED by scripts/gen-experiments.ts from \`claudish --models-top --json\`.`,+ `# Catalog generation ${doc.generationId}, written ${new Date().toISOString()}.`,+ `# Do not edit by hand: re-run the script. ${variants.length} variants.`,+ `description: ${JSON.stringify(`${bench}: every recommended model and subscription command`)}`,+ `bench: ./${bench}.madbench.yaml`,+ ].join("\n");++const variantLines = variants+ .map((v) => ` - name: ${JSON.stringify(v.name)}\n params: { model: ${JSON.stringify(v.model)} }`)+ .join("\n");++// The CLI Bench passes the model as harness.model, so the variants differ there+// and only there. The MCP Bench keeps harness.model fixed (the model under test+// is in the prompt and the Checks), so its staged inputs never differ.+const cli = `${header("cli")}+guard_changes:+ allow: [harness/model]+variants:+${variantLines}+`;+const mcp = `${header("mcp")}+variants:+${variantLines}+`;++writeFileSync(join(BENCH_DIR, "cli.experiment.yaml"), cli);+writeFileSync(join(BENCH_DIR, "mcp.experiment.yaml"), mcp);+console.log(`generation ${doc.generationId}: ${variants.length} variants`);+for (const v of variants) console.log(` ${v.name}`);diff --git a/ai-docs/benches/recommended-models/testdata/cli/slugify.test.ts b/ai-docs/benches/recommended-models/testdata/cli/slugify.test.tsnew file mode 100644index 00000000..821ef134--- /dev/null+++ b/ai-docs/benches/recommended-models/testdata/cli/slugify.test.ts@@ -0,0 +1,9 @@+import { expect, test } from "bun:test";+import { slugify } from "./slugify";++test("slugify", () => {+ expect(slugify("Hello World")).toBe("hello-world");+ expect(slugify("Hello, World!")).toBe("hello-world");+ expect(slugify(" Many spaces here ")).toBe("many-spaces-here");+ expect(slugify("Version 2.0 -- Release")).toBe("version-2-0-release");+});diff --git a/ai-docs/benches/recommended-models/testdata/cli/slugify.ts b/ai-docs/benches/recommended-models/testdata/cli/slugify.tsnew file mode 100644index 00000000..a94db143--- /dev/null+++ b/ai-docs/benches/recommended-models/testdata/cli/slugify.ts@@ -0,0 +1,5 @@+// Turn a title into a URL slug: lower case, ASCII letters and digits only,+// words joined by single hyphens, no hyphen at either end.+export function slugify(title: string): string {+ return title.toLowerCase().replace(/ /g, "-");+}