Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
test(bench): add the recommended-effort madbench
Jack Rudenkoe3cc763f941c
diff --git a/ai-docs/benches/recommended-effort/README.md b/ai-docs/benches/recommended-effort/README.mdnew file mode 100644index 00000000..5c6fe8f1--- /dev/null+++ b/ai-docs/benches/recommended-effort/README.md@@ -0,0 +1,137 @@+# recommended-effort: does the caller pass the recommended effort, and does it reach the wire?++The feature (`ai-docs/architecture/wire.md` § Recommended effort): MCP `list_models` appends a+"Recommended effort by task type" table read from+`~/.claudish/cloud-models-catalog-v3.efforts.json`. The calling agent maps its own task to a+column and passes one effort per model: `team` `efforts: {"<model>": "<level>"}`,+`create_session` `effort`, `run_prompt` `effort`. claudish does not choose a task type.++One Bench, `madbench.yaml`, four Scenarios, and one Experiment, `effort.experiment.yaml`, with+three variants. The prompts never mention effort (`hint` is empty).++| Variant | Staged claudish plugin (`harness.config.plugins`) | How the user names models |+|---|---|---|+| `installed-skill` (baseline) | `plugins/installed/claudish`: the installed `claudish@magus` 2.10.0 `claudish-usage` skill, no effort section | exact ids |+| `worktree-skill` | `plugins/worktree/claudish`: a copy of this worktree's `packages/cli/skills/claudish-usage/SKILL.md` | exact ids |+| `worktree-family` | the same worktree copy | families only ("two different OpenAI GPT models", "a GPT model and a MiniMax model"); no MiniMax model has a row, so its effort must still be omitted |++The two plugin folders hold the same `plugin.json` and differ only in `SKILL.md`, so the guard+(`allow: [harness/config/plugins]`) sees one changed file. **Refresh the worktree copy before+every run** (`cp packages/cli/skills/claudish-usage/SKILL.md+ai-docs/benches/recommended-effort/plugins/worktree/claudish/skills/claudish-usage/SKILL.md`).+It is a real copy because madbench copies a symlink as a symlink, and at `home` the read+confinement would deny its target. Staging replaces the host's plugin set, so neither variant+loads the installed plugin's hooks or any other host plugin. `config.plugins` needs sandbox+level `home`, so the run's real `~/.claudish` is shared back in with `share.paths` (`rw`).++Levels in the table below are generation `g-20261022002624894-85e83b1a`; the Checks read the+generation live, so they stay right when the table changes. The host is native Claude Code+(`{{host}}`, default `claude-sonnet-5-5`, billed to the subscription) with only the claudish MCP+server of THIS worktree's build. The prompts are written like a user's request: they name the+task and (except one) the models, and never mention effort or `list_models`. Whether the agent+finds the table from the tool descriptions and `list_models` output alone is the measurement.+The installed `claudish-usage` skill predates the feature and is not changed.++| Scenario | Tool | Task-type column | Models |+|---|---|---|---|+| `team-frontend` | `team` run | `frontend_ui` | `gpt-6-astra` (xhigh), `gpt-6.1-sol` (high); family variant: two GPT models |+| `session-code-review` | `create_session` | `code_review` | `gpt-6-luna` (xhigh); family variant: a GPT model |+| `run-prompt-pick` | `run_prompt` | `implementation` | the agent picks one recommended OpenAI GPT model (`gpt-6-luna` max, `gpt-6-astra` high, `gpt-6.1-sol` medium) |+| `team-no-entry` | `team` run | `quick_edit` | `gpt-6-astra` (low) and `minimax-m3`, which has NO row: its effort must be omitted; family variant: a GPT and a MiniMax model |++The models are Codex subscription (`cx@`) and OpenCode Go (`zengo@minimax-m3`) routes, checked+live with `bun packages/cli/dist/index.js --probe <model> --json` before the run. All Codex+connections use the Responses API, which echoes the request's `reasoning.effort` in+`response.created`; that echo is the Layer B wire evidence (below).++## What each Check proves++The effort, bare-name and layer-B Checks grade the call that STARTED work: the last start call answered neither with `needs_effort` nor `Error:` (`startedCall`, `checks/effort-lib.ts`). `ConsultedTable` is a readout: `needs_effort` also hands the agent the rows.++Two layers, kept as separate Checks so one can be read without the other.++| Check (metric) | Layer | Proves | Does not prove |+|---|---|---|---|+| `environment:mcp-reachable` | setup | the worktree's MCP server started in the run's sandbox and answered madbench | anything about the agent |+| `environment:skill-registered` `claudish:claudish-usage` | setup | Claude Code registered the staged plugin's skill | which text it holds (the guard diff and the staged folder say that) |+| `session:skill-used` `claudish:claudish-usage` (`SkillUsed`, readout) | A, readout | the agent opened the skill | nothing about the verdict |+| `ts` `checks/needs-effort.ts` (`NeedsEffort`, readout) | A, readout | how many start calls claudish answered with `needs_effort` before the call that started work, whether that call carries the Scenario column's levels, and every column its levels are consistent with | why the agent chose the column |+| `ts` `checks/bare-names.ts` (`BareNames`) | A | every model in the first start call is a bare name: no `provider@` (no prompt names a provider) and no aggregator-qualified `vendor/model` id | that routing chose a subscription |+| `session:tool-sequence` [`list_models`, start tool] (`ConsultedTable`) | A | the agent read `list_models` (where the table is) before it started the work | that it read the table, or the right column |+| `ts` `checks/effort-args.ts` (`EffortArgs`) | A | the FIRST start call passed, for every model it ran, exactly the table entry of the Scenario's column in the newest sidecar generation, and no effort where the table has no entry; for named-model Scenarios, it ran exactly the named models. Score = fraction of models right | why it chose (the reasoning is not graded) |+| `ts` `checks/effort-plumbing.ts` stage `flag` (`EffortFlag`) | B | for each (model, level) the agent passed, a child claudish wrote `[ComposedHandler] --effort <level> pinned for <model>`: the child got `--effort-override <level>` | the request body; see `wire` |+| `ts` `checks/effort-plumbing.ts` stage `wire` (`EffortWire`) | B | the provider's own `response.created` event, logged raw as `[SSE:responses]` in the same process's debug log, echoes `"reasoning":{"effort":"<sent>"}` where `<sent>` is the level, or the level a `[Wire] … effort A → B` note says step 5w mapped it to | anything for a connection that is not the Responses API (no echo) |++Layer B grades whatever effort the agent passed, right or wrong, so a Layer A miss does not hide+a working pipe. The Checks pin no level: they read the newest generation of the sidecar+at grading time, which is the generation `list_models` rendered (`latestEffortRecommendations`+picks the largest `writtenAt`; the Check does the same) and record its id in Evidence.++## Where Layer B's evidence comes from++The MCP server is declared inline with two environment variables:++- `CLAUDISH_BIN=<dist/index.js>`: `team` and `create_session` children run this build+ (`spawn-claudish.ts`), not the `claudish` on PATH.+- `CLAUDISH_DEBUG=1`: the server (`mcpDebugLogEnabled`) and every child (inherited through+ `buildPaneEnv`) write a debug log to `<cwd>/logs/claudish_<time>_<pid>.log`. The cwd of the+ server and of every slot and session is the workspace, so `Session.work_dir/logs/` holds one+ file per process. `run_prompt` runs in the server's own proxy, so its evidence is in the+ server's file; a child's is in the child's file, matched by its pin line.++claudish itself names the effort in a `[Wire]` line only when step 5w changes it (`effort A → B`,+`wire-request.ts` `applyReasoning`); a level the connection publishes is sent unchanged and+logged nowhere. That is why the wire stage reads the provider's echo.++## Running++From the worktree root, after `bun run build`:++```+madbench preflight ai-docs/benches/recommended-effort/madbench.yaml+madbench check ai-docs/benches/recommended-effort/madbench.yaml+madbench ai-docs/benches/recommended-effort/madbench.yaml --report-json <path>+madbench ai-docs/benches/recommended-effort/effort.experiment.yaml --trials 2 --report-json <path>+```++`claudish_js` is an absolute path into this worktree; override it with+`--param claudish_js=<path>` elsewhere. The Bench errors under `--harness mock` (the mock starts+no MCP server, so `environment:mcp-reachable` has nothing to read); `madbench check` lists that+Check as not applicable and is the gate. A graded miss exits 0.++## Why each choice++- `sandbox: home` with `~/.claudish` shared `rw`: `config.plugins` is refused below `home`, and+ `home` hides `~/.claudish` (credentials, catalog, the effort sidecar) unless it is shared. Not+ yet run at this level: macOS Keychain-held keys (for example `OPENCODE_API_KEY` for+ `zengo@minimax-m3`) may not resolve with HOME replaced. The Codex OAuth credential is a file in+ `~/.claudish` and is shared.+- `interactive: true` for the host, with `--permission-mode bypassPermissions`: only an+ interactive run gets madbench's first-run seeding (onboarding, workspace trust) in the+ sandbox's `~/.claude.json`, and the team / session children are interactive Claude Code in the+ same HOME and cwd. Under `--print` a child stopped at the theme picker (`first_run_dialog`).+ `environment:mcp-connected` is not used: on the interactive path it reports "the tool lists no+ MCP server named" although the server works; `environment:mcp-reachable` plus the+ `mcp__claudish__*` calls are the evidence.+- `--allowedTools mcp__claudish`: at `home` the user's own permission settings are gone, and the+ host was refused `mcp__claudish__create_session` ("you haven't granted it yet").+- What `home` needed besides `~/.claudish` (each found by a one-Session probe; see+ `madbench-skill-gap.md`): an absolute MCP `command` (`bun_bin`), live shares of `~/.bun`,+ `~/.local/bin` and `~/.local/share/claude` (the children's `claude`), of+ `<root>/packages/cli` and `<root>/package.json` and `<root>/node_modules` (the bundle imports+ `@opentui/*` at run time), and a dedicated pane socket root (`pane_root`, created once with+ `mkdir -m 700 -p /private/tmp/claudish-bench-mux`) passed as `CLAUDISH_PANE_ROOT`. `ro` shares+ are refused below `container`, so all shares are `rw`.+- Keychain-held keys do not resolve at `home`: `minimax-m3` routed to OpenRouter (metered), not+ `zengo@`. A model whose only funded route is a Keychain key is not measured on its subscription.+- `--strict-mcp-config`: keeps any plugin's MCP server out of the run; only the worktree build serves `mcp__claudish__*`.+- The named models have different levels in their column where possible (`gpt-6-astra` xhigh+ vs `gpt-6.1-sol` high for `frontend_ui`) so a single guessed level cannot pass for both.+- The columns were chosen for tasks with one obvious reading. A pilot run graded a "shell+ one-liner" task as `terminal_ops`; the agent read it as `quick_edit`. That is a fair reading,+ so the pick Scenario now asks for an implementation instead.+- `run-prompt-pick` no longer tells the agent to use the bare name; `BareNames` measures it. In+ the pilot the agent took `oai@gpt-6-luna` from `list_models`' Access line, and the metered+ OpenAI account had no credit.+- Levels were chosen away from `high` where possible, so the wire echo cannot be Claude Code's+ own default level passed through.diff --git a/ai-docs/benches/recommended-effort/RESULTS.md b/ai-docs/benches/recommended-effort/RESULTS.mdnew file mode 100644index 00000000..125c4e3e--- /dev/null+++ b/ai-docs/benches/recommended-effort/RESULTS.md@@ -0,0 +1,844 @@+# recommended-effort: results, 2026-10-23++## run11: rebased on v14.1.0, corrected started-call rule (2026-10-23, 05:54:37–06:00:31Z)++Same Experiment and procedure (`--trials 2`, 24 Sessions, interactive `claude-sonnet-5-5` host,+`sandbox: home`, plugin staged by `harness.config.plugins`). Build: the coordinator's rebuild on+origin/main, `bun packages/cli/dist/index.js --version` → `claudish version 14.1.0` (needs_effort+keys in the caller's spelling for run_prompt/create_session; a rejected recommended cache of another+projection version logged as `[recommended] … has projection version …`; the projection-2 disk+cache readable). Before the run: the worktree skill copy refreshed and `cmp`-identical; the shared+`packages/cli`, root `package.json` and root `node_modules` resolve after the rebase+(`packages/cli/node_modules/@opentui/core` → `node_modules/.bun/@[email protected]+…/…`,+`package.json` present); preflight `PREFLIGHT OK 3 benches · 150 components verified`; check+`93/93 gradable checks failed as required · 0 errored (could not grade) · 0 wrongly passed · 12 not applicable under mock`.+Grading uses the corrected rule: the effort, bare-name and layer-B Checks grade the LAST start call+answered neither with needs_effort nor `Error:`. Effort generation at grading:+`g-20261023054016830-cce2d5b3`.++Report: `ai-docs/sessions/recommended-effort-bench-20261023/reports/run11.json` (gitignored).+Cost: host $2.45 (subscription). Trials 184 s and 161 s.++### Scenario verdicts (`.aggregate.perScenario`)++| Scenario | `installed-skill` | `worktree-skill` | `worktree-family` |+|---|---|---|---|+| `team-frontend` | 2/2 | 2/2 | 2/2 |+| `session-code-review` | 2/2 | 2/2 | 2/2 |+| `run-prompt-pick` | 2/2 | 2/2 | 2/2 |+| `team-no-entry` | 2/2 | 1/2 | 2/2 |+| total | 8/8 | 7/8 | 8/8 |++### Per check++| Check | `installed-skill` | `worktree-skill` | `worktree-family` |+|---|---|---|---|+| `list_models` called first (`ConsultedTable`, readout) | 2/8 | 2/8 | 8/8 |+| efforts equal the table on the started call (`EffortArgs`) | 8/8 | 7/8 | 8/8 |+| bare names on the started call (`BareNames`) | 8/8 | 8/8 | 8/8 |+| skill opened (`SkillUsed`, readout) | 2/8 | 0/8 | 0/8 |+| needs_effort answers before the started call | 6 (6 Sessions) | 5 (5 Sessions) | 0 |+| after needs_effort, the retry carried the Scenario column's levels | 6/6 | 5/5 | n/a |+| `NeedsEffort` readout | 8/8 | 7/8 | 8/8 |+| B1 child pinned the level (`EffortFlag`) | 6/6 | 5/6 | 6/6 |+| B2 provider echoed the level (`EffortWire`) | 8/8 | 7/8 | 8/8 |++The retried levels were consistent with `frontend_ui` (4), `code_review`+`frontend_ui` (4) and+`quick_edit`+`terminal_ops`+`long_context` (3): each set includes the Scenario's own column. In 23+Sessions the started call passed an effort; all 29 (model, level) pairs were pinned (team/session)+and echoed by the provider at that level. No mapping note (`effort A → B`) appeared.++For create_session the needs_effort text now speaks the singular form, e.g. `worktree-skill`/`session-code-review` trial 1:++```+mcp__claudish__create_session {"model":"gpt-6-luna"}+ → needs_effort: nothing was started. Map your task to one task type and call again with effort: "<level>" (effort: "default" keeps Claude Code's own).+ Recommended levels: {"gpt-6-luna": {"implementation": "max", "planning": "max", "research": "max", "code_review": "xhigh", "debugging": "high", "quick_edit": "low", "frontend_ui": "xhigh", "tool_automation": "max", …}}+mcp__claudish__create_session {"effort":"xhigh","model":"gpt-6-luna"}+ → {"session_id":"e8dc4cf8","state":"STARTING"}+```++### Tool calls, verbatim++Passing, family names, `worktree-family`/`team-no-entry` trial 1:++```+mcp__claudish__list_models {}+mcp__claudish__team {"efforts":{"gpt-6.1-sol":"low"},"mode":"run","models":["gpt-6.1-sol","minimax-m3"],"path":"team-typo"}+```++The one failure, `worktree-skill`/`team-no-entry` trial 2: the agent's first and only start call+used the needs_effort **opt-out** unprompted, so claudish started the run with Claude Code's own+effort for both models:++```+mcp__claudish__team {"efforts":{},"mode":"run","models":["gpt-6-astra","minimax-m3"],"path":"team-typo"}+ → {"started": true, "run_id": "team-typo-mvkk7qdz-ba6418", …}+```++`EffortArgs`: "1/2 models right for column quick_edit; gpt-6-astra: passed none, recommended low".+The agent did not open the skill (`SkillUsed` 0/8 in this variant) and said nothing about effort+in its answer; the `efforts: {}` spelling is in the `team` tool's own description and in the+needs_effort text (`efforts: {} keeps Claude Code's own for every model`). The opt-out is+deliberate API, so the agent was free to pass it; for a quick edit it also cost little (the+recommended level was `low`). Whether to keep `{}` as a silent opt-out, or require a reason, is a+product decision.++### Layer B evidence, verbatim++`worktree-skill`/`team-frontend`, trial 1 (after needs_effort):++```+[2026-10-23T05:56:10.410Z] [ComposedHandler] --effort xhigh pinned for cx@gpt-6-astra (catalog clamp skipped)+[2026-10-23T05:56:10.410Z] [Wire] openai-codex@gpt-6-astra → openai/codex-subscription · gpt-6-astra: responses (credential-decided host) [generation g-20261023054016830-cce2d5b3]+[2026-10-23T05:56:12.384Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+[2026-10-23T05:56:10.284Z] [ComposedHandler] --effort high pinned for [email protected] (catalog clamp skipped)+[2026-10-23T05:56:10.284Z] [Wire] [email protected] → openai/codex-subscription · gpt-6.1-sol: responses (credential-decided host) [generation g-20261023054016830-cce2d5b3]+[2026-10-23T05:56:12.520Z] [SSE:responses] response.created … "reasoning":{"effort":"high","summary":"detailed","context":"all_turns","mode":"standard"}+```++`worktree-skill`/`session-code-review`, trial 1:++```+[2026-10-23T05:55:56.186Z] [ComposedHandler] --effort xhigh pinned for cx@gpt-6-luna (catalog clamp skipped)+[2026-10-23T05:55:56.186Z] [Wire] openai-codex@gpt-6-luna → openai/codex-subscription · gpt-6-luna: responses (credential-decided host) [generation g-20261023054016830-cce2d5b3]+[2026-10-23T05:55:57.237Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+```++`worktree-family`/`run-prompt-pick`, trial 1 (run_prompt `effort: "medium"`):++```+[2026-10-23T05:56:51.593Z] [Wire] [email protected] → openai/codex-subscription · gpt-6.1-sol: responses (credential-decided host) [generation g-20261023054016830-cce2d5b3]+[2026-10-23T05:56:53.654Z] [SSE:responses] response.created … "reasoning":{"effort":"medium","summary":"detailed","context":"all_turns","mode":"standard"}+```++### `list_models`, the disk fallback and projection lines++- 12 `list_models` calls, **12 live lists with the effort table**; 0 `No recommended models found`;+ **0 disk-fallback notes** (`_The live catalog could not be read; …_`). No fetch failed while an+ agent was waiting on `list_models`, so the fallback path was not exercised in this run.+- **0 `[recommended] … has projection version …` lines.**++### claudish-mcp log lines written during the run (48 files, 05:54–06:00Z)++All `[mcp]`/`[catalog]`/`[recommended]`/`[effort]`/`[team]` lines, verbatim:++```+[2026-10-23T05:58:00.257Z] [recommended] warm failed: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:58:03.257Z] [catalog] refresh failed: timeout+```++### Not verified++- The disk fallback on the v14.1.0 build (no `list_models` call hit a failed fetch).+- That the `minimax-m3` children ran without a pinned effort; `minimax-m3` still ran on+ OpenRouter at `home` (Keychain keys unavailable). Any host but `claude-sonnet-5-5`.++## run10: needs_effort, no Access line (2026-10-23, 05:36:57–05:43:32Z)++Same Experiment and procedure as run9 (`--trials 2`, 24 Sessions, interactive host+`claude-sonnet-5-5`, `sandbox: home`, plugin staged by `harness.config.plugins`). Build: the+coordinator's `packages/cli/dist/index.js` with needs_effort (a start call without+`efforts`/`effort` for a model that has a row starts nothing and returns the rows), no provider in+`list_models`, 500 kept `claudish-mcp_*.log` files. Worktree skill copy `cmp`-identical.+Effort generation at grading: `g-20261023053138920-f8b4cbb5` (new since run9; the levels used here+are unchanged).++Report: `ai-docs/sessions/recommended-effort-bench-20261023/reports/run10.json` (gitignored).+Cost: host $2.36 (subscription). Trials 169 s and 220 s.++Check changes for this run: the effort, bare-name and layer-B Checks grade the call that+**started** work, not the first call; `ConsultedTable` is a readout (measurement); a new readout+`checks/needs-effort.ts` (`NeedsEffort`) counts needs_effort answers before the started call,+scores whether that call carries the Scenario column's levels, and records every column the+levels are consistent with. Gates: preflight `PREFLIGHT OK 3 benches · 150 components verified`;+check `93/93 gradable checks failed as required · 0 errored (could not grade) · 0 wrongly passed · 12 not applicable under mock`.++**One grading caveat.** run10 picked "the first start call not answered with needs_effort". In+`worktree-family`/`team-no-entry` trial 2 that was a run of the literal family names+(`models: ["gpt","minimax"]`), not the agent's working retry, so that Session's layer-B Checks+graded the wrong call. The rule is now "the last start call answered neither with needs_effort+nor `Error:`" (`checks/effort-lib.ts`), which picks the retry. madbench cannot re-grade `ts` file+Checks offline, so run10's numbers stay as recorded. The counts below say where this changes a+verdict.++### Scenario verdicts (`.aggregate.perScenario`)++| Scenario | `installed-skill` | `worktree-skill` | `worktree-family` |+|---|---|---|---|+| `team-frontend` | 2/2 | 2/2 | 2/2 |+| `session-code-review` | 2/2 | 2/2 | 1/2 |+| `run-prompt-pick` | 2/2 | 2/2 | 2/2 |+| `team-no-entry` | 2/2 | 2/2 | 1/2 |+| total | 8/8 | 8/8 | 6/8 |++### Per check (counted from `.trials[].results[].checks[]`)++| Check | `installed-skill` | `worktree-skill` | `worktree-family` |+|---|---|---|---|+| `list_models` called first (`ConsultedTable`, readout) | 2/8 | 2/8 | 7/8 |+| efforts equal the table on the started call (`EffortArgs`) | 8/8 | 8/8 | 8/8 |+| bare names on the started call (`BareNames`) | 8/8 | 8/8 | 8/8 |+| skill opened (`SkillUsed`, readout) | 1/8 | 0/8 | 0/8 |+| needs_effort answers before the started call | 6 (6 Sessions) | 6 (6 Sessions) | 0 |+| after needs_effort, the retry carried the Scenario column's levels | 6/6 | 6/6 | n/a |+| `NeedsEffort` readout (started call carries the column's levels) | 8/8 | 8/8 | 8/8 |+| B1 child pinned the level (`EffortFlag`) | 6/6 | 6/6 | 4/6 |+| B2 provider echoed the level (`EffortWire`) | 8/8 | 8/8 | 6/8 |++What changed from run9:++- **needs_effort did the job the skill did not.** With exact ids the agent still went straight to+ `team`/`create_session` without `list_models` (12/12 in the two exact-id variants). claudish+ answered all 12 with needs_effort, and all 12 retries carried exactly the Scenario column's+ levels. The columns the retried levels are consistent with always include the Scenario's own:+ `frontend_ui` (only), `code_review` (with `frontend_ui`, same level), `quick_edit` (with+ `terminal_ops`, `long_context`, same level). Every retry was the first retry: one needs_effort+ per Session, never two.+- **The skill still made no measurable difference with exact ids**: `installed-skill` and+ `worktree-skill` made identical calls in 16/16 Sessions, and the worktree skill was never+ opened (0/16).+- **Bare names 24/24 on the started call.** The Access line is gone and no agent sent a+ `provider@` spec.+- **The two `worktree-family` failures are both "no row, no effort, nothing for layer B to+ follow":**+ 1. `session-code-review` trial 1 picked `gpt-5` without `list_models`; it has no row, so no+ needs_effort and no effort (correct), and the session failed `credit_balance_exhausted` on+ metered OpenAI. EffortArgs passed; B1/B2 fail with "passed no effort, so there is nothing to+ follow".+ 2. `team-no-entry` trial 2 first ran `models: ["gpt","minimax"]`, the family words themselves.+ claudish **started** that run (`"started": true`, slot `gpt` spawned as `or@gpt` on+ OpenRouter, FAILED); no needs_effort, since neither word has a row. The agent then called+ `list_models` and retried correctly (`{"efforts":{"gpt-6.1-sol":"low"}, models+ ["gpt-6.1-sol","minimax-m3"]}`, first refused with "Session already exists", then started+ after it deleted the directory). Under the corrected rule this Session's B Checks would grade+ the retry; run10 graded the `gpt`/`minimax` run. Its EffortArgs passed either way.++### Tool calls, verbatim (input and prompt text omitted)++Passing, with needs_effort, `installed-skill`/`team-frontend` trial 1 (identical in all 12+exact-id team/session Sessions, per Scenario):++```+mcp__claudish__team {"mode":"run","models":["gpt-6-astra","gpt-6.1-sol"],"path":"team-ui"}+ → needs_effort (text below)+mcp__claudish__team {"efforts":{"gpt-6-astra":"xhigh","gpt-6.1-sol":"high"},"mode":"run","models":["gpt-6-astra","gpt-6.1-sol"],"path":"team-ui"}+```++The needs_effort text the agent received (this call, verbatim):++```+needs_effort: nothing was started. Map your task to one task type and call again with efforts: {"<model>": "<level>"} (efforts: {} keeps Claude Code's own for every model).+Recommended levels: {+ "gpt-6-astra": {+ "implementation": "high",+ "planning": "max",+ "research": "xhigh",+ "code_review": "xhigh",+ "debugging": "high",+ "quick_edit": "low",+ "frontend_ui": "xhigh",+ "tool_automation": "max",+ "terminal_ops": "low",+ "long_context": "low"+ },+ "gpt-6.1-sol": {+ "implementation": "medium",+ "planning": "max",+ "research": "xhigh",+ "code_review": "xhigh",+ "debugging": "high",+ "quick_edit": "low",+ "frontend_ui": "high",+ "tool_automation": "xhigh",+ "terminal_ops": "xhigh",+ "long_context": "low"+ }+}+```++The other needs_effort pairs: `create_session {"model":"gpt-6-luna"}` → `{"effort":"xhigh","model":"gpt-6-luna"}`;+`team {"models":["gpt-6-astra","minimax-m3"]}` → `{"efforts":{"gpt-6-astra":"low"}, …}` (minimax-m3 left out).++Every failure:++```+worktree-family/session-code-review, trial 1:+mcp__claudish__create_session {"model":"gpt-5"}+ → FAILED prompt_not_read: "credit_balance_exhausted: You have no credits remaining"++worktree-family/team-no-entry, trial 2:+mcp__claudish__team {"mode":"run","models":["gpt","minimax"],"path":"team-typo"}+ → "started": true; slot "gpt" spawnModel "or@gpt", provider OpenRouter, FAILED+mcp__claudish__list_models {}+mcp__claudish__team {"efforts":{"gpt-6.1-sol":"low"},"mode":"run","models":["gpt-6.1-sol","minimax-m3"],"path":"team-typo"}+ → "Error: Session already exists at …/team-typo. Use a new directory path or delete the existing session first."+mcp__claudish__team {"efforts":{"gpt-6.1-sol":"low"},"mode":"run","models":["gpt-6.1-sol","minimax-m3"],"path":"team-typo"}+ → "started": true; gpt-6.1-sol COMPLETED on Openai-codex+```++### Layer B evidence, verbatim++`installed-skill`/`team-frontend`, trial 1 (after needs_effort):++```+[2026-10-23T05:37:16.289Z] [ComposedHandler] --effort xhigh pinned for cx@gpt-6-astra (catalog clamp skipped)+[2026-10-23T05:37:16.289Z] [Wire] openai-codex@gpt-6-astra → openai/codex-subscription · gpt-6-astra: responses (credential-decided host) [generation g-20261023053138920-f8b4cbb5]+[2026-10-23T05:37:18.452Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+[2026-10-23T05:37:16.342Z] [ComposedHandler] --effort high pinned for [email protected] (catalog clamp skipped)+[2026-10-23T05:37:16.342Z] [Wire] [email protected] → openai/codex-subscription · gpt-6.1-sol: responses (credential-decided host) [generation g-20261023053138920-f8b4cbb5]+[2026-10-23T05:37:18.399Z] [SSE:responses] response.created … "reasoning":{"effort":"high","summary":"detailed","context":"all_turns","mode":"standard"}+```++`installed-skill`/`session-code-review`, trial 1 (create_session `effort: "xhigh"` after needs_effort):++```+[2026-10-23T05:37:13.576Z] [ComposedHandler] --effort xhigh pinned for cx@gpt-6-luna (catalog clamp skipped)+[2026-10-23T05:37:15.070Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+```++`worktree-skill`/`run-prompt-pick`, trial 1 (run_prompt `effort: "medium"`, MCP server's proxy):++```+[2026-10-23T05:38:20.676Z] [Wire] [email protected] → openai/codex-subscription · gpt-6.1-sol: responses (credential-decided host) [generation g-20261023053138920-f8b4cbb5]+[2026-10-23T05:38:22.457Z] [SSE:responses] response.created … "reasoning":{"effort":"medium","summary":"detailed","context":"all_turns","mode":"standard"}+```++In all 22 Sessions whose started call passed an effort, every (model, level) pair was echoed at+that level (28 pairs: `echoedEfforts` equal to the level in every `response.created`). No+`[Wire] … effort A → B` mapping note appeared.++### `list_models` answers: 11 calls, 0 disk-fallback notes, 3 without a model list++| Answer | Count |+|---|---|+| live list + effort table | 8 |+| `No recommended models found. Try search_models instead.` + effort table | 3 |+| the disk-fallback note (`_The live catalog could not be read; …_`) | **0** |++The three are trial 2 of `worktree-family` (`team-frontend` 05:42:19.111Z, `run-prompt-pick`+05:42:19.091Z, `session-code-review` 05:42:22.185Z); each matches an+`[mcp] list_models: Unable to load v3 recommended models: The operation timed out.` line below.+**The effort table was appended in all three** (the run2 defect is fixed), but the disk fallback+did not answer them with a model list, although run9 saw it do so twice and+`~/.claudish/recommended-models-cache-v2.json` exists. Not established why; the effort generation+had changed at 05:31 (`g-20261023053138920-f8b4cbb5`). All three agents still passed the table's+levels (they had the table).++### claudish-mcp log lines written during the run (48 files, 05:36–05:44Z, all kept)++```+[2026-10-23T05:37:19.378Z] [recommended] warm failed: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:37:22.378Z] [catalog] refresh failed: timeout+[2026-10-23T05:38:26.774Z] [recommended] warm failed: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:38:28.657Z] [catalog] refresh failed: timeout+[2026-10-23T05:40:12.764Z] [recommended] warm failed: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:41:16.154Z] [recommended] warm failed: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:42:19.087Z] [mcp] list_models: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:42:19.107Z] [mcp] list_models: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:42:22.181Z] [mcp] list_models: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:42:30.641Z] [recommended] warm failed: Unable to load v3 recommended models: The operation timed out.+[2026-10-23T05:42:30.909Z] [catalog] refresh failed: timeout+```++No `[effort]` or `[team]` line was written. No `[mcp] provider removed` line either (no agent sent+a provider).++### Observations++- **`team` starts a run for model names that are not models** (`"gpt"`, `"minimax"` → `or@gpt`,+ FAILED on OpenRouter). needs_effort does not fire for them (no row), so nothing stops the run+ before a slot is spawned. Worth a look in production: a name with no catalog record could be+ refused like needs_effort refuses a missing effort.+- **Keychain-held keys at `home`**: `minimax-m3` again ran on OpenRouter (12 status rows+ `"provider": "OpenRouter"`), not `zengo@`.+- **Catalog/recommended refreshes time out at `home`** in every trial (`[catalog] refresh failed:+ timeout`, `[recommended] warm failed`); the live list still came back 8/11 times.++### Not verified++- Layer B for the corrected started-call rule on `worktree-family`/`team-no-entry` trial 2+ (the workspace is gone; madbench does not re-grade `ts` file Checks).+- Why the disk fallback did not answer the three timed-out `list_models` calls.+- That the `minimax-m3` children ran without a pinned effort; any host but `claude-sonnet-5-5`.++## run9: installed vs worktree skill, exact ids vs families (2026-10-23, 02:48–02:53Z)++`effort.experiment.yaml --trials 2`: 3 variants × 4 Scenarios × 2 trials = 24 Sessions, host+`claude-sonnet-5-5`, interactive, `sandbox: home`, the claudish plugin staged by+`harness.config.plugins`. Build: the coordinator's `packages/cli/dist/index.js` (list_models+disk fallback, provider removal with `user_required_provider`, skill "Model Names" section); the+worktree skill copy `cmp`-identical to `packages/cli/skills/claudish-usage/SKILL.md`.+Report: `ai-docs/sessions/recommended-effort-bench-20261023/reports/run9.json` (gitignored).++Gates: preflight `PREFLIGHT OK 3 benches · 150 components verified`; check+`81/81 gradable checks failed as required · 0 errored (could not grade) · 0 wrongly passed · 12 not applicable under mock`.+Before the sweep, single-Session probes (`--variant worktree-skill --scenario session-code-review`)+found four more things `home` hides, fixed one at a time (README, "Why each choice"; quoted in+`madbench-skill-gap.md`); run6–run8 show `environment:mcp-reachable` passing and+`~/.claudish/logs/claudish-mcp_*.log` appearing. In run9, `environment:mcp-reachable` and+`environment:skill-registered claudish:claudish-usage` passed 24/24.++Cost: host $2.21 (24 Sessions, subscription). Trials 160 s and 126 s. Probes run4–run8: about+$0.25 together. Children: Codex subscription, plus `minimax-m3` on OpenRouter (metered; see below).++### Scenario verdicts (`.aggregate.perScenario`)++| Scenario | `installed-skill` | `worktree-skill` | `worktree-family` |+|---|---|---|---|+| `team-frontend` | 0/2 | 0/2 | 1/2 |+| `session-code-review` | 0/2 | 0/2 | 1/2 |+| `run-prompt-pick` | 2/2 | 2/2 | 2/2 |+| `team-no-entry` | 0/2 | 0/2 | 1/2 |+| total | 2/8 | 2/8 | 5/8 |++### Per check (counted from `.trials[].results[].checks[]`)++| Check | `installed-skill` | `worktree-skill` | `worktree-family` |+|---|---|---|---|+| `list_models` called before the start call (`ConsultedTable`) | 2/8 | 2/8 | 7/8 |+| efforts equal the table (`EffortArgs`) | 2/8 | 2/8 | 8/8 |+| bare model names in the agent's arguments (`BareNames`) | 8/8 | 8/8 | 6/8 |+| skill opened (`SkillUsed`, readout) | 2/8 | 0/8 | 0/8 |+| B1 child pinned the level (`EffortFlag`) | 0/6 (no effort passed) | 0/6 (no effort passed) | 5/6 (the miss passed no effort) |+| B2 provider echoed the level (`EffortWire`) | 2/8 (6 passed no effort) | 2/8 (6 passed no effort) | 7/8 (the miss passed no effort) |++Wherever an effort was passed (11 start calls, 13 models), it equalled the table entry for the+Scenario's column in the newest generation (`g-20261022161717577-cdb3e76a`, which replaced+`g-20261022002624894-85e83b1a` between run2 and run9), and all 13 (model, level) pairs reached a process and+were echoed by the provider at that level. Every model with no row (`minimax-m3`, `gpt-5`) got+no effort.++What the numbers say:++- **The worktree skill did not change behaviour with exact ids.** `installed-skill` and+ `worktree-skill` made identical calls in 16/16 Sessions: team and create_session went+ straight to the start tool with no `list_models` and no effort (12/12); run-prompt-pick called+ `list_models` (it needed a model) and passed the table's level (4/4). The agent opened the+ installed skill twice (both `installed-skill`/`run-prompt-pick`) and never opened the worktree+ skill (0/16), so its effort and "Model Names" text was not read.+- **Naming only a family is what sent the agent to `list_models`** (7/8 against 2/8), and then it+ applied the table every time (8/8, the eighth being `gpt-5`, correctly with no effort).+- **Bare names fail only in the family variant** (2/8): the agent copied `oai@` specs from+ `list_models`' Access lines. claudish removed them and said so (below), the children ran on+ `cx@`, and the efforts still reached the wire.++### Tool calls, verbatim (prompt and input text omitted, shown in the madbench.yaml)++Passing, `worktree-family`/`team-frontend`, trial 1:++```+mcp__claudish__list_models {}+mcp__claudish__team {"efforts":{"gpt-6-luna":"xhigh","gpt-6.1-sol":"high"},"mode":"run","models":["gpt-6.1-sol","gpt-6-luna"],"path":"team-ui"}+```++Passing, a model with no row, `worktree-family`/`team-no-entry`, trial 1:++```+mcp__claudish__list_models {}+mcp__claudish__team {"efforts":{"gpt-6.1-sol":"low"},"mode":"run","models":["gpt-6.1-sol","minimax-m3"],"path":"team-typo"}+```++Every failing start call:++```+installed-skill and worktree-skill, team-frontend, trials 1 and 2 (4 Sessions, identical):+mcp__claudish__team {"mode":"run","models":["gpt-6-astra","gpt-6.1-sol"],"path":"team-ui"}++installed-skill and worktree-skill, session-code-review, trials 1 and 2 (4 Sessions, identical):+mcp__claudish__create_session {"model":"gpt-6-luna"}++installed-skill and worktree-skill, team-no-entry, trials 1 and 2 (4 Sessions, identical):+mcp__claudish__team {"mode":"run","models":["gpt-6-astra","minimax-m3"],"path":"team-typo"}++worktree-family/team-frontend, trial 2 (fails BareNames only):+mcp__claudish__list_models {}+mcp__claudish__team {"efforts":{"oai@gpt-6-luna":"xhigh","[email protected]":"high"},"mode":"run","models":["[email protected]","oai@gpt-6-luna"],"path":"team-ui"}+ claudish: "provider_note": "Provider removed so claudish routing decides: [email protected] -> gpt-6.1-sol, oai@gpt-6-luna -> gpt-6-luna. Pass 'user_required_provider' only when the user named the provider."++worktree-family/team-no-entry, trial 2 (fails BareNames only):+mcp__claudish__list_models {}+mcp__claudish__team {"efforts":{"[email protected]":"low"},"mode":"run","models":["[email protected]","mm@minimax-m3"],"path":"team-typo"}+ claudish: "provider_note": "Provider removed so claudish routing decides: [email protected] -> gpt-6.1-sol, mm@minimax-m3 -> minimax-m3. Pass 'user_required_provider' only when the user named the provider."++worktree-family/session-code-review, trial 2 (no list_models; the family pick was an old model):+mcp__claudish__create_session {"model":"gpt-5"}+ → FAILED prompt_not_read, "credit_balance_exhausted" (gpt-5 has no Codex route; metered OpenAI has no credit)+```++The efforts map kept the agent's `oai@…` keys while `models` was rewritten; the children still+pinned the right levels (B1/B2 below), so the keys were matched before or with the rewrite.++### Layer B evidence, verbatim (from the Checks' Evidence; workspaces are deleted)++`worktree-family`/`team-frontend`, trial 1:++```+[2026-10-23T02:50:25.845Z] [ComposedHandler] --effort xhigh pinned for cx@gpt-6-luna (catalog clamp skipped)+[2026-10-23T02:50:25.845Z] [Wire] openai-codex@gpt-6-luna → openai/codex-subscription · gpt-6-luna: responses (credential-decided host) [generation g-20261022161717577-cdb3e76a]+[2026-10-23T02:50:27.023Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+[2026-10-23T02:50:25.554Z] [ComposedHandler] --effort high pinned for [email protected] (catalog clamp skipped)+[2026-10-23T02:50:25.554Z] [Wire] [email protected] → openai/codex-subscription · gpt-6.1-sol: responses (credential-decided host) [generation g-20261022161717577-cdb3e76a]+[2026-10-23T02:50:28.186Z] [SSE:responses] response.created … "reasoning":{"effort":"high","summary":"detailed","context":"all_turns","mode":"standard"}+```++`worktree-family`/`session-code-review`, trial 1 (create_session `effort: "xhigh"`):++```+[2026-10-23T02:50:22.573Z] [ComposedHandler] --effort xhigh pinned for [email protected] (catalog clamp skipped)+[2026-10-23T02:50:24.719Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+```++`worktree-family`/`team-frontend`, trial 2, after claudish removed `oai@` (agent keys `oai@gpt-6-luna`, `[email protected]`):++```+[2026-10-23T02:52:34.123Z] [ComposedHandler] --effort xhigh pinned for cx@gpt-6-luna (catalog clamp skipped)+[2026-10-23T02:52:35.259Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+[2026-10-23T02:52:33.823Z] [ComposedHandler] --effort high pinned for [email protected] (catalog clamp skipped)+[2026-10-23T02:52:36.501Z] [SSE:responses] response.created … "reasoning":{"effort":"high","summary":"detailed","context":"all_turns","mode":"standard"}+```++`worktree-family`/`team-no-entry`, trial 1 (`low` for the GPT model; no pin for `minimax-m3`+was asserted):++```+[2026-10-23T02:50:24.901Z] [ComposedHandler] --effort low pinned for [email protected] (catalog clamp skipped)+[2026-10-23T02:50:29.098Z] [SSE:responses] response.created … "reasoning":{"effort":"low","summary":"detailed","context":"all_turns","mode":"standard"}+```++run_prompt (all six run-prompt-pick Sessions passed `effort: "medium"` on `gpt-6.1-sol`), e.g.+`installed-skill`, trial 1, in the MCP server's own proxy:++```+[2026-10-23T02:48:36.114Z] [Wire] [email protected] → openai/codex-subscription · gpt-6.1-sol: responses (credential-decided host) [generation g-20261022161717577-cdb3e76a]+[2026-10-23T02:48:38.537Z] [SSE:responses] response.created … "reasoning":{"effort":"medium","summary":"detailed","context":"all_turns","mode":"standard"}+```++No `[Wire] … effort A → B` mapping note appeared.++### `list_models` disk fallback: seen, 2 of 11 calls++All 11 `list_models` answers carried the effort table. Two carried the fallback note, both in+trial 1 of `worktree-family`, 0.3 s apart while four servers started together:++- `session-code-review` (02:50:18.460Z) and `run-prompt-pick` (02:50:18.162Z):+ `_The live catalog could not be read; this list is from generation g-20261022161717577-cdb3e76a, the last one this machine stored._`++The other nine were live. Both fallback Sessions still applied the table (efforts right, 2/2).+The run2 defect (a failed fetch dropping the table) did not recur.++### claudish-mcp log lines written during the run++`~/.claudish/logs` keeps 20 `claudish-mcp_*.log` files, and this run started well over 20 MCP server+processes (madbench's pre-launch probe plus the host's server, per Session), so every file from+02:48–02:51Z had been rotated away when I read the directory. The surviving 20 files (trial 2,+02:51–02:53Z) hold these tagged lines, quoted in full:++```+[2026-10-23T02:52:02.132Z] [catalog] refresh failed: timeout+[2026-10-23T02:52:31.786Z] [mcp] provider removed: [email protected] -> gpt-6.1-sol, oai@gpt-6-luna -> gpt-6-luna+[2026-10-23T02:52:32.092Z] [mcp] provider removed: [email protected] -> gpt-6.1-sol, mm@minimax-m3 -> minimax-m3+[2026-10-23T02:52:39.756Z] [recommended] warm failed: Unable to load v3 recommended models: The operation timed out.+```++No `[effort]` or `[team]` line survived. The two trial-1 fallbacks' own `[recommended]`/`[catalog]`+lines were among the rotated files, so their cause is not quoted.++### The open `home` risks, as observed++- **Keychain-held keys**: not available. `minimax-m3` routed to OpenRouter (team status+ `"provider": "OpenRouter"`, spawn `or@minimax/…`) instead of `zengo@`, whose key lives in the+ Keychain; the slots completed, on metered billing. Codex (`~/.claudish/codex-oauth.json`) worked.+- **Fresh child HOME**: broke the first probe (theme picker); fixed by an interactive host,+ whose seeding the children share. No child hit a dialog in run9.+- **Dropped hooks**: neither variant loads the installed plugin's hooks or any other host+ plugin; nothing in run9 depended on them.++### Not verified++- Whether the `minimax-m3` children ran with Claude Code's own effort (no Check asserts the+ absence of a pin line).+- Cause of the two fallback answers (their log files were rotated).+- Layer B on a non-Responses connection, and the step 5w mapping path (no clamp occurred).+- Any host but `claude-sonnet-5-5`; two trials per variant and Scenario.++## run3: the plugin-staging design at `sandbox: home` (2026-10-23, 02:34–02:36Z) — the MCP server never came up++`effort.experiment.yaml --trials 2`, three variants × four Scenarios × 2 trials = 24 Sessions.+Report: `ai-docs/sessions/recommended-effort-bench-20261023/reports/run3.json` (gitignored).+Gates before it: preflight `PREFLIGHT OK 3 benches · 138 components verified`; check+`93/93 gradable checks failed as required · 0 errored (could not grade) · 0 wrongly passed · 12 not applicable under mock`.+Build: the coordinator's `packages/cli/dist/index.js` (not rebuilt); the worktree skill copy was+`cmp`-identical to `packages/cli/skills/claudish-usage/SKILL.md`.++**Result: 0/24 Scenarios passed, and none of them measured the feature.** In every Session:++- madbench's MCP preflight, from inside the `home` sandbox, 24/24:+ `environment:mcp-reachable: MCP server "claudish" did not answer madbench's pre-launch handshake: the initialize deadline elapsed before the server answered: context deadline exceeded; exit: signal: killed`+- the host Claude Code's own record (`failedMcpServers`), 24/24:+ `{"error":"Connection closed","errorCode":"CONNECTION_CLOSED","name":"claudish"}`+- so no `mcp__claudish__*` call was made (0 in 24 Sessions); every effort, bare-name and+ layer-B Check failed with `no <tool> start call in the Session`. The host said, e.g.: "I+ couldn't run this. The claudish MCP server failed to connect (`CONNECTION_CLOSED`), so its+ `team` tool isn't available in this session."++What did work at `home`: **plugin staging**. `environment:skill-registered claudish:claudish-usage`+passed 24/24, and the skill list the host received names the staged text per variant:+`installed-skill` → "claudish:claudish-usage: Runs models through the claudish MCP tools — team,+create_session, run_prompt — and resolves model IDs against the live …" (the installed 2.10.0+skill); `worktree-*` → "claudish:claudish-usage: CRITICAL - Guide for using Claudish CLI ONLY+through sub-agents …" (the worktree skill). `session:skill-used` (readout) 0/24: never invoked.++**Not established: why the server does not start under `home`.** The Session carries no stderr+from the server. No `~/.claudish/logs/claudish-mcp_*.log` was written during the run (0 files of+that name exist), although `~/.claudish` was shared `rw`: the server either never reached its+logger or did not see the shared path. Its debug log (`CLAUDISH_DEBUG`, `<cwd>/logs/`) was in the+deleted workspace. The `[mcp]`/`[catalog]`/`[recommended]`/`[effort]`/`[team]` grep therefore+has nothing to quote. Candidates, none tested: the macOS read confinement of `home` denying a+file `bun` or the bundle needs outside the allow-set; something blocking on the replaced HOME+(Keychain via `security`, 1Password); the share not resolving. I did not switch back to+`workspace`, as instructed.++Cost and time: host $0.75 (24 Sessions, subscription); trials 59 s and 58 s. A duplicate of the+same command (a keystroke batch the new pane replayed after the first run ended) started at+02:36Z and was stopped with Ctrl-C after one trial; its partial report is+`run3-duplicate-interrupted.json` (host $0.12, same failure) and is not counted.++> These runs used the earlier design: `sandbox: workspace`, the host's installed `claudish@magus`+> plugin and its `claudish-usage` skill (no effort section), and variants `unprompted` / `asked`.+> The Bench now stages the skill with `harness.config.plugins` at `home` (see README). The+> results below are the installed-skill condition only.++Build under test: this worktree, `bun run build`, `packages/cli/dist/index.js` (reports+`claudish version 13.1.0`), uncommitted effort changes included. madbench 0.54.0. Host+`claude-sonnet-5-5` on the Claude Max subscription. Effort sidecar generation+`g-20261022002624894-85e83b1a` in every run.++| Run | What | Report JSON (gitignored session dir) |+|---|---|---|+| run1 | pilot: `madbench.yaml`, one trial, an earlier version of the Scenarios (below) | `ai-docs/sessions/recommended-effort-bench-20261023/reports/run1.json` |+| run2 | `effort.experiment.yaml`, variants `unprompted` and `asked`, `--trials 2` | `ai-docs/sessions/recommended-effort-bench-20261023/reports/run2.json` |++The free gates, before each real run:++```+madbench preflight → PREFLIGHT OK 2 benches · 68 components verified+madbench check → 38/38 gradable checks failed as required · 0 errored (could not grade) · 0 wrongly passed · 8 not applicable under mock+ control holds: every gradable check was graded, and none passed against a harness that did nothing.+madbench grade run2.json → 48/48 checks reproduced · 0 diverged · 0 could not be re-graded · 44 skipped (stated above)+```++`grade` skips every `ts` file Check (madbench skips file-loading script Checks by design), so the+positive control covers only the `session:*` and `environment:*` Checks.++Cost and time: host cost as Claude Code reports it, billed to the subscription: run1 $0.35, run2+$1.59 (16 Sessions). The children ran on the Codex subscription and `zengo@` (team status+`costUsd: 0`). run2 wall time: trial 1 117 s, trial 2 100 s. Each Session took 13 to 63 s.++## Scenario verdicts (run2, from `.aggregate.perScenario`)++| Scenario | `unprompted` | `asked` |+|---|---|---|+| `team-frontend` | 0/2 | 1/2 |+| `session-code-review` | 0/2 | 2/2 |+| `run-prompt-pick` | 2/2 | 1/2 |+| `team-no-entry` | 0/2 | 2/2 |+| total | 2/8 | 6/8 |++## Per layer (run2, counted from `.trials[].results[].checks[]`)++| Check | `unprompted` | `asked` |+|---|---|---|+| A0 `ConsultedTable`: `list_models` before the start call | 2/8 (both `run-prompt-pick`) | 8/8 |+| `list_models` output carried the effort table | 2/2 calls | 6/8 calls (2 returned "No recommended models found") |+| A `EffortArgs`: every model's effort equals its table entry | 2/8 | 6/8 |+| B1 `EffortFlag`: the child pinned the level (team, create_session) | 0/6, no effort passed | 5/6 (the miss passed no effort) |+| B2 `EffortWire`: the provider echoed the level | 2/2 (`run-prompt-pick`); 0/6 elsewhere, no effort passed | 7/8 (the miss passed no effort) |++Per model, wherever the agent passed an effort:++- **Layer A**: 11 of 11 model efforts equalled the table entry, in 8 of 8 start calls. Both+ `team-no-entry` calls in `asked` left `minimax-m3` out of `efforts` (`{"gpt-6-astra":"low"}`).+- **Layer B**: 6 of 6 (model, level) pairs reached a child as a pin line, and 9 of 9 pairs were+ echoed by the provider at the level the agent passed. No `[Wire] … effort A → B` mapping note+ appeared: every level was one the Codex connection publishes.++Every Layer A miss is one of two kinds:++1. **The agent never opened `list_models`** (6/6 `unprompted` team and create_session runs, and+ 3/4 in run1). It loaded only `team` or `create_session` through ToolSearch and called it+ with no `efforts`/`effort`. The `efforts` description ("Take each level from list_models'+ 'Recommended effort by task type' table") was in the schema it loaded. The tool description+ alone did not lead it to the table in 0 of 6 runs. When the agent opened `list_models` for its+ own reason (to pick a model, `run-prompt-pick`), it applied the table without being asked,+ 2/2.+2. **`list_models` returned no table** (2/8 `asked` runs, the production defect below). The+ agent then said so and passed no effort.++## Tool calls, verbatim++Passing, `asked`/`team-frontend`, trial 1:++```+mcp__claudish__list_models {}+mcp__claudish__team {"efforts":{"gpt-6-astra":"xhigh","gpt-6.1-sol":"high"},"input":"Write a small React component `PricingCard` (TypeScript, Tailwind classes) that shows a plan name, a monthly price and a \"Choose plan\" button. Reply with the code only. Do not create or edit any files.","mode":"run","models":["gpt-6-astra","gpt-6.1-sol"],"path":"team-ui"}+```++Passing, a model with no table entry, `asked`/`team-no-entry`, trials 1 and 2 (identical):++```+mcp__claudish__list_models {}+mcp__claudish__team {"efforts":{"gpt-6-astra":"low"},"input":"The line `return \"Helo, \" + name;` has a typo. Reply with the corrected line only. Do not create or edit any files.","mode":"run","models":["gpt-6-astra","minimax-m3"],"path":"team-typo"}+```++Passing without being asked, `unprompted`/`run-prompt-pick`:++```+trial 1: mcp__claudish__run_prompt {"effort":"max","model":"gpt-6-luna","prompt":"Write a TypeScript function `isLeapYear(year: number): boolean` that follows the Gregorian calendar rules (divisible by 4, except centuries unless divisible by 400). Return only the code in a single TypeScript code block, with a brief doc comment."}+trial 2: mcp__claudish__run_prompt {"effort":"medium","model":"gpt-6.1-sol","prompt":"Write a TypeScript function `isLeapYear(year: number): boolean` that follows the Gregorian calendar rules (divisible by 4, except century years, unless divisible by 400). Return only the code in a single TypeScript code block, with a brief doc comment."}+```++Each failing start call (no `list_models` call before it, unless shown):++```+unprompted/team-frontend, trials 1 and 2:+mcp__claudish__team {"input":"Write a small React component `PricingCard` (TypeScript, Tailwind classes) that shows a plan name, a monthly price and a \"Choose plan\" button. Reply with the code only. Do not create or edit any files.","mode":"run","models":["gpt-6-astra","gpt-6.1-sol"],"path":"team-ui"}++unprompted/session-code-review, trials 1 and 2:+mcp__claudish__create_session {"model":"gpt-6-luna","prompt":"Review this TypeScript function for bugs. Reply with a short list of the bugs you find. Do not create or edit any files.\n\nfunction average(xs: number[]): number {\n let sum = 0;\n for (let i = 1; i <= xs.length; i++) sum += xs[i];\n return sum / xs.length;\n}"}++unprompted/team-no-entry, trials 1 and 2:+mcp__claudish__team {"input":"The line `return \"Helo, \" + name;` has a typo. Reply with the corrected line only. Do not create or edit any files.","mode":"run","models":["gpt-6-astra","minimax-m3"],"path":"team-typo"}++asked/team-frontend, trial 2 (list_models returned "No recommended models found. Try search_models instead."):+mcp__claudish__list_models {}+mcp__claudish__team {"input":"Write a small React component `PricingCard` (TypeScript, Tailwind classes) that shows a plan name, a monthly price and a \"Choose plan\" button. Reply with the code only. Do not create or edit any files.","mode":"run","models":["gpt-6-astra","gpt-6.1-sol"],"path":"team-ui"}+ agent: "`list_models` returned "No recommended models found", so I had no effort table to read a level from. I didn't want to guess."++asked/run-prompt-pick, trial 2 (same list_models result):+mcp__claudish__list_models {}+mcp__claudish__run_prompt {"model":"gpt-6-luna","prompt":"Write a TypeScript function `isLeapYear(year: number): boolean` that follows the Gregorian calendar rules (divisible by 4, except centuries unless divisible by 400). Return only the code in one code block, with a brief doc comment."}+ agent: "`list_models` returned "No recommended models found", so claudish had no recommended effort to apply, and I didn't want to invent one."+```++run1 (pilot, `unprompted` wording; the team and session calls match the run2 failures above):++```+run-prompt-pick (then a terminal_ops task: "a shell one-liner that lists the 5 largest files"):+mcp__claudish__list_models {}+mcp__claudish__run_prompt {"effort":"low","model":"oai@gpt-6-luna","prompt":"Give me a single shell one-liner …"}+mcp__claudish__run_prompt {"effort":"low","model":"[email protected]","prompt":"…same…"}+ → both: "[API Error: credit_balance_exhausted You have no credits remaining. …]"+```++The agent read the shell task as `quick_edit` (`low` for both models) where the Scenario expected+`terminal_ops` (`max`, `xhigh`). That is a fair reading, so run2 replaced it with an+implementation task. It took the `oai@` spec from `list_models`' Access line (see+Observations), and the metered OpenAI account had no credit.++## Layer B evidence, verbatim++The lines below are from the debug logs claudish wrote into each workspace, as the Checks+recorded them in Evidence (`.checks[].result.evidence.perModel`). The workspaces are deleted+after each run. `[SSE:responses]` lines are cut to the event type and the `reasoning` object.++`asked`/`team-frontend`, trial 1. Child for `gpt-6-astra`, given `xhigh`:++```+[2026-10-22T15:41:09.213Z] [ComposedHandler] --effort xhigh pinned for cx@gpt-6-astra (catalog clamp skipped)+[2026-10-22T15:41:09.213Z] [Wire] openai-codex@gpt-6-astra → openai/codex-subscription · gpt-6-astra: responses (credential-decided host) [generation g-20261022002624894-85e83b1a]+[2026-10-22T15:41:11.265Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+```++Child for `gpt-6.1-sol`, given `high`:++```+[2026-10-22T15:41:09.447Z] [ComposedHandler] --effort high pinned for [email protected] (catalog clamp skipped)+[2026-10-22T15:41:09.447Z] [Wire] [email protected] → openai/codex-subscription · gpt-6.1-sol: responses (credential-decided host) [generation g-20261022002624894-85e83b1a]+[2026-10-22T15:41:11.710Z] [SSE:responses] response.created … "reasoning":{"effort":"high","summary":"detailed","context":"all_turns","mode":"standard"}+```++`asked`/`session-code-review`, trial 2, create_session `effort: "xhigh"`:++```+[2026-10-22T15:43:01.563Z] [ComposedHandler] --effort xhigh pinned for cx@gpt-6-luna (catalog clamp skipped)+[2026-10-22T15:43:01.563Z] [Wire] openai-codex@gpt-6-luna → openai/codex-subscription · gpt-6-luna: responses (credential-decided host) [generation g-20261022002624894-85e83b1a]+[2026-10-22T15:43:03.362Z] [SSE:responses] response.created … "reasoning":{"effort":"xhigh","summary":"detailed","context":"all_turns","mode":"standard"}+```++`asked`/`team-no-entry`, trial 2. Child for `gpt-6-astra`, given `low`:++```+[2026-10-22T15:43:07.180Z] [ComposedHandler] --effort low pinned for cx@gpt-6-astra (catalog clamp skipped)+[2026-10-22T15:43:07.180Z] [Wire] openai-codex@gpt-6-astra → openai/codex-subscription · gpt-6-astra: responses (credential-decided host) [generation g-20261022002624894-85e83b1a]+[2026-10-22T15:43:09.986Z] [SSE:responses] response.created … "reasoning":{"effort":"low","summary":"detailed","context":"all_turns","mode":"standard"}+```++`unprompted`/`run-prompt-pick`, trial 1. run_prompt `effort: "max"`, in the MCP server's own proxy+(no child, so no pin line; `output_config.effort` went through step 5w):++```+[2026-10-22T15:40:09.641Z] [Wire] openai-codex@gpt-6-luna → openai/codex-subscription · gpt-6-luna: responses (credential-decided host) [generation g-20261022002624894-85e83b1a]+[2026-10-22T15:40:10.883Z] [SSE:responses] response.created … "reasoning":{"effort":"max","summary":"detailed","context":"all_turns","mode":"standard"}+```++`asked`/`run-prompt-pick`, trial 1, `effort: "medium"` on `gpt-6.1-sol`: echoed `"effort":"medium"` at+15:41:15.290Z, after the same `[Wire] [email protected]` line.++Each child's log file echoed the same level in every `response.created` (2 or 3 per child).++## Production defect++**`list_models` drops the effort table whenever the recommendation fetch fails, although the+table is on disk.** `packages/cli/src/mcp-server.ts`, `list_models` handler: when+`getRecommendedModels()` throws, or returns no models, the handler returns+`"No recommended models found. Try search_models instead."` before it reads+`latestEffortRecommendations()`. The sidecar (generation `g-20261022002624894-85e83b1a`) was+present the whole time. In trial 2 of `asked`, four MCP servers (one per Session, concurrency 4)+answered `list_models` within 2.6 s. The first, at 15:42:55.842Z (`session-code-review`), and the+fourth, at 15:42:58.451Z (`team-no-entry`), returned the table. The second, at 15:42:55.883Z+(`run-prompt-pick`), and the third, at 15:42:57.992Z (`team-frontend`), returned only that+sentence. The same four-way start in trial 1 returned four tables. The handler's own comment+says an older sidecar "still shows the last generation this machine read", but on this path+nothing is shown. The `catch {}` also discards the error, and the MCP server writes no always-on+log (`initLogger(true, "debug", true)`). The debug log was in the deleted workspace, so this run+cannot say why the fetch failed. It happened in 2 of 10 `list_models` calls in run2. Effect,+measured: in both cases the agent explicitly declined to pass an effort, and both Scenarios failed+Layer A. Not fixed here, as instructed.++## Observations (not defects of this feature)++- **`list_models`' Access line names the metered route.** For the GPT models it prints+ `oai@gpt-6-luna`, `[email protected]`, `oai@gpt-6-astra`, although `--probe` shows `cx@`+ (subscription) first in the routing chain for each. In run1 the agent used `oai@gpt-6-luna`+ and `[email protected]` from that line, and both calls failed `credit_balance_exhausted`. This+ predates the effort table (`renderRecommendedMarkdown` / `nativeShortcutForVendor`). run2 works+ around it by asking for the bare name.+- **claudish logs the effort it sends only when step 5w changes it.** `applyReasoning`+ (`providers/wire-request.ts`) writes `[Wire] … effort A → B` only for a mapped level, and+ `sentEffort` is never logged. A level the connection publishes leaves no claudish line naming+ it. This Bench therefore proves the wire through the provider's `response.created` echo, which+ exists only on Responses connections. A Chat Completions or Messages connection (`kc@k3`,+ `zengo@minimax-m3`) would leave Layer B unprovable from logs.+- **run_prompt returns an upstream error as an ordinary result**: `"[API Error:+ credit_balance_exhausted …]\n\n---\nTokens: 100 input, 1 output"` (run1). The agent read it+ correctly, but the token footer suggests a completed call.++## Not verified, and why++- **That `minimax-m3` ran with Claude Code's own effort.** The Checks grade efforts the agent+ passed. They do not assert that the `minimax-m3` child has no pin line, and the workspace logs+ are gone. Layer A shows only that no effort was passed for it.+- **Layer B on a connection that is not the Responses API.** Every effort that was passed went to+ a Codex `responses` connection. No clamp (`effort A → B`) case occurred, so the mapping path of+ step 5w was not exercised.+- **A native Claude model.** Effort on a bare Claude name is ignored by design (wire.md). No+ Scenario picked one.+- **Why the two `list_models` fetches failed** (above).+- **Statistics**: 2 trials per variant and Scenario, plus a 1-trial pilot. The "never opens+ `list_models` unprompted" result is 0/6 in run2 and 0/3 in run1 (team and session Scenarios).+ The other rates are small counts, not estimates.+- **Host model**: only `claude-sonnet-5-5`. A different host (Opus, Haiku) was not measured.+- run1 used the earlier Scenario set (`team-implementation`, a `terminal_ops` pick task, and the+ name `team-no-cell` for `team-no-entry`). The checks in `checks/effort-trace.ts` were renamed+ `checks/effort-plumbing.ts` after run1, with the same logic.diff --git a/ai-docs/benches/recommended-effort/checks/bare-names.ts b/ai-docs/benches/recommended-effort/checks/bare-names.tsnew file mode 100644index 00000000..616b91a3--- /dev/null+++ b/ai-docs/benches/recommended-effort/checks/bare-names.ts@@ -0,0 +1,47 @@+/**+ * Layer A — the model names the agent passed. Unless the user named a provider, every+ * model in the start call must be a bare name: no `provider@` shortcut (it pins one+ * route and skips the subscription-first routing chain) and no aggregator-qualified+ * id with a `/` (it pins that aggregator).+ *+ * Score: the fraction of models passed bare. Pass: all of them.+ *+ * config:+ * tool: team | create_session | run_prompt+ * user_named_provider: optional, default false. true: `provider@` is what the user+ * asked for, so only the `/` rule applies+ */+import { requested, startedCall, type Tool } from "./effort-lib.ts";++type Ctx = {+ config?: Record<string, unknown>;+ assertion?: { args?: Record<string, unknown> };+ session?: { calls?: { name: string; args: Record<string, unknown> }[] };+};++export default function evaluate(_output: string, context: Ctx) {+ const cfg = { ...(context.assertion?.args ?? {}), ...((context.assertion?.args?.config as object) ?? {}), ...(context.config ?? {}) } as {+ tool: Tool;+ user_named_provider?: boolean;+ };+ const { call } = startedCall(context.session ?? {}, cfg.tool);+ if (!call) return { pass: false, score: 0, reason: `no ${cfg.tool} start call in the Session` };+ const calls = [call];+ const { models } = requested(call, cfg.tool);+ if (models.length === 0) return { pass: false, score: 0, reason: `the ${cfg.tool} call named no model`, evidence: { args: calls[0].args } };+ const perModel: Record<string, string> = {};+ for (const m of models) {+ if (m.includes("@") && !cfg.user_named_provider) perModel[m] = "provider@ shortcut";+ else if (m.includes("/")) perModel[m] = "aggregator-qualified id";+ else perModel[m] = "bare";+ }+ const bare = Object.values(perModel).filter((v) => v === "bare").length;+ const pass = bare === models.length;+ const offenders = Object.entries(perModel).filter(([, v]) => v !== "bare").map(([m, v]) => `${m} (${v})`);+ return {+ pass,+ score: bare / models.length,+ reason: pass ? `${bare}/${models.length} models passed bare` : `${bare}/${models.length} bare; not bare: ${offenders.join(", ")}`,+ evidence: { models, perModel, userNamedProvider: Boolean(cfg.user_named_provider), userRequiredProviderArg: calls[0].args?.user_required_provider ?? null },+ };+}diff --git a/ai-docs/benches/recommended-effort/checks/effort-args.ts b/ai-docs/benches/recommended-effort/checks/effort-args.tsnew file mode 100644index 00000000..4b232bf6--- /dev/null+++ b/ai-docs/benches/recommended-effort/checks/effort-args.ts@@ -0,0 +1,78 @@+/**+ * Layer A — the calling agent's choice. Did the start call (team run, create_session+ * or run_prompt) pass, for every model it ran, exactly the sidecar's recommended level for the+ * Scenario's task-type column, and no effort for a model with no table entry?+ *+ * Score: the fraction of models whose effort is right. Pass: every model right, and+ * (when the Scenario names models) the call ran exactly those models.+ *+ * config:+ * tool: team | create_session | run_prompt+ * column: the sidecar task type the prompt was written for (e.g. code_review)+ * models: optional; the models the user named, as canonical ids ([] = none named)+ */+import { canonicalOf, newestEfforts, requested, startedCall, type Tool } from "./effort-lib.ts";++type Ctx = {+ config?: Record<string, unknown>;+ assertion?: { args?: Record<string, unknown> };+ session?: { calls?: { name: string; args: Record<string, unknown> }[] };+};++export default function evaluate(_output: string, context: Ctx) {+ const cfg = { ...(context.assertion?.args ?? {}), ...((context.assertion?.args?.config as object) ?? {}), ...(context.config ?? {}) } as {+ tool: Tool;+ column: string;+ models?: string[];+ };+ const { generationId, efforts } = newestEfforts();+ const { call, all: calls, needsEffortBefore } = startedCall(context.session ?? {}, cfg.tool);+ if (!call) {+ return { pass: false, score: 0, reason: `no ${cfg.tool} start call in the Session`, evidence: { generationId } };+ }+ // Graded on the call that started work (after any needs_effort answers).+ const req = requested(call, cfg.tool);+ const perModel: Record<string, { passed: string | null; expected: string | null; ok: boolean }> = {};+ for (const model of Object.keys(req.efforts)) {+ const expected = efforts[canonicalOf(model)]?.[cfg.column]?.effort ?? null;+ const passed = req.efforts[model] ?? null;+ const inModels = req.models.includes(model);+ perModel[model] = { passed, expected, ok: inModels && passed === expected };+ }+ const total = Object.keys(perModel).length;+ const right = Object.values(perModel).filter((m) => m.ok).length;+ const ranCanonical = req.models.map(canonicalOf).sort();+ // An empty list means the user named no exact ids (a family, or "pick one").+ const named = cfg.models && cfg.models.length > 0 ? [...cfg.models].sort() : null;+ const modelsMatch = named === null || JSON.stringify(ranCanonical) === JSON.stringify(named);+ const score = total === 0 ? 0 : right / total;+ const pass = total > 0 && right === total && modelsMatch;+ const wrong = Object.entries(perModel)+ .filter(([, m]) => !m.ok)+ .map(([k, m]) => `${k}: passed ${m.passed ?? "none"}, recommended ${m.expected ?? "none"}`);+ const reason = pass+ ? `${right}/${total} models carry the ${cfg.column} level (generation ${generationId})`+ : [+ `${right}/${total} models right for column ${cfg.column}`,+ ...wrong,+ modelsMatch ? "" : `ran ${ranCanonical.join(", ")}, the user named ${named?.join(", ")}`,+ req.note ?? "",+ ]+ .filter(Boolean)+ .join("; ");+ return {+ pass,+ score,+ reason,+ evidence: {+ generationId,+ column: cfg.column,+ startCalls: calls.length,+ needsEffortBefore: needsEffortBefore.length,+ args: call.args,+ perModel,+ modelsMatch,+ note: req.note,+ },+ };+}diff --git a/ai-docs/benches/recommended-effort/checks/effort-lib.ts b/ai-docs/benches/recommended-effort/checks/effort-lib.tsnew file mode 100644index 00000000..792f1c59--- /dev/null+++ b/ai-docs/benches/recommended-effort/checks/effort-lib.ts@@ -0,0 +1,157 @@+/**+ * Shared by the three Checks of this Bench. Pure reading: the effort sidecar+ * claudish writes, the claudish MCP calls in the Session, and the debug logs the+ * MCP server and its children wrote into the workspace.+ */+import { existsSync, readdirSync, readFileSync } from "node:fs";+import { homedir } from "node:os";+import { join } from "node:path";++export interface Cell {+ effort: string;+}++/**+ * The newest generation of `~/.claudish/cloud-models-catalog-v3.efforts.json`,+ * chosen exactly as `latestEffortRecommendations` chooses it (largest+ * `writtenAt`), so the Check grades against the table `list_models` rendered.+ */+export function newestEfforts(path = join(homedir(), ".claudish", "cloud-models-catalog-v3.efforts.json")): {+ generationId: string;+ efforts: Record<string, Record<string, Cell>>;+} {+ const doc = JSON.parse(readFileSync(path, "utf8")) as {+ generations: Record<string, { writtenAt: number; efforts: Record<string, Record<string, Cell>> }>;+ };+ const [id, gen] = Object.entries(doc.generations).sort(([, a], [, b]) => b.writtenAt - a.writtenAt)[0];+ return { generationId: id, efforts: gen.efforts };+}++/** `cx@gpt-6-astra` → `gpt-6-astra`; a bare name is returned unchanged. */+export function canonicalOf(spec: string): string {+ const at = spec.indexOf("@");+ return at >= 0 ? spec.slice(at + 1) : spec;+}++export type Tool = "team" | "create_session" | "run_prompt";++export interface ToolCall {+ name: string;+ args: Record<string, unknown>;+ tool_call_id?: string;+}++export interface SessionLike {+ calls?: ToolCall[];+ events?: { type?: string; tool_call_id?: string; tool_out?: unknown }[];+}++/** The text the tool returned for one call, from the Session's tool_result Events. */+export function resultText(session: SessionLike, call: ToolCall): string {+ if (!call.tool_call_id) return "";+ const ev = (session.events ?? []).find((e) => e.type === "tool_result" && e.tool_call_id === call.tool_call_id);+ const out = ev?.tool_out;+ return typeof out === "string" ? out : out === undefined ? "" : JSON.stringify(out);+}++/** claudish refused to start because no effort was passed for a model with a row. */+export function isNeedsEffort(session: SessionLike, call: ToolCall): boolean {+ return resultText(session, call).includes("needs_effort:");+}++/**+ * The start call that actually STARTED work: the LAST one whose answer is neither+ * needs_effort nor an `Error:` text (a retry after a failed run supersedes it), else+ * the last call. Also returns the needs_effort answers before it.+ * (run10 graded the FIRST non-needs_effort call; one Session then graded a run of the+ * literal family names "gpt"/"minimax" instead of the agent's working retry.)+ */+export function startedCall(session: SessionLike, tool: Tool): {+ call?: ToolCall;+ all: ToolCall[];+ needsEffortBefore: ToolCall[];+} {+ const all = startCalls(session, tool);+ const ok = (c: ToolCall) => !isNeedsEffort(session, c) && !resultText(session, c).trimStart().startsWith("Error");+ let at = -1;+ for (let i = all.length - 1; i >= 0; i--) if (ok(all[i])) { at = i; break; }+ if (at < 0) at = all.length - 1;+ return { call: all[at], all, needsEffortBefore: all.slice(0, Math.max(at, 0)).filter((c) => isNeedsEffort(session, c)) };+}++/** Every call of `mcp__claudish__<tool>` that starts work (team: mode run / run-and-judge). */+export function startCalls(session: { calls?: ToolCall[] }, tool: Tool): ToolCall[] {+ return (session.calls ?? []).filter((c) => {+ if (c.name !== `mcp__claudish__${tool}`) return false;+ if (tool !== "team") return true;+ const mode = c.args?.mode;+ return mode === "run" || mode === "run-and-judge";+ });+}++/**+ * What one start call asked for: the models it ran and the effort it passed for+ * each (undefined = no effort for that model). `efforts` arriving as a JSON string+ * is parsed and flagged, because the tool schema declares an object.+ */+export function requested(call: ToolCall, tool: Tool): {+ models: string[];+ efforts: Record<string, string | undefined>;+ note?: string;+} {+ const a = call.args ?? {};+ if (tool === "team") {+ const models = Array.isArray(a.models) ? (a.models as unknown[]).map(String) : [];+ let raw = a.efforts as unknown;+ let note: string | undefined;+ if (typeof raw === "string") {+ try {+ raw = JSON.parse(raw);+ note = "efforts arrived as a JSON string, not an object";+ } catch {+ note = `efforts is an unparseable string: ${raw}`;+ raw = undefined;+ }+ }+ const map = raw && typeof raw === "object" ? (raw as Record<string, unknown>) : {};+ const efforts: Record<string, string | undefined> = {};+ for (const m of models) efforts[m] = typeof map[m] === "string" ? (map[m] as string) : undefined;+ for (const k of Object.keys(map)) if (!models.includes(k)) efforts[k] = String(map[k]);+ return { models, efforts, note };+ }+ const model = typeof a.model === "string" ? a.model : "";+ // "default" is the needs_effort opt-out: Claude Code's own effort, i.e. none passed.+ const effort = typeof a.effort === "string" && a.effort !== "" && a.effort !== "default" ? a.effort : undefined;+ return { models: model ? [model] : [], efforts: model ? { [model]: effort } : {} };+}++export interface LogFile {+ path: string;+ lines: string[];+}++/** Every claudish debug log in `<workDir>/logs/` (the MCP server's and each child's). */+export function debugLogs(workDir: string | undefined): LogFile[] {+ if (!workDir) return [];+ const dir = join(workDir, "logs");+ if (!existsSync(dir)) return [];+ return readdirSync(dir)+ .filter((f) => f.startsWith("claudish_") && f.endsWith(".log"))+ .map((f) => ({ path: join(dir, f), lines: readFileSync(join(dir, f), "utf8").split("\n") }));+}++/** The `[Wire] <provider>@<model> → route · wire id: api …` lines that name this model. */+export function wireLinesFor(log: LogFile, model: string): string[] {+ const c = canonicalOf(model);+ return log.lines.filter((l) => {+ const m = l.match(/\[Wire\] ([^@\s]+)@(\S+) → \S+ · (\S+):/);+ if (!m) return false;+ const names = [m[2], m[3]];+ return names.some((n) => n === c || n === model || c.endsWith(n) || n.endsWith(c));+ });+}++/** Clip a log line for Evidence: the timestamp, the head and the matched part. */+export function clip(line: string, max = 400): string {+ return line.length <= max ? line : `${line.slice(0, max)} …(${line.length} chars)`;+}diff --git a/ai-docs/benches/recommended-effort/checks/effort-plumbing.ts b/ai-docs/benches/recommended-effort/checks/effort-plumbing.tsnew file mode 100644index 00000000..60a817b3--- /dev/null+++ b/ai-docs/benches/recommended-effort/checks/effort-plumbing.ts@@ -0,0 +1,129 @@+/**+ * Layer B — the plumbing. For every (model, effort) the agent passed, follow it+ * through the debug logs claudish wrote into the workspace. It grades whatever+ * effort the agent passed, right or wrong, so Layer B is independent of Layer A.+ *+ * config:+ * tool: team | create_session | run_prompt+ * stage: flag | wire+ *+ * stage flag (team, create_session): the child claudish received+ * `--effort-override <level>`, proven by its own debug line+ * `[ComposedHandler] --effort <level> pinned for <model>`.+ * stage wire (all three tools): the level went out on the request, proven by the+ * provider, not by claudish: a Responses `response.created` event echoing+ * `"reasoning":{"effort":"<sent>"…}`, where <sent> is the level, or the level+ * step 5w mapped it to, which claudish names in a `[Wire] … effort A → B` note.+ * For team/create_session the echo must be in the same log file as the pin line+ * (one file per process), so it is the child's own request.+ *+ * Score: the fraction of traced pairs proven. Pass: every pair proven.+ */+import { canonicalOf, clip, debugLogs, type LogFile, requested, startedCall, type Tool, wireLinesFor } from "./effort-lib.ts";++type Ctx = {+ config?: Record<string, unknown>;+ assertion?: { args?: Record<string, unknown> };+ session?: { work_dir?: string; calls?: { name: string; args: Record<string, unknown> }[] };+};++function related(a: string, b: string): boolean {+ const x = canonicalOf(a);+ const y = canonicalOf(b);+ return x === y || x.endsWith(y) || y.endsWith(x);+}++function pinLines(log: LogFile, model: string, level: string): string[] {+ return log.lines.filter((l) => {+ const m = l.match(/\[ComposedHandler\] --effort (\S+) pinned for (\S+)/);+ return m !== null && m[1] === level && related(m[2], model);+ });+}++/** The level step 5w sent: the clamp note's target, else the level itself. */+function sentLevel(log: LogFile, level: string): { sent: string; note?: string } {+ const note = log.lines.find((l) => l.includes("[Wire]") && l.includes(`effort ${level} → `));+ if (!note) return { sent: level };+ const m = note.match(new RegExp(`effort ${level} → (\\S+)`));+ return { sent: m ? m[1] : level, note: clip(note) };+}++function echoes(log: LogFile): { efforts: string[]; sample?: string } {+ const efforts: string[] = [];+ let sample: string | undefined;+ for (const l of log.lines) {+ if (!l.includes("[SSE:responses]") || !l.includes('"response.created"')) continue;+ const m = l.match(/"reasoning":\{"effort":"([a-z]+)"[^}]*\}/);+ if (!m) continue;+ efforts.push(m[1]);+ sample ??= `${l.slice(0, 26)} [SSE:responses] response.created … ${m[0]}`;+ }+ return { efforts, sample };+}++export default function evaluate(_output: string, context: Ctx) {+ const cfg = { ...(context.assertion?.args ?? {}), ...((context.assertion?.args?.config as object) ?? {}), ...(context.config ?? {}) } as {+ tool: Tool;+ stage: "flag" | "wire";+ };+ const session = context.session ?? {};+ const { call: started } = startedCall(session, cfg.tool);+ if (!started) return { pass: false, score: 0, reason: `no ${cfg.tool} start call in the Session` };+ const calls = [started];+ const req = requested(calls[0], cfg.tool);+ const pairs = Object.entries(req.efforts).filter(([, e]) => typeof e === "string") as [string, string][];+ if (pairs.length === 0) {+ return { pass: false, score: 0, reason: `the ${cfg.tool} call passed no effort, so there is nothing to follow`, evidence: { args: calls[0].args } };+ }+ const logs = debugLogs(session.work_dir);+ if (logs.length === 0) {+ return { pass: false, score: 0, reason: `no claudish debug log in ${session.work_dir ?? "(no work dir)"}/logs`, evidence: { pairs } };+ }+ const usesChild = cfg.tool !== "run_prompt";+ const results: Record<string, Record<string, unknown>> = {};+ let proven = 0;+ for (const [model, level] of pairs) {+ const r: Record<string, unknown> = { level };+ // The process that served this model: the child that pinned the level, or (run_prompt)+ // the MCP server's own proxy, whose [Wire] line names the model.+ const files = usesChild+ ? logs.filter((f) => pinLines(f, model, level).length > 0)+ : logs.filter((f) => wireLinesFor(f, model).length > 0);+ r.logFiles = files.map((f) => f.path);+ if (usesChild) r.pinLine = files.length > 0 ? clip(pinLines(files[0], model, level)[0]) : null;+ let ok: boolean;+ if (cfg.stage === "flag") {+ ok = files.length > 0;+ } else {+ ok = false;+ for (const f of files) {+ const { sent, note } = sentLevel(f, level);+ const e = echoes(f);+ r.wireLines = wireLinesFor(f, model).map((l) => clip(l));+ r.clampNote = note ?? null;+ r.sent = sent;+ r.echoedEfforts = e.efforts;+ r.echoSample = e.sample ?? null;+ if (e.efforts.includes(sent)) {+ ok = true;+ break;+ }+ }+ }+ r.proven = ok;+ if (ok) proven++;+ results[model] = r;+ }+ const score = proven / pairs.length;+ const pass = proven === pairs.length;+ const what = cfg.stage === "flag" ? "child pinned --effort-override" : "provider echoed the sent effort";+ const missing = Object.entries(results)+ .filter(([, r]) => !r.proven)+ .map(([m, r]) => `${m} (${r.level}): ${cfg.stage === "flag" ? "no pin line" : `sent ${r.sent ?? "?"}, echoed [${(r.echoedEfforts as string[] | undefined)?.join(",") ?? ""}]`}`);+ return {+ pass,+ score,+ reason: pass ? `${proven}/${pairs.length}: ${what}` : `${proven}/${pairs.length}: ${what}; missing ${missing.join("; ")}`,+ evidence: { stage: cfg.stage, logFiles: logs.map((l) => l.path), perModel: results },+ };+}diff --git a/ai-docs/benches/recommended-effort/checks/needs-effort.ts b/ai-docs/benches/recommended-effort/checks/needs-effort.tsnew file mode 100644index 00000000..714d2616--- /dev/null+++ b/ai-docs/benches/recommended-effort/checks/needs-effort.ts@@ -0,0 +1,56 @@+/**+ * Measurement (used as a readout): did claudish answer a start call with+ * needs_effort, and how did the agent recover?+ *+ * Score 1 when the call that started work carries, for every model, the+ * Scenario column's level: either first time (no needs_effort) or on the retry after+ * needs_effort. Score 0 otherwise, including no start call at all.+ *+ * Evidence: how many needs_effort answers came before the started call, the first+ * needs_effort text (clipped), and which task-type column(s) the started call's+ * levels are consistent with (every model with a row must match that column).+ *+ * config:+ * tool: team | create_session | run_prompt+ * column: the Scenario's task type+ */+import { canonicalOf, clip, newestEfforts, requested, resultText, startedCall, type Tool } from "./effort-lib.ts";++type Ctx = {+ config?: Record<string, unknown>;+ assertion?: { args?: Record<string, unknown> };+ session?: Parameters<typeof startedCall>[0];+};++export default function evaluate(_output: string, context: Ctx) {+ const cfg = { ...(context.assertion?.args ?? {}), ...((context.assertion?.args?.config as object) ?? {}), ...(context.config ?? {}) } as {+ tool: Tool;+ column: string;+ };+ const session = context.session ?? {};+ const { generationId, efforts } = newestEfforts();+ const { call, all, needsEffortBefore } = startedCall(session, cfg.tool);+ if (!call) return { pass: false, score: 0, reason: `no ${cfg.tool} start call in the Session`, evidence: { needsEffortBefore: 0 } };+ const req = requested(call, cfg.tool);+ const withRow = req.models.filter((m) => efforts[canonicalOf(m)]);+ const columns = new Set(withRow.flatMap((m) => Object.keys(efforts[canonicalOf(m)])));+ const consistent = withRow.length === 0 ? [] : [...columns].filter((col) => withRow.every((m) => efforts[canonicalOf(m)]?.[col]?.effort === req.efforts[m]));+ const noRowClean = req.models.filter((m) => !efforts[canonicalOf(m)]).every((m) => req.efforts[m] === undefined);+ const right = withRow.every((m) => efforts[canonicalOf(m)]?.[cfg.column]?.effort === req.efforts[m]) && noRowClean;+ const n = needsEffortBefore.length;+ const story = n === 0 ? "no needs_effort" : `${n} needs_effort answer(s), then ${right ? "a retry with" : "a retry WITHOUT"} the ${cfg.column} levels`;+ return {+ pass: right,+ score: right ? 1 : 0,+ reason: `${story}; levels consistent with column(s): ${consistent.join(", ") || "none"}`,+ evidence: {+ generationId,+ column: cfg.column,+ needsEffortBefore: n,+ startCalls: all.length,+ firstNeedsEffortText: n > 0 ? clip(resultText(session, needsEffortBefore[0]), 1200) : null,+ startedArgs: call.args,+ consistentColumns: consistent,+ },+ };+}diff --git a/ai-docs/benches/recommended-effort/effort.experiment.yaml b/ai-docs/benches/recommended-effort/effort.experiment.yamlnew file mode 100644index 00000000..66c3c062--- /dev/null+++ b/ai-docs/benches/recommended-effort/effort.experiment.yaml@@ -0,0 +1,27 @@+description: "recommended effort: installed vs worktree claudish-usage skill, exact ids vs model families"+bench: ./madbench.yaml+# The variants differ in the staged plugin folder (one file: the claudish-usage+# SKILL.md) and in prompt params, which are not staged inputs.+guard_changes:+ allow: [harness/config/plugins]+variants:+ # Baseline: the installed claudish@magus 2.10.0 skill (no effort section), exact+ # model ids, effort never mentioned.+ - name: installed-skill+ # The skill under test, same prompts.+ - name: worktree-skill+ params:+ plugin: ./plugins/worktree/claudish+ # The skill under test, the user names only model families: the agent chooses the+ # ids (bare, per the bare-name Check), and no MiniMax model has a+ # table row, so its effort must still be omitted.+ - name: worktree-family+ params:+ plugin: ./plugins/worktree/claudish+ ui_models: "two different OpenAI GPT models"+ ui_named: []+ review_model: "an OpenAI GPT model"+ review_named: []+ typo_models: "a GPT model and a MiniMax model"+ typo_named: []+ names_note: ""diff --git a/ai-docs/benches/recommended-effort/madbench-skill-gap.md b/ai-docs/benches/recommended-effort/madbench-skill-gap.mdnew file mode 100644index 00000000..d8458319--- /dev/null+++ b/ai-docs/benches/recommended-effort/madbench-skill-gap.md@@ -0,0 +1,140 @@+# Draft for MadAppGang/madbench: document how to swap ONE plugin in madbench-evals++Status: draft, not filed. It needs Jack's approval first. Classification (madbench-evals+gap rule, step 2): **present but hard to find**. The feature exists and works at preflight. The+skills mention it, but they do not show the case of swapping one installed plugin, or what it+costs a bench that needs host state. This is a skills documentation fix, so no stand-in is+needed.++```+$ madbench version+madbench 0.54.0+$ madbench skills --check --dir ~/.claude/plugins/cache/magus/madbench/0.12.3/skills+skills 0.53.0 · madbench 0.54.0 · stale+```++## Title++madbench-evals: show `harness.config.plugins` as the way to swap one installed plugin, and what+`home` then needs++## The case that hit it++claudish's `recommended-effort` Bench (`ai-docs/benches/recommended-effort/` in+MadAppGang/claudish) has to compare the installed `claudish@magus` plugin's `claudish-usage`+skill with a working-tree version of that skill. Everything else must stay the same. The Bench+ran at `sandbox: workspace` because level `home` hides `~/.claudish`. At `workspace` the host+loaded the installed plugin, so the skill under test was never in the run. Nothing in+madbench-evals told the author which key swaps one plugin. The answer, `harness.config.plugins`,+is in `docs/harness.md` §3a and §4 ("Plugin tree | `config.plugins` (preferred)",+`~/.claude/plugins/cache/magus/dev/3.3.0 # a folder inside your own registry`). The skill has+the key, but three facts that decide whether a bench can use it are missing.++## What madbench-evals lacks (measured against the 0.53.0 skill set)++1. **The swap itself.** `harness-and-sandbox.md` §6 and `schema.md` §1a describe staging a+ plugin folder. Neither says that staging REPLACES the host's plugin set: the registry lives+ in the run's own HOME, so no host plugin loads unless it is staged too. Neither shows the+ use "run against this version of a plugin I have installed" (a cache folder, `~` expanded).+ `docs/harness.md` shows both.+2. **`config.plugins` and host state.** §6 ends "Plugin staging needs sandbox level `home` or+ higher". It does not say what a bench loses at `home` (every host file under `~`), or that+ `sandbox.share.paths` with `access: rw` brings one path back+ (`docs/sandbox-levels.md` § Sharing things into a sandbox). A bench whose tool keeps its+ state in `~/.<tool>` reads the first sentence and stays at `workspace`, where `config.plugins`+ is refused (`docs/harness.md` §3a, Requirements). It ends up running the host's plugin instead.+3. **Symlinks in a staged plugin stay symlinks.** A plugin folder whose skill is a symlink into+ a working tree, the obvious way to keep it current, is copied with the link intact+ (v0.54.0 tree: `internal/sandbox/common.go` `copyDir` recreates `e.Link` with+ `os.Symlink`). At `home` on macOS the read confinement should then deny the target. That+ follows from `docs/sandbox-levels.md` § `home` and was not run. `docs/` does not cover it,+ and preflight cannot see it.++Separately, the skill set is stale against 0.54.0: `harness-and-sandbox.md` §9 says+"`version` | print version — **there is no `--version` flag**". 0.54.0 ships `madbench --version`+and `madbench changelog` (`madbench changelog`: "madbench --version now works and prints what+madbench version prints").++## Proposed patch to `skills/madbench-evals/harness-and-sandbox.md` §6++Append after the paragraph that starts "Use `config.plugins` alone when…":++```markdown+**Swapping one plugin.** Staging is the way to run against a plugin version of your choosing,+an installed one (`~/.claude/plugins/cache/<mkt>/<plugin>/<version>`, `~` expanded) or a+folder beside the bench. Staging REPLACES the host's plugin set: the registry is the run's+own HOME, so a host plugin you did not stage does not load. To A/B one file of a plugin+(a skill's `SKILL.md`), stage two folders that differ only in that file, one per variant+through a param (`plugins: ["{{plugin}}"]`), and declare+`guard_changes: {allow: [harness/config/plugins]}`.++**It needs `home`, so bring back what the tool needs.** At `none` and `workspace` madbench+refuses `config.plugins`. At `home`, every host path under `~` is hidden. A tool that keeps+state there (credentials, caches) gets it back with a live share:++ sandbox:+ level: home+ share:+ paths:+ - { from: ~/.mytool, to: ~/.mytool, access: rw }++**Copy files into the folder; do not symlink them.** A staged plugin is copied with its+symlinks intact. At `home` on macOS the read confinement denies a link target outside the+sandbox, and preflight does not catch it.+```++And in §9's command table, replace the `version` row:++```markdown+| `version` | print version; `madbench --version` prints the same (0.54.0+) |+| `changelog` | print the release notes (0.54.0+) |+```++## Why not a local workaround++A copy of the skill into the workspace (`<testdata>/.claude/skills/…`) is project scope at+`workspace`. The installed plugin's skill still loads beside it at user scope, so the run has+both and measures neither. madbench's own mechanism is the only clean swap.++## Addendum: `home` and tools installed under `~` (measured on the same Bench, 2026-10-23)++A bench that moves to `home` for `config.plugins` hits the read confinement wherever its MCP+server or the processes that server spawns live under the real home. It failed four times in a+row. Each time the cause showed up in a different place:++| Step | What was hidden | What the run reported |+|---|---|---|+| MCP `command: "bun"` → `/Users/jack/.bun/bin/bun` | `~/.bun` | run3, 24/24: `environment:mcp-reachable: MCP server "claudish" did not answer madbench's pre-launch handshake: the initialize deadline elapsed before the server answered: context deadline exceeded; exit: signal: killed`, plus the CLI's `{"error":"Connection closed","errorCode":"CONNECTION_CLOSED","name":"claudish"}` |+| a bundle at `<repo>/packages/cli/dist/index.js` importing `@opentui/*` from `<repo>/packages/cli/node_modules` → `<repo>/node_modules/.bun` | the package root above the declared script directory | run4/run5: `the server closed stdout before answering initialize (the CONNECTION_CLOSED shape); stderr: error: Unexpected while resolving package '@opentui/core' from '…/packages/cli/dist/index.js'`. Sharing only the two `node_modules` trees did not fix it; sharing `packages/cli` and the root `package.json` did |+| the server's own temp root `/tmp/claudish-mux-501` | `/tmp` outside the allow-set | run6: the server logged `[mcp] stale pane sweep failed: EPERM: operation not permitted, lstat '/tmp/claudish-mux-501'` |+| an interactive Claude Code child the server starts in the same HOME | madbench's first-run seeding only on an interactive host | run7: the child stopped at "Welcome to Claude Code v2.1.309 … Let's get started. Choose the text style…" (claudish: `first_run_dialog`) because the host was `interactive: false` |++Separately, `access: ro` is refused at `home`, and only at preflight: `madbench list` loads the+bench. The message is `sandbox: level home cannot share "/Users/jack/.bun" read-only — a local process writes anything it can read; use access: copy to snapshot it, or level container`.+`copy` is not usable for `~/.bun` (29 GB), so the shares are `rw`.++Two things to report:++1. **Bug-shaped (needs a reproduction from madbench):** in the first case the failed-server row+ carried no stderr and no hint that the `command` itself was unreadable. `docs/harness.md`+ says a failed row carries "the server's own words — JSON-RPC error, stderr tail, exit+ status". A binary that cannot be executed produces no words. Preflight could flag a bare+ `command` that resolves on PATH into a directory the `home` profile will hide. It already+ resolves `command` on this machine (§ MCP declarations).+2. **Skills text (proposed, `harness-and-sandbox.md` §4 or §7a):**++```markdown+**At `home`, a server's toolchain is part of the server.** The profile declares an MCP+server's absolute `command`, `args` and `cwd`, and the directory its script sits in. It does+NOT declare:+- a bare `command` that PATH resolves under `~` (`bun` in `~/.bun/bin`, `claude` in+ `~/.local/bin`): write it absolute, and share the install directory (`access: rw`; `ro` is+ container-only, and `copy` duplicates the whole tree);+- the package root above the script (`node_modules`, `package.json`): share it;+- temp or socket directories the server uses (`/tmp/<tool>-<uid>`): point the tool at a+ dedicated directory and share that.+Interactive children the server starts in the run's HOME need madbench's first-run seeding,+which only an interactive host gets: make the host `interactive: true` with+`--permission-mode bypassPermissions`. Prove the server with one single-trial Scenario+(`environment:mcp-reachable`) before a sweep.+```diff --git a/ai-docs/benches/recommended-effort/madbench.yaml b/ai-docs/benches/recommended-effort/madbench.yamlnew file mode 100644index 00000000..e35f445d--- /dev/null+++ b/ai-docs/benches/recommended-effort/madbench.yaml@@ -0,0 +1,337 @@+description: "claudish MCP: does the caller pass list_models' recommended effort, and does it reach the wire?"++# The table under test is generation-dependent: the Checks read the newest+# generation of ~/.claudish/cloud-models-catalog-v3.efforts.json at grading time+# (the same one list_models renders), so no recommended level is pinned in this file.+params:+ # The calling agent. It reaches the models under test only through claudish.+ host: claude-sonnet-5-5+ # THIS worktree's build. The installed claudish (and its plugin's MCP server)+ # predates the effort table, so the run must not touch it.+ claudish_js: /Users/jack/mag/claudish/.claude/worktrees/claudish-effort-level/packages/cli/dist/index.js+ # The bun that runs it, ABSOLUTE: at sandbox level `home` the harness declares an absolute+ # MCP `command` readable, while a bare `bun` resolves into ~/.bun, which the level hides+ # (run3: the server never executed). Override with --param bun_bin=<path>.+ bun_bin: /Users/jack/.bun/bin/bun+ # The worktree root. The bundle in claudish_js imports @opentui/* at run time from+ # packages/cli/node_modules, whose entries link into <root>/node_modules/.bun; the+ # `home` read confinement declares only dist/ (run4 probe: "error: Unexpected while+ # resolving package '@opentui/core'"), so both node_modules trees are shared below.+ claudish_root: /Users/jack/mag/claudish/.claude/worktrees/claudish-effort-level+ # The pane socket root for team / create_session children (CLAUDISH_PANE_ROOT). The+ # default, /tmp/claudish-mux-<uid>, is outside the `home` read allow-set (run6 probe:+ # "[mcp] stale pane sweep failed: EPERM: operation not permitted, lstat+ # '/tmp/claudish-mux-501'"), and it holds the user's own live panes. A dedicated+ # mode-700 directory, created once by hand (`mkdir -m 700 -p <path>`), is shared below.+ pane_root: /private/tmp/claudish-bench-mux+ # The last line of every prompt. Empty (the default): the user never mentions+ # effort, and only the tool descriptions, the skill and list_models' output lead+ # the agent to the table.+ hint: ""+ # The claudish plugin the host is GIVEN, staged by harness.config.plugins. The two+ # folders differ in one file, skills/claudish-usage/SKILL.md:+ # ./plugins/installed/claudish the installed claudish@magus 2.10.0 skill (no effort section)+ # ./plugins/worktree/claudish this worktree's packages/cli/skills/claudish-usage/SKILL.md+ plugin: ./plugins/installed/claudish+ # How the user names the models. The defaults are exact ids; the `worktree-family`+ # variant names only families ("a GPT model and a MiniMax model"), with [] as the+ # named list so the Checks look up whatever the agent chose.+ ui_models: "gpt-6-astra and gpt-6.1-sol"+ ui_named: [gpt-6-astra, gpt-6.1-sol]+ review_model: "gpt-6-luna"+ review_named: [gpt-6-luna]+ typo_models: "gpt-6-astra and minimax-m3"+ typo_named: [gpt-6-astra, minimax-m3]+ names_note: "Use the model names exactly as I wrote them."++harness:+ type: claude-code+ model: "{{host}}"+ use_subscription: required+ # madbench's mechanism for giving the run ONE plugin of our choosing: the folder is+ # staged as an installed, user-scoped, enabled registry in the run's own HOME, so the+ # host's installed claudish@magus (and every other host plugin) is not loaded.+ config:+ plugins: ["{{plugin}}"]+ # The MCP server is declared inline (a file path would resolve in the workspace)+ # and --strict-mcp-config keeps any plugin's MCP server out.+ # CLAUDISH_BIN team / create_session children run the same build, not PATH's claudish+ # CLAUDISH_DEBUG the server and every child write a debug log to <workspace>/logs/,+ # which Layer B reads (pin line, [Wire] lines, raw SSE)+ # --allowedTools "Bash(sleep:*)" mcp__claudish: the host polls team / list_sessions and needs a+ # wait between polls; under --print an un-allowed Bash call is denied.+ args:+ - --mcp-config+ - '{"mcpServers":{"claudish":{"command":"{{bun_bin}}","args":["{{claudish_js}}","--mcp"],"env":{"CLAUDISH_BIN":"{{claudish_js}}","CLAUDISH_DEBUG":"1","CLAUDISH_PANE_ROOT":"{{pane_root}}"}}}}'+ - --strict-mcp-config+ # Interactive: Bash parks at an approval menu nobody answers unless the mode is bypass.+ - --permission-mode+ - bypassPermissions+ - --allowedTools+ - "Bash(sleep:*)"+ # At `home` the user's own permission settings are gone, and --print then denies an+ # MCP tool call (run6 probe: "Claude requested permissions to use+ # mcp__claudish__create_session, but you haven't granted it yet"). Allow the server.+ - mcp__claudish++defaults:+ # home, because config.plugins stages its registry into the run's own HOME (madbench+ # refuses it at workspace). home hides the real ~/.claudish (credentials, catalog,+ # the effort sidecar), so it is shared back in live. The MCP server, its children and+ # the Checks then read the same ~/.claudish as before.+ sandbox:+ level: home+ share:+ paths:+ - { from: ~/.claudish, to: ~/.claudish, access: rw }+ # Tool install dirs under the real home, which `home` hides. `ro` is refused+ # below level container and `copy` would copy 29 GB (~/.bun), so they are live+ # (rw) shares: the run CAN write there. bun runs the MCP server and its children;+ # ~/.local/bin/claude -> ~/.local/share/claude/versions/<v> is what every team /+ # session child launches.+ - { from: ~/.bun, to: ~/.bun, access: rw }+ - { from: ~/.local/bin, to: ~/.local/bin, access: rw }+ - { from: ~/.local/share/claude, to: ~/.local/share/claude, access: rw }+ - { from: "{{claudish_root}}/packages/cli", to: ~/.bench-share/cli, access: rw }+ - { from: "{{claudish_root}}/package.json", to: ~/.bench-share/root-package.json, access: rw }+ - { from: "{{claudish_root}}/node_modules", to: ~/.bench-share/root-node_modules, access: rw }+ - { from: "{{pane_root}}", to: ~/.bench-share/pane-root, access: rw }+ # Interactive, because only an interactive run gets madbench's first-run seeding+ # (hasCompletedOnboarding, workspace trust) in the sandbox's ~/.claude.json, and the+ # team / session children are interactive Claude Code in the SAME HOME and cwd. Under+ # --print at `home` a child stopped at the theme picker (run7 probe: FAILED+ # "first_run_dialog", "Welcome to Claude Code v2.1.309 … Let's get started.").+ interactive: true+ timeout: 720s+ capture: log++defaultScenario:+ metrics:+ - "metrics.usd = cost"++metrics:+ - metrics.consultedTable = mean(consultedScore)+ - metrics.effortArgs = mean(argsScore)+ - metrics.effortFlag = mean(flagScore)+ - metrics.effortWire = mean(wireScore)+ - metrics.bareNames = mean(bareScore)+ - metrics.skillUsed = mean(skillScore)+ - metrics.needsEffortRight = mean(needsEffortScore)+ - metrics.needsEffortAnswers = sum(needsEffortCount)+ - 'metrics.spend = { value: sum(usd), unit: "usd", better: "lower" }'++scenarios:+ # ── frontend_ui, team, two named models with different levels ───────────────+ - name: team-frontend+ description: "team run on two named models for a frontend UI task"+ prompt: |+ Use claudish's `team` tool to have {{ui_models}} each do this frontend task+ independently:++ Write a small React component `PricingCard` (TypeScript, Tailwind classes)+ that shows a plan name, a monthly price and a "Choose plan" button. Reply+ with the code only. Do not create or edit any files.++ Use the path `team-ui`. {{names_note}} Wait for both to finish (run `sleep 15` with Bash between status checks), then show me+ each model's answer. {{hint}}+ checks:+ - { type: environment:mcp-reachable, value: claudish }+ # Setup: the staged plugin's skill is the one Claude Code registered.+ - { type: environment:skill-registered, value: "claudish:claudish-usage" }+ # Readout: whether the agent opened the skill at all.+ - { type: session:skill-used, value: "claudish:claudish-usage", metric: SkillUsed, readout: true, metrics: ["metrics.skillScore = score"] }+ # A: bare model names; no prompt here names a provider.+ - type: ts+ metric: BareNames+ metrics: ["metrics.bareScore = score"]+ value: file://checks/bare-names.ts+ config: { tool: team }+ # A0: the table is in list_models' output, so reading it must come first.+ - type: session:tool-sequence+ metric: ConsultedTable+ readout: true # a measurement: needs_effort now supplies the rows too+ metrics: ["metrics.consultedScore = score"]+ value: [mcp__claudish__list_models, mcp__claudish__team]+ # A: each model's effort == its frontend_ui level.+ - type: ts+ metric: EffortArgs+ metrics: ["metrics.argsScore = score"]+ value: file://checks/effort-args.ts+ config: { tool: team, column: frontend_ui, models: "{{ui_named}}" }+ # Readout: needs_effort answers before the started call, and whether the retry+ # carried this column's levels (Evidence names the column(s) it matched).+ - type: ts+ metric: NeedsEffort+ readout: true+ metrics: ["metrics.needsEffortScore = score", "metrics.needsEffortCount = evidence.needsEffortBefore"]+ value: file://checks/needs-effort.ts+ config: { tool: team, column: frontend_ui }+ # B1: each slot's child claudish pinned the level it was given.+ - type: ts+ metric: EffortFlag+ metrics: ["metrics.flagScore = score"]+ value: file://checks/effort-plumbing.ts+ config: { tool: team, stage: flag }+ # B2: the provider echoed that level on the child's request.+ - type: ts+ metric: EffortWire+ metrics: ["metrics.wireScore = score"]+ value: file://checks/effort-plumbing.ts+ config: { tool: team, stage: wire }++ # ── code_review, create_session, one named model ─────────────────────────────+ - name: session-code-review+ description: "create_session on one named model for a code review"+ prompt: |+ Use claudish's `create_session` tool to have {{review_model}} review this+ function for bugs. Tell it to reply with a short list of the bugs it finds and not to+ create or edit any files.++ function average(xs: number[]): number {+ let sum = 0;+ for (let i = 1; i <= xs.length; i++) sum += xs[i];+ return sum / xs.length;+ }++ {{names_note}} Wait for the session to finish (run `sleep 15` with Bash+ between list_sessions checks), then show me its review. {{hint}}+ checks:+ - { type: environment:mcp-reachable, value: claudish }+ # Setup: the staged plugin's skill is the one Claude Code registered.+ - { type: environment:skill-registered, value: "claudish:claudish-usage" }+ # Readout: whether the agent opened the skill at all.+ - { type: session:skill-used, value: "claudish:claudish-usage", metric: SkillUsed, readout: true, metrics: ["metrics.skillScore = score"] }+ # A: bare model names; no prompt here names a provider.+ - type: ts+ metric: BareNames+ metrics: ["metrics.bareScore = score"]+ value: file://checks/bare-names.ts+ config: { tool: create_session }+ - type: session:tool-sequence+ metric: ConsultedTable+ readout: true # a measurement: needs_effort now supplies the rows too+ metrics: ["metrics.consultedScore = score"]+ value: [mcp__claudish__list_models, mcp__claudish__create_session]+ - type: ts+ metric: EffortArgs+ metrics: ["metrics.argsScore = score"]+ value: file://checks/effort-args.ts+ config: { tool: create_session, column: code_review, models: "{{review_named}}" }+ # Readout: needs_effort answers before the started call, and whether the retry+ # carried this column's levels (Evidence names the column(s) it matched).+ - type: ts+ metric: NeedsEffort+ readout: true+ metrics: ["metrics.needsEffortScore = score", "metrics.needsEffortCount = evidence.needsEffortBefore"]+ value: file://checks/needs-effort.ts+ config: { tool: create_session, column: code_review }+ - type: ts+ metric: EffortFlag+ metrics: ["metrics.flagScore = score"]+ value: file://checks/effort-plumbing.ts+ config: { tool: create_session, stage: flag }+ - type: ts+ metric: EffortWire+ metrics: ["metrics.wireScore = score"]+ value: file://checks/effort-plumbing.ts+ config: { tool: create_session, stage: wire }++ # ── implementation, run_prompt, the agent picks the model ────────────────────+ - name: run-prompt-pick+ description: "run_prompt on a model the agent picks, for an implementation task"+ prompt: |+ I need a TypeScript function `isLeapYear(year: number): boolean` that follows+ the Gregorian calendar rules. Ask one of claudish's recommended OpenAI GPT+ models to write it, with the `run_prompt` tool: pick whichever one you think+ fits this kind of task best. Show me the code it returns. {{hint}}+ checks:+ - { type: environment:mcp-reachable, value: claudish }+ # Setup: the staged plugin's skill is the one Claude Code registered.+ - { type: environment:skill-registered, value: "claudish:claudish-usage" }+ # Readout: whether the agent opened the skill at all.+ - { type: session:skill-used, value: "claudish:claudish-usage", metric: SkillUsed, readout: true, metrics: ["metrics.skillScore = score"] }+ # A: bare model names; no prompt here names a provider.+ - type: ts+ metric: BareNames+ metrics: ["metrics.bareScore = score"]+ value: file://checks/bare-names.ts+ config: { tool: run_prompt }+ - type: session:tool-sequence+ metric: ConsultedTable+ readout: true # a measurement: needs_effort now supplies the rows too+ metrics: ["metrics.consultedScore = score"]+ value: [mcp__claudish__list_models, mcp__claudish__run_prompt]+ # No `models:`: the agent chose. The expected level is looked up for whatever it ran.+ - type: ts+ metric: EffortArgs+ metrics: ["metrics.argsScore = score"]+ value: file://checks/effort-args.ts+ config: { tool: run_prompt, column: implementation }+ # Readout: needs_effort answers before the started call, and whether the retry+ # carried this column's levels (Evidence names the column(s) it matched).+ - type: ts+ metric: NeedsEffort+ readout: true+ metrics: ["metrics.needsEffortScore = score", "metrics.needsEffortCount = evidence.needsEffortBefore"]+ value: file://checks/needs-effort.ts+ config: { tool: run_prompt, column: implementation }+ # run_prompt has no child: only the wire stage applies (the MCP server's own proxy).+ - type: ts+ metric: EffortWire+ metrics: ["metrics.wireScore = score"]+ value: file://checks/effort-plumbing.ts+ config: { tool: run_prompt, stage: wire }++ # ── quick_edit, team, one named model in the table and one not ──────────────+ - name: team-no-entry+ description: "team run where one named model has no table entry: its effort must be omitted"+ prompt: |+ Use claudish's `team` tool to have {{typo_models}} each make this one-line+ fix independently:++ The line `return "Helo, " + name;` has a typo. Reply with the corrected+ line only. Do not create or edit any files.++ Use the path `team-typo`. {{names_note}} Wait for both to finish (run `sleep 15` with Bash between status checks), then show me+ each model's answer. {{hint}}+ checks:+ - { type: environment:mcp-reachable, value: claudish }+ # Setup: the staged plugin's skill is the one Claude Code registered.+ - { type: environment:skill-registered, value: "claudish:claudish-usage" }+ # Readout: whether the agent opened the skill at all.+ - { type: session:skill-used, value: "claudish:claudish-usage", metric: SkillUsed, readout: true, metrics: ["metrics.skillScore = score"] }+ # A: bare model names; no prompt here names a provider.+ - type: ts+ metric: BareNames+ metrics: ["metrics.bareScore = score"]+ value: file://checks/bare-names.ts+ config: { tool: team }+ - type: session:tool-sequence+ metric: ConsultedTable+ readout: true # a measurement: needs_effort now supplies the rows too+ metrics: ["metrics.consultedScore = score"]+ value: [mcp__claudish__list_models, mcp__claudish__team]+ # A: the GPT model gets its quick_edit level; no MiniMax model has a row in the table, so no key for it.+ - type: ts+ metric: EffortArgs+ metrics: ["metrics.argsScore = score"]+ value: file://checks/effort-args.ts+ config: { tool: team, column: quick_edit, models: "{{typo_named}}" }+ # Readout: needs_effort answers before the started call, and whether the retry+ # carried this column's levels (Evidence names the column(s) it matched).+ - type: ts+ metric: NeedsEffort+ readout: true+ metrics: ["metrics.needsEffortScore = score", "metrics.needsEffortCount = evidence.needsEffortBefore"]+ value: file://checks/needs-effort.ts+ config: { tool: team, column: quick_edit }+ - type: ts+ metric: EffortFlag+ metrics: ["metrics.flagScore = score"]+ value: file://checks/effort-plumbing.ts+ config: { tool: team, stage: flag }+ - type: ts+ metric: EffortWire+ metrics: ["metrics.wireScore = score"]+ value: file://checks/effort-plumbing.ts+ config: { tool: team, stage: wire }diff --git a/ai-docs/benches/recommended-effort/plugins/installed/claudish/.claude-plugin/plugin.json b/ai-docs/benches/recommended-effort/plugins/installed/claudish/.claude-plugin/plugin.jsonnew file mode 100644index 00000000..59c408c4--- /dev/null+++ b/ai-docs/benches/recommended-effort/plugins/installed/claudish/.claude-plugin/plugin.json@@ -0,0 +1,12 @@+{+ "name": "claudish",+ "version": "2.10.0",+ "description": "MCP runtime for external AI models. Proxies OpenRouter, Ollama and LM Studio, runs blind team voting, manages async sessions, and shows the session's runs live above the prompt, with stop and live-screen tabs. Required by code-search, dev, multimodel and designer.",+ "author": {+ "name": "Jack Rudenko",+ "email": "[email protected]",+ "company": "MadAppGang"+ },+ "license": "MIT",+ "skills": "./skills/"+}diff --git a/ai-docs/benches/recommended-effort/plugins/installed/claudish/skills/claudish-usage/SKILL.md b/ai-docs/benches/recommended-effort/plugins/installed/claudish/skills/claudish-usage/SKILL.mdnew file mode 100644index 00000000..d96f9859--- /dev/null+++ b/ai-docs/benches/recommended-effort/plugins/installed/claudish/skills/claudish-usage/SKILL.md@@ -0,0 +1,1103 @@+---+name: claudish-usage+description: Runs models through the claudish MCP tools — team, create_session, run_prompt — and resolves model IDs against the live catalog. Use when the user mentions claudish, OpenRouter, or external AI models.+user-invocable: false+---++# Claudish Usage Skill++**Purpose:** How agents run models — external and native — through the claudish MCP tools+**Status:** Production Ready++## Everything runs through the MCP tools++There is no CLI tier and no "direct usage" tier. **Every model run goes through a claudish+MCP tool** — one model or a panel, external or native, inside `/team` and `/delegate` or+anywhere else. `Bash("claudish ...")` is not a fallback, not a shortcut for a small task,+and not acceptable "just this once".++| You need | Tool |+|---|---|+| A panel of models on one prompt, in parallel | `team` |+| One model, long task, its own tools and working directory | `create_session` |+| One model, one completion, no tools and no session | `run_prompt` |+| What models exist right now | `list_models`, `search_models` |+| Debugging claudish: the same prompt through several models' APIs | `compare_models` (debug tool) |+| Report a failure to claudish's developers | `report_error` |++Inside a session that claudish started, `team`, `create_session`, the session tools and+`report_error` are absent. See "Inside a claudish session (child mode)" below.++Two properties are why: MCP sessions run in their own process, so a model's transcript+never enters your context window; and every tool returns a **structured per-slot result**+with status and errors, so a failure is data you can act on rather than text you have to+parse. Both are lost the moment you shell out.++The only CLI that survives is a set of three read-only diagnostics — see+"Diagnostics: the only CLI left". They investigate the runtime. They never run a task.++## Inside a claudish session (child mode)++**How to tell.** Run exactly `echo "CLAUDISH_PANE_CHILD=${CLAUDISH_PANE_CHILD:-unset}"`,+with nothing appended. If it prints `CLAUDISH_PANE_CHILD=1`, claudish started this session,+and its MCP server runs in nested mode. `CLAUDISH_PANE_CHILD=unset` means it did not. That+variable is the deciding signal. Never read an exit status as its value: `printenv` exits 1+when the variable is unset, which once read as "child mode" and skipped a whole panel.++A missing tool is not enough on its own. Claude Code can defer MCP tools behind tool+search, so a parent session can list `create_session` only as a deferred name, and a+session without claudish has no claudish tools at all. Neither one is child mode. The tool+shape that does match child mode is `list_models` present while `create_session` and `team`+are absent, with deferred names counted as present. If claudish has no tools at all, claudish+is unreachable: follow the workflow's own path for that case, not this section.++**What it means.** You are the delegate. claudish allows one level of nesting, so no+session tier exists below you. The session that started you is waiting for your result.++**What to do.** Do the task yourself, with your own tools and subagents. If a workflow step+says to delegate to an external model, `create_session` or `team`, you already satisfy that+step by running it. Where the workflow names an internal path, take it (for example the+internal writer in `/dev:qa`), and say in your report that you took it because the session+is nested.++**What you still have.** `run_prompt` gives a one-shot second opinion from another model.+`list_models` and `search_models` still resolve IDs. `compare_models` is also there, but+it is a debug tool for claudish's developers, not a second opinion.++**Why the internal path is the right one.** Another agent CLI started from a shell or a+terminal pane (`codex exec`, `claude -p`, `claudish -p`) rebuilds the nesting that the limit+exists to prevent. It also hides the work from the session that asked for it: that session+sees neither the transcript nor a structured result. Your own tools keep the work visible+and the result in one place.++## Quick Start++### Step 1: Install the runtime (once)++```bash+npm install -g claudish # or: bun add -g claudish+```++This installs the **runtime**, not a way to run tasks. The plugin's `.mcp.json` starts the+MCP server by launching this binary with `--mcp`, so it must be on `PATH` or the tools+below do not exist at all. Restart the session after installing so the server registers.++### Step 2: Find out what models exist right now++```+list_models() → current recommended set, pricing, capabilities, access lines+search_models({ query: "kimi" }) → every live variant in one family+```++Never skip this. Model IDs from memory are dead IDs — see "Model Alias Resolution".++### Step 3: Run something++```+// One completion, no tools, no session lifecycle+run_prompt(model="grok", prompt="Review this diff for security issues")++// A real Claude Code session with tools, in a working directory+create_session(model="grok", prompt=TASK_PROMPT, timeout_seconds=300,+ agent="dev:developer", work_dir=WORK_DIR)+→ the bounded wait fires ("Waiting for a run to end") → get_output(ID)++// A whole panel. `run` STARTS it and returns immediately — it does not wait.+team(mode="run", path=SESSION_DIR, models=["internal", "grok", "gemini"],+ input_file=SESSION_DIR + "/input.md", require_pattern="```vote", agent="dev:researcher")+→ wait until settled ("Waiting for a run to end") → team(mode="status") → read response-NN.md+```++## Model Alias Resolution++All commands that use external models (/team, /delegate, /dev:fix, etc.) MUST resolve model names through this three-step chain before calling claudish.++**The catalog is live, never committed.** Model IDs come from the `list_models` /+`search_models` MCP tools, which claudish serves from its own Firebase-backed+catalog with a 24-hour cache. There is no model-aliases file in this repo, and+you must not resolve model IDs from memory — training data carries dead IDs.++### Three-Step Resolution Chain++```+Step 1: INTERPRET (Claude Code LLM)+ User says anything → Claude infers what family/capability they mean+ "use Elon's model" → xAI family+ "the Google one" → Google family+ "kimi3" → Moonshot family, major version 3+ "latest gpt" → OpenAI family, newest version++Step 2: RESOLVE (live catalog lookup — list_models / search_models)+ Family/intent → an ID that EXISTS in the live catalog right now+ xAI family → whatever grok-* the catalog currently lists+ "kimi3" → kimi-k3 (matched in the catalog, not guessed)+ "latest gpt" → highest gpt-* version the catalog lists++Step 3: ROUTE (Claudish)+ Live model ID → correct provider API endpoint+```++### Resolving a model name++1. Call `list_models` first — it is cheap, cached, and returns the current+ recommended set with pricing, capabilities and access prefixes.+2. If the user's request isn't covered there, call `search_models` with the+ family name (e.g. `search_models("kimi")`) to see every live variant.+3. Check `.claude/multimodel-team.json` → `customAliases` for a user-defined+ shorthand. A custom alias always wins on key conflict — but if it maps to an+ ID the catalog no longer lists, say so instead of using it silently.+4. `"internal"` and `"default"` select the host Claude tier, and `"opus"`/`"sonnet"`/+ `"haiku"` a specific one. They ARE sent to claudish and run through its native+ passthrough on the user's own subscription — no API key, no provider prefix, no+ translation. They are not catalog IDs, so `list_models` will not list them and a+ catalog check must not reject them. Requires `claudish >= 8.0.0` — native names have+ been runnable since 7.65.0, but the `team` contract in this skill needs 8.0.0.++### Use the resolver — do not do this by hand++`resolve-models.ts` performs the whole check and prints the disclosure. It ships **in this+plugin**, beside this skill, so `${CLAUDE_PLUGIN_ROOT}` resolves to it on every channel+claudish publishes to. Call `list_models` first, then hand it the IDs:++```bash+bun "${CLAUDE_PLUGIN_ROOT}/scripts/resolve-models.ts" \+ --catalog "<comma-separated ids from list_models>" [--context review] [--json]+```++It verifies every model-bearing field, drops dead IDs individually, computes+provenance, and emits a receipt — **print that receipt verbatim.** Exit `3` means+nothing survived; `0` means proceed with what it selected.++Doing this in your head is what the rest of this section explains. That is the fallback for+paths the resolver does not cover, and for consumers that depend on claudish without+`multimodel` installed beside it. Measured over 30 benchmark runs, prose alone produced+the disclosure at best 14/15 times; the resolver produces it every time, because it is code.++### Verify every field of the preferences file++**Every field of the preferences file is untrusted.** `customAliases` is not the only place+a dead ID hides, and in practice it is the least likely — a file found in the wild had+`customAliases: {}` and six decommissioned IDs sitting in `defaultModels`. **Verify every+ID you take from this file against the live catalog, whichever field it came from:**++| Field | Verify? |+|---|---|+| `defaultModels` | yes |+| `contextPreferences[*]` | yes |+| `customAliases` values | yes |++No field is exempt. Drop each ID the catalog does not list, name the dropped IDs+in your reply, and **carry on with the survivors.**++**A dead entry invalidates that entry, never the request.** Resolving is the next+step, not a fallback:++- A stale `customAliases` mapping means *the alias* is wrong. If the user named a+ version, resolve that intent against the catalog and use what you find —+ `kimi3` with a dead `kimi3 → kimi-k2.5` alias still resolves to `kimi-k3` when+ the catalog lists it.+- Dead entries in `defaultModels` or `contextPreferences[*]` mean *those entries*+ are wrong. Run with whatever survives.++Returning "no models" is correct only when the catalog genuinely offers nothing+that satisfies the request. Refusing a run while a live model sits in the catalog+is the same failure as using a dead one — it just fails in the other direction.++### Report what the check found, not what the file claims++When you report your model choice, **state the result of the catalog check**:++> `3 of 7 saved model IDs are no longer in the catalog: <id-a>, <id-b>, <id-c>`++Report it every run, including when nothing was dropped — `all 5 saved IDs are+still live` is the same disclosure with a different value.++That count is derived from the comparison you just performed, so it **cannot be+silently wrong**. It is the disclosure that matters: the user's question is "are+my models alive and what did you actually use", not "what date is in my file".++### File age is secondary, and `lastUpdated` cannot carry it++If you state an age, take it from the **file's modification time** and say so:+`preferences file modified 12 days ago (filesystem mtime)`.++- **Never present `lastUpdated` as the file's age.** It is declared metadata and+ is not maintained by every write path — a file has been seen reporting March+ while its own `history[0].date` said July. Quoting it as an age is false+ precision.+- If `lastUpdated` and the newest `history[].date` disagree, report+ `freshness metadata inconsistent` and name both. Do not pick the newer one.+- If no trustworthy source exists, `freshness unknown` is a complete answer.+- `mtime` has its own limits — a checkout or copy resets it — which is exactly+ why the source is always labelled.++**Age never gates.** It never rejects a model (a 157-day-old file whose IDs are+all live is fine — use it) and never approves one (a file written today can be+entirely dead). Catalog membership is what decides; age is context for the human.++### Version intent is a hard constraint, not a hint++When the user names a version — `kimi3`, `gpt-5.6`, `sonnet 5` — that version is+a **requirement**. Resolve it against the catalog and use what you find.++- If the exact version exists → use it.+- If it does not exist → **say so and show the live alternatives.** Ask which+ one they want.+- **NEVER** fall back to a lower version because its name is closer as a string.+ `kimi3` resolving to `kimi-k2.5` is a bug, not a near-miss: string distance+ cannot tell a version bump from a typo, and silently downgrading a model is+ worse than erroring.++### Interpreting User Intent (Step 1)++| User says | Resolve by | Notes |+|---|---|---|+| "grok" | `search_models("grok")` | Take the current flagship, not a remembered ID |+| "Elon's AI" / "xAI model" | `search_models("grok")` | Company association |+| "Google's model" | `search_models("gemini")` | Company association |+| "the cheap one" | `list_models` → Quick picks → Budget | Cost intent |+| "something fast for coding" | `list_models` → Fast variants | Capability intent |+| "biggest context" | `list_models` → Quick picks → Large context | Capability intent |+| "kimi3" | `search_models("kimi")`, require major v3 | Version is a constraint |+| "LATEST_MINIMAX_MODEL" | Verify it's in the catalog, then pass through | Already a full ID |++When uncertain, show the live candidates and ask the user to pick. Listing real+options is always better than guessing one.++### Identity vs routing address++Every catalog record carries the model's **identity** and several **addresses** for+reaching it. Only the identity is the model. Addresses live in sibling fields, so it is+easy to copy the wrong one out of the same record:++| In the record | Example | What it is |+|---|---|---|+| `id` | `kimi-k3` | **the identity — this is the model** |+| `openrouterId` | `moonshotai/kimi-k3` | an address: route via OpenRouter |+| Access line | `kimi@kimi-k3` · `kc@kimi-k3` | addresses: same model, different accounts |++**Bare means no `@` AND no `/`.** `moonshotai/kimi-k3` is not a bare ID — the vendor+slug is a route, not part of the name. Both prefix forms pin the request to one+provider and bypass the subscription-aware backend selection and fallback that passing+`id` gives you. `z-ai/glm-5.2` is as wrong as `[email protected]`, for the same reason.++- If the user names an address (`cx@LATEST_GPT_MODEL`), pass it through **verbatim**.+- Otherwise pass `id`, and let claudish pick the backend.+- Never assemble an address yourself, and never substitute one field for another —+ "the catalog reports it" is not a licence to send it, because the catalog reports+ every address too.++### Responsibility Boundaries++| Responsibility | Owner |+|---|---|+| Understanding user intent → family/capability | **Claude Code** (LLM heuristic) |+| Which model IDs exist right now | **Claudish** (live catalog, 24h cache) |+| User custom aliases | **Magus** (`.claude/multimodel-team.json` `customAliases`) |+| Model ID → API endpoint | **Claudish** (provider routing) |+| API keys, backend fallbacks | **Claudish** |++### Rules++- ALWAYS resolve against the live catalog; NEVER from memory or a committed file+- NEVER invent a model ID — if nothing matches, show live options and ask+- NEVER silently downgrade to an older version than the user asked for+- ALWAYS send the catalog's `id`. NEVER send an address (`vendor/model`,+ `provider@model`) where a model belongs — not even one the catalog reports, since it+ reports `openrouterId` and every Access route alongside `id`. An address goes through+ only when the user named it themselves+- User `customAliases` override, but flag any that the catalog no longer lists++## MCP Tool Reference++Parameter names below are verified against claudish 8.0.0. Use them exactly.++### `team` — a panel of models, started in one call++**`run` starts the panel and returns immediately. It does not wait for the models, and+their answers are not in its response.** You start the run, poll `status` until it settles,+then read each slot's answer off disk. Requires **claudish >= 8.0.0**.++```+team(mode, path, models, judges, input, input_file,+ require_pattern, min_output_bytes, agent, claude_flags, slot)+```++| Parameter | What it does |+|---|---|+| `mode` | `"run"` · `"status"` · `"cancel"` · `"judge"` · `"run-and-judge"` |+| `path` | Session directory. **Must be within the current working directory.** |+| `models` | The panel. Native names are **ordinary entries** — see below. |+| `judges` | Models that judge the collected responses (`judge` / `run-and-judge`) |+| `input` | The prompt every slot receives, inline |+| `input_file` | The same prompt, read from a file. **Prefer this.** Passing both is a hard error. |+| `require_pattern` | Regex the response MUST match, or the slot is reported FAILED |+| `min_output_bytes` | Floor below which a response counts as empty |+| `agent` | Subagent every child runs as. Applies to EVERY child — there is no per-model form. |+| `claude_flags` | Any other Claude Code flags, space-separated. An `--agent` here loses to `agent`. |+| `slot` | `cancel` only — cancel one slot instead of the whole run |++**There is no `timeout` any more, and passing one is silently ignored.** The schema does+not set `additionalProperties: false`, so a leftover `timeout=180` raises no error — it+just does nothing, while the call still reads as though it set a deadline. Nothing+terminates a slot on a timer; see "Deciding whether a quiet slot is stuck" below.++**Prefer `input_file` over `input`.** A panel prompt is routinely 100+ lines, and an inline+`input` echoes the whole thing verbatim in the user's terminal, burying every other+argument in the call. Write the prompt to `<path>/input.md` first, then name that file. The+path must be inside the working directory.++**Native names are ordinary slots.** `internal` / `default` select the host Claude tier;+`opus` / `sonnet` / `haiku` / `claude-*` select a specific one. They go in the same `models`+array as Grok and Gemini and run on the user's Claude subscription through the native+passthrough — no API key, no translation. Because they go through the tool, they are+covered by `require_pattern` like every other slot.++**`require_pattern` is the success oracle, not the exit code.** A slot that exits 0 having+never produced the shape the prompt demanded is reported FAILED (state EMPTY, reason+`shape_mismatch`). Set it whenever the prompt mandates an output shape. Omitting it is how+a vote-less response gets counted as a vote.++**An `agent` with its own report format can override the shape your prompt asks for.** The+agent's definition is the child's system prompt, so its format wins over a format stated in+`input.md`. Measured: a `dev:reviewer` child answered in its reviewer format+(`**Verdict**: PASS`) and failed `require_pattern="VERDICT:"`, so a full review was reported+`shape_mismatch` (EMPTY), while two other slots happened to comply. Before you pair an `agent`+with `require_pattern`, read the agent's output section and make the pattern match a line+that format writes. If the two cannot agree, keep the pattern and run that panel without+`agent`. `shape_mismatch` therefore means "wrong format", not "no answer": the answer is+still in `response-<slot>.md`.++#### The three-step lifecycle++**Every caller of `mode="run"` follows these three steps.** There is no shortcut that skips+step 2 — a workflow that reads results straight out of the `run` response reads nothing.++**Step 1 — start the run.**++```+team(mode="run", path=SESSION_DIR,+ models=[...resolved models, native names included...],+ input_file=`${SESSION_DIR}/input.md`,+ require_pattern=<regex for the shape the prompt mandates>,+ agent=RESOLVED_AGENT)+```++It returns a slot map, not results:++```json+{+ "started": true,+ "team_session_id": "team-20260827-0015",+ "session_path": "/abs/path/to/SESSION_DIR",+ "monitor_record": "team-5e6f7a8b",+ "slots": { "<model-a>": "01", "<model-b>": "02", "internal": "03" },+ "next": { "status": "...", "cancel": "...", "judge": "..." }+}+```++`monitor_record` names the run's end record, the file step 2 waits on. It is not+`team_session_id`, though both start with `team-`. A claudish older than 10.4.0 returns no+`monitor_record`, and step 2 then waits on `<session_path>/status.json`. `slots` maps each+display model name to its anonymised slot id. Keep it — that id addresses everything else+on disk for that model:++| Path | Contents |+|---|---|+| `<session_path>/response-<slot>.md` | the model's answer — parse it from here |+| `<session_path>/stats/<slot>.json` | tokens, cost, tool counts |+| `<session_path>/errors/<slot>.log` | stderr and diagnostics, on failure |+| `<session_path>/errors/<slot>-upstream.jsonl` | raw provider error bodies, when any |++**Step 2 — wait until the run settles, then read `status`.** The wait is the record wait in+**Waiting for a run to end**, or its `status.json` wait when the result has no+`monitor_record`; then make one call:++```+team(mode="status", path=SESSION_PATH)+```++```json+{+ "startedAt": "...",+ "models": {+ "01": { "state": "COMPLETED", "exitCode": 0, "outputSize": 10988 },+ "02": { "state": "RUNNING", "exitCode": null }+ },+ "idle_seconds_by_slot": { "02": 94 },+ "activity_by_slot": { "02": "tool_executing" },+ "summary": "<rendered result card — present ONLY once the run has settled>"+}+```++**Settled means the run's end record exists** — the wait's line says so. With no `monitor_record` it means no slot in `status.json` is `PENDING` or+`RUNNING`. A slot going quiet is not the run settling.++`summary` is the rendered result card the old blocking `run` used to return — `N/M+succeeded`, `reason=shape_mismatch`, and the rest. A workflow that matched on that text+keeps working; it just reads it from a settled `status` instead.++**Bound the wait and fail loudly.** There is no server-side deadline any more, so an+unbounded wait is unbounded. Pick a wall-clock ceiling that suits the work, and on hitting it+report what is still running rather than waiting on. Wait with the bounded background wait in+**Waiting for a run to end**, never a foreground `sleep`.++**Step 3 — read the answers off disk.** For each entry in the `slots` map from step 1, read+`<session_path>/response-<slot>.md`. Slot ids are shuffled, so responses can be read blind+before the mapping is consulted. **Never write an example naming a file like+`grok-result.md`** — the tool does not produce those, and a workflow that expects one reads+nothing.++A slot whose `models[<slot>].state` is `FAILED` or `EMPTY` produced no usable answer;+`error.reason` says why.++#### Deciding whether a quiet slot is stuck++Nothing cancels on your behalf. `status` gives you two fields for this, and they are only+meaningful **read together**:++- `idle_seconds_by_slot` — seconds since that slot's child last wrote anything.+- `activity_by_slot` — `starting`, `running`, `tool_executing`, `finishing`, or a terminal+ state. `starting`: the child is still starting up. `finishing`: the slot's+ answer is complete and its child is exiting; nothing to do.++Ninety seconds of silence in `tool_executing` is a build or a test suite, and is completely+normal. The same ninety seconds in `running` is a model that stopped mid-answer. The reaper+this replaced had no activity signal at all, which is exactly why it killed slots that were+working: in one real session it killed three of five actively-working slots, because its+only progress signal was token flow — and token flow stops during a local tool call. A+model running `go test ./...` looked identical to a hung one.++To cancel, do it deliberately:++```+team(mode="cancel", path=SESSION_PATH, slot="02") # one slot+team(mode="cancel", path=SESSION_PATH) # the whole run+```++A cancelled slot records `error.reason === "cancelled"`, which is distinct from+`nonzero_exit`. Report it as a decision, not a crash.++**Default: do not cancel automatically.** Report the slot as still running and let the user+decide. Losing a vote to an impatient auto-cancel is the same failure the old deadline+caused.++#### `run-and-judge` — the blocking alternative++`mode="run-and-judge"` still blocks and still returns the verdict, so a panel that would+rather not poll can use it as a drop-in. The trade is that it holds the MCP tool call open+for the whole run, which is the shape that hit the client's 1800s idle abort. It also only+fits work that wants a judging stage — **a plain generation call has no blocking+equivalent and must poll.**++### `create_session` — one model, full session++```+create_session(model, prompt, timeout_seconds, agent, claude_flags, work_dir)+ → session_id+```++The child runs a full Claude Code session — plugins, skills, tools, its own working+directory — in its own process. Its transcript never reaches your context window. This is+the tool that replaces every hand-rolled instruction-file-plus-result-file pattern.++#### Waiting for a run to end++`cancel_session(ID)` stops a session; `list_sessions(include_completed=true)` lists them+all. `run_prompt` writes no session record, so there is nothing to wait on.++1. Before ending a turn with a run in flight, start a **bounded** wait on the run's end+ record: the `meta.json` claudish writes once, when the run ends, in its sessions directory.+ Use the `Monitor` tool (`timeout_ms`, at most 1800000), or for a ceiling under ten minutes+ `Bash` with `run_in_background` (`timeout`, at most 600000). Run:++ ```bash+ # a session: SESSION_ID is create_session's session_id+ f="${CLAUDISH_SESSIONS_DIR:-$HOME/.claudish/sessions}/SESSION_ID/meta.json"+ until grep -Eqs '"status": *"(completed|failed|timeout|cancelled)"' "$f"; do sleep 10; done; echo "claudish session SESSION_ID ended"+ # a team run: MONITOR_RECORD is the `monitor_record` field of the `run` result+ f="${CLAUDISH_SESSIONS_DIR:-$HOME/.claudish/sessions}/MONITOR_RECORD/meta.json"+ until grep -Eqs '"status": *"(completed|failed|cancelled)"' "$f"; do sleep 10; done; echo "claudish team run MONITOR_RECORD ended"+ ```++ The test is the end state, not the file's existence: a file that is missing, half+ written or unreadable fails it and is read again ten seconds later. `MONITOR_RECORD` is+ `monitor_record`, not `team_session_id`: both start with `team-`, and the wrong one names a+ directory that never appears, so the wait runs to its ceiling. So does a+ `CLAUDISH_SESSIONS_DIR` set only in the claudish server's `env` block: the wait expands the+ variable in Claude Code's environment, so it moves where claudish writes and not where the+ wait looks. While the result+ carries `monitor_record`, never wait on the run's `status.json`: it is rewritten in place,+ and a slot can stay `RUNNING` in it after the run has ended.++ **A `run` result with no `monitor_record`** comes from a claudish older than 10.4.0, which+ writes no end record. Do not stop waiting: run the same bounded wait, with the same tools+ and ceiling, on the run's `status.json` instead. It needs `bun` on `PATH`:++ ```bash+ # no monitor_record: SESSION_PATH is the `session_path` field of the `run` result+ f="SESSION_PATH/status.json"+ until F="$f" bun --env-file=/dev/null -e 'try{const m=Object.values(JSON.parse(require("fs").readFileSync(process.env.F,"utf8")).models??{});process.exit(m.length>0&&m.every(s=>!/^(PENDING|RUNNING)$/.test(s?.state))?0:1)}catch{process.exit(1)}'; do sleep 10; done; echo "claudish team run settled"+ ```++ It settles when the file parses and no slot in `models` is `PENDING` or `RUNNING`; a file+ that is missing, half written or unparseable is read again ten seconds later. On this path+ a start that failed or a whole-run cancel can leave a slot `RUNNING`, and the wait then runs+ to its ceiling. Tell the user a claudish of 10.4.0 or later ends the wait exactly, but+ collect the results either way.++ Ceilings: a team run, 30 minutes. A session, its `timeout_seconds` plus two minutes. Above+ 30 minutes the wait cannot cover that in one go: when it times out and `get_output` still+ says the session is running, start the same wait again. The session's own timeout ends it.++2. When the wait's line arrives, make the one+ settled call: `get_output` / `get_diagnostics`, or `team(mode="status")`. A wait that+ times out is the ceiling: report what is still running.++Never `sleep` in the foreground between status calls: Claude Code blocks it, and it holds+the turn without waking anything.++claudish pushes nothing into the conversation. A `<channel source="claudish">` block never+arrives, so never wait for one. The one live view is+`list_sessions(include_completed=true)`: each row carries `state`, `activity`, `idle_seconds` and, on a failed or+timed-out row, `reason_detail`. One call every 10 seconds or so is enough.++A live row also carries `recent`: the current turn's last 20 steps, oldest first, each+`{at, kind, tool, summary}`. `kind` is `text`, `thinking`, `tool` or `ask`; `summary` is one+line naming the target (the file, the first line of a Bash command, the pattern or url), at+most 100 characters. It shows what the child did between two polls, which `activity` alone+cannot. It is `null` before the first step and on rows read from disk, which have no live+transcript to follow.++### `run_prompt` — one completion, no session++```+run_prompt(model, prompt, system_prompt)+```++One completion, straight to one model's API. No tools, no working directory, no session,+so it is never subagent work: a task that must read files, run commands or write code goes+through `create_session`, and a panel through `team`. Use `run_prompt` when you want an+opinion, not work. There is no `max_tokens` argument; never pass one.++### Catalog and reporting++```+list_models() // current recommended set — call this first, always+search_models(query) // every live variant in a family+report_error(error_type, model, stderr_snippet, session_path, additional_context)+```++`compare_models` is a **debug** tool for claudish's own developers. It sends one prompt to+several models' APIs side by side. It never stands in for `team`: it runs no tools, keeps+no record and collects no votes.++### Tool groups++`CLAUDISH_MCP_TOOLS` in the server's environment limits which tools it exposes. The plugin+sets none, so every tool is there.++| Value | Tools |+|---|---|+| unset / `all` | all 14 |+| `low-level` | `list_models`, `run_prompt`, `search_models` |+| `agentic` | `preflight`, `report_error`, `team` |+| `session` | `cancel_session`, `capture_session`, `create_session`, `get_diagnostics`, `get_output`, `list_sessions`, `send_input` |+| `debug` | `compare_models` |++`channel` is not a group. A server given it warns once on stderr and exposes all 14 tools.++## Choosing between the tools++```+Does the task need a panel of independent opinions?+ → team(mode="run", ...) with require_pattern+ → poll team(mode="status", ...) until settled → read response-<slot>.md++Does it need one model to DO work — read files, write code, run commands?+ → create_session(...) → get_output(session_id)++Do you just need one model's answer to one question?+ → run_prompt(...)++Do you need to know what models exist?+ → list_models() / search_models(query)+```++Task size is not a factor in that choice, and never was. A "simple" prompt and a+multi-phase workflow both go through the same tools; a long prompt is just a long `prompt`+or `input` argument. There is no size at which shelling out becomes correct.++## 🤖 Agent Selection Guide++### Step 1: Find the Right Agent++**When user requests Claudish task, follow this process:**++1. **Check for existing agents** that suit the task type+2. **If no suitable agent exists:**+ - Suggest creating a new agent for this task type+ - Offer to proceed with the generic `general-purpose` agent if user declines+3. **If user declines agent creation:**+ - Proceed with `general-purpose` and say so++### Step 2: Agent Type Selection Matrix++> **Note:** Every model — native and external — is invoked via claudish MCP tools (`team`,+> `create_session`). The orchestrator resolves the agent and passes it in the tool's `agent`+> parameter, which applies to every child in the run. Task context still travels in the prompt.++| Task Type | Recommended Agent | Alternatives | Notes |+|-----------|----------------------|--------------|-------|+| **Investigation** | `code-search:analyze` | `dev:debugger` | Read-only code tracing; `dev:researcher` is for web research |+| **Code review** | `dev:reviewer` | — | Security, correctness, maintainability passes |+| **Architecture** | `dev:architect` | — | Design and planning tasks |+| **Implementation** | `dev:developer` | — | Building features |+| **Testing** | `dev:qa-engineer` | — | Test strategy and coverage |+| **Debugging** | `dev:debugger` | — | Error analysis and tracing |+| **Documentation** | `dev:docs` | `dev:researcher` | Writing or auditing documentation |+| **UI/Design** | `dev:frontend-developer` | `designer` plugin | Visual and UX tasks |++### Step 3: Agent Creation Offer (When No Agent Exists)++**Template response:**+```+I notice you want to use [Model Name] for [task type].++RECOMMENDATION: Create a specialized [task type] agent.++This would:+✅ Provide better task-specific guidance+✅ Reusable for future [task type] tasks+✅ Optimized prompting for [Model Name]++Options:+1. Create specialized agent (recommended) - takes 2-3 minutes+2. Use generic general-purpose agent - works but less optimized++Which would you prefer?+```++### Step 4: Example Agent Selection++**Example 1: User says "use Grok to implement authentication"**+```+Task: Code implementation (authentication)++Decision:+1. Resolve "grok" via list_models / search_models+2. Pick the agent: implementation → dev:developer+3. create_session(model=<resolved id>, prompt=TASK,+ timeout_seconds=300, agent="dev:developer", work_dir=REPO)+4. On completed → get_output(session_id) → summarise+```++**Example 2: User says "ask GPT to review my API design"**+```+Task: Code review (API design)++Decision:+1. Resolve the GPT family via search_models("gpt") — had the user named a version, it would be a hard constraint+2. Pick the agent: review → dev:reviewer+3. One reviewer → create_session(model=<resolved id>, prompt=REVIEW_PROMPT,+ agent="dev:reviewer", timeout_seconds=300)+ A panel → team(mode="run", path=SESSION_DIR, models=[...],+ input_file=SESSION_DIR + "/input.md", agent="dev:reviewer")+ → poll status until settled → read response-NN.md+```++**Example 3: User says "use Gemini to refactor this component"**+```+Task: Refactoring (component)++Decision:+1. Resolve "gemini" via search_models("gemini")+2. No specialised refactoring agent exists → offer to create one+3. User declines → agent="dev:frontend-developer", else "general-purpose"+4. create_session(model=<resolved id>, prompt=TASK, agent=<chosen>, work_dir=REPO)+```++## Team Mode Integration++When used with the `/team` command for multi-model blind voting, every model — native and+external — is started through the `team` MCP tool in a single call. Write the vote prompt+to `input.md` first, then:++```+team(mode="run", path=SESSION_DIR, models=["internal", "grok", "gemini"],+ input_file=`${SESSION_DIR}/input.md`, require_pattern="```vote", agent=RESOLVED_AGENT)+```++**That call starts the panel and returns a slot map. It does not return votes.** Poll+`team(mode="status", path=SESSION_DIR)` until no slot is `RUNNING`, then read each vote from+`response-<slot>.md`. The `slots` map in the `run` response gives the model-name-to-slot-id+mapping, so responses can still be read blind. Full procedure: **The three-step lifecycle**+under `team` above.++## Overview++**Claudish** is an external-model runtime: it runs Claude Code against any model a provider+catalog offers (Grok, GPT, MiniMax, Gemini, Kimi, …) by proxying through a local+Anthropic-API-compatible server, and it exposes that capability to Claude Code as an **MCP+server**. Magus talks to the MCP server. It does not talk to the binary.++Claudish:+- ✅ Runs Claude Code with **any catalogued model**, not just Anthropic's+- ✅ Supports **multiple backends** (OpenRouter, Gemini Direct, OpenAI Direct, Ollama, …)+- ✅ Runs native Claude tiers through a passthrough on the user's own subscription+- ✅ Supports 100% of Claude Code features inside a session+- ✅ Tracks cost and returns it in the session record+- ✅ Enables multi-model workflows via `team`++**Use Cases:**+- Run tasks with different models (fast coding, deep reasoning, vision)+- Compare model performance on the same task+- Reduce cost by routing simple tasks to cheaper models+- Reach models with specialised capabilities++## Routing is Claudish's, not ours++**Magus implements no routing.** Not provider prefixes, not fallback chains, not API-key+detection, not pre-flight reachability checks. Send the catalog's `id` (see "Identity vs+routing address" above) and let claudish resolve it. That is the whole contract.++**Never document a provider/prefix/env-var table or a routing troubleshooting guide here**,+for two reasons:++1. **Ownership.** The Responsibility Boundaries table above assigns "Model ID → API+ endpoint" and "API keys, backend fallbacks" to Claudish. A routing guide here+ contradicts that and invites plugin code to compensate for provider behaviour.+2. **Drift.** Routing detail copied into this repo goes stale and starts teaching the wrong+ thing — the backend separator is `@`, and which providers carry which models changes+ most releases.++**A model that will not route is a claudish bug, not a magus workaround.** Do not add+retry, probe, or fallback logic to a command, agent, or skill in this repo to route around+it. Report it instead — `report_error` via the claudish MCP server, which anonymises paths+and keys.++## Requirements++- **Claudish >= 8.0.0 on `PATH`** — `bun add -g claudish` (or `npm install -g claudish`).+ It is the runtime the MCP server runs inside; without it there are no tools.+- **Claude Code** — must be installed.++**8.0.0 is a hard floor for `team`, not a preference.** Below it, `run` blocks instead of+returning a handle and `input_file` does not exist, so a workflow written to this skill+starts a run and reads nothing. Check with `claudish --version`.++**Credentials are Claudish's, and are deliberately not listed here.** There is no single+required key: which providers a model can reach, and which env var each one reads, is+claudish's routing concern and changes most releases. Naming any of them here is how this+skill ends up teaching wrong routing.++The authoritative, always-current list is the ENVIRONMENT VARIABLES section of+`claudish --help` (a diagnostic — see below).++If a model fails for want of a credential, that is a claudish diagnosis. Do not add key+checks or provider detection to this repo.++## Diagnostics: the only CLI left++Exactly three CLI invocations remain permitted, and **only for investigating the runtime.**+They are read-only, they produce no work, and none of them may appear in a workflow that is+trying to accomplish a task:++| Command | Answers |+|---|---|+| `claudish --help` | Flags, and the authoritative ENVIRONMENT VARIABLES list |+| `claudish --version` | Which runtime is installed |+| `claudish --models [query]` | Ad-hoc catalog lookup **while debugging the catalog itself** |++**These are for a human debugging session, not for orchestration.** In particular,+`--models` is NOT how a workflow discovers models — `list_models` and `search_models` are,+because they return structured records the workflow can act on. Reaching for `--models` in+a task is the CLI habit sneaking back in through the one door left open.++Anything not on this list — `--model`, `--stdin`, `--json`, `--agent`, `--port`,+`--interactive`, `--quiet` — has an MCP equivalent above and must be used through it.++Do not encode a diagnostic's output as procedure in this repo. It describes claudish's+behaviour at one moment; restating it here is how the routing table went stale.++## Delegation Patterns++### Pattern 1: One model does a piece of work++```+// Resolve first (list_models / search_models), then:+create_session(model=RESOLVED_ID, prompt=TASK_PROMPT, timeout_seconds=300,+ agent="dev:developer", work_dir=REPO_PATH)+→ session_id : report "Delegating to {MODEL}…"+→ the bounded wait fires → get_output(ID) says completed : get_output(ID, tail_lines=200) → summarise for the user+→ get_output(ID) says `failed` / `timeout` after the wait : get_diagnostics(ID) → report → stop+```++A session started with a `prompt` never waits for input, so this pattern has no+`send_input` step.++The session's own transcript stays out of your context — that is the point, and it is why+no instruction file, no result file and no `/tmp` scratch is involved. Do not add any.++### Pattern 2: A long or structured prompt++Put it in the `prompt` (or `input`) argument. There is no command-line length limit to work+around, because there is no command line.++```+PROMPT = "# Code Review Task\n"+ + "\n## Files\n" + files.map(f => "- " + f).join("\n")+ + "\n\n## Criteria\nCorrectness, maintainability, performance, security"+ + "\n\n## Required output shape\nEnd with a fenced `review` block containing "+ + "VERDICT and ISSUES."++create_session(model=RESOLVED_ID, prompt=PROMPT, timeout_seconds=300,+ agent="dev:reviewer", work_dir=REPO_PATH)+```++If the prompt mandates a shape, run it through `team` instead with+`require_pattern="```review"` so a shapeless response is reported FAILED rather than+accepted.++### Pattern 3: Several models on the same prompt++One `team` call starts them all. Not a loop, not one session per model — the tool+parallelises internally. The call returns a slot map immediately; you poll, then read.++```+team(mode="run", path=SESSION_DIR, models=["internal", "grok", "gemini"],+ input_file=`${SESSION_DIR}/input.md`, require_pattern="```review", agent="dev:reviewer")+```++Then poll `team(mode="status", path=SESSION_DIR)` until no slot is `RUNNING`, and read+`response-<slot>.md` for each slot in the `run` response's `slots` map. Per-slot status+comes back in `status.models`: check every one. See **The three-step lifecycle** above.++## Failure Handling++Failures arrive as **structured data**, not as text on stderr. There are exactly two shapes.++### From `team`: the per-model result object++Each slot reports its own status. Read every one — a run where three of five slots came+back is a partial success you must disclose, not a success.++Read them from `status.models[<slot>]` once the run has settled — not from the `run`+response, which returns before any slot has finished.++| What you see | What it means |+|---|---|+| state `FAILED` | The slot errored. `error.reason` carries the cause. |+| state EMPTY, reason `shape_mismatch` | The slot exited 0 but never matched `require_pattern`. Treat as failed. |+| reason `nonzero_exit` | The child process died. Check `errors/<slot>.log`. |+| reason `cancelled` | **You** cancelled it. Report it as a decision, not a crash. |+| below `min_output_bytes` | Effectively empty. Treat as failed. |+| state `RUNNING` at your poll ceiling | Not a failure. Report it as still running and let the user decide. |++Show failed slots as FAILED in your results table, proceed with the survivors, and name+what failed. No retry, no substitution.++### From `create_session`: the end state++The row's `state` in `list_sessions`, or `get_output`'s `status` after the wait, names the end+state. A failed or timed-out row also carries `reason_detail`: the provider's error message, or+the first line of the kept detail, at most 160 characters. Quote it in your report; it is `null`+when nothing readable was kept.++| End state | Meaning | Next call |+|---|---|---|+| `failed` | The child exited with an error, or claudish died before recording an end | `get_diagnostics(session_id)` |+| `timeout` | It hit `timeout_seconds` and was killed | `get_diagnostics(session_id)` |+| `completed` with empty or surprising output | Suspect a silent failure | `get_diagnostics(session_id)` |++`get_diagnostics` returns the child's stderr, upstream error bodies, recent event frames,+the resolved model chain, and paths to the full records — with no re-run and no debug flag.++### The escalation rule: STOP and REPORT++Never silently substitute. If the user asked for Gemini and Gemini failed, do not quietly+run GPT, and do not quietly fall back to the embedded Claude.++```+"{Model} failed.++What happened:+1. Tool: {team | create_session}+ Error: {error from the result object or from get_diagnostics after the failed line}++Options:+(1) Retry the same model+(2) Use a different model+(3) Skip this model, continue with the others+(4) Cancel+(5) Report this error to claudish developers++Which do you prefer?"+```++On (5), call `report_error(error_type, model, stderr_snippet, session_path,+additional_context)`. `error_type` is one of `provider_failure`, `adapter_error`,+`stream_error`, `team_failure`. **Consent is required** — always ask first. Data is+sanitised (keys, paths, emails stripped) before sending.++## Best Practices++### 1. ✅ Resolve models live, every run++Call `list_models` before you name a model, and `search_models` for a specific family.+Never a remembered ID, never a committed list.++### 2. ✅ Choose the tool by task shape, not task size++Panel → `team`. Work → `create_session`. An answer → `run_prompt`. Size never makes+shelling out correct.++### 3. ✅ Set `require_pattern` whenever the prompt demands a shape++It is the only thing standing between "exited 0" and "actually produced a vote".++### 4. ✅ Pass the agent in the `agent` parameter++Not in the prompt, and not via `claude_flags` — an `--agent` in `claude_flags` loses to+`agent`. It applies to every child in the run; there is no per-model form.++### 5. ✅ Let sessions carry the context++A `create_session` child runs in its own process, so its transcript costs you nothing.+Summarise its `get_output` for the user rather than pasting it.++### 6. ✅ Check every slot's status before you believe a result++Structured per-slot status is the reason to use these tools. Ignoring it throws away the+advantage. For `team` that status lives in a **settled** `status` response, never in the+`run` response.++### 7. ✅ Poll `team` to completion; never read results from the `run` response++`run` returns before any model has answered. A workflow that parses its response finds no+results and reports whatever its empty-case is — for a vote panel, INCONCLUSIVE on every+run. Poll `status` until no slot is `RUNNING`, bound the loop, then read off disk.++### 8. ✅ Write long prompts to `input.md` and pass `input_file`++An inline `input` echoes the entire prompt verbatim in the user's terminal, burying the+model list, the agent and the shape check inside a wall of text.++## Anti-Patterns (Avoid These)++### ❌ Don't Ignore Model Selection++**Wrong:** taking whatever the default is for every task.++**Right:** pick for the task — fast variants for iteration, reasoning models for analysis,+large-context models for whole-repo work — and resolve the choice against `list_models`.++### ❌ Don't Hardcode Model Lists++**Wrong:**+```typescript+const MODELS = ["grok", "gpt"];+```++**Right:** resolve from the live catalog at call time.++```+list_models() → current recommended set+search_models({ query: "kimi" }) → every live variant in a family+```++### ✅ Do Accept Custom Models From Users++**Problem:** the user provides a model ID that `list_models` does not return.++**Wrong (rejecting it):**+```typescript+const shortlist = ["grok", "gpt"];+if (!shortlist.includes(userModel)) {+ throw new Error("Model not in my shortlist"); // ❌ DON'T DO THIS+}+```++**Right:** pass it to the tool and let claudish validate it.++```+create_session(model=userModel, prompt=TASK, timeout_seconds=300)+```++Users legitimately have access to beta models, private fine-tunes, newly released models+not yet in the recommended set, and regional or enterprise endpoints. Say that the ID is+not in the recommended set if it is not, then run it anyway — claudish decides whether it+resolves.++### ❌ Don't Skip Error Handling++```+// On get_output reporting `failed` after the wait → get_diagnostics(ID), then STOP and REPORT+// "Grok failed: {error content}. Options: (1) Retry, (2) Different model, (3) Skip, (4) Cancel"+// On `completed` → get_output(ID)+```++**❌ NEVER do silent fallback.** If `create_session` fails for Gemini, do not silently run+with the embedded Claude instead. Report the failure and let the user decide.++## Troubleshooting++### The claudish MCP tools are not available++The plugin's `.mcp.json` starts the server by launching the claudish binary, so the binary+must be installed (`npm install -g claudish`) and on `PATH`. Restart the session so the MCP+server registers. Confirm with `claudish --version`.++### A slot reported FAILED, or came back empty++Read the error from the per-model result (`team`) or call `get_diagnostics` (`create_session`).+Common causes: rate limits, an ID the catalog no longer lists, a missing credential for that+provider. An empty response that exited 0 usually means `require_pattern` did its job.++### Every slot failed with `prompt_not_accepted`++`prompt_not_accepted` means the child's Claude Code never took its launch prompt: the prompt+sits typed in the input box and no transcript starts. In one slot it is a one-off. In every+slot it is the child's input mode, not the models, and a retry fails the same way.++The known cause is `"editorMode": "vim"` in the user's Claude Code settings. claudish 10.7.0+fixes it by starting every child in normal mode. Check with `claudish --version`. On an older+claudish, upgrade it, or pass `claude_flags='--settings {"editorMode":"normal"}'` to the+`team` or `create_session` call. The value holds no space, so the whitespace split of+`claude_flags` leaves it intact.++On claudish 10.7.0 or later, every slot failing this way is a claudish bug: report it with+`report_error` and stop.++### A model will not route++Report it with `report_error` and pick a different model from `list_models`. Do not+investigate the routing chain — routing is claudish's, and a caller that hands it a bare+name has no business knowing which provider was tried.++Note for the report, not for you to act on: a 401 from a provider often means "this+provider does not carry this model" rather than "your key is wrong". Either way the fix+belongs in claudish, never in a workaround here.++### The response is slow, or costs more than expected++Pick a different model — resolve alternatives with `list_models`, which carries pricing and+capability data, and check context sizes there rather than assuming. Cost per session comes+back in the session record; you do not need to parse anything to get it.++### Context window exceeded inside a session++Use a larger-context model from `list_models`, or split the task into several+`create_session` calls. Do not try to trim by moving work into files — the session already+keeps its own transcript out of your context.++## Additional Resources++- Claudish GitHub: https://github.com/MadAppGang/claudish+- Install the runtime: `npm install -g claudish`+- OpenRouter: https://openrouter.ai · Models: https://openrouter.ai/models++---++**Maintained by:** MadAppGang+**Last Updated:** August 22, 2026diff --git a/ai-docs/benches/recommended-effort/plugins/worktree/claudish/.claude-plugin/plugin.json b/ai-docs/benches/recommended-effort/plugins/worktree/claudish/.claude-plugin/plugin.jsonnew file mode 100644index 00000000..59c408c4--- /dev/null+++ b/ai-docs/benches/recommended-effort/plugins/worktree/claudish/.claude-plugin/plugin.json@@ -0,0 +1,12 @@+{+ "name": "claudish",+ "version": "2.10.0",+ "description": "MCP runtime for external AI models. Proxies OpenRouter, Ollama and LM Studio, runs blind team voting, manages async sessions, and shows the session's runs live above the prompt, with stop and live-screen tabs. Required by code-search, dev, multimodel and designer.",+ "author": {+ "name": "Jack Rudenko",+ "email": "[email protected]",+ "company": "MadAppGang"+ },+ "license": "MIT",+ "skills": "./skills/"+}diff --git a/ai-docs/benches/recommended-effort/plugins/worktree/claudish/skills/claudish-usage/SKILL.md b/ai-docs/benches/recommended-effort/plugins/worktree/claudish/skills/claudish-usage/SKILL.mdnew file mode 100644index 00000000..a559271d--- /dev/null+++ b/ai-docs/benches/recommended-effort/plugins/worktree/claudish/skills/claudish-usage/SKILL.md@@ -0,0 +1,1347 @@+---+name: claudish-usage+description: CRITICAL - Guide for using Claudish CLI ONLY through sub-agents to run Claude Code with any AI model (OpenRouter, Gemini, OpenAI, local models). NEVER run Claudish directly in main context unless user explicitly requests it. Use when user mentions external AI models, Claudish, OpenRouter, Gemini, OpenAI, Ollama, or alternative models. Includes mandatory sub-agent delegation patterns, agent selection guide, file-based instructions, and strict rules to prevent context window pollution.+---++# Claudish Usage Skill++**Version:** 2.0.0+**Purpose:** Guide AI agents on how to use Claudish CLI to run Claude Code with any AI model+**Status:** Production Ready++## ⚠️ CRITICAL RULES - READ FIRST++### 🚫 NEVER Run Claudish from Main Context++**Claudish MUST ONLY be run through sub-agents** unless the user **explicitly** requests direct execution.++**Why:**+- Running Claudish directly pollutes main context with 10K+ tokens (full conversation + reasoning)+- Destroys context window efficiency+- Makes main conversation unmanageable++**When you can run Claudish directly:**+- ✅ User explicitly says "run claudish directly" or "don't use a sub-agent"+- ✅ User is debugging and wants to see full output+- ✅ User specifically requests main context execution++**When you MUST use sub-agent:**+- ✅ User says "use Grok to implement X" (delegate to sub-agent)+- ✅ User says "ask GPT-5.3 to review X" (delegate to sub-agent)+- ✅ User mentions any model name without "directly" (delegate to sub-agent)+- ✅ Any production task (always delegate)++### 📋 Workflow Decision Tree++```+User Request+ ↓+Does it mention Claudish/OpenRouter/model name? → NO → Don't use this skill+ ↓ YES+ ↓+Does user say "directly" or "in main context"? → YES → Run in main context (rare)+ ↓ NO+ ↓+Find appropriate agent or create one → Delegate to sub-agent (default)+```++### 🏷️ Model Names++claudish routing picks the provider from the user's setup: their configured providers,+subscriptions first, their preferences and manual overrides. Do not choose it yourself.++1. **Pass bare names** (`gpt-6.1-sol`, `minimax-m3`), never `provider@model`, unless the+ user named the provider. A `provider@` on a catalog model is removed by the MCP tools,+ and their answer says so.+2. **User named the provider** ("run Kimi on OpenRouter"): pass the spec and list it in+ `user_required_provider` (`team`: an array of those `models`/`judges` entries;+ `create_session` / `run_prompt`: `true`).+3. **User named exact models:** pass them as given. No `preflight` or `search_models`+ call is needed to check them: claudish routes and checks them. You still read+ `list_models` once for their efforts (next section).+4. **User named only a family or vendor** ("a GPT model", "minimax", "latest Grok"): call+ `list_models` and take the newest model of that family from it, as a bare name.++### 🎚️ Reasoning Effort per Model++Before `team`, `create_session` or `run_prompt` on an external model:++1. Call `list_models`. Its last section, **Recommended effort by task type**, has a row per+ model and a column per task type.+2. Map YOUR task to one column (planning, implementation, code_review, debugging, …). You+ know the task; claudish does not guess it.+3. For each model you run, take its cell. This works the same when the models were chosen+ with the user and when the task already fixed them.+4. Pass it: `team(..., models: [...], efforts: {"<model>": "<level>"})` with keys exactly+ as in `models`, or `create_session(..., effort: "<level>")`, or+ `run_prompt(..., effort: "<level>")`.++A model with no row, or `-` in your column, gets no entry: it keeps Claude Code's own effort.++If you call without any effort while a model has a row, nothing starts: the tool answers+`needs_effort:` with each model's levels by task type. Pick your column and call again. To keep+Claude Code's own level on purpose, pass `efforts: {}` (`team`) or `effort: "default"`.++## 🤖 Agent Selection Guide++### Step 1: Find the Right Agent++**When user requests Claudish task, follow this process:**++1. **Check for existing agents** that support proxy mode or external model delegation+2. **If no suitable agent exists:**+ - Suggest creating a new proxy-mode agent for this task type+ - Offer to proceed with generic `general-purpose` agent if user declines+3. **If user declines agent creation:**+ - Warn about context pollution+ - Ask if they want to proceed anyway++### Step 2: Agent Type Selection Matrix++| Task Type | Recommended Agent | Fallback | Notes |+|-----------|------------------|----------|-------|+| **Code implementation** | Create coding agent with proxy mode | `general-purpose` | Best: custom agent for project-specific patterns |+| **Code review** | Use existing code review agent + proxy | `general-purpose` | Check if plugin has review agent first |+| **Architecture planning** | Use existing architect agent + proxy | `general-purpose` | Look for `architect` or `planner` agents |+| **Testing** | Use existing test agent + proxy | `general-purpose` | Look for `test-architect` or `tester` agents |+| **Refactoring** | Create refactoring agent with proxy | `general-purpose` | Complex refactors benefit from specialized agent |+| **Documentation** | `general-purpose` | - | Simple task, generic agent OK |+| **Analysis** | Use existing analysis agent + proxy | `general-purpose` | Check for `analyzer` or `detective` agents |+| **Other** | `general-purpose` | - | Default for unknown task types |++### Step 3: Agent Creation Offer (When No Agent Exists)++**Template response:**+```+I notice you want to use [Model Name] for [task type].++RECOMMENDATION: Create a specialized [task type] agent with proxy mode support.++This would:+✅ Provide better task-specific guidance+✅ Reusable for future [task type] tasks+✅ Optimized prompting for [Model Name]++Options:+1. Create specialized agent (recommended) - takes 2-3 minutes+2. Use generic general-purpose agent - works but less optimized+3. Run directly in main context (NOT recommended - pollutes context)++Which would you prefer?+```++### Step 4: Common Agents by Plugin++**Frontend Plugin:**+- `typescript-frontend-dev` - Use for UI implementation with external models+- `frontend-architect` - Use for architecture planning with external models+- `senior-code-reviewer` - Use for code review (can delegate to external models)+- `test-architect` - Use for test planning/implementation++**Bun Backend Plugin:**+- `backend-developer` - Use for API implementation with external models+- `api-architect` - Use for API design with external models++**Code Analysis Plugin:**+- `codebase-detective` - Use for investigation tasks with external models++**No Plugin:**+- `general-purpose` - Default fallback for any task++### Step 5: Example Agent Selection++**Example 1: User says "use Grok to implement authentication"**+```+Task: Code implementation (authentication)+Plugin: Bun Backend (if backend) or Frontend (if UI)++Decision:+1. Check for backend-developer or typescript-frontend-dev agent+2. Found backend-developer? → Use it with Grok proxy+3. Not found? → Offer to create custom auth agent+4. User declines? → Use general-purpose with file-based pattern+```++**Example 2: User says "ask GPT-5.3 to review my API design"**+```+Task: Code review (API design)+Plugin: Bun Backend++Decision:+1. Check for api-architect or senior-code-reviewer agent+2. Found? → Use it with GPT-5.3 proxy+3. Not found? → Use general-purpose with review instructions+4. Never run directly in main context+```++**Example 3: User says "use Gemini to refactor this component"**+```+Task: Refactoring (component)+Plugin: Frontend++Decision:+1. No specialized refactoring agent exists+2. Offer to create component-refactoring agent+3. User declines? → Use typescript-frontend-dev with proxy+4. Still no agent? → Use general-purpose with file-based pattern+```++## Overview++**Claudish** is a CLI tool that allows running Claude Code with any AI model via prefix-based routing. Supports OpenRouter (100+ models), direct Google Gemini API, direct OpenAI API, and local models (Ollama, LM Studio, vLLM, MLX).++**Key Principle:** **ALWAYS** use Claudish through sub-agents with file-based instructions to avoid context window pollution.++## What is Claudish?++Claudish (Claude-ish) is a proxy tool that:+- ✅ Runs Claude Code with **any AI model** via prefix-based routing+- ✅ Supports OpenRouter, Gemini, OpenAI, and local models+- ✅ Uses local API-compatible proxy server+- ✅ Supports 100% of Claude Code features+- ✅ Provides cost tracking and model selection+- ✅ Enables multi-model workflows++## Model Routing++| You write | claudish does | Example |+|-----------|---------------|---------|+| bare name | routes through the user's setup (subscription first) | `gpt-6.1-sol` |+| `provider@model` | that provider only (MCP tools: only with `user_required_provider`) | `or@moonshotai/kimi-k3` |+| `ollama@` `lmstudio@` | a local model, never stripped | `[email protected]` |+| `http://...` | a custom endpoint, never stripped | `http://localhost:8000/model` |++**Use Cases:**+- Run tasks with different AI models (Grok for speed, GPT-5.3 for reasoning, Gemini for large context)+- Use direct APIs for lower latency (Gemini, OpenAI)+- Use local models for free, private inference (Ollama, LM Studio)+- Compare model performance on same task+- Reduce costs with cheaper models for simple tasks++## Requirements++### System Requirements+- **Claudish CLI** - Install with: `npm install -g claudish` or `bun install -g claudish`+- **Claude Code** - Must be installed+- **At least one API key** (see below)++### Environment Variables++```bash+# API Keys (at least one required)+export OPENROUTER_API_KEY='sk-or-v1-...' # OpenRouter (100+ models)+export GEMINI_API_KEY='...' # Direct Gemini API (g/ prefix)+export OPENAI_API_KEY='sk-...' # Direct OpenAI API (oai/ prefix)++# Placeholder (required to prevent Claude Code dialog)+export ANTHROPIC_API_KEY='sk-ant-api03-placeholder'++# Custom endpoints (optional)+export GEMINI_BASE_URL='https://...' # Custom Gemini endpoint+export OPENAI_BASE_URL='https://...' # Custom OpenAI/Azure endpoint+export OLLAMA_BASE_URL='http://...' # Custom Ollama server+export LMSTUDIO_BASE_URL='http://...' # Custom LM Studio server++# Default model (optional)+export CLAUDISH_MODEL='openai/gpt-5.3' # Default model+```++**Get API Keys:**+- OpenRouter: https://openrouter.ai/keys (free tier available)+- Gemini: https://aistudio.google.com/apikey+- OpenAI: https://platform.openai.com/api-keys+- Local models: No API key needed++## Quick Start Guide++### Step 1: Install Claudish++```bash+# With npm (works everywhere)+npm install -g claudish++# With Bun (faster)+bun install -g claudish++# Verify installation+claudish --version+```++### Step 2: Get Available Models++```bash+# List ALL OpenRouter models grouped by provider+claudish --models++# Fuzzy search models by name, ID, or description+claudish --models gemini+claudish --models "grok code"++# Show top recommended programming models (curated list)+claudish --top-models++# JSON output for parsing+claudish --models --json+claudish --top-models --json++# Force update from OpenRouter API+claudish --models --force-update+```++### Step 3: Run Claudish++**Interactive Mode (default):**+```bash+# Shows model selector, persistent session+claudish+```++**Single-shot Mode:**+```bash+# One task and exit (requires --model)+claudish --model x-ai/grok-code-fast-1 "implement user authentication"+```++**With stdin for large prompts:**+```bash+# Read prompt from stdin (useful for git diffs, code review)+git diff | claudish --stdin --model openai/gpt-5.3-codex "Review these changes"+```++## Recommended Models++**Top Models for Development (v3.1.1):**++| Model | Provider | Best For |+|-------|----------|----------|+| `openai/gpt-5.3` | OpenAI | **Default** - Most advanced reasoning |+| `minimax/minimax-m2.1` | MiniMax | Budget-friendly, fast |+| `z-ai/glm-4.7` | Z.AI | Balanced performance |+| `google/gemini-3-pro-preview` | Google | 1M context window |+| `moonshotai/kimi-k2-thinking` | MoonShot | Extended thinking |+| `deepseek/deepseek-v3.2` | DeepSeek | Code specialist |+| `qwen/qwen3-vl-235b-a22b-thinking` | Alibaba | Vision + reasoning |++**Direct API Options (lower latency):**++| Model | Backend | Best For |+|-------|---------|----------|+| `g/gemini-2.0-flash` | Gemini | Fast tasks, large context |+| `oai/gpt-4o` | OpenAI | General purpose |+| `ollama/llama3.2` | Local | Free, private |++**Get Latest Models:**+```bash+# List all models (auto-updates every 2 days)+claudish --models++# Search for specific models+claudish --models grok+claudish --models "gemini flash"++# Show curated top models+claudish --top-models++# Force immediate update+claudish --models --force-update+```++## NEW: Direct Agent Selection (v2.1.0)++**Use `--agent` flag to invoke agents directly without the file-based pattern:**++```bash+# Use specific agent (prepends @agent- automatically)+claudish --model x-ai/grok-code-fast-1 --agent frontend:developer "implement React component"++# Claude receives: "Use the @agent-frontend:developer agent to: implement React component"++# List available agents in project+claudish --list-agents+```++**When to use `--agent` vs file-based pattern:**++**Use `--agent` when:**+- Single, simple task that needs agent specialization+- Direct conversation with one agent+- Testing agent behavior+- CLI convenience++**Use file-based pattern when:**+- Complex multi-step workflows+- Multiple agents needed+- Large codebases+- Production tasks requiring review+- Need isolation from main conversation++**Example comparisons:**++**Simple task (use `--agent`):**+```bash+claudish --model x-ai/grok-code-fast-1 --agent frontend:developer "create button component"+```++**Complex task (use file-based):**+```typescript+// multi-phase-workflow.md+Phase 1: Use api-architect to design API+Phase 2: Use backend-developer to implement+Phase 3: Use test-architect to add tests+Phase 4: Use senior-code-reviewer to review++then:+claudish --model x-ai/grok-code-fast-1 --stdin < multi-phase-workflow.md+```++## Best Practice: File-Based Sub-Agent Pattern++### ⚠️ CRITICAL: Don't Run Claudish Directly from Main Conversation++**Why:** Running Claudish directly in main conversation pollutes context window with:+- Entire conversation transcript+- All tool outputs+- Model reasoning (can be 10K+ tokens)++**Solution:** Use file-based sub-agent pattern++### File-Based Pattern (Recommended)++**Step 1: Create instruction file**+```markdown+# /tmp/claudish-task-{timestamp}.md++## Task+Implement user authentication with JWT tokens++## Requirements+- Use bcrypt for password hashing+- Generate JWT with 24h expiration+- Add middleware for protected routes++## Deliverables+Write implementation to: /tmp/claudish-result-{timestamp}.md++## Output Format+```markdown+## Implementation++[code here]++## Files Created/Modified+- path/to/file1.ts+- path/to/file2.ts++## Tests+[test code if applicable]++## Notes+[any important notes]+```+```++**Step 2: Run Claudish with file instruction**+```bash+# Read instruction from file, write result to file+claudish --model x-ai/grok-code-fast-1 --stdin < /tmp/claudish-task-{timestamp}.md > /tmp/claudish-result-{timestamp}.md+```++**Step 3: Read result file and provide summary**+```typescript+// In your agent/command:+const result = await Read({ file_path: "/tmp/claudish-result-{timestamp}.md" });++// Parse result+const filesModified = extractFilesModified(result);+const summary = extractSummary(result);++// Provide short feedback to main agent+return `✅ Task completed. Modified ${filesModified.length} files. ${summary}`;+```++### Complete Example: Using Claudish in Sub-Agent++```typescript+/**+ * Example: Run code review with Grok via Claudish sub-agent+ */+async function runCodeReviewWithGrok(files: string[]) {+ const timestamp = Date.now();+ const instructionFile = `/tmp/claudish-review-instruction-${timestamp}.md`;+ const resultFile = `/tmp/claudish-review-result-${timestamp}.md`;++ // Step 1: Create instruction file+ const instruction = `# Code Review Task++## Files to Review+${files.map(f => `- ${f}`).join('\n')}++## Review Criteria+- Code quality and maintainability+- Potential bugs or issues+- Performance considerations+- Security vulnerabilities++## Output Format+Write your review to: ${resultFile}++Use this format:+\`\`\`markdown+## Summary+[Brief overview]++## Issues Found+### Critical+- [issue 1]++### Medium+- [issue 2]++### Low+- [issue 3]++## Recommendations+- [recommendation 1]++## Files Reviewed+- [file 1]: [status]+\`\`\`+`;++ await Write({ file_path: instructionFile, content: instruction });++ // Step 2: Run Claudish with stdin+ await Bash(`claudish --model x-ai/grok-code-fast-1 --stdin < ${instructionFile}`);++ // Step 3: Read result+ const result = await Read({ file_path: resultFile });++ // Step 4: Parse and return summary+ const summary = extractSummary(result);+ const issueCount = extractIssueCount(result);++ // Step 5: Clean up temp files+ await Bash(`rm ${instructionFile} ${resultFile}`);++ // Step 6: Return concise feedback+ return {+ success: true,+ summary,+ issueCount,+ fullReview: result // Available if needed, but not in main context+ };+}++function extractSummary(review: string): string {+ const match = review.match(/## Summary\s*\n(.*?)(?=\n##|$)/s);+ return match ? match[1].trim() : "Review completed";+}++function extractIssueCount(review: string): { critical: number; medium: number; low: number } {+ const critical = (review.match(/### Critical\s*\n(.*?)(?=\n###|$)/s)?.[1].match(/^-/gm) || []).length;+ const medium = (review.match(/### Medium\s*\n(.*?)(?=\n###|$)/s)?.[1].match(/^-/gm) || []).length;+ const low = (review.match(/### Low\s*\n(.*?)(?=\n###|$)/s)?.[1].match(/^-/gm) || []).length;++ return { critical, medium, low };+}+```++## Sub-Agent Delegation Pattern++When running Claudish from an agent, use the Task tool to create a sub-agent:++### Pattern 1: Simple Task Delegation++```typescript+/**+ * Example: Delegate implementation to Grok via Claudish+ */+async function implementFeatureWithGrok(featureDescription: string) {+ // Use Task tool to create sub-agent+ const result = await Task({+ subagent_type: "general-purpose",+ description: "Implement feature with Grok",+ prompt: `+Use Claudish CLI to implement this feature with Grok model:++${featureDescription}++INSTRUCTIONS:+1. Search for available models:+ claudish --models grok++2. Run implementation with Grok:+ claudish --model x-ai/grok-code-fast-1 "${featureDescription}"++3. Return ONLY:+ - List of files created/modified+ - Brief summary (2-3 sentences)+ - Any errors encountered++DO NOT return the full conversation transcript or implementation details.+Keep your response under 500 tokens.+ `+ });++ return result;+}+```++### Pattern 2: File-Based Task Delegation++```typescript+/**+ * Example: Use file-based instruction pattern in sub-agent+ */+async function analyzeCodeWithGemini(codebasePath: string) {+ const timestamp = Date.now();+ const instructionFile = `/tmp/claudish-analyze-${timestamp}.md`;+ const resultFile = `/tmp/claudish-analyze-result-${timestamp}.md`;++ // Create instruction file+ const instruction = `# Codebase Analysis Task++## Codebase Path+${codebasePath}++## Analysis Required+- Architecture overview+- Key patterns used+- Potential improvements+- Security considerations++## Output+Write analysis to: ${resultFile}++Keep analysis concise (under 1000 words).+`;++ await Write({ file_path: instructionFile, content: instruction });++ // Delegate to sub-agent+ const result = await Task({+ subagent_type: "general-purpose",+ description: "Analyze codebase with Gemini",+ prompt: `+Use Claudish to analyze codebase with Gemini model.++Instruction file: ${instructionFile}+Result file: ${resultFile}++STEPS:+1. Read instruction file: ${instructionFile}+2. Run: claudish --model google/gemini-2.5-flash --stdin < ${instructionFile}+3. Wait for completion+4. Read result file: ${resultFile}+5. Return ONLY a 2-3 sentence summary++DO NOT include the full analysis in your response.+The full analysis is in ${resultFile} if needed.+ `+ });++ // Read full result if needed+ const fullAnalysis = await Read({ file_path: resultFile });++ // Clean up+ await Bash(`rm ${instructionFile} ${resultFile}`);++ return {+ summary: result,+ fullAnalysis+ };+}+```++### Pattern 3: Multi-Model Comparison++```typescript+/**+ * Example: Run same task with multiple models and compare+ */+async function compareModels(task: string, models: string[]) {+ const results = [];++ for (const model of models) {+ const timestamp = Date.now();+ const resultFile = `/tmp/claudish-${model.replace('/', '-')}-${timestamp}.md`;++ // Run task with each model+ await Task({+ subagent_type: "general-purpose",+ description: `Run task with ${model}`,+ prompt: `+Use Claudish to run this task with ${model}:++${task}++STEPS:+1. Run: claudish --model ${model} --json "${task}"+2. Parse JSON output+3. Return ONLY:+ - Cost (from total_cost_usd)+ - Duration (from duration_ms)+ - Token usage (from usage.input_tokens and usage.output_tokens)+ - Brief quality assessment (1-2 sentences)++DO NOT return full output.+ `+ });++ results.push({+ model,+ resultFile+ });+ }++ return results;+}+```++## Common Workflows++### Workflow 1: Quick Code Generation with Grok++```bash+# Fast, agentic coding with visible reasoning+claudish --model x-ai/grok-code-fast-1 "add error handling to api routes"+```++### Workflow 2: Complex Refactoring with GPT-5.3++```bash+# Advanced reasoning for complex tasks+claudish --model openai/gpt-5 "refactor authentication system to use OAuth2"+```++### Workflow 3: UI Implementation with Qwen (Vision)++```bash+# Vision-language model for UI tasks+claudish --model qwen/qwen3-vl-235b-a22b-instruct "implement dashboard from figma design"+```++### Workflow 4: Code Review with Gemini++```bash+# State-of-the-art reasoning for thorough review+git diff | claudish --stdin --model google/gemini-2.5-flash "Review these changes for bugs and improvements"+```++### Workflow 5: Multi-Model Consensus++```bash+# Run same task with multiple models+for model in "x-ai/grok-code-fast-1" "google/gemini-2.5-flash" "openai/gpt-5.3"; do+ echo "=== Testing with $model ==="+ claudish --model "$model" "find security vulnerabilities in auth.ts"+done+```++## Claudish CLI Flags Reference++### Essential Flags++| Flag | Description | Example |+|------|-------------|---------|+| `--model <model>` | OpenRouter model to use | `--model x-ai/grok-code-fast-1` |+| `--stdin` | Read prompt from stdin | `git diff \| claudish --stdin --model grok` |+| `--models` | List all models or search | `claudish --models` or `claudish --models gemini` |+| `--top-models` | Show top recommended models | `claudish --top-models` |+| `--json` | JSON output (implies --quiet) | `claudish --json "task"` |+| `--help-ai` | Print AI agent usage guide | `claudish --help-ai` |++### Advanced Flags++| Flag | Description | Default |+|------|-------------|---------|+| `--interactive` / `-i` | Interactive mode | Auto (no prompt = interactive) |+| `--quiet` / `-q` | Suppress log messages | Quiet in single-shot |+| `--verbose` / `-v` | Show log messages | Verbose in interactive |+| `--debug` / `-d` | Enable debug logging to file | Disabled |+| `--port <port>` | Proxy server port | Random (3000-9000) |+| `--no-auto-approve` | Require permission prompts | Auto-approve enabled |+| `--dangerous` | Disable sandbox | Disabled |+| `--monitor` | Proxy to real Anthropic API (debug) | Disabled |+| `--force-update` | Force refresh model cache | Auto (>2 days) |++### Output Modes++1. **Quiet Mode (default in single-shot)**+ ```bash+ claudish --model grok "task"+ # Clean output, no [claudish] logs+ ```++2. **Verbose Mode**+ ```bash+ claudish --verbose "task"+ # Shows all [claudish] logs for debugging+ ```++3. **JSON Mode**+ ```bash+ claudish --json "task"+ # Structured output: {result, cost, usage, duration}+ ```++## Cost Tracking++Claudish automatically tracks costs in the status line:++```+directory • model-id • $cost • ctx%+```++**Example:**+```+my-project • x-ai/grok-code-fast-1 • $0.12 • 67%+```++Shows:+- 💰 **Cost**: $0.12 USD spent in current session+- 📊 **Context**: 67% of context window remaining++**JSON Output Cost:**+```bash+claudish --json "task" | jq '.total_cost_usd'+# Output: 0.068+```++## Error Handling++### Error 1: OPENROUTER_API_KEY Not Set++**Error:**+```+Error: OPENROUTER_API_KEY environment variable is required+```++**Fix:**+```bash+export OPENROUTER_API_KEY='sk-or-v1-...'+# Or add to ~/.zshrc or ~/.bashrc+```++### Error 2: Claudish Not Installed++**Error:**+```+command not found: claudish+```++**Fix:**+```bash+npm install -g claudish+# Or: bun install -g claudish+```++### Error 3: Model Not Found++**Error:**+```+Model 'invalid/model' not found+```++**Fix:**+```bash+# List available models+claudish --models++# Use valid model ID+claudish --model x-ai/grok-code-fast-1 "task"+```++### Error 4: OpenRouter API Error++**Error:**+```+OpenRouter API error: 401 Unauthorized+```++**Fix:**+1. Check API key is correct+2. Verify API key at https://openrouter.ai/keys+3. Check API key has credits (free tier or paid)++### Error 5: Port Already in Use++**Error:**+```+Error: Port 3000 already in use+```++**Fix:**+```bash+# Let Claudish pick random port (default)+claudish --model grok "task"++# Or specify different port+claudish --port 8080 --model grok "task"+```++## Best Practices++### 1. ✅ Use File-Based Instructions++**Why:** Avoids context window pollution++**How:**+```bash+# Write instruction to file+echo "Implement feature X" > /tmp/task.md++# Run with stdin+claudish --stdin --model grok < /tmp/task.md > /tmp/result.md++# Read result+cat /tmp/result.md+```++### 2. ✅ Choose Right Model for Task++**Fast Coding:** `x-ai/grok-code-fast-1`+**Complex Reasoning:** `google/gemini-2.5-flash` or `openai/gpt-5`+**Vision/UI:** `qwen/qwen3-vl-235b-a22b-instruct`++### 3. ✅ Use --json for Automation++**Why:** Structured output, easier parsing++**How:**+```bash+RESULT=$(claudish --json "task" | jq -r '.result')+COST=$(claudish --json "task" | jq -r '.total_cost_usd')+```++### 4. ✅ Delegate to Sub-Agents++**Why:** Keeps main conversation context clean++**How:**+```typescript+await Task({+ subagent_type: "general-purpose",+ description: "Task with Claudish",+ prompt: "Use claudish --model grok '...' and return summary only"+});+```++### 5. ✅ Update Models Regularly++**Why:** Get latest model recommendations++**How:**+```bash+# Auto-updates every 2 days+claudish --models++# Search for specific models+claudish --models deepseek++# Force update now+claudish --models --force-update+```++### 6. ✅ Use --stdin for Large Prompts++**Why:** Avoid command line length limits++**How:**+```bash+git diff | claudish --stdin --model grok "Review changes"+```++## Anti-Patterns (Avoid These)++### ❌❌❌ NEVER Run Claudish Directly in Main Conversation (CRITICAL)++**This is the #1 mistake. Never do this unless user explicitly requests it.**++**WRONG - Destroys context window:**+```typescript+// ❌ NEVER DO THIS - Pollutes main context with 10K+ tokens+await Bash("claudish --model grok 'implement feature'");++// ❌ NEVER DO THIS - Full conversation in main context+await Bash("claudish --model gemini 'review code'");++// ❌ NEVER DO THIS - Even with --json, output is huge+const result = await Bash("claudish --json --model gpt-5 'refactor'");+```++**RIGHT - Always use sub-agents:**+```typescript+// ✅ ALWAYS DO THIS - Delegate to sub-agent+const result = await Task({+ subagent_type: "general-purpose", // or specific agent+ description: "Implement feature with Grok",+ prompt: `+Use Claudish to implement the feature with Grok model.++CRITICAL INSTRUCTIONS:+1. Create instruction file: /tmp/claudish-task-${Date.now()}.md+2. Write detailed task requirements to file+3. Run: claudish --model x-ai/grok-code-fast-1 --stdin < /tmp/claudish-task-*.md+4. Read result file and return ONLY a 2-3 sentence summary++DO NOT return full implementation or conversation.+Keep response under 300 tokens.+ `+});++// ✅ Even better - Use specialized agent if available+const result = await Task({+ subagent_type: "backend-developer", // or frontend-dev, etc.+ description: "Implement with external model",+ prompt: `+Use Claudish with x-ai/grok-code-fast-1 model to implement authentication.+Follow file-based instruction pattern.+Return summary only.+ `+});+```++**When you CAN run directly (rare exceptions):**+```typescript+// ✅ Only when user explicitly requests+// User: "Run claudish directly in main context for debugging"+if (userExplicitlyRequestedDirect) {+ await Bash("claudish --model grok 'task'");+}+```++### ❌ Don't Ignore Model Selection++**Wrong:**+```bash+# Always using default model+claudish "any task"+```++**Right:**+```bash+# Choose appropriate model+claudish --model x-ai/grok-code-fast-1 "quick fix"+claudish --model google/gemini-2.5-flash "complex analysis"+```++### ❌ Don't Parse Text Output++**Wrong:**+```bash+OUTPUT=$(claudish --model grok "task")+COST=$(echo "$OUTPUT" | grep cost | awk '{print $2}')+```++**Right:**+```bash+# Use JSON output+COST=$(claudish --json --model grok "task" | jq -r '.total_cost_usd')+```++### ❌ Don't Hardcode Model Lists++**Wrong:**+```typescript+const MODELS = ["x-ai/grok-code-fast-1", "openai/gpt-5.3"];+```++**Right:**+```typescript+// Query dynamically+const { stdout } = await Bash("claudish --models --json");+const models = JSON.parse(stdout).models.map(m => m.id);+```++### ✅ Do Accept Custom Models From Users++**Problem:** User provides a custom model ID that's not in --top-models++**Wrong (rejecting custom models):**+```typescript+const availableModels = ["x-ai/grok-code-fast-1", "openai/gpt-5.3"];+const userModel = "custom/provider/model-123";++if (!availableModels.includes(userModel)) {+ throw new Error("Model not in my shortlist"); // ❌ DON'T DO THIS+}+```++**Right (accept any valid model ID):**+```typescript+// Claudish accepts ANY valid OpenRouter model ID, even if not in --top-models+const userModel = "custom/provider/model-123";++// Validate it's a non-empty string with provider format+if (!userModel.includes("/")) {+ console.warn("Model should be in format: provider/model-name");+}++// Use it directly - Claudish will validate with OpenRouter+await Bash(`claudish --model ${userModel} "task"`);+```++**Why:** Users may have access to:+- Beta/experimental models+- Private/custom fine-tuned models+- Newly released models not yet in rankings+- Regional/enterprise models+- Cost-saving alternatives++**Always accept user-provided model IDs** unless they're clearly invalid (empty, wrong format).++### ✅ Do Handle User-Preferred Models++**Scenario:** User says "use my custom model X" and expects it to be remembered++**Solution 1: Environment Variable (Recommended)**+```typescript+// Set for the session+process.env.CLAUDISH_MODEL = userPreferredModel;++// Or set permanently in user's shell profile+await Bash(`echo 'export CLAUDISH_MODEL="${userPreferredModel}"' >> ~/.zshrc`);+```++**Solution 2: Session Cache**+```typescript+// Store in a temporary session file+const sessionFile = "/tmp/claudish-user-preferences.json";+const prefs = {+ preferredModel: userPreferredModel,+ lastUsed: new Date().toISOString()+};+await Write({ file_path: sessionFile, content: JSON.stringify(prefs, null, 2) });++// Load in subsequent commands+const { stdout } = await Read({ file_path: sessionFile });+const prefs = JSON.parse(stdout);+const model = prefs.preferredModel || defaultModel;+```++**Solution 3: Prompt Once, Remember for Session**+```typescript+// In a multi-step workflow, ask once+if (!process.env.CLAUDISH_MODEL) {+ const { stdout } = await Bash("claudish --models --json");+ const models = JSON.parse(stdout).models;++ const response = await AskUserQuestion({+ question: "Select model (or enter custom model ID):",+ options: models.map((m, i) => ({ label: m.name, value: m.id })).concat([+ { label: "Enter custom model...", value: "custom" }+ ])+ });++ if (response === "custom") {+ const customModel = await AskUserQuestion({+ question: "Enter OpenRouter model ID (format: provider/model):"+ });+ process.env.CLAUDISH_MODEL = customModel;+ } else {+ process.env.CLAUDISH_MODEL = response;+ }+}++// Use the selected model for all subsequent calls+const model = process.env.CLAUDISH_MODEL;+await Bash(`claudish --model ${model} "task 1"`);+await Bash(`claudish --model ${model} "task 2"`);+```++**Guidance for Agents:**+1. ✅ **Accept any model ID** user provides (unless obviously malformed)+2. ✅ **Don't filter** based on your "shortlist" - let Claudish handle validation+3. ✅ **Offer to set CLAUDISH_MODEL** environment variable for session persistence+4. ✅ **Explain** that --top-models shows curated recommendations, --models shows all+5. ✅ **Validate format** (should contain "/") but not restrict to known models+6. ❌ **Never reject** a user's custom model with "not in my shortlist"++### ❌ Don't Skip Error Handling++**Wrong:**+```typescript+const result = await Bash("claudish --model grok 'task'");+```++**Right:**+```typescript+try {+ const result = await Bash("claudish --model grok 'task'");+} catch (error) {+ console.error("Claudish failed:", error.message);+ // Fallback to embedded Claude or handle error+}+```++## Agent Integration Examples++### Example 1: Code Review Agent++```typescript+/**+ * Agent: code-reviewer (using Claudish with multiple models)+ */+async function reviewCodeWithMultipleModels(files: string[]) {+ const models = [+ "x-ai/grok-code-fast-1", // Fast initial scan+ "google/gemini-2.5-flash", // Deep analysis+ "openai/gpt-5.3" // Final validation+ ];++ const reviews = [];++ for (const model of models) {+ const timestamp = Date.now();+ const instructionFile = `/tmp/review-${model.replace('/', '-')}-${timestamp}.md`;+ const resultFile = `/tmp/review-result-${model.replace('/', '-')}-${timestamp}.md`;++ // Create instruction+ const instruction = createReviewInstruction(files, resultFile);+ await Write({ file_path: instructionFile, content: instruction });++ // Run review with model+ await Bash(`claudish --model ${model} --stdin < ${instructionFile}`);++ // Read result+ const result = await Read({ file_path: resultFile });++ // Extract summary+ reviews.push({+ model,+ summary: extractSummary(result),+ issueCount: extractIssueCount(result)+ });++ // Clean up+ await Bash(`rm ${instructionFile} ${resultFile}`);+ }++ return reviews;+}+```++### Example 2: Feature Implementation Command++```typescript+/**+ * Command: /implement-with-model+ * Usage: /implement-with-model "feature description"+ */+async function implementWithModel(featureDescription: string) {+ // Step 1: Get available models+ const { stdout } = await Bash("claudish --models --json");+ const models = JSON.parse(stdout).models;++ // Step 2: Let user select model+ const selectedModel = await promptUserForModel(models);++ // Step 3: Create instruction file+ const timestamp = Date.now();+ const instructionFile = `/tmp/implement-${timestamp}.md`;+ const resultFile = `/tmp/implement-result-${timestamp}.md`;++ const instruction = `# Feature Implementation++## Description+${featureDescription}++## Requirements+- Write clean, maintainable code+- Add comprehensive tests+- Include error handling+- Follow project conventions++## Output+Write implementation details to: ${resultFile}++Include:+- Files created/modified+- Code snippets+- Test coverage+- Documentation updates+`;++ await Write({ file_path: instructionFile, content: instruction });++ // Step 4: Run implementation+ await Bash(`claudish --model ${selectedModel} --stdin < ${instructionFile}`);++ // Step 5: Read and present results+ const result = await Read({ file_path: resultFile });++ // Step 6: Clean up+ await Bash(`rm ${instructionFile} ${resultFile}`);++ return result;+}+```++## Troubleshooting++### Issue: Slow Performance++**Symptoms:** Claudish takes long time to respond++**Solutions:**+1. Use faster model: `x-ai/grok-code-fast-1` or `minimax/minimax-m2`+2. Reduce prompt size (use --stdin with concise instructions)+3. Check internet connection to OpenRouter++### Issue: High Costs++**Symptoms:** Unexpected API costs++**Solutions:**+1. Use budget-friendly models (check pricing with `--models` or `--top-models`)+2. Enable cost tracking: `--cost-tracker`+3. Use --json to monitor costs: `claudish --json "task" | jq '.total_cost_usd'`++### Issue: Context Window Exceeded++**Symptoms:** Error about token limits++**Solutions:**+1. Use model with larger context (Gemini: 1000K, Grok: 256K)+2. Break task into smaller subtasks+3. Use file-based pattern to avoid conversation history++### Issue: Model Not Available++**Symptoms:** "Model not found" error++**Solutions:**+1. Update model cache: `claudish --models --force-update`+2. Check OpenRouter website for model availability+3. Use alternative model from same category++## Additional Resources++**Documentation:**+- Full README: `mcp/claudish/README.md` (in repository root)+- AI Agent Guide: Print with `claudish --help-ai`+- Model Integration: `skills/claudish-integration/SKILL.md` (in repository root)++**External Links:**+- Claudish GitHub: https://github.com/MadAppGang/claude-code+- OpenRouter: https://openrouter.ai+- OpenRouter Models: https://openrouter.ai/models+- OpenRouter API Docs: https://openrouter.ai/docs++**Version Information:**+```bash+claudish --version+```++**Get Help:**+```bash+claudish --help # CLI usage+claudish --help-ai # AI agent usage guide+```++---++**Maintained by:** MadAppGang+**Last Updated:** January 5, 2026+**Skill Version:** 2.0.0