Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
test(advisor): add the advisor-subscription madbench bench
Jack Rudenko03a094600748
diff --git a/ai-docs/benches/advisor-subscription/README.md b/ai-docs/benches/advisor-subscription/README.mdnew file mode 100644index 00000000..bf06d8b5--- /dev/null+++ b/ai-docs/benches/advisor-subscription/README.md@@ -0,0 +1,133 @@+# advisor-subscription: `--advisor` panel members on the subscription++A re-runnable live check of subscription-aware advisor routing (session+`dev-feature-advisor-subscription-20261020-233354-f7e4`, live checks L1–L7). Through v12.1.2+every advisor panel call was one bare `fetch` with a metered key: a Codex subscriber paid per+token, and `cx@` went to OpenRouter. Each panel member is now planned by the function main+routing is derived from and called through the same handler machinery+(`ai-docs/architecture/advisor.md`, "Panel routing and billing").++Real Claude Code drives this worktree's claudish build against the real subscription backends:+ChatGPT Codex (`cx@`, OAuth), Kimi Coding (`kc@`) and the Claude Code login. There is no mock+upstream, and **no metered key may serve a live call**. A correct answer proves nothing here:+claudish's debug log says who served each panel call and how it billed.++## What the Checks read++`claudish-under-test` passes `--debug-claudish`, so claudish writes `logs/claudish_*.log` into+the workspace. Three kinds of line are graded (`checks/advisor-log.ts`, one `section` per Check):++| Line | Carries | Section |+|---|---|---|+| `[advisor-origin] {json}` | one `advisor_call` per panel member, one `advisor_collector_call` per collector run: `origin`, `stubPath`, `upstreamStatus`, `servedBy {provider, displayName, arm, billing}`, `credentialSource` | `records` |+| `[advisor-notice] <line>` | the startup notice: each member's first hop and its billing | `notice` |+| `… Calling API: <url>` | every upstream request; `[advisor] Calling API: … (role, auth, source)` is the Claude boundary line | `hosts`, `secrets` |++`records` requires at least one record per expected member and requires EVERY record of that+member to match, so one metered record for a subscription member fails it. `notice` also+compares the notice's first hop (`<model> -> <displayName> (`) with every upstream record's+`servedBy.displayName` (L4: the notice promised what ran).++## The Scenarios++The Bench's `harness.args` carry `--permission-mode bypassPermissions`; each Scenario adds its+own `--advisor` (and, for `claude-login`, `--model`) through its own `harness: {args: […]}`,+which madbench appends after the Bench's. claudish keeps the last `--model`;+`claudish-provenance.txt` records the argv.++| Scenario | Main model | `--advisor` | Passes when |+|---|---|---|---|+| `routed-codex` (L1, L2, L4, L6) | `[email protected]`, code mode auto | `gpt-6.1-sol,[email protected],kc@k3:` | the answer carries `REF: heron-5521` (only in `plan.md`) and an `ADVICE:` line; `plan.md` was read; every record of `gpt-6.1-sol` and `[email protected]` is `upstream` and served by `openai-codex`, `servedBy.arm` `oauth`, billing `subscription`; every `kc@k3` record is `upstream`, `kimi-coding`, `subscription`; each member's notice line says `(subscription` and names the provider that served; no notice line says "never use a subscription"; every `Calling API:` goes to `chatgpt.com/backend-api/codex/` or the Kimi Coding host (`api.kimi.ai/coding/`, what the catalog's kimi-coding connection publishes; `api.kimi.com/coding/` is the built-in default), both present; no `was served by … not … as the notice said` line; code mode on from discovery and every relayed exec resumed |+| `routed-grok` (L3 grok) | `[email protected]` | `grok-4.7,[email protected]:` | the answer has an `ADVICE:` line; every record of both members is `upstream`, served by `grok-subscription`, billing `subscription`; both notice lines say `Grok Build (subscription` and name the provider that served; every `Calling API:` goes to `chatgpt.com/backend-api/codex/` or `cli-chat-proxy.grok.com`, both present, none to `api.x.ai` or OpenRouter; no served-unlike line |+| `rejected-sub` (L3 gemini) | `[email protected]` | `gemini-4-argon` | **an observation, not a verdict.** What an expired subscription key should do to the chain is not settled, so only two Checks grade: the answer has a `STATUS:` line (the session completed) and an `advisor_call` record exists. Readouts record the rest: `chain_observed` (hops tried in order, statuses, servedBy, stub reason, `[advisor]` warnings, the `Calling API:` sequence), `stubbed_on_401`, `hosts_subscription_only` and `notice_observed` (the earlier expectation: a stub carrying 401, no metered host). On 2026-10-21 the plan had no Zen Go hop at all (`content-use opt-in id only`), and the call was served metered by Gemini |+| `claude-login` (L7) | `claude-sonnet-5-5`, native: claudish controls the child's auth | `[email protected],sonnet:haiku` | the answer has an `ADVICE:` line; `[email protected]` served by `openai-codex` on `oauth`; `sonnet` and the collector `haiku` are `upstream`, `servedBy.arm` `oauth`, billing `subscription`, `credentialSource: claude-code-login`; the notice says `sonnet -> api.anthropic.com (subscription: your Claude Code login` and has a collector line for `haiku` on the subscription, and never the uncontrolled form `metered, billed per token: ANTHROPIC_API_KEY`; one `[advisor] Calling API: https://api.anthropic.com/v1/messages (role: panel, auth: bearer, source: claude-code-login` line and one with `role: collector`; no `auth: bearer` line to any other host; no log line matches `Bearer sk-ant`, `sk-ant-oat` or `sk-ant-api` |+| `refusal` (L5) | `[email protected]`, no advisor | none in the session | see below |++Every Scenario also carries **`provenance`**: the shim wrote `claudish-provenance.txt` naming+the `dist/index.js` it ran, its `claudish version`, the git head and the argv.++`routed-codex` also has the readout `advisor_tool_seen` (`session:tool-used: advisor`): whether+the captured Session records the advisor tool call at all. The log records are the graded+evidence; the readout only says what the transcript shows.++### Why `routed-codex` names no collector++A panel of two or more with no `:` gets the default collector `haiku`. On a `cx@` launch there is+no Claude login, so `haiku` would sign with an `ANTHROPIC_API_KEY` if claudish finds one (env,+config, Keychain, 1Password): metered. The trailing `:` names no collector, so the Scenario+stays subscription-only. The Claude collector is covered by `claude-login`.++### `refusal`: a startup refusal as a real Check++A refused launch exits 1 before Claude Code starts, so it cannot be the Scenario's own session (madbench would+record an error, never a grade). Instead the Scenario sets `ADVISOR_REFUSAL_PROBE=1` in its+`sandbox.env`, and the shim, before it starts the (advisor-free) session, runs++```+bun <repo>/packages/cli/dist/index.js --debug-claudish --log-level debug --advisor "<spec>" -p "hi"+```++with a 60 s `timeout`, once for `ag@gemini-4-argon` and once for `opusplan`, each in its own+`refusal/<name>/` directory, recording `exit.txt`, `stdout.txt`, `stderr.txt` and its own+`logs/`. The Checks pass when the exit code is 1, stderr carries `--advisor cannot work in this+launch` and names the member and its reason (`does not serve` / no route for Antigravity;+`switches tier` for the `opusplan` Claude selector), and that directory's log has no `Calling+API:` line. The mock harness never runs the shim, so under `madbench check` the files are absent+and both Checks fail.++## Grok needs a live login++`routed-grok` needs a current SuperGrok login (`claudish login grok`). A revoked refresh token+makes both members fail; trust a fresh claudish process, not a long-lived MCP server's preflight,+which can hold a stale token in memory.++## `rejected-sub` can spend++When bare `gemini-4-argon` has no usable subscription candidate (the 2026-10-21 run: Antigravity+does not serve it, Zen Go is skipped as `content-use opt-in id only`), its chain is metered and+the call bills the Google key. Skip it with `--scenario` when that must not happen.++## Running it++```bash+madbench list ai-docs/benches/advisor-subscription/madbench.yaml+madbench preflight ai-docs/benches/advisor-subscription/madbench.yaml+madbench check ai-docs/benches/advisor-subscription/madbench.yaml # 33/33 failed as required+ADVISOR_SUB_KEEP_LOGS=<dir> madbench ai-docs/benches/advisor-subscription/madbench.yaml --report-json <out.json>+```++Run it in a visible terminal and never pipe it: a pipe turns on `--plain`. `use_subscription:+required` makes madbench mask any `ANTHROPIC_API_KEY` it resolves, so claudish never receives one+from madbench; preflight says so.++## Keeping the logs++madbench deletes the workspace after grading and has no retention flag; its documented answer+is to have a Check copy what it needs out (madbench `docs/sandbox-and-testdata.md`, "No workspace+retention flag"). The `provenance` Check's first line does that: with+`ADVISOR_SUB_KEEP_LOGS=<dir>` set it copies `logs/claudish_*.log`, `claudish-provenance.txt` and+`refusal/` to `<dir>/<workspace name>/`. Match a folder to its Scenario with the report's+`.results[].session.work_dir`. A passing `exec` Check also keeps its own output in the report's+`evidence.output` (madbench 0.53.0).++## Reading a failure++Read `.summary.errors` before `.summary.failed`: an errored Scenario never graded.++```bash+jq -c '.summary' out.json+jq -r '.results[] | "\(.scenario_id) \(.status)", (.checks[] | select(.result.pass==false) | " \(.spec.metric): \(.result.reason)")' out.json+```++| Failed Check | Means | Look at |+|---|---|---|+| `panel_served` with `servedBy.billing=metered` | a panel call was billed per token: the defect this Bench exists for | the record's `hopsTried`, `servedBy` in the Check's evidence |+| `panel_served` with `no advisor_call record` | the main model never called the advisor, or the decorator never captured it | `advisor_tool_seen`; `[advisor-swap]` lines in the log |+| `panel_stubbed` with `origin=upstream` | a rejected subscription member was served by another hop | the record's `servedBy`; `hosts` will name the host |+| `notice_matches` with `notice said X, served by Y` | L4: the notice promised one provider and another served | the notice lines and records in the evidence |+| `notice_matches` with `metered, billed per token: ANTHROPIC_API_KEY` | `claude-login`: the launch was not one claudish controls (settings or env carry a credential) | the notice text names the reason; `childAuthControl` |+| `hosts` | a request went to a host outside the subscription set | the `calling_api` tally in the evidence |+| `no_bearer_value` | a Claude login value reached the log | the matched pattern and line count |+| `refusal_*` | the launch was not refused, refused for another reason, or made a request first | the Check's output: spec, exit code, stderr |+| `code_mode_on` / `resumed` | the main session's code mode broke alongside the advisor | `codex-code-mode` README |+| `provenance` | the shim did not run, so the build under test is unknown | `harness.binary`; `bun run build` |diff --git a/ai-docs/benches/advisor-subscription/checks/advisor-log.ts b/ai-docs/benches/advisor-subscription/checks/advisor-log.tsnew file mode 100644index 00000000..d6cd80b8--- /dev/null+++ b/ai-docs/benches/advisor-subscription/checks/advisor-log.ts@@ -0,0 +1,280 @@+// Grades what claudish's debug log says about the advisor, one section per Check.+//+// claudish-under-test passes --debug-claudish, so claudish writes+// logs/claudish_*.log into the workspace. Three kinds of line matter+// (ai-docs/architecture/advisor.md):+//+// [advisor-origin] {json} one record per panel model (`advisor_call`), per+// collector run (`advisor_collector_call`) and per+// call (`advisor_rewrite`): origin, stubPath,+// requestedModel, upstreamStatus, servedBy+// (provider, displayName, the signing+// credential, billing), ...+// [advisor-notice] <line> the startup notice, one line each+// ... Calling API: <url> every upstream request; `[advisor] Calling API:`+// is the Claude member/collector boundary line+//+// `config.section` picks what this Check grades:+//+// records every `members[]` (and `collector`) entry has at least one record,+// and EVERY record of that model matches the expected fields. A+// single metered record for a subscription member fails it.+// notice each `lines[]` entry is matched by some notice line that starts+// (trimmed) with its `startsWith`, when given, and carries all of+// its `contains`; no notice line carries a `forbid` string; for+// each model in `servedByMatches`, the notice's first hop+// (`<model> -> <displayName> (`) equals every upstream record's+// servedBy.displayName (L4: the notice promised what ran).+// hosts every `Calling API:` URL matches one of `allow`; each `require`+// pattern matches some line; no line matches a `forbid` pattern.+// secrets the log carries advisor records (so it is the right log), and no+// line of any kept file matches a `forbid` pattern.+// observe no expectation beyond "a record exists per member": the evidence+// records the hops tried, statuses, servedBy, stub reason, the+// `[advisor]` warnings and the `Calling API:` sequence.+//+// Every section fails when no log exists, so the mock harness fails them all.+import { readdirSync, readFileSync } from "node:fs";+import { join } from "node:path";++type Expect = {+ model: string;+ origin?: string;+ provider?: string;+ /** Compared with the record's servedBy signing field (claudish names it SIGNING_KEY). */+ signedBy?: string;+ billing?: string;+ credentialSource?: string;+ stubPath?: string[];+ status?: number;+};+type Config = {+ section: "records" | "notice" | "hosts" | "secrets" | "observe";+ members?: Expect[];+ collector?: Expect;+ lines?: { startsWith?: string; contains: string[] }[];+ forbid?: string[];+ servedByMatches?: string[];+ allow?: string[];+ require?: string[];+};+type Rec = Record<string, any>;++/** claudish's own field name for the credential that signed (`oauth` = the subscription). */+const SIGNING_KEY = "arm";++const fail = (reason: string, evidence: Record<string, unknown> = {}) => ({ pass: false, score: 0, reason, evidence });++function readLogs(wd: string): { names: string[]; text: string } {+ const dir = join(wd, "logs");+ let names: string[] = [];+ try {+ names = readdirSync(dir).filter((n) => /^claudish_.*\.log$/.test(n)).sort();+ } catch {+ return { names: [], text: "" };+ }+ return { names, text: names.map((n) => readFileSync(join(dir, n), "utf8")).join("\n") };+}++function records(text: string): Rec[] {+ const out: Rec[] = [];+ for (const line of text.split("\n")) {+ const i = line.indexOf("[advisor-origin] ");+ if (i < 0) continue;+ try {+ out.push(JSON.parse(line.slice(i + "[advisor-origin] ".length)));+ } catch {+ // a truncated line is not a record+ }+ }+ return out;+}++function noticeLines(text: string): string[] {+ const out: string[] = [];+ for (const line of text.split("\n")) {+ const i = line.indexOf("[advisor-notice] ");+ if (i >= 0) out.push(line.slice(i + "[advisor-notice] ".length));+ }+ return out;+}++/** The short form of a record that an evidence reader needs. */+function brief(r: Rec) {+ return {+ kind: r.kind,+ requestedModel: r.requestedModel,+ origin: r.origin,+ stubPath: r.stubPath,+ upstreamStatus: r.upstreamStatus,+ servedBy: r.servedBy,+ credentialSource: r.credentialSource,+ loginSkipped: r.loginSkipped,+ systemHead: r.systemHead,+ shapeRefusal: r.shapeRefusal,+ hopsTried: r.hopsTried,+ reason: typeof r.reason === "string" ? r.reason.slice(0, 400) : r.reason,+ };+}++function mismatches(r: Rec, e: Expect): string[] {+ const bad: string[] = [];+ if (e.origin && r.origin !== e.origin) bad.push(`origin=${r.origin}`);+ if (e.provider && r.servedBy?.provider !== e.provider) bad.push(`servedBy.provider=${r.servedBy?.provider}`);+ if (e.signedBy && r.servedBy?.[SIGNING_KEY] !== e.signedBy) {+ bad.push(`servedBy.${SIGNING_KEY}=${r.servedBy?.[SIGNING_KEY]}`);+ }+ if (e.billing && r.servedBy?.billing !== e.billing) bad.push(`servedBy.billing=${r.servedBy?.billing}`);+ if (e.credentialSource && r.credentialSource !== e.credentialSource) bad.push(`credentialSource=${r.credentialSource}`);+ if (e.stubPath && !e.stubPath.includes(r.stubPath)) bad.push(`stubPath=${r.stubPath}`);+ if (e.status !== undefined) {+ const text = `${r.upstreamStatus ?? ""} ${r.reason ?? ""}`;+ if (r.upstreamStatus !== e.status && !text.includes(String(e.status))) bad.push(`upstreamStatus=${r.upstreamStatus}`);+ }+ return bad;+}++function gradeRecords(recs: Rec[], cfg: Config) {+ const problems: string[] = [];+ const perModel: Record<string, unknown> = {};+ const check = (kind: string, e: Expect) => {+ const mine = recs.filter((r) => r.kind === kind && r.requestedModel === e.model);+ perModel[`${kind}:${e.model}`] = mine.map(brief);+ if (mine.length === 0) {+ problems.push(`${e.model}: no ${kind} record`);+ return;+ }+ mine.forEach((r, i) => {+ const bad = mismatches(r, e);+ if (bad.length) problems.push(`${e.model} ${kind} #${i + 1}: ${bad.join(", ")}`);+ });+ };+ for (const e of cfg.members ?? []) check("advisor_call", e);+ if (cfg.collector) check("advisor_collector_call", cfg.collector);+ const counts = Object.fromEntries(Object.entries(perModel).map(([k, v]) => [k, (v as unknown[]).length]));+ return { problems, evidence: { counts, records: perModel } };+}++function gradeNotice(text: string, recs: Rec[], cfg: Config) {+ const lines = noticeLines(text);+ const problems: string[] = [];+ if (lines.length === 0) problems.push("no [advisor-notice] line in the log");+ for (const want of cfg.lines ?? []) {+ const ok = lines.some(+ (l) => (!want.startsWith || l.trim().startsWith(want.startsWith)) && want.contains.every((c) => l.includes(c)),+ );+ if (!ok) {+ problems.push(+ `no notice line ${want.startsWith ? `starting ${JSON.stringify(want.startsWith)} and ` : ""}carrying all of ${JSON.stringify(want.contains)}`,+ );+ }+ }+ for (const f of cfg.forbid ?? []) {+ const hit = lines.find((l) => l.includes(f));+ if (hit) problems.push(`notice carries forbidden text ${JSON.stringify(f)}: ${hit.trim()}`);+ }+ const served: Record<string, unknown> = {};+ for (const model of cfg.servedByMatches ?? []) {+ // A displayName may itself carry parentheses ("Grok Build (subscription)"),+ // so the notice's first hop is compared as a prefix, never cut by regex.+ const line = lines.find((l) => l.trim().startsWith(`${model} -> `))?.trim();+ const ran = recs.filter((r) => r.kind === "advisor_call" && r.requestedModel === model && r.origin === "upstream");+ const names = [...new Set(ran.map((r) => r.servedBy?.displayName))];+ served[model] = { notice: line ?? null, servedBy: names };+ if (!line) problems.push(`${model}: no notice line '${model} -> <provider> (' found`);+ else if (ran.length === 0) problems.push(`${model}: no upstream record to compare the notice with`);+ else {+ const unlike = names.filter((n) => !line.startsWith(`${model} -> ${n} (`) && line !== `${model} -> ${n}`);+ if (unlike.length) problems.push(`${model}: notice said "${line}", served by ${unlike.join(", ")}`);+ }+ }+ return { problems, evidence: { notice: lines, servedBy_vs_notice: served } };+}++function gradeHosts(text: string, cfg: Config) {+ const calls = text.split("\n").filter((l) => l.includes("Calling API:")).map((l) => l.replace(/^\[[^\]]*\] /, ""));+ const problems: string[] = [];+ if (calls.length === 0) problems.push("no 'Calling API:' line in the log");+ const allow = (cfg.allow ?? []).map((p) => new RegExp(p));+ for (const c of calls) {+ const url = c.slice(c.indexOf("Calling API:") + "Calling API:".length).trim();+ if (!allow.some((re) => re.test(url))) problems.push(`not allowed: ${c}`);+ }+ for (const p of cfg.require ?? []) {+ if (!calls.some((c) => new RegExp(p).test(c))) problems.push(`no 'Calling API:' line matches /${p}/`);+ }+ for (const p of cfg.forbid ?? []) {+ const hit = calls.find((c) => new RegExp(p).test(c));+ if (hit) problems.push(`forbidden /${p}/: ${hit}`);+ }+ const tally: Record<string, number> = {};+ for (const c of calls) tally[c] = (tally[c] ?? 0) + 1;+ return { problems: [...new Set(problems)], evidence: { calling_api: tally } };+}++/**+ * observe: records what happened for `members[]` without judging it. Passes+ * when every member has at least one record (so a readout built on it still+ * fails under the mock); the evidence is the point: each record's hopsTried,+ * upstreamStatus, servedBy and stub reason, every `[advisor]` warning line,+ * every routing/fallback line, and the `Calling API:` sequence in order.+ */+function observe(text: string, recs: Rec[], cfg: Config) {+ const problems: string[] = [];+ const perModel: Record<string, unknown> = {};+ for (const e of cfg.members ?? []) {+ const mine = recs.filter((r) => r.kind === "advisor_call" && r.requestedModel === e.model);+ if (mine.length === 0) problems.push(`${e.model}: no advisor_call record`);+ perModel[e.model] = mine.map((r) => ({ ...brief(r), route: r.route, routedModel: r.routedModel, billing: r.billing, upstreamStatusSource: r.upstreamStatusSource, latencyMs: r.latencyMs }));+ }+ const strip = (l: string) => l.replace(/^\[[^\]]*\] /, "");+ const all = text.split("\n");+ const warnings = all.filter((l) => /\[advisor\]/.test(l) && !/Calling API:/.test(l)).map(strip).slice(0, 40);+ const routing = all+ .filter((l) => /\[Fallback|terminal credential|\[advisor-route|\[Routing\]/.test(l))+ .map((l) => strip(l).slice(0, 400))+ .slice(0, 40);+ const callingApi = all.filter((l) => l.includes("Calling API:")).map(strip);+ return { problems, evidence: { records: perModel, advisor_warnings: warnings, routing_lines: routing, calling_api_in_order: callingApi } };+}++function gradeSecrets(wd: string, text: string, recs: Rec[], cfg: Config) {+ const problems: string[] = [];+ if (recs.length === 0) problems.push("no [advisor-origin] record, so this is not an advisor log");+ const hits: Record<string, number> = {};+ for (const p of cfg.forbid ?? []) {+ const n = text.split("\n").filter((l) => new RegExp(p).test(l)).length;+ hits[p] = n;+ if (n > 0) problems.push(`${n} log line(s) match /${p}/`);+ }+ let redacted = 0;+ for (const l of text.split("\n")) if (l.includes("[redacted]")) redacted++;+ return { problems, evidence: { forbidden_hits: hits, redacted_markers: redacted, records: recs.length, work_dir: wd } };+}++export default function evaluate(_output: string, context: { config?: Config; session?: { work_dir?: string } }) {+ const cfg = context.config;+ if (!cfg?.section) throw new Error("advisor-log: config.section is required");+ const wd = context.session?.work_dir;+ if (!wd) return fail("the Session carries no work_dir, so the claudish log cannot be read");+ const { names, text } = readLogs(wd);+ if (names.length === 0) return fail("no logs/claudish_*.log in the workspace (claudish never ran with --debug-claudish)");+ const recs = records(text);+ const r =+ cfg.section === "records"+ ? gradeRecords(recs, cfg)+ : cfg.section === "notice"+ ? gradeNotice(text, recs, cfg)+ : cfg.section === "hosts"+ ? gradeHosts(text, cfg)+ : cfg.section === "observe"+ ? observe(text, recs, cfg)+ : gradeSecrets(wd, text, recs, cfg);+ const pass = r.problems.length === 0;+ return {+ pass,+ score: pass ? 1 : 0,+ reason: pass ? `${cfg.section}: every expectation held (${names.join(", ")})` : `${cfg.section}: ${r.problems.join("; ")}`,+ evidence: { logs: names, ...r.evidence },+ };+}diff --git a/ai-docs/benches/advisor-subscription/claudish-under-test b/ai-docs/benches/advisor-subscription/claudish-under-testnew file mode 100755index 00000000..961f2e81--- /dev/null+++ b/ai-docs/benches/advisor-subscription/claudish-under-test@@ -0,0 +1,48 @@+#!/bin/sh+# harness.binary for this Bench: THIS worktree's built claudish+# (packages/cli/dist/index.js, the output of `bun run build`), never the+# claudish on PATH. The path is derived from this file's own location, so a+# run cannot pick up a global install by accident.+#+# Before handing over it writes claudish-provenance.txt into the workspace+# (the agent's cwd); the `provenance` Check reads it back, so the report+# records which build ran and the exact argv madbench handed over (the+# Scenario's own `harness.args`, e.g. --advisor, come last).+#+# ADVISOR_REFUSAL_PROBE=1 (set by the `refusal` Scenario's sandbox.env): before+# the session starts, run two launches that claudish must REFUSE at startup,+# each in its own directory under refusal/, recording exit code, stdout,+# stderr and its own debug log. The `refusal_*` Checks grade those files. The+# mock harness never runs this file, so under `madbench check` the files are+# absent and the Checks fail, as they must.+#+# Then it adds the debug-log flags every Check that reads logs/claudish_*.log+# depends on, and `-q`, which returns before claudish's update prompt.+set -eu+here=$(cd "$(dirname "$0")" && pwd)+root=$(cd "$here/../../.." && pwd)+dist="$root/packages/cli/dist/index.js"+{+ echo "dist=$dist"+ echo "version=$(bun "$dist" --version 2>/dev/null | head -n 1)"+ echo "head=$(git -C "$root" rev-parse HEAD 2>/dev/null || echo unknown)"+ echo "dist_mtime=$(stat -f %Sm -t %Y-%m-%dT%H:%M:%S "$dist")"+ echo "argv=$*"+} > claudish-provenance.txt++if [ "${ADVISOR_REFUSAL_PROBE:-}" = "1" ]; then+ probe() {+ name=$1+ spec=$2+ mkdir -p "refusal/$name"+ rc=0+ (cd "refusal/$name" && timeout 60 bun "$dist" --debug-claudish --log-level debug \+ --advisor "$spec" -p "hi" > stdout.txt 2> stderr.txt < /dev/null) || rc=$?+ echo "$rc" > "refusal/$name/exit.txt"+ echo "spec=$spec" > "refusal/$name/spec.txt"+ }+ probe ag-gemini "ag@gemini-4-argon"+ probe opusplan "opusplan"+fi++exec bun "$dist" --debug-claudish --log-level debug -q "$@"diff --git a/ai-docs/benches/advisor-subscription/madbench.yaml b/ai-docs/benches/advisor-subscription/madbench.yamlnew file mode 100644index 00000000..be036905--- /dev/null+++ b/ai-docs/benches/advisor-subscription/madbench.yaml@@ -0,0 +1,328 @@+description: "advisor-subscription: --advisor panel members route like main requests, on the subscription, and the notice matches what served"++# Real Claude Code -> this worktree's claudish build (--advisor) -> the REAL+# subscription backends: ChatGPT Codex (cx@, OAuth), Kimi Coding (kc@), the+# Claude Code login. No mock upstream, no metered key may serve a live call.+# README.md explains each Check and how to read a failure.+#+# Every claim that matters is in claudish's debug log, not in the answer: the+# `[advisor-origin]` records say who served each panel call and how it billed,+# the `[advisor-notice]` lines say what the launch promised, and the+# `Calling API:` lines say where every request went. The Checks read+# logs/claudish_*.log, which claudish writes into the workspace because+# claudish-under-test passes --debug-claudish.++harness:+ type: claude-code+ # This worktree's packages/cli/dist/index.js, via a shim that records the+ # build and argv it ran (claudish-provenance.txt). A path with a `/`+ # resolves against this file, so the claudish on PATH is never used.+ binary: ./claudish-under-test+ model: [email protected]+ # The run bills this machine's Claude Code login or does not run; an+ # ANTHROPIC_API_KEY madbench resolves is masked rather than handed to+ # claudish, where it would make a Claude advisor member metered.+ use_subscription: required+ # Interactive (the default): Bash parks at an approval menu nobody answers+ # unless the mode is bypassPermissions. claudish forwards it to claude.+ args: ["--permission-mode", "bypassPermissions"]++defaults:+ # workspace, not home: home replaces HOME and hides ~/.claudish, so the+ # Codex OAuth credential, the Kimi Coding key and the Claude login would+ # be gone and every member could only sign metered (or not at all).+ sandbox: workspace+ # Every prompt is an order, so a question is a defect: fail it by name.+ driver: false+ # One advisor call fans out to every panel member (60 s budget each) and a+ # collector (30 s); an interactive cold start adds Claude Code's boot.+ timeout: 480s++scenarios:+ # ── routed-codex: L1 (bare gpt-*), L2 (cx@), L4 (notice), L6 (code mode) ──+ - name: routed-codex+ description: "main cx@ in code mode; panel gpt-6.1-sol, [email protected], kc@k3 all served on their subscriptions"+ testdata: ./testdata/routed-codex+ # The trailing ':' names NO collector. Without it a 3-member panel gets+ # the default collector haiku, which on a cx@ launch has no Claude login+ # and would sign with an ANTHROPIC_API_KEY if one is found: metered.+ harness:+ args: ["--advisor", "gpt-6.1-sol,[email protected],kc@k3:"]+ prompt: "Use the Read tool to read plan.md. Then call the advisor tool exactly once, asking whether the plan in plan.md is sound. Then reply in exactly two lines. The first line starts with the word REF and a colon, followed by the reference code written in plan.md. The second line starts with the word ADVICE and a colon, followed by the first sentence of the advice the advisor returned."+ checks:+ - type: regex+ metric: answer+ value: 'REF:\s*heron-5521[\s\S]*ADVICE:\s*\S'+ - type: session:file-read+ metric: read_plan+ value: plan.md+ # One advisor_call record per member, and every record of that member+ # served upstream by the named subscription provider, signed with the+ # OAuth credential (`signedBy` is compared with the record's+ # servedBy signing field).+ - type: ts+ metric: panel_served+ value: "file://./checks/advisor-log.ts"+ config:+ section: records+ members:+ - { model: gpt-6.1-sol, origin: upstream, provider: openai-codex, signedBy: oauth, billing: subscription }+ - { model: [email protected], origin: upstream, provider: openai-codex, signedBy: oauth, billing: subscription }+ - { model: kc@k3, origin: upstream, provider: kimi-coding, billing: subscription }+ # L4: the notice names each member's provider as a subscription, never+ # says panel calls skip the subscription, and promised what served.+ - type: ts+ metric: notice_matches+ value: "file://./checks/advisor-log.ts"+ config:+ section: notice+ lines:+ - { startsWith: "gpt-6.1-sol -> ", contains: ["(subscription"] }+ - { startsWith: "[email protected] -> ", contains: ["(subscription"] }+ - { startsWith: "kc@k3 -> ", contains: ["(subscription"] }+ forbid: ["never use a subscription"]+ servedByMatches: [gpt-6.1-sol, [email protected], kc@k3]+ # Every upstream request (main and panel) went to the Codex backend or+ # the Kimi Coding host; nothing to api.openai.com or OpenRouter. The+ # catalog's kimi-coding connection publishes api.kimi.ai/coding/v1+ # (measured 2026-10-21); the built-in default is api.kimi.com/coding/v1.+ - type: ts+ metric: hosts+ value: "file://./checks/advisor-log.ts"+ config:+ section: hosts+ allow: ['^https://chatgpt\.com/backend-api/codex/', '^https://api\.kimi\.(ai|com)/coding/']+ require: ['Calling API: https://chatgpt\.com/backend-api/codex/', 'Calling API: https://api\.kimi\.(ai|com)/coding/']+ # No routed call was served by a hop unlike the notice's (stickiness+ # warning). A positive Check: the log must hold an advisor_rewrite.+ - ¬_served_unlike+ type: exec+ metric: no_served_unlike+ cmd: 'L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" && grep -h "\[advisor-origin\].*\"advisor_rewrite\"" $L | head -n 3 | cut -c1-600 && ! grep -h "was served by .* not .* as the notice said" $L'+ - &code_mode_on+ type: exec+ metric: code_mode_on+ # auto chose code mode from Codex /models discovery (codex-code-mode bench).+ cmd: 'L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" && grep -h "\[CodeMode\] on (" $L && grep -q "\[CodeMode\] on ([a-z]*, discovery: " $L'+ - &resumed+ type: exec+ metric: resumed+ # Every exec relayed to Claude Code as tool_use blocks later logs+ # `exec <id>: resumed;` and no script took the lost-state path+ # (codex-code-mode bench, the D1 detector).+ cmd: |+ L=$(ls logs/claudish_*.log 2>/dev/null); test -n "$L" || exit 1+ grep -h -e "relayed as" -e ": resumed;" -e "script lost" -e "answered as Script failed" $L | sed "s/^\[[^]]*\] //"+ awk '+ /\[ResponsesTurn\] exec call_[A-Za-z0-9]+ relayed as/ { id=$0; sub(/.*\] exec /,"",id); sub(/ .*/,"",id); r[id]=1 }+ /\[CodeMode\] exec call_[A-Za-z0-9]+: resumed;/ { id=$0; sub(/.*\] exec /,"",id); sub(/:.*/,"",id); if (id in r) s[id]=1 }+ /script lost|answered as Script failed/ { lost++ }+ END { n=0; m=0; for (i in r) { n++; if (i in s) m++ }+ printf "relayed=%d resumed=%d lost=%d\n", n, m, lost+0+ exit (n > 0 && m == n && lost == 0) ? 0 : 1 }' $L+ - &provenance+ type: exec+ metric: provenance+ # The shim ran, so this worktree's dist/index.js is what claudish was.+ #+ # The first line keeps the evidence: madbench deletes the workspace+ # after grading and has no retention flag; its documented answer is+ # "have a Check copy it out" (madbench docs/sandbox-and-testdata.md,+ # "No workspace retention flag"). With ADVISOR_SUB_KEEP_LOGS=<dir> set,+ # the logs, the provenance file and refusal/ are copied to+ # <dir>/<workspace name>/ (map it to a Scenario by the report's+ # session.work_dir). Unset, it does nothing.+ cmd: |+ if [ -n "${ADVISOR_SUB_KEEP_LOGS:-}" ]; then d="$ADVISOR_SUB_KEEP_LOGS/$(basename "$PWD")"; mkdir -p "$d" && cp claudish-provenance.txt logs/claudish_*.log "$d/" 2>/dev/null; [ -d refusal ] && cp -R refusal "$d/" 2>/dev/null; true; fi+ cat claudish-provenance.txt && grep -q "^dist=/.*/packages/cli/dist/index.js$" claudish-provenance.txt && grep -q "^version=claudish version " claudish-provenance.txt+ # Measures whether Claude Code recorded an advisor tool call at all; the+ # log records above are the graded evidence.+ - type: session:tool-used+ metric: advisor_tool_seen+ value: advisor+ readout: true++ # ── routed-grok: L3 grok — bare grok-4.7 and [email protected] on the SuperGrok subscription ──+ - name: routed-grok+ description: "main cx@; panel grok-4.7 (bare) and [email protected] (explicit) both served by Grok Build on the subscription"+ testdata: ./testdata/plain+ harness:+ args: ["--advisor", "grok-4.7,[email protected]:"]+ prompt: "Read question.md, then call the advisor tool exactly once with that question. Then reply with one line that starts with the word ADVICE and a colon, followed by the first sentence of the advice the advisor returned."+ checks:+ - type: regex+ metric: answer+ value: 'ADVICE:\s*\S'+ - type: ts+ metric: panel_served+ value: "file://./checks/advisor-log.ts"+ config:+ section: records+ members:+ - { model: grok-4.7, origin: upstream, provider: grok-subscription, billing: subscription }+ - { model: [email protected], origin: upstream, provider: grok-subscription, billing: subscription }+ - type: ts+ metric: notice_matches+ value: "file://./checks/advisor-log.ts"+ config:+ section: notice+ lines:+ - { startsWith: "grok-4.7 -> ", contains: ["Grok Build (subscription"] }+ - { startsWith: "[email protected] -> ", contains: ["Grok Build (subscription"] }+ forbid: ["never use a subscription"]+ servedByMatches: [grok-4.7, [email protected]]+ # Main on the Codex backend, panel on the SuperGrok host; never the+ # metered x.ai API or OpenRouter.+ - type: ts+ metric: hosts+ value: "file://./checks/advisor-log.ts"+ config:+ section: hosts+ allow: ['^https://chatgpt\.com/backend-api/codex/', '^https://cli-chat-proxy\.grok\.com/']+ require: ['Calling API: https://chatgpt\.com/backend-api/codex/', 'Calling API: https://cli-chat-proxy\.grok\.com/']+ forbid: ['api\.x\.ai', 'openrouter']+ - *not_served_unlike+ - *provenance++ # ── rejected-sub: L3 gemini — a rejected subscription login is never moved onto a metered hop ──+ - name: rejected-sub+ description: "panel gemini-4-argon plans OpenCode Zen Go (subscription) first and its key is expired (401): OBSERVED, what the chain does next is recorded, not graded"+ testdata: ./testdata/plain+ harness:+ args: ["--advisor", "gemini-4-argon"]+ prompt: "Read question.md, then call the advisor tool exactly once with that question. Then reply with one line that starts with the word STATUS and a colon, saying whether the advisor returned advice or an error, and quoting the first sentence of what it returned."+ # What an expired subscription key should do to the chain is not settled+ # (2026-10-21: the owner says it must NOT block the chain; the design note+ # says a rejected subscription login is never routed onto a metered hop).+ # So only "the session completed" and "the call was recorded" grade; the+ # rest are readouts that record what happened.+ checks:+ - type: regex+ metric: answer+ value: 'STATUS:\s*\S'+ # Graded: an advisor_call record exists for the member.+ - type: ts+ metric: panel_recorded+ value: "file://./checks/advisor-log.ts"+ config:+ section: records+ members:+ - { model: gemini-4-argon }+ # Readout: hops tried in order, each status, which hop served or the+ # stub reason, servedBy billing, every [advisor] warning line.+ - type: ts+ metric: chain_observed+ readout: true+ value: "file://./checks/advisor-log.ts"+ config:+ section: observe+ members:+ - { model: gemini-4-argon }+ # Readout: did the design note's expectation hold (stub carrying the 401,+ # no metered host)? Recorded either way, never failing the Scenario.+ - type: ts+ metric: stubbed_on_401+ readout: true+ value: "file://./checks/advisor-log.ts"+ config:+ section: records+ members:+ - { model: gemini-4-argon, origin: stub, stubPath: [PANEL_ERROR, ALL_PANEL_FAILED], status: 401 }+ - type: ts+ metric: hosts_subscription_only+ readout: true+ value: "file://./checks/advisor-log.ts"+ config:+ section: hosts+ allow: ['^https://chatgpt\.com/backend-api/codex/', '^https://opencode\.ai/zen/go/']+ require: ['Calling API: https://chatgpt\.com/backend-api/codex/', 'Calling API: https://opencode\.ai/zen/go/']+ forbid: ['generativelanguage\.googleapis\.com', 'openrouter', 'opencode\.ai/zen/v1', 'api\.openai\.com']+ - type: ts+ metric: notice_observed+ readout: true+ value: "file://./checks/advisor-log.ts"+ config:+ section: notice+ lines:+ - { startsWith: "gemini-4-argon -> ", contains: ["(subscription"] }+ forbid: ["never use a subscription"]+ - *provenance++ # ── claude-login: L7 — Claude member and collector on the Claude Code login ──+ - name: claude-login+ description: "main claude-sonnet-5-5 (claudish controls the child's auth); panel [email protected] + sonnet, collector haiku, both Claude calls on the login"+ testdata: ./testdata/plain+ # --model here comes after the Bench's and wins (claudish keeps the last+ # --model); the provenance file records the argv.+ harness:+ args: ["--model", "claude-sonnet-5-5", "--advisor", "[email protected],sonnet:haiku"]+ prompt: "Read question.md, then call the advisor tool exactly once with that question. Then reply with one line that starts with the word ADVICE and a colon, followed by the first sentence of the advice the advisor returned."+ checks:+ - type: regex+ metric: answer+ value: 'ADVICE:\s*\S'+ - type: ts+ metric: panel_served+ value: "file://./checks/advisor-log.ts"+ config:+ section: records+ members:+ - { model: [email protected], origin: upstream, provider: openai-codex, signedBy: oauth, billing: subscription }+ - { model: sonnet, origin: upstream, signedBy: oauth, billing: subscription, credentialSource: claude-code-login }+ collector: { model: haiku, origin: upstream, signedBy: oauth, billing: subscription, credentialSource: claude-code-login }+ - type: ts+ metric: notice_matches+ value: "file://./checks/advisor-log.ts"+ config:+ section: notice+ lines:+ - { startsWith: "sonnet -> ", contains: ["api.anthropic.com (subscription: your Claude Code login"] }+ - { startsWith: "[email protected] -> ", contains: ["(subscription"] }+ - { startsWith: "collector:", contains: ["haiku", "subscription"] }+ # The uncontrolled-launch form of a Claude member reads+ # "(metered, billed per token: ANTHROPIC_API_KEY; ...)".+ forbid: ["never use a subscription", "metered, billed per token: ANTHROPIC_API_KEY"]+ servedByMatches: [[email protected]]+ # The Claude calls carry the login bearer to api.anthropic.com and only+ # there; one boundary line per role.+ - type: ts+ metric: hosts+ value: "file://./checks/advisor-log.ts"+ config:+ section: hosts+ allow: ['^https://chatgpt\.com/backend-api/codex/', '^https://api\.anthropic\.com/v1/messages']+ require:+ - '\[advisor\] Calling API: https://api\.anthropic\.com/v1/messages \(role: panel, auth: bearer, source: claude-code-login'+ - '\[advisor\] Calling API: https://api\.anthropic\.com/v1/messages \(role: collector, auth: bearer, source: claude-code-login'+ forbid: ['Calling API: (?!https://api\.anthropic\.com/)\S+ \(.*auth: bearer']+ # No bearer value in any log line or origin record.+ - type: ts+ metric: no_bearer_value+ value: "file://./checks/advisor-log.ts"+ config:+ section: secrets+ forbid: ['Bearer sk-ant', 'sk-ant-oat', 'sk-ant-api']+ - *not_served_unlike+ - *provenance++ # ── refusal: L5 — a panel model nothing can serve is refused at launch ──+ # No advisor in the session itself: the shim runs the two launches that+ # must be refused (ADVISOR_REFUSAL_PROBE=1) before the session starts, from+ # madbench's clean environment, and records each one under refusal/.+ - name: refusal+ description: "--advisor ag@gemini-4-argon and --advisor opusplan exit 1 before spawning, naming the member and the reason"+ testdata: ./testdata/plain+ sandbox:+ level: workspace+ env:+ ADVISOR_REFUSAL_PROBE: "1"+ timeout: 300s+ prompt: "Reply with the single word ready, in capital letters, and nothing else."+ checks:+ - type: exec+ metric: refusal_antigravity+ cmd: 'd=refusal/ag-gemini; test -f $d/exit.txt && cat $d/spec.txt && echo "exit=$(cat $d/exit.txt)" && cat $d/stderr.txt && test "$(cat $d/exit.txt)" = 1 && grep -q -- "--advisor cannot work in this launch" $d/stderr.txt && grep -q "gemini-4-argon" $d/stderr.txt && grep -Eqi "does not serve|no route|not served" $d/stderr.txt && ! grep -rqs "Calling API:" $d/logs'+ - type: exec+ metric: refusal_claude_selector+ cmd: 'd=refusal/opusplan; test -f $d/exit.txt && cat $d/spec.txt && echo "exit=$(cat $d/exit.txt)" && cat $d/stderr.txt && test "$(cat $d/exit.txt)" = 1 && grep -q -- "--advisor cannot work in this launch" $d/stderr.txt && grep -q "opusplan" $d/stderr.txt && grep -q "switches tier" $d/stderr.txt && ! grep -rqs "Calling API:" $d/logs'+ - *provenancediff --git a/ai-docs/benches/advisor-subscription/testdata/plain/question.md b/ai-docs/benches/advisor-subscription/testdata/plain/question.mdnew file mode 100644index 00000000..93da1f8d--- /dev/null+++ b/ai-docs/benches/advisor-subscription/testdata/plain/question.md@@ -0,0 +1,4 @@+# Question++Should a CLI tool that caches its parsed config in `/tmp` check the source file's+modification time before reusing the cache?diff --git a/ai-docs/benches/advisor-subscription/testdata/routed-codex/plan.md b/ai-docs/benches/advisor-subscription/testdata/routed-codex/plan.mdnew file mode 100644index 00000000..f52ec3b4--- /dev/null+++ b/ai-docs/benches/advisor-subscription/testdata/routed-codex/plan.md@@ -0,0 +1,7 @@+# Cache plan++Reference: heron-5521++We want to cache the parsed config of a CLI tool between runs. The plan is to write the+parsed JSON to `/tmp/cli-config-cache.json` and reuse it whenever the file exists, without+checking whether the source config changed since the cache was written.