Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
docs(reports): record the advisor subscription-routing build and bench
Jack Rudenko77a999b77b2a
diff --git a/ai-docs/reports/advisor-subscription-routing-2026-10.md b/ai-docs/reports/advisor-subscription-routing-2026-10.mdnew file mode 100644index 00000000..dd3774bd--- /dev/null+++ b/ai-docs/reports/advisor-subscription-routing-2026-10.md@@ -0,0 +1,88 @@+# `--advisor` on subscription-aware routing — build and validation report (2026-10-20 … 2026-10-22)++Branch `worktree-investgiage-gpt-file`, released as claudish 14.0.0. ROADMAP trigger 1 ("migrate all to+subscription", owner, 2026-10-20). Rationale lives in `ai-docs/architecture/advisor.md` (panel routing,+the advisor handler set, the Claude rule) and `adapters.md` (signing-value registry, call abort, client+cancel). This file records what was built, what was measured, and what is still open.++## What changed for users++- Every `--advisor` panel member and the collector go through the same routing chain a main model+ would (`planAdvisorModel` + `buildAdvisorChains`, before the port). A bare `gpt-*` or `cx@` member is+ served by the Codex subscription, `grok-*` / `gk@` by the Grok subscription, `kc@k3` by Kimi Coding,+ `ag@gemini-*` by Antigravity. The old metered bare-fetch routes (`advisorRouteFor`) are gone+ (`feat(advisor)!`, `cb8c6bba`).+- A Claude member or collector signs with the user's Claude Code login, only on a launch where claudish+ controls the child's auth (`childAuthControl`), and only to `api.anthropic.com`.+- The startup notice and the runtime read the same plan; a call served by a hop with different billing+ warns (`5621fd46`).+- Billing follows the arm that signed each request (`RequestAuth.arm`), including `openai-codex` and+ retry legs (`da614153`, `7f47a921`).+- Every credential a request signed with is redacted by value in every log sink, telemetry and every+ body sent to Claude Code (signing-value registry, leases released when the body ends).+- An aborted internal call stops its fetch, queue and retry waits and does not advance the chain; a+ client cancel is recorded as `cancelled`, not as a truncation.+- **A rejected subscription key moves the routing chain on to the next hop, metered included** (owner,+ 2026-10-21: an expired key must not block the chain; a configured metered key is consent), with a+ stderr notice naming the next hop and its billing (`9ace3fe5`, `2f282101`). A claudish connection+ fault still never advances onto a metered hop (`f8e09504`).+- The availability check resolves Antigravity ids the way the transport does, so `[email protected]`+ keeps its hop when the account lists only variants (`8452d275`).++## Measured++Live, subscriptions only, real Claude Code under madbench (`ai-docs/benches/advisor-subscription/`), on+the build merged with main 13.4.0 (run 4, 2026-10-22, one trial per Scenario):++| Scenario | Checks | Served by |+|---|---|---|+| routed-codex | 9/9 | `gpt-6.1-sol`, `[email protected]`: openai-codex, oauth, subscription; `kc@k3`: kimi-coding, subscription; main-session code mode resumed |+| routed-grok | 6/6 | `grok-4.7`, `[email protected]`: grok-subscription, subscription |+| claude-login | 7/7 | `sonnet` panel and `haiku` collector: native-anthropic, oauth, `claude-code-login`; 0 bearer values in the log |+| refusal | 3/3 | `ag@gemini-0-nonexistent` and `opusplan` refused with exit 1 before any `Calling API:` |+| routed-antigravity | 8/8 | `[email protected]`: antigravity, subscription; sent `gemini-3.8-flash-high` from a served list of four variants |++Hosts called across all five: `chatgpt.com/backend-api/codex`, `api.kimi.ai/coding/v1`,+`cli-chat-proxy.grok.com`, `api.anthropic.com/v1/messages`, `daily-cloudcode-pa.googleapis.com`. None to+generativelanguage.googleapis.com, api.openai.com, api.x.ai, OpenRouter or OpenCode Zen.++One earlier run (2026-10-21, `rejected-sub`) made one metered call: bare `gemini-4-argon` had no usable+subscription candidate on this machine (Antigravity did not serve it then; Zen Go was skipped as+`content-use opt-in id only`), so it was served by the Gemini API on the user's key, as the notice said+beforehand. The Scenario stays in the bench with a README warning; it is skipped in subscription-only runs.++Tests (all written blind by an external GPT writer through claudish on `cx@`, run by a separate runner):+114 tests for the core work (13/13 mutations red, negative control), plus 52 for fix round 4+(`fallback-handler.credential-advance`, `advisor-notice-text`, `model-availability.antigravity-resolve`),+every marked group mutation-proven. After the merge with main 13.4.0, 53 targeted test files: 704 + 121+child-process tests pass, 10 skip, 0 fail; typecheck and lint clean.++## Found on the way++- **The connection-fault 400 could advance onto a metered hop after D7** (blind writer T5, G2:+ `Expected [1, 0] Received [1, 1]`). `FallbackHandler.handle()` checked the D7 branch before the+ `x-claudish-connection-error` marker, which only `isRetryableError` read. Latent (today's producers put+ no upstream 401 in that body), fixed in `f8e09504`.+- **A log-flush race in a test**: the logger writes with async `appendFile`, so a log read right after a+ fake-timer flush saw 0 lines once in about 10 runs. The test now polls with a 2 s deadline.+- **Equivalent mutant**: removing the notice's own `redactForClient` call changes nothing observable,+ because `log` and `logStderr` redact signing values at the sink (`logger.ts`). Recorded, not a gap.+- **The Antigravity account's model list changes**: on 2026-10-21 it served only+ `gemini-3.8-flash-tiered`; on 2026-10-22 four variants and `gemini-4-argon-high/-medium/-low`. Two bench+ Checks that pinned ids broke; they now grade against the served list the transport logs.+- madbench 0.53.1 keeps only the last of several `--scenario` flags, silently; run one Scenario per+ invocation.+- The merge commit `982c1169` says "13.3.0" in its subject, but it contains main up to 13.4.0: another+ session's fetch moved the shared `origin/main` ref between the preflight and the merge.++## Open++- The `advisor_call` record's `routedModel` holds the planned wire id, not the id the Antigravity+ transport sent (`advisor.md` says so since `a6b27640`). Recording the sent id needs the transport to+ report it back.+- Code-mode replay after a proxy restart is abandoned when a hook repaired a nested tool call (from+ merging main 13.2.0's hooks layer); needs a hooks API change.+- From `model-failures-2026-10-21.md`: D1 (restart long-lived claudish MCP servers that hold a stale+ Grok token and old routing), D4, D5, D6 (preflight probes only hop 1), D9 (`go@` reported as missing+ credentials); renew the Zen Go key. D3 was fixed on main (`56df54b5`).+- A reasoning-only panel reply is still a stub (ROADMAP).diff --git a/ai-docs/reports/model-failures-2026-10-21.md b/ai-docs/reports/model-failures-2026-10-21.mdindex 807d8005..9286f964 100644--- a/ai-docs/reports/model-failures-2026-10-21.md+++ b/ai-docs/reports/model-failures-2026-10-21.md@@ -7,6 +7,17 @@ behind `origin/main` 13.2.0; the traced logic is the same on main, only line numre-run with a fresh process (`bun packages/cli/dist/index.js --probe …`). Owner review on the sameday reclassified three first-draft "account" rows as claudish behaviour; this version records that.+## Status (2026-10-22)++| Item | Status |+|---|---|+| D3 | fixed on main: `56df54b5 fix(muse): route bare muse-wonder-full to the muse@ subscription` (13.3.0) |+| D7 | fixed in 14.0.0: `9ace3fe5`, `2f282101`; the connection-fault exclusion `f8e09504` |+| D8 | fixed in 14.0.0: `8452d275`; verified live 2026-10-22 (`[email protected]` served by Antigravity, sent `gemini-3.8-flash-high`) |+| D1, D2, D4, D5, D6, D9 | open |++Details: `advisor-subscription-routing-2026-10.md`.+## Every failure| Model / spec | preflight answer | Cause | Item |