Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
docs: revision 3 of the remaining-work plan
Jack Rudenko1a6817cc45bc
diff --git a/ai-docs/reports/REMAINING_WORK_claudish-20260920.md b/ai-docs/reports/REMAINING_WORK_claudish-20260920.mdindex 9cf872b1..b26310e5 100644--- a/ai-docs/reports/REMAINING_WORK_claudish-20260920.md+++ b/ai-docs/reports/REMAINING_WORK_claudish-20260920.md@@ -1,200 +1,184 @@# Remaining work on the claudish side-**Revision 2, 2026-09-21.** Branch `fix/v3-reader-gaps`, 47 commits on top of PR #266. Nothing pushed,-CI has never run on them, nothing released. Full suite green (3,608 pass, 0 fail, 28 live tests-skipped). Test All: 24 providers ready.+**Revision 3, 2026-09-21.** Branch `fix/v3-reader-gaps`. Nothing pushed, CI has never run on this+branch, nothing released.-What changed since revision 1: models-index delivered modalities, the membership rename, route prices-and the Vertex reason on generation `g-20260920154425586-418ad3dd`, all validated live. That unblocks-Phase 1. Two items are still theirs: Poe's verified probe pick, and renaming the pricing-discriminator `shape` to `type`.+What changed since revision 2: the backend shipped BOTH remaining items — the pricing discriminator+is now `type` and Poe has a verified probe pick — so Phase 1 is finished, not merely unblocked. The+Vertex ADC refactor's first three items are done and live-verified. One defect was found that was+not in any plan: the catalog quotes a metered price of zero on 167 connections, which would have+made routing prefer them.Every decision below is Jack's unless marked **DISCUSS** (needs a decision) or **RESEARCH** (needs ameasurement). Sizes are agent-hours including live verification.----## Phase 1 — consume the catalog data the backend just shipped+## Done since revision 2-The data exists; claudish still guesses. This phase is small, independent, and removes guesses.+All verified against live data on generation `g-20260921062451697-f490edba`, not against fixtures.-### 1.1 Modalities replace the name-shaped guess (M, ~3h)+| Item | Commit | Evidence |+|---|---|---|+| 1.1 Modalities decide chat capability | `7fce457` | 543 models chat and 270 excluded by catalog evidence, 51 still unknown; 16 models that write text stopped being hidden by their names (`gpt-5-image`, `gemini-3.1-flash-image`, `qwen-audio-3.0-realtime-plus`) |+| 1.2 Comparable price per connection | `7fce457` | 923 of 1,734 mapped connections priced; `qwen3.7-flash` reads `$0.16/M` from its first tier, never the `0.553` mean |+| 1.3 `shape` → `type` | `7fce457` | Backend cut over; `shape` is gone from the wire entirely, so claudish reads `type` with no alias, as agreed |+| 1.4 Poe probe pick | verified, no code change | `poe/gateway` → `gemini-3.8-flash` reaches our probe path; `App.tsx:1342` already prefers the catalog pick over discovery |+| Zero-price defect (NEW) | `9bedcf7b` | A `flat`/`tiered` price summing to zero is unknown, not free; connections reading as free fell from 197 to exactly the 30 that declare it |+| 3.1–3.3 Vertex on ADC | `ae785275` | `buildsOwnEndpoint` keeps Vertex in the registry; project resolves from env → ADC `quota_project_id` → `gcloud config`; Express deleted across 17 references; live token obtained with no env var set |+| The routing gate | `182116eb` | `scripts/route-table-snapshot.ts`; baseline of all 1,123 models captured on the current generation |-**Today.** `classifyChatCapability()` in `providers/transport/probe-discovery.ts` decides "is this a-chat model" from the catalog's `videoOutput` flag, and then from a list of name patterns-(`\bimage\b`, `-tts`, `t2v`, …). The name rule is the last guess left in that path.--**Change.** Persist `inputModalities` and `outputModalities` through the reader and decide from them.--- `providers/all-models-cache.ts`: add both fields to `SlimModelEntry`, optional.-- `providers/catalog-client.ts`: carry them through the slim projection into the cache.-- `providers/transport/probe-discovery.ts`: a row whose `outputModalities` exists and lacks `"text"`- is not a chat model; a row that has them and includes `"text"` is; only a row with none of them- (absent, `null` or `[]`, all meaning unknown) falls through to `videoOutput` and then the name rule.-- Keep the name rule for models the catalog does not know at all (local servers, custom endpoints).--**Acceptance.** Against the live cache: the 211 rows with a non-text output are excluded by catalog-evidence, not by name; no row with `outputModalities: ["text"]` is excluded; the count of models the-picker offers changes only in the direction of removing non-chat models. Tests by Codex, including a-row whose name looks like an image model but whose catalog output is text.--### 1.2 Prices feed the routing order (M, ~3h; needed by Phase 2)--**Today.** Nothing reads `pricing`. Ordering inside a tier is arbitrary.--**Change.**-- `providers/all-models-cache.ts` and `catalog-client.ts`: persist the connection's `pricing` object- verbatim, including `shape`/`type`, `tiers`, `input`, `output`, `cachedRead`.-- New helper (`providers/connection-price.ts`): return one comparable number per connection, and a- label. Rules, from the agreement with the backend: `flat` uses `input`+`output`; `tiered` uses the- FIRST tier, which the catalog guarantees equals the top-level numbers; `free` is zero; `unavailable`- and a missing object are unknown and sort after every priced connection. Never average tiers, never- guess a tier from request size.--**Acceptance.** A unit test over the real fixture rows: 28 tiered connections rank by their first-tier; an `unavailable` connection never outranks a priced one. No behaviour change until Phase 2 uses-it.--### 1.3 Switch `shape` to `type` when the backend names the generation (S, ~30m)--One rename in the reader plus the fixture; no alias, same day as their cutover. Blocked on them.--### 1.4 Prefer Poe's verified pick when it appears (S, ~30m)--No code change is expected: the catalog pick already wins over discovery when present. Verify on the-generation that publishes it, and keep our account-list discovery as the fallback. Blocked on them.+Tests were authored by Codex with a mutation proof on every load-bearing case: each was shown to+fail with its fix reverted, and the implementation restored byte-for-byte afterwards.----## Phase 2 — the routing redesign (the large one)--**Decided.** One order: user rules → subscriptions (catalog membership) → dynamic subscriptions (the-account's own list) → native API → gateways → fallback. Inside a tier: the model's own vendor first,-then cheapest by catalog price, then the rest. No local preference list, no local state. Vendor-first-orders only WITHIN a tier. A spent limit is not remembered; the request moves to the next hop.+## What the gate measured, and why Phase 2 matters more than we thought-### 2.1 Candidate gathering (M, ~4h)+Today's routing sends **417 chat models to a final hop the catalog never mapped.** Only 492 of 1,123+models have an OpenRouter connection, yet 1,109 chains end at an OpenRouter-style hop, because the+hand-written table carries `"*": ["openrouter"]` (`default-routing-rules.ts:152`). For those models+the catalog already names a provider that does serve them: Poe 241, Together 163, Alibaba metered+100, OpenAI 74, Google 22.-New module, for example `providers/route-candidates.ts`, returning candidates with their tier,-provider, wire id and price. Sources, all already in the cache: plan membership-(`subscriptionPlanIds` plus plan `inclusions`), connections (`aggregators[]` with-`routeStatus: "mapped"`), and each dynamic subscription's own model list. No hand-written table.+`qwen3.8-max` is the worked example. It has four mapped connections, none of them OpenRouter, and+today's chain is `qtoken -> zengo -> qpay -> openrouter`, whose last hop cannot succeed. Fireworks,+which the catalog says serves it, is absent entirely.-### 2.2 Tier assignment and ordering (M, ~3h)+Two consequences for the design:-A `tier` on each provider definition: `subscription`, `dynamic-subscription`, `native`, `gateway`,-`fallback`. Sort by tier, then vendor-first (the catalog's `provider` field equals the candidate's-vendor), then by the price from 1.2, then by provider name for determinism.+1. The fallback MUST default to OpenRouter when `defaultProvider` is unset, or deleting the table+ removes the last hop from 1,109 models at once. Jack's decision already says the fallback exists,+ is optional and can be any aggregator; this makes its default explicit.+2. Rule 2.4 is real and now has a precise form. `providerServesModel("openrouter", "qwen3.8-max")`+ returns `unknown`, so the dead hop survives the availability filter. The asymmetry ("only a+ positive no removes a hop") is right for a subscription, whose membership we cannot enumerate,+ and wrong for a gateway whose complete connection list the backend publishes. The catalog itself+ distinguishes them: a route marked `client_model_selection_required` is account-selected, so+ absence stays unknown; any other route is backend-owned, so absence IS denial.-### 2.3 Credential filter and exact wire ids (done)--Already true on this branch: hops without a credential are dropped, and ids come from the connection.+----### 2.4 The fallback rule (S, ~1h)+## Phase 2 — the routing redesign-`defaultProvider` already accepts any provider and is skipped when empty, so "any aggregator, or none"-works today. What is missing: do not append the fallback when the catalog positively says that-aggregator does not serve the model. Then `qwen3.8-max` either reaches Fireworks or returns a clear-error instead of a hop that cannot succeed.+**Decided.** One order: user rules → subscriptions → dynamic subscriptions → native API → gateways →+fallback. Inside a tier: the model's own vendor first, then cheapest by catalog price, then the+larger context window, then provider name for determinism. No local preference list, no local state.-### 2.5 Delete the hand-written table (M, ~2h)+Two facts found today remove work from this phase:-Remove `providers/default-routing-rules.ts`, `buildCatalogRoutingRules`,-`retainKnownCatalogRoutingRules` and the four-way `mergeRoutingRules`. Only the user's own rules-remain, used verbatim. This is where the "exact-key beats glob" hazard disappears.+- **Subscriptions are already connections.** `aggregators[]` carries `moonshotai/kimi-code-subscription`,+ `openai/codex-subscription`, `z-ai/glm-coding-subscription`, `qwen/qwencloud-token-plan` and+ `opencode/go-subscription` beside the gateways. One source covers every tier; only the tier itself+ comes from claudish's provider table.+- **The vendor needs no local table.** `route.routeId === entry.provider` identifies the vendor's own+ route. True for 412 models; the other 709 are served only by gateways, which is a fact about those+ models rather than a gap.-### 2.6 No catalog means local only (S, ~1h)+### 2A Candidate gathering (in progress)+A `tier` on every provider definition plus one pure module `providers/route-candidates.ts`, gathering+from catalog connections and from namespace claims for the four dynamic subscriptions the catalog+cannot publish. Nothing is wired in; `scripts/route-candidates-preview.ts` compares what it WOULD do+against today's chain for all 1,123 models.-Without a readable catalog claudish serves local providers and explicit `provider@model` specs, and-never guesses a renamed id. A bare name returns an error naming the refresh command.+### 2B Wiring, and deleting the table (L, ~6h)+Replace the first step of `routeBare` — matching the merged table — with candidate gathering. The+credential filter and the availability filter after it are unchanged and must not be duplicated.+Delete `default-routing-rules.ts`, `buildCatalogRoutingRules`, `retainKnownCatalogRoutingRules` and+`mergeRoutingRules`; `loadRoutingRules()` then returns the user's own rules verbatim.-### 2.7 The gate (M, ~3h)+Known scope beyond routing: the TUI shows which user rules "override a default"+(`App.tsx:650-670`, `RoutingContent.tsx:382-385`). With no defaults left, that concept disappears+and those views need to show the effective chain instead.-A before-and-after route table over every model in the catalog, produced by a script and read by-hand. Every difference must be a deliberate addition, or a removal this design names. No route may-disappear silently. Then Test All and a real session on two subscriptions.+### 2C The gate (M, ~3h)+Capture after, diff against the committed baseline, review by hand. Every difference must be a+deliberate addition or a removal this design names. Then Test All, a cold-cache run, and a real+session on two subscriptions.-**DISCUSS before 2.2 lands:** what claudish should do when two candidates in one tier tie on price-and neither is the vendor (today: provider name, alphabetical).+**DISCUSS:** nothing outstanding. The tie-break inside a tier was decided on 2026-09-21: larger+context window, then provider name.----## Phase 3 — Vertex: refactor to Application Default Credentials--**Decided.** ADC is Google's way in. The Express API-key path is deleted, not kept.+## Phase 3 — Vertex, the rest-| # | Work | Files | Size |-|---|---|---|---|-| 3.1 | A provider declares that its transport builds its own endpoint, and stays in the registry; the excluded-provider error stops reading as a missing credential | `provider-definitions.ts`, `providers/remote-provider-registry.ts`, `proxy-server.ts` | S, ~1h |-| 3.2 | Resolve project and location from ADC `quota_project_id`, then `gcloud config get project`; `VERTEX_PROJECT`/`VERTEX_LOCATION` still win | `auth/vertex-auth.ts` | S, ~1h |-| 3.3 | Delete the Express path: `VERTEX_API_KEY`, the alias on `apiKeyEnvVar`, the `express` arm of `selectVertexAuthMode`, the express branch of `vertexProfile`, and the docs that describe it | `provider-definitions.ts`, `provider-profiles.ts`, `auth/vertex-auth.ts`, `docs/` | S, ~1h |-| 3.4 | Readiness and the Providers tab report an ADC credential, "needs a project", or "run `gcloud auth application-default login`" | `auth/credentials/*vertex*`, `tui/` | M, ~3h |-| 3.5 | Probe pick from the project's publisher models, ranked newest first | `transport/vertex-oauth.ts`, `providers/model-discovery.ts` | M, ~3h |-| 3.6 | Live validation and tests: a Gemini publisher model, an Anthropic publisher model, streaming, the 401 refresh | tests, `docs/` | M, ~3h |+| # | Work | Size |+|---|---|---|+| 3.4 | Readiness and the Providers tab. The credential authority is already correct via `resolveVertexConfig`; what remains is `--probe`'s provenance display, which still reports on `VERTEX_PROJECT` alone and so can call a working Vertex unconfigured (`api-key-map.ts:58`, consumed in `cli.ts`) | M, ~3h |+| 3.5 | Probe pick from the project's publisher models, newest first. The catalog will never supply one: Vertex is `client_model_selection_required` by agreement, which is correct | M, ~3h |+| 3.6 | Live validation and tests: a Gemini publisher model, an Anthropic publisher model, streaming, the 401 refresh | M, ~3h |-3.1 and 3.2 together turn Vertex from unreachable into working on ADC.--**RESEARCH.** Service-account credentials are a shape we have never exercised here; per-location model-availability means a pick verified on this project may 404 on another; Anthropic-on-Vertex uses-`rawPredict` with its own payload shape.+**RESEARCH.** Service-account credentials are a shape never exercised here; per-location model+availability means a pick verified on one project may 404 on another; Anthropic-on-Vertex uses+`rawPredict` with its own payload. Codex also reported four branches of project resolution that+cannot be tested without an injected ADC reader and command runner — worth adding those seams.---## Phase 4 — tests and debt-### 4.1 Tests for the eleven fixes of 2026-09-19/20 (M, ~3h, Codex writes)--The Gemini probe effort, the parameter-versus-model hint, Mistral's model list, Poe's pick and-transport method, the discovery retries, the 403 "access denied" hint and its classification, the-DeepSeek effort-versus-thinking conflict, the local 401 message, the newest-first ranking, and the-Alibaba PAYG effort enum. One brief, one Codex run, each test verified to fail when its fix is-reverted.+### 4.1 The eleven fixes of 2026-09-19/20 (M, ~3h, Codex writes)+Still outstanding. Today's Codex runs covered the NEW work only. The eleven are: the Gemini probe+effort, the parameter-versus-model hint, Mistral's model list, Poe's pick and transport method, the+discovery retries, the 403 access-denied hint and its classification, the DeepSeek+effort-versus-thinking conflict, the local 401 message, the newest-first ranking, and the Alibaba+metered effort enum.### 4.2 An incomplete list must stay visible (S, ~1h)+Unchanged from revision 2: return the incompleteness rather than only recording it.++### 4.3 Vocabulary (S, ~2h) — sequence AFTER 2B+"served set" has 24 sites, and `routing-rules.ts` and `default-routing-rules.ts` hold most of them.+The second file is deleted by 2B, so renaming words in it now is wasted work.-**Decided:** return the incompleteness, do not only record it. Add a detailed call-(`{ models, incomplete?: { reason, droppedRows?, hasMore? } }`) with the array function kept as a thin-wrapper; `providerServesModel` refuses to deny while the marker is set; the picker keeps the rows and-can surface the state later.+### 4.4 Two small defects (S, ~1h)+Redact `readinessDetail` before anything displays it. The `[object Object]` probe error was hunted+today and is NOT in `probe-discovery.ts`, `probe-catalog.ts` or `probe-live.ts` — every failure path+there builds a string. It needs its reproduction (the Mistral candidate walk) to locate.-### 4.3 Vocabulary (S, ~2h)+----Retire "served set" in the code (33 uses) and "roster" in `ai-docs/architecture` (about 13 files) and-`docs/settings-reference.md:747`. Extend `scripts/no-retired-terms.test.ts` to both words afterwards.+## Phase 5 — release 9.8.0-### 4.4 Two small defects (S, ~1h)+Unchanged: finish the phases, suite green, Test All re-run, push onto PR #266 so CI runs for the+first time, bump both manifests, regenerate `version.ts`, tag the merge commit by explicit ref, watch+the release workflow, verify the published version. Then delete the `QWEN_CLOUD_PLAN_API_KEY`+Keychain item, remove the 5 duplicate lines in `.env`, and remove the four `worktree-agent-*`+worktrees.-Redact `readinessDetail` before anything displays it (nothing does today). Find where a probe error-rendered `[object Object]`, seen on the Mistral candidate walk.+**Before the worktree is reaped:** `ai-docs/sessions/dev-feature-catalog-phase1-20260921-0015/`+holds the implementation notes and probe scripts from today and is gitignored. Anything durable in+it must be promoted to `ai-docs/` or `docs/` first. The measurements themselves are already in this+file and in the commit messages.----## Phase 5 — release 9.8.0+## Outstanding with the backend++Written up in `ai-docs/reports/BACKEND_zero_prices_models_index-20260921.md`:-1. Finish Phases 1 to 4, suite green, Test All re-run with all keys.-2. Push `fix/v3-reader-gaps` onto PR #266's branch as a fast-forward, let CI run for the first time.-3. Bump `package.json` and `packages/cli/package.json`, regenerate `version.ts`, merge, tag the merge- commit with an explicit ref, watch the release workflow, verify the published version.-4. After the release: delete the `QWEN_CLOUD_PLAN_API_KEY` Keychain item, remove the 5 duplicate lines- in `/Users/jack/mag/claudish/.env`, and remove the four `worktree-agent-*` worktrees.+1. 167 connections quote `type: "flat"` with `input: 0, output: 0`; 163 are `together-ai/gateway` on+ models Together AI charges for. Zero is the strongest possible price, so a missing measurement+ PROMOTES those connections. Please publish `unavailable` instead. claudish's workaround is meant+ to be deleted, not kept.+2. `opencode/systemone` (2 connections) resolves to no claudish provider — is it a new profile we+ should map? The other two unmapped routes need nothing: `qwen/realtime-websocket` is a WebSocket+ API with no chat transport here, and `openrouter/decisions` publishes `["decisions"]` output.---## Deferred, with the reason-- **Alibaba PAYG** until after the v3 release. Nine of nine request shapes and models are denied on- `dashscope-intl` while the same key lists 169 models; the account's model access is the cause, not- claudish and not the catalog.-- **The public-list rule (old group D).** Catalog membership already gates the Coding Plan, and the- public list's denial is what keeps an unlisted id away from a `400 Model not exist` that does not- advance the chain.+- **Alibaba metered** until after the v3 release. Nine of nine request shapes and models are denied+ on `dashscope-intl` while the same key lists 169 models; the account's model access is the cause.+- **The public-list rule (old group D).** Catalog membership already gates the Coding Plan.- **Alibaba Coding Plan** while the stored key is the Token Plan key.---## Suggested order-Phase 1.1 and 1.2 first: they are small, they remove guesses, and 1.2 is a prerequisite for the-ordering in Phase 2. Then Phase 3.1 to 3.3, which is two or three hours and makes Vertex usable. Then-Phase 2, the largest piece, with its gate. Phase 4 can interleave whenever a Codex run is free. Phase-5 last. Total, excluding anything blocked on the backend: roughly 30 to 35 agent-hours.+2A's preview, reviewed by hand, then 2B behind its gate, then 4.3 (whose targets 2B rewrites), then+Vertex 3.4 to 3.6, then 4.1, 4.2 and 4.4, then the release. Roughly 20 to 25 agent-hours remain,+down from 30 to 35 at revision 2.