Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

docs: six validated tasks for models-index

Jack Rudenko7df52b8d775c
diff --git a/ai-docs/reports/BACKEND_TASKS_models_index-20260920.md b/ai-docs/reports/BACKEND_TASKS_models_index-20260920.md
new file mode 100644
index 00000000..77411df6
--- /dev/null
+++ b/ai-docs/reports/BACKEND_TASKS_models_index-20260920.md
@@ -0,0 +1,174 @@
+# Tasks for models-index, from claudish (2026-09-20)
+
+Every item below was measured against the **live** service, not a cache. Unless stated otherwise the
+generation is `g-20260919132927305-5a035947` (893 slim rows), read with
+`Accept: application/vnd.models-index.catalog+json;version=3`.
+
+Reproduce any line with:
+
+```
+BASE=https://us-central1-claudish-6da10.cloudfunctions.net
+ACCEPT='Accept: application/vnd.models-index.catalog+json;version=3'
+curl -sH "$ACCEPT" "$BASE/queryModels?status=all&catalog=slim&includeRouteVariants=true&limit=1000"
+curl -sH "$ACCEPT" "$BASE/queryModels?status=all&catalog=full&limit=5"
+curl -sH "$ACCEPT" "$BASE/queryPlans?limit=100"
+curl -sH "$ACCEPT" "$BASE/probeModels"
+```
+
+Context: claudish now reads contract 3, sends exact wire ids, and decides chat capability from
+`videoOutput`. The tasks below are the gaps that remain on the publishing side. None of them is
+urgent for correctness of what claudish already ships; each removes a guess claudish would otherwise
+have to make, and claudish will not work around any of them locally.
+
+---
+
+## T1. Publish input and output modalities on every projection
+
+**What we see.** The full projection's `capabilities` object carries, in this generation:
+`audioInput, audioOutput, batchApi, citations, codeExecution, fineTuning, imageOutput, jsonMode,
+pdfInput, promptCaching, streaming, structuredOutput, tools, videoInput, videoOutput, vision`.
+No row carries any key matching `/modal/i`, on any projection. The slim projection carries only
+`supportsVision`, `videoInput` and `videoOutput`.
+
+**Why it matters.** claudish must decide whether a model is a chat model, that is text in and text
+out. Today it infers that from `videoOutput` plus a name-shaped rule for rows the catalog does not
+cover. Both are inferences about something the catalog knows.
+
+**Asked for.** `inputModalities` and `outputModalities` per model, as explicit lists (for example
+`["text","image"]` and `["text"]`), on the slim and full projections alike, for every model
+including route variants.
+
+**Acceptance.** A slim row for a text model reports `outputModalities: ["text"]`; an image generator
+reports `["image"]`; a model that reads video but answers in text reports video in the input list and
+only text in the output list.
+
+---
+
+## T2. Rename the retired "roster" vocabulary on the wire
+
+**What we see.** `queryPlans` publishes `rosterRequirements[]` (with `rosterId`, `authority`,
+`requiredForActivation`, `maxAgeHours`) and `rosterCoverage` (`status`, `observedAt`, `expiresAt`),
+and these `routeReason` values: `authenticated_account_roster_required`,
+`authenticated_variant_roster_required`, `no_exact_callable_roster`,
+`authoritative_roster_identity_unresolved`. The slim projection carries the word inside collector
+ids, for example `"sourceCollectorId":"checked-roster:anthropic-claude-code-checked"`. `probeModels`
+is already clean.
+
+**Why it matters.** Both projects agreed one name per concept: a **dynamic models catalog** is the
+list one credential can see, and a **membership** is what a plan publishes. "Roster" was used for
+both, which is how the two got confused in the first place. claudish has removed the word from its
+own source (0 occurrences) and quotes these wire names verbatim until they change.
+
+**Asked for.**
+- `rosterCoverage` → `membershipCoverage`; `rosterRequirements` → `membershipRequirements`;
+ `rosterId` → `membershipId` (or `sourceId` where it names the collector, not a membership).
+- `authenticated_account_roster_required` → `dynamic_models_catalog_required` (agreed with Jack).
+- `authenticated_variant_roster_required`, `no_exact_callable_roster` and
+ `authoritative_roster_identity_unresolved` → the same vocabulary; propose names and we will follow.
+- Collector ids: rename at your convenience; they are opaque to claudish.
+
+**Acceptance.** `curl … | grep -i roster` returns nothing for `queryModels`, `queryPlans` and
+`probeModels`. Please tell us the generation in which the rename lands, so claudish can switch on the
+same day: claudish keeps no alias for a renamed value.
+
+---
+
+## T3. Publish pricing for metered route profiles
+
+**What we see.** 23 of 24 route profiles have mapped connections without `pricing`. Whole metered
+profiles have none at all:
+
+| Route profile | Priced / mapped |
+|---|---|
+| `qwen/dashscope-direct` | 0 / 141 |
+| `openai/direct-api` | 0 / 130 |
+| `google/direct-api` | 0 / 46 |
+| `fireworks/gateway` | 0 / 26 |
+| `mistralai/direct-api` | 0 / 20 |
+| `opencode/zen` | 61 / 78 |
+| `openrouter/gateway` | 401 / 526 |
+| `together-ai/gateway` | 243 / 244 |
+| `ollama/cloud` | 20 / 24 |
+
+Where pricing exists it is `{input, output, cachedRead}`, which is the shape claudish wants.
+
+**Why it matters.** Jack decided claudish's routing order on 2026-09-20: inside one tier, the model's
+own vendor first, then **cheapest by the catalog's price**, then the rest, with no local table and no
+local state. Without a price for a metered connection, claudish cannot order it and must fall back to
+"the rest", which puts an expensive gateway ahead of a cheap one by accident.
+
+**Asked for.** `pricing` on every **metered** mapped connection (`direct-api` and `gateway`
+profiles). Subscription profiles need none, and should carry none rather than a zero.
+
+**Acceptance.** Each metered profile above reports priced == mapped, or names the models it cannot
+price and why.
+
+---
+
+## T4. Probe picks must be models an ordinary account can call
+
+**What we see.** Two of the 24 picks fail that test.
+
+- `mistralai/direct-api` → `labs-leanstral-1-5`. Measured 2026-09-19 with a working Mistral key:
+ `403 "Model labs-leanstral-1-5 is a Labs model. To use Labs models, an admin must enable them in
+ your organization settings at https://admin.mistral.ai/plateforme/privacy."` The same account's
+ `/v1/models` returns 53 models, of which 51 are not Labs.
+- `qwen/dashscope-direct` → `qwen3.8-omni-flash`, an omni model, as the pick for a text chat probe.
+
+**Why it matters.** The pick is the first model Test All tries. A pick no ordinary account may call
+reads to the user as "this provider is broken". claudish now walks to the next candidate where the
+provider publishes a model list, but the first impression is still a failure, and a provider without
+a list has no second candidate.
+
+**Asked for.** Pick a generally available text chat model: exclude Labs, preview and allow-listed
+models, and prefer a plain text model over an omni, audio, image or realtime variant.
+
+**Acceptance.** Every pick in `probeModels` is a model whose access needs nothing beyond a valid key
+for that product, and whose output modality is text (see T1).
+
+---
+
+## T5. Split `ollama/cloud` into a subscription profile and a metered profile
+
+**What we see.** One profile, `ollama/cloud`. The plan `ollama-cloud` binds to it
+(`routeStatus: supported`), and 20 of its 24 mapped connections also carry per-token `pricing`.
+OpenCode is already split the way we mean: `opencode/go-subscription` and `opencode/zen`.
+
+**Why it matters.** Jack's rule of 2026-09-20: metered and subscription are always separate products
+in claudish, even when one key serves both, so that spending money is explicit and each lands in its
+own tier of the routing chain. With one profile, claudish cannot tell which one a request would bill.
+
+**Asked for.** Two profiles under `ollama`, following the OpenCode precedent: one for the Cloud
+subscription that the `ollama-cloud` plan binds to, and one metered. Tell us the profile ids; claudish
+adds the second provider only once they exist, and will not invent the identity.
+
+**Acceptance.** `queryPlans` binds `ollama-cloud` to the subscription profile, and the metered
+profile carries the pricing.
+
+---
+
+## T6. Poe and Vertex: `no_verified_probe_model`
+
+**What we see.** `poe/gateway` and `vertex/google-cloud` are the only two unavailable picks that are
+not `client_model_selection_required`. Poe answers normally with a key: `/v1/models` returns 341
+models and a chat request returns 200 (measured 2026-09-19).
+
+**Why it matters.** Low. claudish now discovers a Poe model from the account's own list and probes it
+(`gpt-5.4`, live). Vertex is moving to Google's Application Default Credentials on our side.
+
+**Asked for.** Either verify a pick for these two, or mark them `client_model_selection_required`,
+which is what they behave like. Today's reason says the backend intends to have a pick and does not,
+which reads as a temporary gap rather than a decision.
+
+**Acceptance.** Each of the two carries a verified pick, or the reason states that the client selects.
+
+---
+
+## Not a backend task, recorded to close the question
+
+**Alibaba PAYG "Model access denied" is not a catalog fault.** The catalog's `qwen/dashscope-direct`
+wire ids match the account's own list, spelled the same. The account's `GET
+/compatible-mode/v1/models` returns 200 with 169 models, and a chat request for an id taken from that
+list (`qwen-plus-character`) answers `403 Model.AccessDenied`, exactly as the catalog's ids do. The
+account's model permissions are the cause. claudish ships a script for this,
+`scripts/validate-dashscope-key.ts`, which separates "key rejected" from "access denied".