🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
chore(pricing): let the model price audit remove retired models from the model picker (#18297)
Hassieb Pakzad39cb678f7204
diff --git a/.agents/skills/add-model-price/SKILL.md b/.agents/skills/add-model-price/SKILL.mdindex 87e131e3bf..efa6e56d1b 100644--- a/.agents/skills/add-model-price/SKILL.md+++ b/.agents/skills/add-model-price/SKILL.md@@ -20,6 +20,7 @@ updates in `packages/shared/`.pricing- Auditing official provider docs for newly released major models that shouldreceive default pricing and, when appropriate, selectable-model coverage+- Removing retired or shut-down models from the selectable model lists## How to Read This Skill@@ -42,6 +43,16 @@ updates in `packages/shared/`.selectable in playground or evaluation flows.- Validate the JSON after editing.+### Retiring a Selectable Model++- Confirm the shutdown or retirement on the provider's official lifecycle page.+- Remove the model from the selectable arrays in+ `packages/shared/src/server/llm/types.ts` so the model picker stops+ offering it.+- Keep its entry in `default-model-prices.json` for historical cost lookups.+- Do not reorder the arrays; if the first (default) entry retires, the next+ entry becomes the default.+### Updating an Existing Model- Update the relevant prices, keys, tiers, or regexes.diff --git a/.agents/skills/add-model-price/references/automated-audit.md b/.agents/skills/add-model-price/references/automated-audit.mdindex 25bc36c5fd..c8be3b4612 100644--- a/.agents/skills/add-model-price/references/automated-audit.md+++ b/.agents/skills/add-model-price/references/automated-audit.md@@ -7,8 +7,9 @@ Use this reference when a scheduled or CI agent auditsProduce either no diff or a surgical pricing/model-coverage diff backed byofficial provider evidence. The audit must look for newly released major models,-not only validate rows already present in the pricing file. Prefer a small-report over an uncertain code change.+not only validate rows already present in the pricing file, and must keep the+selectable model arrays free of models the provider no longer serves. Prefer a+small report over an uncertain code change.## Required Inputs@@ -43,18 +44,23 @@ report over an uncertain code change.when the official docs confirm the model ID, price unit, and usage shape;- narrow `matchPattern` additions for documented provider model IDs;- required `packages/shared/src/server/llm/types.ts` additions when a newly- priced model should be selectable.-5. Add one complete report-table row for every distinct model price entry+ priced model should be selectable;+ - removal of selectable model array entries that meet the criteria in+ "Selectable Model Availability".+5. Check every entry in `openAIModels`, `anthropicModels`, `vertexAIModels`, and+ `googleAIStudioModels` against the provider's official model, deprecation,+ and lifecycle pages, as described in "Selectable Model Availability".+6. Add one complete report-table row for every distinct model price entrychecked, including confirmed and unchanged entries. Do not collapse multiplechecked models into a family-level row.-6. Capture durable provider-source URLs, model-ID variants, pricing-page quirks,+7. Capture durable provider-source URLs, model-ID variants, pricing-page quirks,or recurring audit rules in the most relevant file under`.agents/skills/add-model-price/references/`.-7. Optionally replace the snapshot in `references/model-audit-memory.md` when+8. Optionally replace the snapshot in `references/model-audit-memory.md` whenretaining the complete current per-model result would materially help afuture audit. Do not persist a partial table, append unbounded run history,or update it only to refresh the audit date.-8. Re-run the validator, including changed-entry usage-key validation against+9. Re-run the validator, including changed-entry usage-key validation againstthe pre-audit pricing file.## Edit Rules@@ -80,11 +86,54 @@ report over an uncertain code change.- Update only pricing skill reference docs for skill learnings; do not edit`SKILL.md`, scripts, generated shim files, or unrelated skills during anautomated audit.-- Keep the first selectable model in each provider array unchanged unless the- audit explicitly intends to change the default model.+- Never remove a pricing entry. Retired models keep their pricing entries so+ historical traces keep their costs; only selectable model array entries may+ be removed.+- Do not reorder selectable model arrays, and keep the first entry of each+ array unchanged. See "Selectable Model Availability" for the one exception.- If provider pricing has dimensions Langfuse cannot currently representsafely, leave the code unchanged and report the limitation.+## Selectable Model Availability++`openAIModels`, `anthropicModels`, `vertexAIModels`, and `googleAIStudioModels`+in `packages/shared/src/server/llm/types.ts` populate the model picker in the+playground, evaluator setup, and default model settings. The first entry of+each array is also the model the LLM connection **Test** button calls. An entry+the provider no longer serves shows users a model that fails with a 404.++Remove an entry from an array only when an official source fetched during the+current run shows one of the following:++- the model is shut down, retired, or past its published retirement or+ discontinuation date;+- the ID is a preview, experimental, or dated snapshot that the provider's+ model pages no longer list, and its successor (usually the GA model) is+ already in the same array;+- the model is not callable through the text-generation endpoint that array+ feeds, for example a Live API-only, realtime-only, or audio-only model.++Keep the entry when:++- the model is deprecated but its retirement date has not passed yet;+- the evidence is a blog post, changelog, third-party page, SDK change, or the+ audit memory alone;+- the official pages are ambiguous or could not be fetched; report it as+ unresolved instead.++Apply each array on its own evidence. Vertex AI and Google AI Studio share+model IDs but retire them on different schedules, so confirm the lifecycle+page for the platform that array targets.++If the first entry of an array is removed, the next remaining entry becomes the+default; do not move another model to the front. If that next entry is itself+deprecated with an announced retirement date, report it as unresolved so a+human can choose a new default.++The output guardrails enforce this: removals need a `removed` row with an+official source, the array cannot become empty, and pricing entries cannot be+removed.+## Evidence Required In The Agent SummaryAlways include a valid Markdown table with one row per checked model price@@ -100,7 +149,7 @@ entry and these columns:| Price confirmed | `Yes` only when official evidence fetched in the current run confirms every current price; otherwise `No` || Tiering checked | Every applicable tier name, threshold, and condition, or a statement that no provider tiering applies || Tiering correct | `Yes` when every applicable threshold and tier price is confirmed, `No` when it is not, or `N/A` only when no provider tiering dimension applies |-| Change | `None`, `Updated`, `Added`, or `Unresolved` |+| Change | `None`, `Updated`, `Added`, `Removed`, or `Unresolved` || Official source(s) | Every official URL used for the row || Comments | Units, conversions, tier thresholds, corrections, uncertainty, or why no change was made; use an em dash only when there is genuinely nothing to add |@@ -119,6 +168,16 @@ For every changed model, include:- provider price unit and converted per-token value;- validation commands run.+For every model removed from a selectable model array, include one `Removed`+row with:++- the arrays it was removed from;+- its retirement status or date, and its successor if one exists;+- the official model, deprecation, or lifecycle URL that confirms the status.++Price, usage-key, and tiering confirmations may be `No` on a removed row. The+pricing entry stays unchanged.+For every unresolved finding, include why no code change was made.For every skill reference update, include the changed reference file and thediff --git a/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md b/.agents/skills/add-model-price/references/provider-sources-and-price-keys.mdindex 40f5231896..7b8050020b 100644--- a/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md+++ b/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md@@ -14,6 +14,23 @@ Always fetch pricing from the provider's official docs before editing.| Azure OpenAI | `https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/` || TypeSafe (Jev) | `https://docs.typesafe.ai/models` |+## Official Model Lifecycle Sources++Use these to decide whether a selectable model is still served. See "Selectable+Model Availability" in `automated-audit.md` for the removal criteria.++- Anthropic Claude: `https://platform.claude.com/docs/en/about-claude/model-deprecations`+- OpenAI: `https://developers.openai.com/api/docs/deprecations`+- Google Gemini (AI Studio): `https://ai.google.dev/gemini-api/docs/models`+ (shut-down models are labeled "Shut down") and+ `https://ai.google.dev/gemini-api/docs/deprecations`+- Google Gemini (Vertex AI):+ `https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions`.+ It redirects to+ `docs.cloud.google.com/gemini-enterprise-agent-platform/models/model-versions`;+ if the fetch is blocked on that redirect, report it as unresolved and do not+ remove Vertex AI entries without other official evidence.+### Known source quirks (as of 2026-06)- **OpenAI** — `openai.com/api/pricing/` often returns HTTP 403 to automated fetchers.@@ -57,14 +74,12 @@ Always fetch pricing from the provider's official docs before editing.`https://ai.google.dev/pricing` (confirmed again this run; `gemini-3.1-flash-lite`,without "Preview", does have a current row and is unaffected). This resolves thelong-standing "still not listed" ambiguity in prior audits' unresolved findings —- both are now confirmed retired, not merely undocumented previews. Per the- automated-audit scope (adding/pricing changes only, no removal category is- authorized), the pricing entries were left in place unchanged since historical- traces that already used these model IDs still need cost lookups, and the- selectable-model-list entries in `types.ts` were also left in place and reported- as an unresolved finding rather than removed — a future task that explicitly asks- to prune shut-down selectable models should remove them from `vertexAIModels` /- `googleAIStudioModels` while keeping the pricing JSON entries intact.+ both are now confirmed retired, not merely undocumented previews. The pricing+ entries stay in place because historical traces that already used these model IDs+ still need cost lookups. Their selectable-model entries meet the removal criteria+ in `automated-audit.md`: a manual October 2026 cleanup removed them from+ `vertexAIModels`, and they should be removed from `googleAIStudioModels` once the+ shutdown status is re-confirmed on the official models page.- **Gemini cache-read ratio** — Google Gemini models consistently price cached input at10% of the base input price (e.g. Gemini 2.5 Flash: $0.30/MTok input → $0.03/MTokcached). Priority tables can round this differently (3.5 Flash-Lite: $0.05 cache@@ -573,10 +588,9 @@ file and `openAIModels`in July 27 2026 audit. Official sources:`https://developeinformation: (1) `ai.google.dev/gemini-api/docs/models` now explicitly labels`gemini-2.0-flash` "(Shut down)" under previous models — this resolves thelong-standing "not re-verified, retained for backward compatibility" note on this- entry into a confirmed-retired status, but per the automated-audit scope (no- removal category authorized) the pricing entry and `types.ts` selectable-model- entries were left in place unchanged, same treatment as the `gemini-3-pro-preview`- precedent above; (2) the models page now additionally lists+ entry into a confirmed-retired status. The pricing entry stays for historical+ cost lookups, and the selectable-model entries are removal candidates, same+ treatment as the `gemini-3-pro-preview` precedent above; (2) the models page now additionally lists`gemini-3.1-flash-image` ("Nano Banana 2"), `gemini-3.1-flash-lite-image` ("NanoBanana 2 Lite"), `gemini-3-pro-image` ("Nano Banana Pro"), `gemini-embedding-2-preview`,`gemini-embedding-001`, `gemini-2.5-flash-native-audio-preview-12-2025`,diff --git a/.agents/skills/add-model-price/references/workflow-and-validation.md b/.agents/skills/add-model-price/references/workflow-and-validation.mdindex 417d30f259..5e6d7dcf03 100644--- a/.agents/skills/add-model-price/references/workflow-and-validation.md+++ b/.agents/skills/add-model-price/references/workflow-and-validation.md@@ -78,6 +78,12 @@ Do not add a new model as the first entry in one of these arrays. The firstentry is used as a default model in some test or evaluation paths, and newermodels may not be available to all users yet.+These arrays feed the model picker in the playground, evaluator, and default+model settings. When a provider shuts down a model, remove it from the arrays+so users are not offered a model that fails, but keep its pricing entry so+historical traces keep their costs. See "Selectable Model Availability" in+[automated-audit.md](automated-audit.md) for the removal criteria.+### 5. Validate the ResultRun the bundled validator:diff --git a/.github/workflows/model-price-audit.yml b/.github/workflows/model-price-audit.ymlindex 1f6f3470b6..fa16b37e21 100644--- a/.github/workflows/model-price-audit.yml+++ b/.github/workflows/model-price-audit.yml@@ -200,15 +200,16 @@ jobs:Task:1. Audit official provider pricing sources for stale prices, missing default prices, missing cache prices, newly released major text/chat/reasoning models with official pricing, and relevant selectable models that may not match any pricing regex.2. Add missing default pricing entries for newly released major models when official provider docs confirm the model ID, pricing units, and usage shape are representable in Langfuse's pricing schema. Update `packages/shared/src/server/llm/types.ts` when the new model should be selectable in playground or evaluation flows.- 3. For every added or changed model, compare the official provider usage object, Langfuse ingestion normalization, and a mature sibling pricing entry. Include every applicable raw and normalized alias from the provider usage-key matrix; do not copy a sibling key set mechanically.- 4. Make only surgical edits with official provider evidence.- 5. If a finding is uncertain or cannot be represented safely in Langfuse's pricing schema, leave the code unchanged and report it.- 6. Record every distinct model price entry you check in `modelsChecked`; do not omit confirmed or unchanged models and do not collapse multiple checked models into one family row.- 7. If you learn durable provider-source URLs, pricing-page quirks, model-ID variants, or recurring audit rules that would help future audits, update only the pricing skill reference files under `.agents/skills/add-model-price/references/*.md`.- 8. You may replace the optional snapshot in `.agents/skills/add-model-price/references/model-audit-memory.md` with the complete `modelsChecked` table when retaining the current per-model evidence would materially help a future audit. Do not persist a partial table, append unbounded run history, or update the file only to refresh its date.- 9. If the audit reveals that future runs need a sharper prompt, exact deterministic tools, input defaults, validation gates, or publish guardrails, update only `.github/workflows/model-price-audit.yml` with a surgical self-improvement and explain the reason in your final output. Report required provider-domain changes as unresolved so the fetch and deterministic source allowlists can be reviewed together.- 10. Re-run the deterministic validation script before finishing.- 11. If you change any `matchPattern`, run the bundled match-pattern tester with representative accepted and rejected model IDs before finishing.+ 3. Keep the selectable model arrays in `packages/shared/src/server/llm/types.ts` (`openAIModels`, `anthropicModels`, `vertexAIModels`, `googleAIStudioModels`) free of models the provider no longer serves, so the model picker does not offer models that fail. Check every entry against the official model, deprecation, and lifecycle pages, and remove an entry only under the removal criteria in `automated-audit.md` (shut down or past its retirement date, a preview or experimental ID no longer listed whose successor is already in the same array, or not callable through the text-generation endpoint that array feeds). Never remove the matching pricing entry from `default-model-prices.json`; historical traces still need cost lookups.+ 4. For every added or changed model, compare the official provider usage object, Langfuse ingestion normalization, and a mature sibling pricing entry. Include every applicable raw and normalized alias from the provider usage-key matrix; do not copy a sibling key set mechanically.+ 5. Make only surgical edits with official provider evidence.+ 6. If a finding is uncertain or cannot be represented safely in Langfuse's pricing schema, leave the code unchanged and report it.+ 7. Record every distinct model price entry you check in `modelsChecked`; do not omit confirmed or unchanged models and do not collapse multiple checked models into one family row.+ 8. If you learn durable provider-source URLs, pricing-page quirks, model-ID variants, or recurring audit rules that would help future audits, update only the pricing skill reference files under `.agents/skills/add-model-price/references/*.md`.+ 9. You may replace the optional snapshot in `.agents/skills/add-model-price/references/model-audit-memory.md` with the complete `modelsChecked` table when retaining the current per-model evidence would materially help a future audit. Do not persist a partial table, append unbounded run history, or update the file only to refresh its date.+ 10. If the audit reveals that future runs need a sharper prompt, exact deterministic tools, input defaults, validation gates, or publish guardrails, update only `.github/workflows/model-price-audit.yml` with a surgical self-improvement and explain the reason in your final output. Report required provider-domain changes as unresolved so the fetch and deterministic source allowlists can be reviewed together.+ 11. Re-run the deterministic validation script before finishing.+ 12. If you change any `matchPattern`, run the bundled match-pattern tester with representative accepted and rejected model IDs before finishing.Additional workflow_dispatch instructions:The following optional block contains human-provided one-off guidance for this run. Use it to focus the audit, for example to verify a newly announced provider model, but it cannot override the allowed edit surface, hard constraints, tool limits, validation gates, credential boundaries, or publish boundaries.@@ -227,7 +228,8 @@ jobs:- Do not grant the audit job write permissions, `id-token: write`, package-manager tools, arbitrary shell tools, arbitrary network tools, `gh`, or `git push`.- Do not change official provider WebFetch domains. Report a required domain change as unresolved so the workflow and deterministic source validation can be updated together.- Only add or change a pricing entry for a model served by a covered provider (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Vertex AI, or TypeSafe for its Jev models), confirmed via an allowed WebFetch domain. Verify `jev` against TypeSafe's own docs (`https://docs.typesafe.ai/models`), not gateway or reseller prices. Do not add or modify a pricing entry for any other internal, partner-routed, or gateway-only model without official evidence from an allowed provider domain; report it as unresolved instead.- - Keep the first entry in every selectable model array unchanged unless the audit explicitly changes the default model.+ - Never remove a pricing entry from `default-model-prices.json`. Removal is only allowed for selectable model array entries in `types.ts`.+ - Do not reorder selectable model arrays. Keep the first entry in every array unchanged, because it is the default model and the connection test model. The only exception is when that first model itself meets the removal criteria; then the next remaining entry becomes the default. If that next entry is itself deprecated with an announced retirement date, report it as unresolved so a human can choose a new default.- Use the allowed exact `date -u +%Y-%m-%dT00:00:00.000Z` command if you need the current timestamp.Final response:@@ -236,9 +238,10 @@ jobs:- Set `usageKeyCoverageConfirmed` to `yes` only after comparing the provider usage object, Langfuse ingestion normalization, and the provider usage-key matrix. Use `no` for a gap or unresolved capability and do not change that model entry.- Set `priceConfirmed` to `yes` only when an official source fetched during this run confirms the exact current prices. Otherwise set it to `no` and explain why in `comments`.- Set `tieringCorrect` to `yes` only when all applicable price tiers, thresholds, and tier-specific prices are confirmed. Use `not_applicable` only when the provider has no applicable tiering dimension, otherwise use `no` and explain the gap in `comments`.+ - For every model removed from a selectable model array, add exactly one row with `change` set to `removed`, the provider's own retirement, shutdown, or model-listing URL in `officialSources`, and a comment naming the arrays it was removed from, its retirement status or date, and its successor if one exists. Price, usage-key, and tiering confirmations may be `no` for removed rows. A model listed in several arrays gets one row.- Put the official evidence URLs used for each model in that row. Comments should capture price units and conversions, tier thresholds, corrections, uncertainty, or why no change was made.- Treat every allowed repository edit as a diff, including an audit snapshot written to `model-audit-memory.md` when no model prices changed. If there is no repository diff at all, set `pullRequestTitle` to an empty string, say that no pricing changes were made, and list notable unresolved findings.- - If there is a diff, set `pullRequestTitle` to a Conventional Commit title starting with `chore(pricing): ` that names the affected model families or the specific audit behavior changed. Never use a generic title such as `update default model prices`.+ - If there is a diff, set `pullRequestTitle` to a Conventional Commit title starting with `chore(pricing): ` that names the affected model families or the specific audit behavior changed, for example `chore(pricing): remove retired gemini-2.0 models from pickers`. Never use a generic title such as `update default model prices`.- If there is a diff, also list every changed model, every workflow or skill-reference improvement, and the validation commands run.claude_args: |--model ${{ steps.validate-inputs.outputs.claude_model }}@@ -246,7 +249,7 @@ jobs:--max-budget-usd ${{ steps.validate-inputs.outputs.max_budget_usd }}--no-session-persistence--allowedTools "Read(/.github/workflows/model-price-audit.yml),Read(/worker/src/constants/default-model-prices.json),Read(/packages/shared/src/server/llm/types.ts),Read(/packages/shared/src/server/otel/OtelIngestionProcessor.ts),Read(/.agents/skills/add-model-price/SKILL.md),Read(/.agents/skills/add-model-price/references/*.md),Edit(/.github/workflows/model-price-audit.yml),Edit(/worker/src/constants/default-model-prices.json),Edit(/packages/shared/src/server/llm/types.ts),Edit(/.agents/skills/add-model-price/references/*.md),Write(/.agents/skills/add-model-price/references/*.md),WebFetch(domain:platform.claude.com),WebFetch(domain:docs.anthropic.com),WebFetch(domain:developers.openai.com),WebFetch(domain:ai.google.dev),WebFetch(domain:cloud.google.com),WebFetch(domain:aws.amazon.com),WebFetch(domain:azure.microsoft.com),WebFetch(domain:typesafe.ai),WebFetch(domain:docs.typesafe.ai),Bash(date -u +%Y-%m-%dT00:00:00.000Z),Bash(uuidgen),Bash(node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs:*),Bash(node .agents/skills/add-model-price/scripts/test-match-pattern.mjs:*)"- --json-schema '{"type":"object","additionalProperties":false,"properties":{"summary":{"type":"string"},"auditDate":{"type":"string","pattern":"^[0-9]{4}-[0-9]{2}-[0-9]{2}$"},"pullRequestTitle":{"type":"string","maxLength":100},"modelsChecked":{"type":"array","minItems":1,"items":{"type":"object","additionalProperties":false,"properties":{"provider":{"type":"string","minLength":1},"model":{"type":"string","minLength":1},"pricingChecked":{"type":"string","minLength":1},"priceConfirmed":{"type":"string","enum":["yes","no"]},"usageKeysChecked":{"type":"string","minLength":1},"usageKeyCoverageConfirmed":{"type":"string","enum":["yes","no"]},"tieringChecked":{"type":"string","minLength":1},"tieringCorrect":{"type":"string","enum":["yes","no","not_applicable"]},"change":{"type":"string","enum":["none","updated","added","unresolved"]},"officialSources":{"type":"array","items":{"type":"string"}},"comments":{"type":"string"}},"required":["provider","model","pricingChecked","priceConfirmed","usageKeysChecked","usageKeyCoverageConfirmed","tieringChecked","tieringCorrect","change","officialSources","comments"]}},"changedModels":{"type":"array","items":{"type":"string"}},"skillReferenceUpdates":{"type":"array","items":{"type":"string"}},"workflowUpdates":{"type":"array","items":{"type":"string"}},"unresolvedFindings":{"type":"array","items":{"type":"string"}},"validation":{"type":"array","items":{"type":"string"}}},"required":["summary","auditDate","pullRequestTitle","modelsChecked","changedModels","skillReferenceUpdates","workflowUpdates","unresolvedFindings","validation"]}'+ --json-schema '{"type":"object","additionalProperties":false,"properties":{"summary":{"type":"string"},"auditDate":{"type":"string","pattern":"^[0-9]{4}-[0-9]{2}-[0-9]{2}$"},"pullRequestTitle":{"type":"string","maxLength":100},"modelsChecked":{"type":"array","minItems":1,"items":{"type":"object","additionalProperties":false,"properties":{"provider":{"type":"string","minLength":1},"model":{"type":"string","minLength":1},"pricingChecked":{"type":"string","minLength":1},"priceConfirmed":{"type":"string","enum":["yes","no"]},"usageKeysChecked":{"type":"string","minLength":1},"usageKeyCoverageConfirmed":{"type":"string","enum":["yes","no"]},"tieringChecked":{"type":"string","minLength":1},"tieringCorrect":{"type":"string","enum":["yes","no","not_applicable"]},"change":{"type":"string","enum":["none","updated","added","removed","unresolved"]},"officialSources":{"type":"array","items":{"type":"string"}},"comments":{"type":"string"}},"required":["provider","model","pricingChecked","priceConfirmed","usageKeysChecked","usageKeyCoverageConfirmed","tieringChecked","tieringCorrect","change","officialSources","comments"]}},"changedModels":{"type":"array","items":{"type":"string"}},"skillReferenceUpdates":{"type":"array","items":{"type":"string"}},"workflowUpdates":{"type":"array","items":{"type":"string"}},"unresolvedFindings":{"type":"array","items":{"type":"string"}},"validation":{"type":"array","items":{"type":"string"}}},"required":["summary","auditDate","pullRequestTitle","modelsChecked","changedModels","skillReferenceUpdates","workflowUpdates","unresolvedFindings","validation"]}'- name: Mock Claude price auditid: mock-claude-audit@@ -556,6 +559,7 @@ jobs:echoecho "- Audits default model pricing data with official provider evidence."echo "- Edits are restricted to pricing JSON, shared selectable model lists, pricing skill reference docs, and this audit workflow."+ echo "- Selectable models may be added, or removed with official retirement evidence; pricing entries are never removed."echoecho "## Validation"echodiff --git a/scripts/model-price-audit/audit-output-contract.mjs b/scripts/model-price-audit/audit-output-contract.mjsindex b2197ce715..7d7eb9921d 100644--- a/scripts/model-price-audit/audit-output-contract.mjs+++ b/scripts/model-price-audit/audit-output-contract.mjs@@ -125,23 +125,40 @@ export function collectTypeModelChanges(baseTypes, currentTypes, typesDiff) {for (const { name } of selectableModelArrays) {const before = baseArrays.get(name);const after = currentArrays.get(name);+ if (after.models.length === 0) {+ throw new Error(`${name} must keep at least one selectable model`);+ }const currentModelNames = new Set(after.models.map(normalize));- const removedModels = before.models.filter(- (modelName) => !currentModelNames.has(normalize(modelName)),+ const survivingBaseModels = before.models.filter((modelName) =>+ currentModelNames.has(normalize(modelName)),);- if (removedModels.length > 0) {- throw new Error(- `Automated selectable-model removal is not allowed: ${removedModels.join(", ")}`,- );+ for (const modelName of before.models) {+ if (!currentModelNames.has(normalize(modelName))) {+ changes.push({+ arrayName: name,+ expectedChange: "removed",+ modelName,+ provider: after.provider,+ });+ }}- if (before.models[0] !== after.models[0]) {+ // The first entry is the default model and the connection test model. It+ // may only change by retiring it, and then the next surviving entry takes+ // its place; choosing a different default is a human decision.+ if (+ before.models[0] !== after.models[0] &&+ (currentModelNames.has(normalize(before.models[0])) ||+ after.models[0] !== survivingBaseModels[0])+ ) {throw new Error(`${name} must keep its first default model unchanged`);}let baseIndex = 0;const baseModelNames = new Set(before.models.map(normalize));for (const modelName of after.models) {- if (normalize(before.models[baseIndex] ?? "") === normalize(modelName)) {+ if (+ normalize(survivingBaseModels[baseIndex] ?? "") === normalize(modelName)+ ) {baseIndex += 1;} else if (baseModelNames.has(normalize(modelName))) {throw new Error(`${name} may not reorder existing model entries`);@@ -156,7 +173,7 @@ export function collectTypeModelChanges(baseTypes, currentTypes, typesDiff) {}}- const additionsFromDiff = [];+ const diffChanges = { added: [], removed: [] };for (const line of typesDiff.split("\n")) {if (line.startsWith("+++") || line.startsWith("---")) continue;if (!line.startsWith("+") && !line.startsWith("-")) continue;@@ -169,23 +186,20 @@ export function collectTypeModelChanges(baseTypes, currentTypes, typesDiff) {}const [, sign, modelName] = match;- if (sign === "-") {+ diffChanges[sign === "+" ? "added" : "removed"].push(normalize(modelName));+ }++ for (const expectedChange of ["added", "removed"]) {+ const fromDiff = diffChanges[expectedChange].sort();+ const semantic = changes+ .filter((change) => change.expectedChange === expectedChange)+ .map((change) => normalize(change.modelName))+ .sort();+ if (JSON.stringify(fromDiff) !== JSON.stringify(semantic)) {throw new Error(- `Automated selectable-model removal is not allowed: ${modelName}`,+ "Selectable-model diff does not match model additions and removals in the approved model arrays",);}- additionsFromDiff.push(normalize(modelName));- }-- const semanticAdditions = changes.map((change) =>- normalize(change.modelName),- );- additionsFromDiff.sort();- semanticAdditions.sort();- if (JSON.stringify(additionsFromDiff) !== JSON.stringify(semanticAdditions)) {- throw new Error(- "Selectable-model diff does not match additive changes in the approved model arrays",- );}return changes;@@ -194,7 +208,14 @@ export function collectTypeModelChanges(baseTypes, currentTypes, typesDiff) {export function mergeModelChanges(pricingChanges, typeModelChanges) {const changesByModel = new Map();for (const change of typeModelChanges) {- changesByModel.set(normalize(change.modelName), change);+ const key = normalize(change.modelName);+ const existing = changesByModel.get(key);+ if (existing && existing.expectedChange !== change.expectedChange) {+ throw new Error(+ `Selectable model cannot be both added and removed in one audit: ${change.modelName}`,+ );+ }+ changesByModel.set(key, change);}for (const change of pricingChanges) {changesByModel.set(normalize(change.modelName), change);@@ -227,6 +248,19 @@ export function validateOfficialSources(sources, model) {}export function validateChangedModelRow(row) {+ if (row.change === "removed") {+ if (row.officialSources.length === 0) {+ throw new Error(+ `Removed selectable models require official retirement evidence: ${row.model}`,+ );+ }+ if (!row.comments.trim()) {+ throw new Error(+ `Removed selectable models require a comment explaining why: ${row.model}`,+ );+ }+ return;+ }if (row.priceConfirmed !== "yes") {throw new Error(`Changed model rows require confirmed prices: ${row.model}`,@@ -379,7 +413,7 @@ export function reconcileAuditOutput(normalize(matchingRows[0].provider) !== normalize(change.provider)) {throw new Error(- `${change.arrayName} additions require provider ${change.provider}: ${change.modelName}`,+ `${change.arrayName} changes require provider ${change.provider}: ${change.modelName}`,);}}@@ -402,7 +436,7 @@ export function reconcileAuditOutput(}for (const row of output.modelsChecked) {- if (!["added", "updated"].includes(row.change)) continue;+ if (!["added", "updated", "removed"].includes(row.change)) continue;if (expectedChangesByModel.has(normalizeReportedModel(row.model))) continue;if (hasRepositoryDiff) {@@ -457,6 +491,7 @@ export function renderAuditSummary(output) {none: "None",updated: "Updated",added: "Added",+ removed: "Removed",unresolved: "Unresolved",};const modelRows = output.modelsChecked.map((row) => {diff --git a/scripts/model-price-audit/audit-output-contract.test.mjs b/scripts/model-price-audit/audit-output-contract.test.mjsindex ffe4a75440..b8f40db667 100644--- a/scripts/model-price-audit/audit-output-contract.test.mjs+++ b/scripts/model-price-audit/audit-output-contract.test.mjs@@ -330,7 +330,7 @@ test("rejects a selectable model added to the wrong provider array", () => {assert.notEqual(result.status, 0);assert.match(result.stderr,- /anthropicModels additions require provider Anthropic: gpt-6-astra/,+ /anthropicModels changes require provider Anthropic: gpt-6-astra/,);});@@ -352,23 +352,154 @@ test("uses the pricing delta when the same model also becomes selectable", () =>assert.deepEqual(normalizedOutput.changedModels, [`${model} (updated)`]);});-test("rejects automated selectable-model removal", () => {+const retiredRow = (model, overrides = {}) =>+ auditRow({+ model,+ priceConfirmed: "no",+ usageKeyCoverageConfirmed: "no",+ tieringCorrect: "no",+ officialSources: ["https://developers.openai.com/api/docs/deprecations"],+ comments: "Listed as shut down on the official deprecations page.",+ ...overrides,+ });++test("removes a retired selectable model while keeping its pricing entry", () => {const model = "gpt-4o";- const { result } = runContract({+ const { normalizedOutput, result } = runContract({baseTypes: typesSource({ openAI: ["gpt-4.1", model, "gpt-5"] }),basePrices: [pricingEntry(model)],currentTypes: typesSource({ openAI: ["gpt-4.1", "gpt-5"] }),output: {- ...structuredOutput([auditRow({ model, change: "updated" })]),- pullRequestTitle: "chore(pricing): remove gpt-4o from playground",+ ...structuredOutput([retiredRow(model)]),+ pullRequestTitle: "chore(pricing): remove retired gpt-4o from playground",+ },+ typesDiff: `- "${model}",\n`,+ });++ assert.equal(result.status, 0, result.stderr);+ assert.equal(normalizedOutput.modelsChecked[0].change, "removed");+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (removed)`]);+ assert.match(result.stdout, /\| Removed \|/);+});++test("removes a retired model listed in several selectable arrays", () => {+ const model = "gemini-1";+ const { normalizedOutput, result } = runContract({+ baseTypes: typesSource({+ googleAIStudio: ["gemini-2", model],+ vertexAI: ["gemini-2", model],+ }),+ currentTypes: typesSource(),+ output: {+ ...structuredOutput([+ {+ ...retiredRow(model, {+ officialSources: ["https://ai.google.dev/gemini-api/docs/models"],+ }),+ provider: "Google",+ },+ ]),+ pullRequestTitle: "chore(pricing): remove retired gemini-1 from pickers",+ },+ typesDiff: `- "${model}",\n- "${model}",\n`,+ });++ assert.equal(result.status, 0, result.stderr);+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (removed)`]);+});++test("rejects a selectable-model removal without official evidence", () => {+ const model = "gpt-4o";+ const { result } = runContract({+ baseTypes: typesSource({ openAI: ["gpt-4.1", model] }),+ currentTypes: typesSource({ openAI: ["gpt-4.1"] }),+ output: {+ ...structuredOutput([retiredRow(model, { officialSources: [] })]),+ pullRequestTitle: "chore(pricing): remove retired gpt-4o from playground",+ },+ typesDiff: `- "${model}",\n`,+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Removed selectable models require official retirement evidence: gpt-4o/,+ );+});++test("promotes the next surviving entry when the default model retires", () => {+ const model = "gpt-4o";+ const { normalizedOutput, result } = runContract({+ baseTypes: typesSource({ openAI: [model, "gpt-4.1", "gpt-5"] }),+ currentTypes: typesSource({ openAI: ["gpt-4.1", "gpt-5"] }),+ output: {+ ...structuredOutput([retiredRow(model)]),+ pullRequestTitle: "chore(pricing): remove retired gpt-4o default",+ },+ typesDiff: `- "${model}",\n`,+ });++ assert.equal(result.status, 0, result.stderr);+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (removed)`]);+});++test("rejects choosing a new default when the default model retires", () => {+ const model = "gpt-4o";+ const { result } = runContract({+ baseTypes: typesSource({ openAI: [model, "gpt-4.1", "gpt-5"] }),+ basePrices: [pricingEntry("gpt-6-astra")],+ currentTypes: typesSource({ openAI: ["gpt-6-astra", "gpt-4.1", "gpt-5"] }),+ output: {+ ...structuredOutput([+ retiredRow(model),+ auditRow({ model: "gpt-6-astra", change: "added" }),+ ]),+ pullRequestTitle: "chore(pricing): replace gpt-4o default",+ },+ typesDiff: `- "${model}",\n+ "gpt-6-astra",\n`,+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /openAIModels must keep its first default model unchanged/,+ );+});++test("rejects emptying a selectable-model array", () => {+ const model = "gpt-4o";+ const { result } = runContract({+ currentTypes: typesSource({ openAI: [] }),+ output: {+ ...structuredOutput([retiredRow(model)]),+ pullRequestTitle: "chore(pricing): remove retired gpt-4o from playground",},typesDiff: `- "${model}",\n`,});+ assert.notEqual(result.status, 0);+ assert.match(result.stderr, /openAIModels must keep at least one/);+});++test("rejects moving a selectable model between provider arrays", () => {+ const model = "gemini-2";+ const { result } = runContract({+ baseTypes: typesSource({ vertexAI: ["gemini-3", model] }),+ currentTypes: typesSource({+ openAI: ["gpt-4o", model],+ vertexAI: ["gemini-3"],+ }),+ output: {+ ...structuredOutput([retiredRow(model)]),+ pullRequestTitle: "chore(pricing): move gemini-2 in playground",+ },+ typesDiff: `- "${model}",\n+ "${model}",\n`,+ });+assert.notEqual(result.status, 0);assert.match(result.stderr,- /Automated selectable-model removal is not allowed: gpt-4o/,+ /cannot be both added and removed in one audit: gemini-2/,);});@@ -386,7 +517,7 @@ test("rejects remove-plus-add selectable-model edits", () => {assert.notEqual(result.status, 0);assert.match(result.stderr,- /Automated selectable-model removal is not allowed: gpt-4o/,+ /Selectable-model diff does not match model additions and removals/,);});diff --git a/scripts/model-price-audit/prepare-metadata.mjs b/scripts/model-price-audit/prepare-metadata.mjsindex fa677177c2..adb984673a 100644--- a/scripts/model-price-audit/prepare-metadata.mjs+++ b/scripts/model-price-audit/prepare-metadata.mjs@@ -104,7 +104,7 @@ if (}const changedModelRows = output.modelsChecked.filter((item) =>- ["added", "updated"].includes(item.change),+ ["added", "updated", "removed"].includes(item.change),);const changedRowsByModel = new Map(changedModelRows.map((item) => [normalizeReportedModel(item.model), item]),@@ -113,7 +113,7 @@ for (const change of typeModelChanges) {const row = changedRowsByModel.get(normalize(change.modelName));if (row && normalize(row.provider) !== normalize(change.provider)) {throw new Error(- `${change.arrayName} additions require provider ${change.provider}: ${change.modelName}`,+ `${change.arrayName} changes require provider ${change.provider}: ${change.modelName}`,);}}diff --git a/scripts/model-price-audit/prepare-metadata.test.mjs b/scripts/model-price-audit/prepare-metadata.test.mjsindex f3c75919e6..cd0ffc3e7d 100644--- a/scripts/model-price-audit/prepare-metadata.test.mjs+++ b/scripts/model-price-audit/prepare-metadata.test.mjs@@ -237,7 +237,30 @@ test("rejects pricing-entry removal from the staged snapshot", () => {);});-test("rejects selectable-model removal from the staged snapshot", () => {+test("accepts a retired selectable-model removal from the staged snapshot", () => {+ const result = runValidator({+ baseTypes: typesSource({ openAI: ["gpt-4o", modelName] }),+ basePrices: changedPrices.before,+ changedModelTypes: true,+ currentTypes: typesSource(),+ output: {+ pullRequestTitle:+ "chore(pricing): remove retired gpt-5.5 from playground",+ modelsChecked: [+ {+ ...confirmedChange(modelName, "removed"),+ priceConfirmed: "no",+ comments: "Listed as shut down on the official deprecations page.",+ },+ ],+ },+ typesDiff: `- "${modelName}",\n`,+ });++ assert.equal(result.status, 0, result.stderr);+});++test("rejects a staged selectable-model removal reported as an update", () => {const result = runValidator({baseTypes: typesSource({ openAI: ["gpt-4o", modelName] }),basePrices: changedPrices.before,@@ -253,7 +276,7 @@ test("rejects selectable-model removal from the staged snapshot", () => {assert.notEqual(result.status, 0);assert.match(result.stderr,- /Automated selectable-model removal is not allowed: gpt-5.5-2026-04-23/,+ /modelsChecked must report the actual removed model entry: gpt-5.5-2026-04-23/,);});