🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
fix(pricing): reconcile model audit output (#17074)
Hassieb Pakzad40507586a6fb
diff --git a/.github/workflows/model-price-audit.yml b/.github/workflows/model-price-audit.ymlindex af24478865..c2c94bb30c 100644--- a/.github/workflows/model-price-audit.yml+++ b/.github/workflows/model-price-audit.yml@@ -70,6 +70,9 @@ jobs:node-version: "24"package-manager-cache: false+ - name: Test audit guardrails+ run: node --test scripts/model-price-audit/*.test.mjs+- name: Validate workflow inputsid: validate-inputsenv:@@ -203,7 +206,7 @@ jobs:6. Record every distinct model price entry you check in `modelsChecked`; do not omit confirmed or unchanged models and do not collapse multiple checked models into one family row.7. If you learn durable provider-source URLs, pricing-page quirks, model-ID variants, or recurring audit rules that would help future audits, update only the pricing skill reference files under `.agents/skills/add-model-price/references/*.md`.8. You may replace the optional snapshot in `.agents/skills/add-model-price/references/model-audit-memory.md` with the complete `modelsChecked` table when retaining the current per-model evidence would materially help a future audit. Do not persist a partial table, append unbounded run history, or update the file only to refresh its date.- 9. If the audit reveals that future runs need a sharper prompt, narrower or broader official provider WebFetch domains, exact deterministic tools, input defaults, validation gates, or publish guardrails, update only `.github/workflows/model-price-audit.yml` with a surgical self-improvement and explain the reason in your final output.+ 9. If the audit reveals that future runs need a sharper prompt, exact deterministic tools, input defaults, validation gates, or publish guardrails, update only `.github/workflows/model-price-audit.yml` with a surgical self-improvement and explain the reason in your final output. Report required provider-domain changes as unresolved so the fetch and deterministic source allowlists can be reviewed together.10. Re-run the deterministic validation script before finishing.11. If you change any `matchPattern`, run the bundled match-pattern tester with representative accepted and rejected model IDs before finishing.@@ -222,7 +225,7 @@ jobs:- Do not edit `.agents/skills/add-model-price/SKILL.md` or `.agents/skills/add-model-price/scripts/**`.- Workflow self-improvements must preserve the schedule/manual triggers, repo guard, read-only audit permissions, explicit read-only GitHub token, separate publish job, secret names, input validation, diff allowlist, hook-disabled commit path, staged-blob checks, artifact handoff, and non-fatal reviewer request.- Do not grant the audit job write permissions, `id-token: write`, package-manager tools, arbitrary shell tools, arbitrary network tools, `gh`, or `git push`.- - If official provider source domains change, keep the WebFetch tool allowlist and the audit report's `officialSourceHosts` validation list synchronized.+ - Do not change official provider WebFetch domains. Report a required domain change as unresolved so the workflow and deterministic source validation can be updated together.- Keep the first entry in every selectable model array unchanged unless the audit explicitly changes the default model.- Use the allowed exact `date -u +%Y-%m-%dT00:00:00.000Z` command if you need the current timestamp.@@ -287,7 +290,7 @@ jobs:usageKeysChecked: "Not checked",usageKeyCoverageConfirmed: "no",tieringChecked: "Not checked",- tieringCorrect: "not_applicable",+ tieringCorrect: "no",change: "none",officialSources: [],comments: "Claude and provider-source checks were skipped in dry-run mode.",@@ -323,6 +326,19 @@ jobs:echo "JSON"} >> "$GITHUB_OUTPUT"+ - name: Re-test audit guardrails after agent edits+ run: node --test scripts/model-price-audit/*.test.mjs++ - name: Discard pricing-only formatting changes+ run: |+ set -euo pipefail++ base_pricing_file="$RUNNER_TEMP/default-model-prices-before-audit.json"+ git show "HEAD:worker/src/constants/default-model-prices.json" > "$base_pricing_file"+ node scripts/model-price-audit/discard-formatting-only-pricing-diff.mjs \+ --base "$base_pricing_file" \+ --current worker/src/constants/default-model-prices.json+- name: Write audit summaryenv:STRUCTURED_OUTPUT: ${{ steps.claude-audit.outputs.structured_output || steps.mock-claude-audit.outputs.structured_output }}@@ -335,167 +351,27 @@ jobs:git ls-files --others --exclude-standard} | sort -u)+ HAS_REPOSITORY_DIFF=false+ if [ "${#changed_files[@]}" -gt 0 ]; then+ HAS_REPOSITORY_DIFF=true+ fi+MEMORY_SNAPSHOT_ONLY=falseif [ "${#changed_files[@]}" -eq 1 ] && [ "${changed_files[0]}" = ".agents/skills/add-model-price/references/model-audit-memory.md" ]; thenMEMORY_SNAPSHOT_ONLY=truefi+ BASE_PRICING_FILE="$RUNNER_TEMP/default-model-prices-before-audit.json"+ BASE_TYPES_FILE="$RUNNER_TEMP/llm-types-before-audit.ts"+ git show "HEAD:packages/shared/src/server/llm/types.ts" > "$BASE_TYPES_FILE"+ CURRENT_PRICING_FILE="worker/src/constants/default-model-prices.json"+ CURRENT_TYPES_FILE="packages/shared/src/server/llm/types.ts"STRUCTURED_OUTPUT_PATH="$RUNNER_TEMP/claude-model-price-audit.json"- export MEMORY_SNAPSHOT_ONLY STRUCTURED_OUTPUT_PATH- node <<'NODE' | tee "$RUNNER_TEMP/claude-model-price-audit.txt"- const fs = require("node:fs");- const output = JSON.parse(process.env.STRUCTURED_OUTPUT);- if (- process.env.MEMORY_SNAPSHOT_ONLY === "true" &&- output.pullRequestTitle.trim() === ""- ) {- output.pullRequestTitle =- "chore(pricing): record model price audit snapshot";- }- fs.writeFileSync(process.env.STRUCTURED_OUTPUT_PATH, JSON.stringify(output));- const escapeCell = (value) =>- String(value ?? "")- .replace(/&/g, "&")- .replace(/</g, "<")- .replace(/>/g, ">")- .replace(/\\/g, "\\\\")- .replace(/\|/g, "\\|")- .replace(/`/g, "\\`")- .replace(/\[/g, "\\[")- .replace(/\]/g, "\\]")- .replace(/\r?\n/g, "<br>");- const list = (items) =>- Array.isArray(items) && items.length > 0- ? items.map((item) => `- ${String(item).replace(/\r?\n/g, " ")}`).join("\n")- : "- None";- const officialSourceHosts = [- "ai.google.dev",- "aws.amazon.com",- "azure.microsoft.com",- "cloud.google.com",- "developers.openai.com",- "docs.anthropic.com",- "platform.claude.com",- ];- const sourceLinks = (sources, model) => {- if (!Array.isArray(sources)) {- throw new Error(`officialSources must be an array for ${model}`);- }-- return sources.length > 0- ? sources- .map((source, index) => {- const url = new URL(source);- if (url.protocol !== "https:") {- throw new Error(`Unsupported source URL protocol for ${model}: ${source}`);- }- if (- !officialSourceHosts.some(- (host) => url.hostname === host || url.hostname.endsWith(`.${host}`),- )- ) {- throw new Error(`Source URL is not on an approved official domain for ${model}: ${source}`);- }- const href = url.href.replace(/\(/g, "%28").replace(/\)/g, "%29");- return `[${index + 1}](${href})`;- })- .join(" ")- : "—";- };-- if (!Array.isArray(output.modelsChecked) || output.modelsChecked.length === 0) {- throw new Error("modelsChecked must contain every model price checked during the audit");- }-- const modelKeys = new Set();- const modelRows = output.modelsChecked.map((item) => {- const key = `${item.provider}\u0000${item.model}`.toLowerCase();- if (modelKeys.has(key)) {- throw new Error(`Duplicate modelsChecked row: ${item.provider} / ${item.model}`);- }- modelKeys.add(key);-- if (- (item.priceConfirmed === "yes" ||- item.usageKeyCoverageConfirmed === "yes" ||- item.tieringCorrect === "yes") &&- item.officialSources.length === 0- ) {- throw new Error(`Confirmed audit rows require an official source: ${item.model}`);- }- if (- (item.priceConfirmed === "no" ||- item.usageKeyCoverageConfirmed === "no" ||- item.tieringCorrect === "no") &&- !item.comments.trim()- ) {- throw new Error(`Unconfirmed audit rows require comments: ${item.model}`);- }- if (- ["added", "updated"].includes(item.change) &&- item.usageKeyCoverageConfirmed !== "yes"- ) {- throw new Error(`Changed model rows require confirmed usage-key coverage: ${item.model}`);- }-- const priceConfirmed = item.priceConfirmed === "yes" ? "Yes" : "No";- const usageKeyCoverageConfirmed =- item.usageKeyCoverageConfirmed === "yes" ? "Yes" : "No";- const tieringCorrect =- item.tieringCorrect === "not_applicable"- ? "N/A"- : item.tieringCorrect === "yes"- ? "Yes"- : "No";- const changeLabels = {- none: "None",- updated: "Updated",- added: "Added",- unresolved: "Unresolved",- };-- return `| ${escapeCell(item.provider)} | ${escapeCell(item.model)} | ${escapeCell(item.pricingChecked)} | ${priceConfirmed} | ${escapeCell(item.usageKeysChecked)} | ${usageKeyCoverageConfirmed} | ${escapeCell(item.tieringChecked)} | ${tieringCorrect} | ${changeLabels[item.change]} | ${sourceLinks(item.officialSources, item.model)} | ${escapeCell(item.comments) || "—"} |`;- });- const proposedTitle = output.pullRequestTitle- ? `\`${output.pullRequestTitle.replace(/`/g, "\\`")}\``- : "_No repository diff proposed_";-- process.stdout.write(- [- `**Audit date:** ${output.auditDate}`,- `**Proposed pull request title:** ${proposedTitle}`,- "",- output.summary,- "",- "### Models checked",- "",- "| Provider | Model / pricing entry | Pricing checked | Price confirmed | Usage keys checked | Usage-key coverage | Tiering checked | Tiering correct | Change | Official source(s) | Comments |",- "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",- ...modelRows,- "",- "### Changed models",- "",- list(output.changedModels),- "",- "### Skill reference updates",- "",- list(output.skillReferenceUpdates),- "",- "### Workflow updates",- "",- list(output.workflowUpdates),- "",- "### Unresolved findings",- "",- list(output.unresolvedFindings),- "",- "### Validation",- "",- list(output.validation),- "",- ].join("\n"),- );- NODE+ TYPES_DIFF_FILE="$RUNNER_TEMP/llm-types-before-prepare.diff"+ git diff --unified=0 -- "packages/shared/src/server/llm/types.ts" > "$TYPES_DIFF_FILE"+ export BASE_PRICING_FILE BASE_TYPES_FILE CURRENT_PRICING_FILE CURRENT_TYPES_FILE HAS_REPOSITORY_DIFF MEMORY_SNAPSHOT_ONLY STRUCTURED_OUTPUT_PATH TYPES_DIFF_FILE+ node scripts/model-price-audit/validate-and-render-audit-output.mjs \+ | tee "$RUNNER_TEMP/claude-model-price-audit.txt"{echo "## Model price audit summary"@@ -503,16 +379,6 @@ jobs:cat "$RUNNER_TEMP/claude-model-price-audit.txt"} >> "$GITHUB_STEP_SUMMARY"- - name: Discard pricing-only formatting changes- run: |- set -euo pipefail-- base_pricing_file="$RUNNER_TEMP/default-model-prices-before-audit.json"- git show "HEAD:worker/src/constants/default-model-prices.json" > "$base_pricing_file"- node scripts/model-price-audit/discard-formatting-only-pricing-diff.mjs \- --base "$base_pricing_file" \- --current worker/src/constants/default-model-prices.json-- name: Validate audit diffrun: |set -euo pipefail@@ -646,15 +512,20 @@ jobs:staged_pricing_file="$RUNNER_TEMP/default-model-prices-staged.json"git show ":worker/src/constants/default-model-prices.json" > "$staged_pricing_file"+ staged_types_file="$RUNNER_TEMP/llm-types-staged.ts"+ git show ":packages/shared/src/server/llm/types.ts" > "$staged_types_file"git diff --cached --binary > "$RUNNER_TEMP/model-price-audit-diagnostic.patch"test -s "$RUNNER_TEMP/model-price-audit-diagnostic.patch"CURRENT_PRICING_FILE="$staged_pricing_file"+ BASE_TYPES_FILE="$RUNNER_TEMP/llm-types-base.ts"+ git show "HEAD:packages/shared/src/server/llm/types.ts" > "$BASE_TYPES_FILE"+ CURRENT_TYPES_FILE="$staged_types_file"TYPES_DIFF_FILE="$RUNNER_TEMP/llm-types.diff"git diff --cached --unified=0 -- "packages/shared/src/server/llm/types.ts" > "$TYPES_DIFF_FILE"STRUCTURED_OUTPUT_PATH="$RUNNER_TEMP/claude-model-price-audit.json"PR_TITLE_PATH="$RUNNER_TEMP/model-price-audit-pr-title.txt"- export BASE_PRICING_FILE CHANGED_MODEL_TYPES CHANGED_PRICING_JSON CURRENT_PRICING_FILE STRUCTURED_OUTPUT_PATH TYPES_DIFF_FILE+ export BASE_PRICING_FILE BASE_TYPES_FILE CHANGED_MODEL_TYPES CHANGED_PRICING_JSON CURRENT_PRICING_FILE CURRENT_TYPES_FILE STRUCTURED_OUTPUT_PATH TYPES_DIFF_FILEnode scripts/model-price-audit/prepare-metadata.mjs > "$PR_TITLE_PATH"PR_TITLE="$(cat "$PR_TITLE_PATH")"diff --git a/scripts/model-price-audit/audit-output-contract.mjs b/scripts/model-price-audit/audit-output-contract.mjsnew file mode 100644index 0000000000..367835a7b8--- /dev/null+++ b/scripts/model-price-audit/audit-output-contract.mjs@@ -0,0 +1,510 @@+const officialSourceHosts = [+ "ai.google.dev",+ "aws.amazon.com",+ "azure.microsoft.com",+ "cloud.google.com",+ "developers.openai.com",+ "docs.anthropic.com",+ "platform.claude.com",+];++const normalize = (value) => value.trim().toLowerCase();++export const normalizeReportedModel = (value) =>+ normalize(value).replace(/\s+\((?:also\s+)?matches\b[^)]*\)$/u, "");++const canonicalize = (value) => {+ if (Array.isArray(value)) return value.map(canonicalize);+ if (value !== null && typeof value === "object") {+ return Object.fromEntries(+ Object.keys(value)+ .sort()+ .map((key) => [key, canonicalize(value[key])]),+ );+ }+ return value;+};++const indexUniquePricingModels = (prices, label) => {+ const modelsByName = new Map();+ for (const item of prices) {+ const key = normalize(item.modelName);+ if (modelsByName.has(key)) {+ throw new Error(+ `${label} pricing contains duplicate normalized modelName: ${item.modelName}`,+ );+ }+ modelsByName.set(key, item);+ }+ return modelsByName;+};++export function collectPricingModelChanges(basePrices, currentPrices) {+ const baseByName = indexUniquePricingModels(basePrices, "Base");+ const currentByName = indexUniquePricingModels(currentPrices, "Current");+ const modelNames = Array.from(baseByName.keys()).concat(+ Array.from(currentByName.keys()),+ );+ const changes = [];++ for (const modelName of new Set(modelNames)) {+ const before = baseByName.get(modelName);+ const after = currentByName.get(modelName);+ if (!after) {+ changes.push({ modelName: before.modelName, expectedChange: "removed" });+ } else if (!before) {+ changes.push({ modelName: after.modelName, expectedChange: "added" });+ } else if (+ JSON.stringify(canonicalize(before)) !==+ JSON.stringify(canonicalize(after))+ ) {+ changes.push({ modelName: after.modelName, expectedChange: "updated" });+ }+ }++ return changes;+}++const selectableModelArrays = [+ {+ name: "openAIModels",+ pattern: /export const openAIModels = \[([\s\S]*?)\] as const;/u,+ provider: "OpenAI",+ },+ {+ name: "anthropicModels",+ pattern: /export const anthropicModels = \[([\s\S]*?)\] as const;/u,+ provider: "Anthropic",+ },+ {+ name: "vertexAIModels",+ pattern: /export const vertexAIModels = \[([\s\S]*?)\] as const;/u,+ provider: "Google",+ },+ {+ name: "googleAIStudioModels",+ pattern: /export const googleAIStudioModels = \[([\s\S]*?)\] as const;/u,+ provider: "Google",+ },+];++const readSelectableModelArrays = (source, label) =>+ new Map(+ selectableModelArrays.map(({ name, pattern, provider }) => {+ const match = source.match(pattern);+ if (!match) {+ throw new Error(`${label} types file is missing ${name}`);+ }+ const models = match[1]+ .split("\n")+ .map((line) => line.trim())+ .filter((line) => line && !line.startsWith("//"))+ .map((line) => {+ const modelMatch = line.match(/^"([^"]+)",(?:\s*\/\/.*)?$/u);+ if (!modelMatch) {+ throw new Error(+ `${label} ${name} contains an unsupported entry: ${line}`,+ );+ }+ return modelMatch[1];+ });+ const normalizedModels = models.map(normalize);+ if (new Set(normalizedModels).size !== normalizedModels.length) {+ throw new Error(`${label} ${name} contains duplicate model entries`);+ }+ return [name, { models, provider }];+ }),+ );++export function collectTypeModelChanges(baseTypes, currentTypes, typesDiff) {+ const baseArrays = readSelectableModelArrays(baseTypes, "Base");+ const currentArrays = readSelectableModelArrays(currentTypes, "Current");+ const changes = [];++ for (const { name } of selectableModelArrays) {+ const before = baseArrays.get(name);+ const after = currentArrays.get(name);+ const currentModelNames = new Set(after.models.map(normalize));+ const removedModels = before.models.filter(+ (modelName) => !currentModelNames.has(normalize(modelName)),+ );+ if (removedModels.length > 0) {+ throw new Error(+ `Automated selectable-model removal is not allowed: ${removedModels.join(", ")}`,+ );+ }+ if (before.models[0] !== after.models[0]) {+ throw new Error(`${name} must keep its first default model unchanged`);+ }++ let baseIndex = 0;+ const baseModelNames = new Set(before.models.map(normalize));+ for (const modelName of after.models) {+ if (normalize(before.models[baseIndex] ?? "") === normalize(modelName)) {+ baseIndex += 1;+ } else if (baseModelNames.has(normalize(modelName))) {+ throw new Error(`${name} may not reorder existing model entries`);+ } else {+ changes.push({+ arrayName: name,+ expectedChange: "added",+ modelName,+ provider: after.provider,+ });+ }+ }+ }++ const additionsFromDiff = [];+ for (const line of typesDiff.split("\n")) {+ if (line.startsWith("+++") || line.startsWith("---")) continue;+ if (!line.startsWith("+") && !line.startsWith("-")) continue;++ const match = line.match(/^([+-])\s*"([^"]+)",(?:\s*\/\/.*)?$/u);+ if (!match) {+ throw new Error(+ `Selectable-model diff contains an unsupported changed line: ${line}`,+ );+ }++ const [, sign, modelName] = match;+ if (sign === "-") {+ throw new Error(+ `Automated selectable-model removal is not allowed: ${modelName}`,+ );+ }+ additionsFromDiff.push(normalize(modelName));+ }++ const semanticAdditions = changes.map((change) =>+ normalize(change.modelName),+ );+ additionsFromDiff.sort();+ semanticAdditions.sort();+ if (JSON.stringify(additionsFromDiff) !== JSON.stringify(semanticAdditions)) {+ throw new Error(+ "Selectable-model diff does not match additive changes in the approved model arrays",+ );+ }++ return changes;+}++export function mergeModelChanges(pricingChanges, typeModelChanges) {+ const changesByModel = new Map();+ for (const change of typeModelChanges) {+ changesByModel.set(normalize(change.modelName), change);+ }+ for (const change of pricingChanges) {+ changesByModel.set(normalize(change.modelName), change);+ }+ return Array.from(changesByModel.values());+}++export function validateOfficialSources(sources, model) {+ if (!Array.isArray(sources)) {+ throw new Error(`officialSources must be an array for ${model}`);+ }++ for (const source of sources) {+ const url = new URL(source);+ if (url.protocol !== "https:") {+ throw new Error(+ `Unsupported source URL protocol for ${model}: ${source}`,+ );+ }+ if (+ !officialSourceHosts.some(+ (host) => url.hostname === host || url.hostname.endsWith(`.${host}`),+ )+ ) {+ throw new Error(+ `Source URL is not on an approved official domain for ${model}: ${source}`,+ );+ }+ }+}++export function validateChangedModelRow(row) {+ if (row.priceConfirmed !== "yes") {+ throw new Error(+ `Changed model rows require confirmed prices: ${row.model}`,+ );+ }+ if (row.usageKeyCoverageConfirmed !== "yes") {+ throw new Error(+ `Changed model rows require confirmed usage-key coverage: ${row.model}`,+ );+ }+ if (!["yes", "not_applicable"].includes(row.tieringCorrect)) {+ throw new Error(+ `Changed model rows require confirmed or inapplicable tiering: ${row.model}`,+ );+ }+ if (row.officialSources.length === 0) {+ throw new Error(+ `Changed model rows require an official source: ${row.model}`,+ );+ }+}++const appendComment = (row, message) => {+ const separator = row.comments.trim() ? " " : "";+ row.comments = `${row.comments.trim()}${separator}${message}`;+};++export function reconcileAuditOutput(+ rawOutput,+ {+ basePrices,+ currentPrices,+ baseTypes,+ currentTypes,+ hasRepositoryDiff,+ memorySnapshotOnly,+ typesDiff,+ },+) {+ const output = structuredClone(rawOutput);+ const warnings = [];++ if (+ !Array.isArray(output.modelsChecked) ||+ output.modelsChecked.length === 0+ ) {+ throw new Error(+ "modelsChecked must contain every model price checked during the audit",+ );+ }+ if (!Array.isArray(output.unresolvedFindings)) {+ throw new Error("unresolvedFindings must be an array");+ }++ if (memorySnapshotOnly && output.pullRequestTitle.trim() === "") {+ output.pullRequestTitle =+ "chore(pricing): record model price audit snapshot";+ }++ const pricingChanges = collectPricingModelChanges(basePrices, currentPrices);+ const typeModelChanges = collectTypeModelChanges(+ baseTypes,+ currentTypes,+ typesDiff,+ );+ const removedPricingModels = pricingChanges.filter(+ (change) => change.expectedChange === "removed",+ );+ if (removedPricingModels.length > 0) {+ throw new Error(+ `Automated pricing-entry removal is not allowed: ${removedPricingModels+ .map((change) => change.modelName)+ .join(", ")}`,+ );+ }+ const expectedModelChanges = mergeModelChanges(+ pricingChanges,+ typeModelChanges,+ );+ const changedModelKeys = new Set(+ expectedModelChanges.map((change) => normalize(change.modelName)),+ );++ const modelKeys = new Set();+ for (const row of output.modelsChecked) {+ const key = `${row.provider}\u0000${row.model}`.toLowerCase();+ if (modelKeys.has(key)) {+ throw new Error(+ `Duplicate modelsChecked row: ${row.provider} / ${row.model}`,+ );+ }+ modelKeys.add(key);+ validateOfficialSources(row.officialSources, row.model);++ const unsupportedConfirmations = [];+ if (row.officialSources.length === 0) {+ if (row.priceConfirmed === "yes") unsupportedConfirmations.push("price");+ if (row.usageKeyCoverageConfirmed === "yes") {+ unsupportedConfirmations.push("usage-key coverage");+ }+ if (["yes", "not_applicable"].includes(row.tieringCorrect)) {+ unsupportedConfirmations.push("tiering applicability");+ }+ }++ if (unsupportedConfirmations.length > 0) {+ if (changedModelKeys.has(normalizeReportedModel(row.model))) {+ throw new Error(+ `Confirmed audit rows require an official source: ${row.model}`,+ );+ }+ if (row.priceConfirmed === "yes") row.priceConfirmed = "no";+ if (row.usageKeyCoverageConfirmed === "yes") {+ row.usageKeyCoverageConfirmed = "no";+ }+ if (["yes", "not_applicable"].includes(row.tieringCorrect)) {+ row.tieringCorrect = "no";+ }+ const warning = `${row.model}: ${unsupportedConfirmations.join(+ ", ",+ )} confirmation downgraded because no approved official source was provided.`;+ appendComment(row, warning);+ output.unresolvedFindings.push(warning);+ warnings.push(warning);+ }++ if (+ (row.priceConfirmed === "no" ||+ row.usageKeyCoverageConfirmed === "no" ||+ row.tieringCorrect === "no") &&+ !row.comments.trim()+ ) {+ throw new Error(`Unconfirmed audit rows require comments: ${row.model}`);+ }+ }++ const rowsByModel = new Map();+ for (const row of output.modelsChecked) {+ const key = normalizeReportedModel(row.model);+ rowsByModel.set(key, [...(rowsByModel.get(key) ?? []), row]);+ }+ const expectedChangesByModel = new Map(+ expectedModelChanges.map((change) => [normalize(change.modelName), change]),+ );++ for (const change of typeModelChanges) {+ const matchingRows = rowsByModel.get(normalize(change.modelName)) ?? [];+ if (+ matchingRows.length === 1 &&+ normalize(matchingRows[0].provider) !== normalize(change.provider)+ ) {+ throw new Error(+ `${change.arrayName} additions require provider ${change.provider}: ${change.modelName}`,+ );+ }+ }++ for (const change of expectedChangesByModel.values()) {+ const matchingRows = rowsByModel.get(normalize(change.modelName)) ?? [];+ if (matchingRows.length === 0) {+ throw new Error(+ `modelsChecked must report the actual ${change.expectedChange} model entry: ${change.modelName}`,+ );+ }+ if (matchingRows.length > 1) {+ throw new Error(+ `modelsChecked must report exactly one row for changed model: ${change.modelName}`,+ );+ }+ const [row] = matchingRows;+ row.change = change.expectedChange;+ validateChangedModelRow(row);+ }++ for (const row of output.modelsChecked) {+ if (!["added", "updated"].includes(row.change)) continue;+ if (expectedChangesByModel.has(normalizeReportedModel(row.model))) continue;++ if (hasRepositoryDiff) {+ throw new Error(+ `modelsChecked reports a model without a matching pricing or selectable-model diff: ${row.model}`,+ );+ }+ warnings.push(+ `${row.model}: reported ${row.change} without a repository diff; change reset to none.`,+ );+ row.change = "none";+ }++ output.changedModels = expectedModelChanges.map(+ (change) => `${change.modelName} (${change.expectedChange})`,+ );++ return { output, pricingChanges, typeModelChanges, warnings };+}++const escapeCell = (value) =>+ String(value ?? "")+ .replace(/&/g, "&")+ .replace(/</g, "<")+ .replace(/>/g, ">")+ .replace(/\\/g, "\\\\")+ .replace(/\|/g, "\\|")+ .replace(/`/g, "\\`")+ .replace(/\[/g, "\\[")+ .replace(/\]/g, "\\]")+ .replace(/\r?\n/g, "<br>");++const list = (items) =>+ Array.isArray(items) && items.length > 0+ ? items.map((item) => `- ${String(item).replace(/\r?\n/g, " ")}`).join("\n")+ : "- None";++const sourceLinks = (sources) =>+ sources.length > 0+ ? sources+ .map((source, index) => {+ const href = new URL(source).href+ .replace(/\(/g, "%28")+ .replace(/\)/g, "%29");+ return `[${index + 1}](${href})`;+ })+ .join(" ")+ : "—";++export function renderAuditSummary(output) {+ const changeLabels = {+ none: "None",+ updated: "Updated",+ added: "Added",+ unresolved: "Unresolved",+ };+ const modelRows = output.modelsChecked.map((row) => {+ const priceConfirmed = row.priceConfirmed === "yes" ? "Yes" : "No";+ const usageKeyCoverageConfirmed =+ row.usageKeyCoverageConfirmed === "yes" ? "Yes" : "No";+ const tieringCorrect =+ row.tieringCorrect === "not_applicable"+ ? "N/A"+ : row.tieringCorrect === "yes"+ ? "Yes"+ : "No";+ return `| ${escapeCell(row.provider)} | ${escapeCell(row.model)} | ${escapeCell(row.pricingChecked)} | ${priceConfirmed} | ${escapeCell(row.usageKeysChecked)} | ${usageKeyCoverageConfirmed} | ${escapeCell(row.tieringChecked)} | ${tieringCorrect} | ${changeLabels[row.change]} | ${sourceLinks(row.officialSources)} | ${escapeCell(row.comments) || "—"} |`;+ });+ const proposedTitle = output.pullRequestTitle+ ? `\`${output.pullRequestTitle.replace(/\\/g, "\\\\").replace(/`/g, "\\`")}\``+ : "_No repository diff proposed_";++ return [+ `**Audit date:** ${output.auditDate}`,+ `**Proposed pull request title:** ${proposedTitle}`,+ "",+ output.summary,+ "",+ "### Models checked",+ "",+ "| Provider | Model / pricing entry | Pricing checked | Price confirmed | Usage keys checked | Usage-key coverage | Tiering checked | Tiering correct | Change | Official source(s) | Comments |",+ "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",+ ...modelRows,+ "",+ "### Changed models",+ "",+ list(output.changedModels),+ "",+ "### Skill reference updates",+ "",+ list(output.skillReferenceUpdates),+ "",+ "### Workflow updates",+ "",+ list(output.workflowUpdates),+ "",+ "### Unresolved findings",+ "",+ list(output.unresolvedFindings),+ "",+ "### Validation",+ "",+ list(output.validation),+ "",+ ].join("\n");+}diff --git a/scripts/model-price-audit/audit-output-contract.test.mjs b/scripts/model-price-audit/audit-output-contract.test.mjsnew file mode 100644index 0000000000..459c70dab0--- /dev/null+++ b/scripts/model-price-audit/audit-output-contract.test.mjs@@ -0,0 +1,497 @@+import assert from "node:assert/strict";+import fs from "node:fs";+import os from "node:os";+import path from "node:path";+import { spawnSync } from "node:child_process";+import test from "node:test";+import {+ collectTypeModelChanges,+ renderAuditSummary,+} from "./audit-output-contract.mjs";++const scriptPath = path.join(+ import.meta.dirname,+ "validate-and-render-audit-output.mjs",+);+const workflowPath = path.join(+ import.meta.dirname,+ "../../.github/workflows/model-price-audit.yml",+);+const repositoryTypesPath = path.join(+ import.meta.dirname,+ "../../packages/shared/src/server/llm/types.ts",+);++const officialSource = "https://developers.openai.com/api/docs/pricing";++const pricingEntry = (modelName, inputPrice = 1) => ({+ modelName,+ pricingTiers: [{ prices: { input: inputPrice } }],+});++const auditRow = ({+ model,+ change = "none",+ priceConfirmed = "yes",+ usageKeyCoverageConfirmed = "yes",+ tieringCorrect = "not_applicable",+ officialSources = [officialSource],+ comments = "Confirmed from the official pricing page.",+}) => ({+ provider: "OpenAI",+ model,+ pricingChecked: "Input pricing",+ priceConfirmed,+ usageKeysChecked: "input",+ usageKeyCoverageConfirmed,+ tieringChecked: "No tiering applies",+ tieringCorrect,+ change,+ officialSources,+ comments,+});++const structuredOutput = (modelsChecked) => ({+ summary: "Audit complete.",+ auditDate: "2026-09-04",+ pullRequestTitle: "",+ modelsChecked,+ changedModels: [],+ skillReferenceUpdates: [],+ workflowUpdates: [],+ unresolvedFindings: [],+ validation: [],+});++const typesSource = ({+ anthropic = ["claude-3"],+ googleAIStudio = ["gemini-2"],+ openAI = ["gpt-4o"],+ vertexAI = ["gemini-2"],+} = {}) => `+export const openAIModels = [+${openAI.map((model) => ` "${model}",`).join("\n")}+] as const;+export const anthropicModels = [+${anthropic.map((model) => ` "${model}",`).join("\n")}+] as const;+export const vertexAIModels = [+${vertexAI.map((model) => ` "${model}",`).join("\n")}+] as const;+export const googleAIStudioModels = [+${googleAIStudio.map((model) => ` "${model}",`).join("\n")}+] as const;+`;++function runContract({+ baseTypes = typesSource(),+ basePrices = [],+ currentTypes = baseTypes,+ currentPrices = basePrices,+ output,+ typesDiff = "",+}) {+ const fixtureDirectory = fs.mkdtempSync(+ path.join(os.tmpdir(), "model-price-audit-output-"),+ );+ const basePath = path.join(fixtureDirectory, "base.json");+ const currentPath = path.join(fixtureDirectory, "current.json");+ const baseTypesPath = path.join(fixtureDirectory, "base-types.ts");+ const currentTypesPath = path.join(fixtureDirectory, "current-types.ts");+ const normalizedOutputPath = path.join(fixtureDirectory, "output.json");+ const typesDiffPath = path.join(fixtureDirectory, "types.diff");+ fs.writeFileSync(basePath, JSON.stringify(basePrices));+ fs.writeFileSync(currentPath, JSON.stringify(currentPrices));+ fs.writeFileSync(baseTypesPath, baseTypes);+ fs.writeFileSync(currentTypesPath, currentTypes);+ fs.writeFileSync(typesDiffPath, typesDiff);++ const result = spawnSync(process.execPath, [scriptPath], {+ encoding: "utf8",+ env: {+ ...process.env,+ BASE_PRICING_FILE: basePath,+ BASE_TYPES_FILE: baseTypesPath,+ CURRENT_PRICING_FILE: currentPath,+ CURRENT_TYPES_FILE: currentTypesPath,+ HAS_REPOSITORY_DIFF: String(+ JSON.stringify(basePrices) !== JSON.stringify(currentPrices) ||+ typesDiff.length > 0,+ ),+ MEMORY_SNAPSHOT_ONLY: "false",+ STRUCTURED_OUTPUT: JSON.stringify(output),+ STRUCTURED_OUTPUT_PATH: normalizedOutputPath,+ TYPES_DIFF_FILE: typesDiffPath,+ },+ });+ const normalizedOutput = fs.existsSync(normalizedOutputPath)+ ? JSON.parse(fs.readFileSync(normalizedOutputPath, "utf8"))+ : null;+ fs.rmSync(fixtureDirectory, { force: true, recursive: true });+ return { normalizedOutput, result };+}++test("downgrades unsupported confirmation claims on a no-diff audit", () => {+ const model = "gpt-4o-2024-05-13";+ const { normalizedOutput, result } = runContract({+ output: structuredOutput([+ auditRow({+ model,+ priceConfirmed: "no",+ usageKeyCoverageConfirmed: "yes",+ officialSources: [],+ comments: "Not independently re-fetched during this run.",+ }),+ ]),+ });++ assert.equal(result.status, 0, result.stderr);+ assert.equal(+ normalizedOutput.modelsChecked[0].usageKeyCoverageConfirmed,+ "no",+ );+ assert.equal(normalizedOutput.modelsChecked[0].tieringCorrect, "no");+ assert.match(+ normalizedOutput.modelsChecked[0].comments,+ /downgraded because no approved official source was provided/i,+ );+ assert.match(normalizedOutput.unresolvedFindings[0], new RegExp(model));+ assert.match(result.stderr, /::warning::/);+ assert.match(result.stdout, /gpt-4o-2024-05-13/);+});++test("rejects a changed pricing entry without confirmed prices", () => {+ const model = "gpt-6-astra";+ const { result } = runContract({+ basePrices: [],+ currentPrices: [pricingEntry(model)],+ output: {+ ...structuredOutput([+ auditRow({ model, change: "added", priceConfirmed: "no" }),+ ]),+ pullRequestTitle: "chore(pricing): add gpt-6-astra pricing",+ },+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Changed model rows require confirmed prices: gpt-6-astra/,+ );+});++test("derives changed pricing metadata from the semantic diff", () => {+ const model = "gpt-6-astra";+ const { normalizedOutput, result } = runContract({+ basePrices: [],+ currentPrices: [pricingEntry(model)],+ output: {+ ...structuredOutput([auditRow({ model, change: "none" })]),+ pullRequestTitle: "chore(pricing): add gpt-6-astra pricing",+ },+ });++ assert.equal(result.status, 0, result.stderr);+ assert.equal(normalizedOutput.modelsChecked[0].change, "added");+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (added)`]);+});++test("rejects automated pricing-entry removal", () => {+ const model = "gpt-4o";+ const { result } = runContract({+ basePrices: [pricingEntry(model)],+ currentPrices: [],+ output: {+ ...structuredOutput([auditRow({ model, change: "updated" })]),+ pullRequestTitle: "chore(pricing): remove gpt-4o pricing",+ },+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Automated pricing-entry removal is not allowed: gpt-4o/,+ );+});++test("ignores object-key ordering when deriving changed pricing metadata", () => {+ const unchangedBefore = {+ modelName: "gpt-4o",+ pricingTiers: [{ prices: { input: 1, output: 2 }, priority: 0 }],+ };+ const unchangedAfter = {+ pricingTiers: [{ priority: 0, prices: { output: 2, input: 1 } }],+ modelName: "gpt-4o",+ };+ const changedBefore = pricingEntry("gpt-6-astra", 1);+ const changedAfter = pricingEntry("gpt-6-astra", 2);+ const { normalizedOutput, result } = runContract({+ basePrices: [unchangedBefore, changedBefore],+ currentPrices: [unchangedAfter, changedAfter],+ output: {+ ...structuredOutput([+ auditRow({ model: "gpt-4o" }),+ auditRow({ model: "gpt-6-astra", change: "updated" }),+ ]),+ pullRequestTitle: "chore(pricing): update gpt-6-astra pricing",+ },+ });++ assert.equal(result.status, 0, result.stderr);+ assert.deepEqual(normalizedOutput.changedModels, ["gpt-6-astra (updated)"]);+});++test("reconciles a selectable-model-only addition", () => {+ const model = "gpt-6-astra";+ const { normalizedOutput, result } = runContract({+ basePrices: [pricingEntry(model)],+ currentTypes: typesSource({ openAI: ["gpt-4o", model] }),+ output: {+ ...structuredOutput([auditRow({ model, change: "none" })]),+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",+ },+ typesDiff: `+ "${model}",\n`,+ });++ assert.equal(result.status, 0, result.stderr);+ assert.equal(normalizedOutput.modelsChecked[0].change, "added");+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (added)`]);+});++test("rejects changing a selectable array default model", () => {+ const model = "gpt-6-astra";+ const { result } = runContract({+ basePrices: [pricingEntry(model)],+ currentTypes: typesSource({ openAI: [model, "gpt-4o"] }),+ output: {+ ...structuredOutput([auditRow({ model, change: "added" })]),+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",+ },+ typesDiff: `+ "${model}",\n`,+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /openAIModels must keep its first default model unchanged/,+ );+});++test("rejects a selectable model added to the wrong provider array", () => {+ const model = "gpt-6-astra";+ const { result } = runContract({+ basePrices: [pricingEntry(model)],+ currentTypes: typesSource({ anthropic: ["claude-3", model] }),+ output: {+ ...structuredOutput([auditRow({ model, change: "added" })]),+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",+ },+ typesDiff: `+ "${model}",\n`,+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /anthropicModels additions require provider Anthropic: gpt-6-astra/,+ );+});++test("uses the pricing delta when the same model also becomes selectable", () => {+ const model = "gpt-6-astra";+ const { normalizedOutput, result } = runContract({+ basePrices: [pricingEntry(model, 1)],+ currentPrices: [pricingEntry(model, 2)],+ currentTypes: typesSource({ openAI: ["gpt-4o", model] }),+ output: {+ ...structuredOutput([auditRow({ model, change: "added" })]),+ pullRequestTitle: "chore(pricing): update and expose gpt-6-astra",+ },+ typesDiff: `+ "${model}",\n`,+ });++ assert.equal(result.status, 0, result.stderr);+ assert.equal(normalizedOutput.modelsChecked[0].change, "updated");+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (updated)`]);+});++test("rejects automated selectable-model removal", () => {+ const model = "gpt-4o";+ const { result } = runContract({+ baseTypes: typesSource({ openAI: ["gpt-4.1", model, "gpt-5"] }),+ basePrices: [pricingEntry(model)],+ currentTypes: typesSource({ openAI: ["gpt-4.1", "gpt-5"] }),+ output: {+ ...structuredOutput([auditRow({ model, change: "updated" })]),+ pullRequestTitle: "chore(pricing): remove gpt-4o from playground",+ },+ typesDiff: `- "${model}",\n`,+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Automated selectable-model removal is not allowed: gpt-4o/,+ );+});++test("rejects remove-plus-add selectable-model edits", () => {+ const model = "gpt-4o";+ const { result } = runContract({+ basePrices: [pricingEntry(model)],+ output: {+ ...structuredOutput([auditRow({ model, change: "updated" })]),+ pullRequestTitle: "chore(pricing): reorder gpt-4o in playground",+ },+ typesDiff: `- "${model}",\n+ "${model}",\n`,+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Automated selectable-model removal is not allowed: gpt-4o/,+ );+});++test("rejects unrelated changes in the selectable-model file", () => {+ const model = "gpt-6-astra";+ const { result } = runContract({+ basePrices: [pricingEntry(model)],+ currentTypes: typesSource({ openAI: ["gpt-4o", model] }),+ output: {+ ...structuredOutput([auditRow({ model, change: "added" })]),+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",+ },+ typesDiff: `+ "${model}",\n+export const unrelated = true;\n`,+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Selectable-model diff contains an unsupported changed line/,+ );+});++test("rejects case-insensitive pricing model collisions", () => {+ const model = "gpt-4o";+ const { result } = runContract({+ basePrices: [pricingEntry(model)],+ currentPrices: [pricingEntry(model), pricingEntry("GPT-4O")],+ output: {+ ...structuredOutput([auditRow({ model, change: "updated" })]),+ pullRequestTitle: "chore(pricing): update gpt-4o pricing",+ },+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Current pricing contains duplicate normalized modelName: GPT-4O/,+ );+});++test("rejects ambiguous evidence rows for a changed model", () => {+ const model = "gpt-6-astra";+ const { result } = runContract({+ currentPrices: [pricingEntry(model)],+ output: {+ ...structuredOutput([+ auditRow({ model }),+ { ...auditRow({ model }), provider: "Azure OpenAI" },+ ]),+ pullRequestTitle: "chore(pricing): add gpt-6-astra pricing",+ },+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /modelsChecked must report exactly one row for changed model: gpt-6-astra/,+ );+});++test("clears model-controlled changedModels for non-model diffs", () => {+ const model = "gpt-4o";+ const { normalizedOutput, result } = runContract({+ basePrices: [pricingEntry(model)],+ output: {+ ...structuredOutput([+ auditRow({+ model,+ officialSources: [],+ priceConfirmed: "no",+ usageKeyCoverageConfirmed: "yes",+ }),+ ]),+ changedModels: ["invented-model (added)"],+ pullRequestTitle: "chore(pricing): tighten audit workflow contract",+ },+ typesDiff: "diff --git a/reference.md b/reference.md\n",+ });++ assert.equal(result.status, 0, result.stderr);+ assert.deepEqual(normalizedOutput.changedModels, []);+ assert.equal(+ normalizedOutput.modelsChecked[0].usageKeyCoverageConfirmed,+ "no",+ );+});++test("parses the checked-in selectable-model arrays", () => {+ const types = fs.readFileSync(repositoryTypesPath, "utf8");+ assert.deepEqual(collectTypeModelChanges(types, types, ""), []);+});++test("tolerates pre-existing comments inside selectable-model arrays", () => {+ const types = typesSource().replace(+ ' "gpt-4o",',+ ' // Current flagship\n "gpt-4o",',+ );+ assert.deepEqual(collectTypeModelChanges(types, types, ""), []);+});++test("escapes backslashes and backticks in the proposed title", () => {+ const output = {+ ...structuredOutput([auditRow({ model: "gpt-4o" })]),+ pullRequestTitle: "chore(pricing): check \\ path and `code`",+ };++ assert.match(+ renderAuditSummary(output),+ /chore\(pricing\): check \\\\ path and \\`code\\`/,+ );+});++test("runs formatting cleanup before the extracted output contract", () => {+ const workflow = fs.readFileSync(workflowPath, "utf8");+ const cleanupIndex = workflow.indexOf(+ " - name: Discard pricing-only formatting changes",+ );+ const summaryIndex = workflow.indexOf(" - name: Write audit summary");++ assert.ok(cleanupIndex >= 0, "workflow must clean up formatting-only diffs");+ assert.ok(+ cleanupIndex < summaryIndex,+ "semantic cleanup must run before report reconciliation",+ );+ assert.match(+ workflow,+ /node scripts\/model-price-audit\/validate-and-render-audit-output\.mjs/,+ );+ assert.match(+ workflow,+ /run: node --test scripts\/model-price-audit\/\*\.test\.mjs/,+ );+ assert.equal(+ workflow.match(+ /run: node --test scripts\/model-price-audit\/\*\.test\.mjs/g,+ ).length,+ 2,+ );+ assert.ok(+ workflow.lastIndexOf(+ "run: node --test scripts/model-price-audit/*.test.mjs",+ ) > workflow.indexOf(" - name: Run Claude price audit"),+ "guardrail tests must rerun after permitted workflow self-edits",+ );+ assert.doesNotMatch(workflow, /node <<'NODE' \| tee/);+});diff --git a/scripts/model-price-audit/prepare-metadata.mjs b/scripts/model-price-audit/prepare-metadata.mjsindex 3fba18281a..fa677177c2 100644--- a/scripts/model-price-audit/prepare-metadata.mjs+++ b/scripts/model-price-audit/prepare-metadata.mjs@@ -2,6 +2,14 @@import { createHash } from "node:crypto";import fs from "node:fs";+import {+ collectPricingModelChanges,+ collectTypeModelChanges,+ mergeModelChanges,+ normalizeReportedModel,+ validateChangedModelRow,+ validateOfficialSources,+} from "./audit-output-contract.mjs";process.on("uncaughtException", (error) => {console.error(`::error::${error.message}`);@@ -10,7 +18,9 @@ process.on("uncaughtException", (error) => {const requiredEnvironment = ["BASE_PRICING_FILE",+ "BASE_TYPES_FILE","CURRENT_PRICING_FILE",+ "CURRENT_TYPES_FILE","STRUCTURED_OUTPUT_PATH","TYPES_DIFF_FILE",];@@ -26,8 +36,6 @@ const output = JSON.parse();const title = output.pullRequestTitle.trim();const normalize = (value) => value.trim().toLowerCase();-const normalizeReportedModel = (value) =>- normalize(value).replace(/\s+\((?:also\s+)?matches\b[^)]*\)$/u, "");if (!/^chore\(pricing\): [^\r\n]+$/.test(title) || title.length > 100) {throw new Error(@@ -46,44 +54,33 @@ if ();}-const basePricingText = fs.readFileSync(- process.env.BASE_PRICING_FILE,- "utf8",-);+const basePricingText = fs.readFileSync(process.env.BASE_PRICING_FILE, "utf8");const currentPricingText = fs.readFileSync(process.env.CURRENT_PRICING_FILE,"utf8",);const basePrices = JSON.parse(basePricingText);const currentPrices = JSON.parse(currentPricingText);-const basePricesByName = new Map(- basePrices.map((item) => [normalize(item.modelName), item]),+const pricingModelChanges = collectPricingModelChanges(+ basePrices,+ currentPrices,);-const currentPricesByName = new Map(- currentPrices.map((item) => [normalize(item.modelName), item]),+const removedPricingModels = pricingModelChanges.filter(+ (change) => change.expectedChange === "removed",);-const pricingModelChanges = [];-for (const modelName of new Set([- ...basePricesByName.keys(),- ...currentPricesByName.keys(),-])) {- const before = basePricesByName.get(modelName);- const after = currentPricesByName.get(modelName);- if (JSON.stringify(before) !== JSON.stringify(after)) {- pricingModelChanges.push({- modelName: after?.modelName ?? before.modelName,- expectedChange: before ? "updated" : "added",- });- }+if (removedPricingModels.length > 0) {+ throw new Error(+ `Automated pricing-entry removal is not allowed: ${removedPricingModels+ .map((change) => change.modelName)+ .join(", ")}`,+ );}-const typeModelChanges = new Set();-for (const line of fs- .readFileSync(process.env.TYPES_DIFF_FILE, "utf8")- .split("\n")) {- const match = line.match(/^[+-]\s*"([^"]+)"(?:,|:)/);- if (match) typeModelChanges.add(match[1]);-}+const typeModelChanges = collectTypeModelChanges(+ fs.readFileSync(process.env.BASE_TYPES_FILE, "utf8"),+ fs.readFileSync(process.env.CURRENT_TYPES_FILE, "utf8"),+ fs.readFileSync(process.env.TYPES_DIFF_FILE, "utf8"),+);if (process.env.CHANGED_PRICING_JSON === "true" &&@@ -97,7 +94,10 @@ if (`base sha256: ${digest(basePricingText)}, current sha256: ${digest(currentPricingText)})`,);}-if (process.env.CHANGED_MODEL_TYPES === "true" && typeModelChanges.size === 0) {+if (+ process.env.CHANGED_MODEL_TYPES === "true" &&+ typeModelChanges.length === 0+) {throw new Error("Selectable-model types changed without a concrete model identifier change",);@@ -109,27 +109,33 @@ const changedModelRows = output.modelsChecked.filter((item) =>const changedRowsByModel = new Map(changedModelRows.map((item) => [normalizeReportedModel(item.model), item]),);-for (const change of pricingModelChanges) {+for (const change of typeModelChanges) {+ const row = changedRowsByModel.get(normalize(change.modelName));+ if (row && normalize(row.provider) !== normalize(change.provider)) {+ throw new Error(+ `${change.arrayName} additions require provider ${change.provider}: ${change.modelName}`,+ );+ }+}+const expectedModelChanges = mergeModelChanges(+ pricingModelChanges,+ typeModelChanges,+);+for (const change of expectedModelChanges) {const row = changedRowsByModel.get(normalize(change.modelName));if (!row || row.change !== change.expectedChange) {throw new Error(- `modelsChecked must report the actual ${change.expectedChange} pricing entry: ${change.modelName}`,+ `modelsChecked must report the actual ${change.expectedChange} model entry: ${change.modelName}`,);}+ validateOfficialSources(row.officialSources, row.model);+ validateChangedModelRow(row);}const affectedModels = new Map();-for (const change of pricingModelChanges) {+for (const change of expectedModelChanges) {affectedModels.set(normalize(change.modelName), change.modelName);}-for (const modelName of typeModelChanges) {- affectedModels.set(normalize(modelName), modelName);- if (!changedRowsByModel.has(normalize(modelName))) {- throw new Error(- `modelsChecked must report the selectable-model change: ${modelName}`,- );- }-}for (const row of changedModelRows) {if (!affectedModels.has(normalizeReportedModel(row.model))) {throw new Error(diff --git a/scripts/model-price-audit/prepare-metadata.test.mjs b/scripts/model-price-audit/prepare-metadata.test.mjsindex bcedc0b376..f3c75919e6 100644--- a/scripts/model-price-audit/prepare-metadata.test.mjs+++ b/scripts/model-price-audit/prepare-metadata.test.mjs@@ -11,8 +11,30 @@ const workflowPath = path.join("../../.github/workflows/model-price-audit.yml",);+const typesSource = ({+ anthropic = ["claude-3"],+ googleAIStudio = ["gemini-2"],+ openAI = ["gpt-4o"],+ vertexAI = ["gemini-2"],+} = {}) => `+export const openAIModels = [+${openAI.map((model) => ` "${model}",`).join("\n")}+] as const;+export const anthropicModels = [+${anthropic.map((model) => ` "${model}",`).join("\n")}+] as const;+export const vertexAIModels = [+${vertexAI.map((model) => ` "${model}",`).join("\n")}+] as const;+export const googleAIStudioModels = [+${googleAIStudio.map((model) => ` "${model}",`).join("\n")}+] as const;+`;+function runValidator({+ baseTypes = typesSource(),basePrices = [],+ currentTypes = baseTypes,currentPrices = basePrices,output,typesDiff = "",@@ -24,12 +46,16 @@ function runValidator({);const files = {base: path.join(fixtureDirectory, "base.json"),+ baseTypes: path.join(fixtureDirectory, "base-types.ts"),current: path.join(fixtureDirectory, "current.json"),+ currentTypes: path.join(fixtureDirectory, "current-types.ts"),output: path.join(fixtureDirectory, "output.json"),typesDiff: path.join(fixtureDirectory, "types.diff"),};fs.writeFileSync(files.base, JSON.stringify(basePrices));+ fs.writeFileSync(files.baseTypes, baseTypes);fs.writeFileSync(files.current, JSON.stringify(currentPrices));+ fs.writeFileSync(files.currentTypes, currentTypes);fs.writeFileSync(files.output, JSON.stringify(output));fs.writeFileSync(files.typesDiff, typesDiff);@@ -38,9 +64,11 @@ function runValidator({env: {...process.env,BASE_PRICING_FILE: files.base,+ BASE_TYPES_FILE: files.baseTypes,CHANGED_MODEL_TYPES: String(changedModelTypes),CHANGED_PRICING_JSON: String(changedPricingJson),CURRENT_PRICING_FILE: files.current,+ CURRENT_TYPES_FILE: files.currentTypes,STRUCTURED_OUTPUT_PATH: files.output,TYPES_DIFF_FILE: files.typesDiff,},@@ -50,6 +78,15 @@ function runValidator({}const modelName = "gpt-5.5-2026-04-23";+const confirmedChange = (model, change) => ({+ provider: "OpenAI",+ model,+ change,+ priceConfirmed: "yes",+ usageKeyCoverageConfirmed: "yes",+ tieringCorrect: "not_applicable",+ officialSources: ["https://developers.openai.com/api/docs/pricing"],+});const changedPrices = {before: [{ modelName, pricingTiers: [{ prices: { input: 5 } }] }],after: [{ modelName, pricingTiers: [{ prices: { input: 10 } }] }],@@ -66,7 +103,7 @@ test("accepts an added pricing entry from the staged snapshot", () => {changedPricingJson: true,output: {pullRequestTitle: "chore(pricing): add OpenAI gpt-6-astra pricing",- modelsChecked: [{ model: addedModel.modelName, change: "added" }],+ modelsChecked: [confirmedChange(addedModel.modelName, "added")],},});@@ -81,10 +118,7 @@ test("accepts a unique trailing match-alias annotation", () => {output: {pullRequestTitle: "chore(pricing): update gpt-5.5 large-context tier",modelsChecked: [- {- model: `${modelName} (also matches gpt-5.5)`,- change: "updated",- },+ confirmedChange(`${modelName} (also matches gpt-5.5)`, "updated"),],},});@@ -96,6 +130,22 @@ test("accepts a unique trailing match-alias annotation", () => {);});+test("accepts a selectable-model-only addition with confirmed evidence", () => {+ const selectableModel = "gpt-6-astra";+ const result = runValidator({+ basePrices: [{ modelName: selectableModel }],+ changedModelTypes: true,+ currentTypes: typesSource({ openAI: ["gpt-4o", selectableModel] }),+ output: {+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",+ modelsChecked: [confirmedChange(selectableModel, "added")],+ },+ typesDiff: `+ "${selectableModel}",\n`,+ });++ assert.equal(result.status, 0, result.stderr);+});+test("accepts a concrete title for a reference-only diff", () => {const result = runValidator({output: {@@ -127,7 +177,7 @@ test("rejects an unrelated decorated model row", () => {assert.notEqual(result.status, 0);assert.match(result.stderr,- /modelsChecked must report the actual updated pricing entry/,+ /modelsChecked must report the actual updated model entry/,);});@@ -146,6 +196,67 @@ test("rejects a generic pull request title", () => {);});+test("revalidates changed-entry evidence from the staged snapshot", () => {+ const result = runValidator({+ basePrices: changedPrices.before,+ currentPrices: changedPrices.after,+ changedPricingJson: true,+ output: {+ pullRequestTitle: "chore(pricing): update gpt-5.5 pricing",+ modelsChecked: [+ {+ ...confirmedChange(modelName, "updated"),+ priceConfirmed: "no",+ },+ ],+ },+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Changed model rows require confirmed prices: gpt-5.5-2026-04-23/,+ );+});++test("rejects pricing-entry removal from the staged snapshot", () => {+ const result = runValidator({+ basePrices: changedPrices.before,+ currentPrices: [],+ changedPricingJson: true,+ output: {+ pullRequestTitle: "chore(pricing): remove gpt-5.5 pricing",+ modelsChecked: [confirmedChange(modelName, "updated")],+ },+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Automated pricing-entry removal is not allowed: gpt-5.5-2026-04-23/,+ );+});++test("rejects selectable-model removal from the staged snapshot", () => {+ const result = runValidator({+ baseTypes: typesSource({ openAI: ["gpt-4o", modelName] }),+ basePrices: changedPrices.before,+ changedModelTypes: true,+ currentTypes: typesSource(),+ output: {+ pullRequestTitle: "chore(pricing): remove gpt-5.5 from playground",+ modelsChecked: [confirmedChange(modelName, "updated")],+ },+ typesDiff: `- "${modelName}",\n`,+ });++ assert.notEqual(result.status, 0);+ assert.match(+ result.stderr,+ /Automated selectable-model removal is not allowed: gpt-5.5-2026-04-23/,+ );+});+test("validates metadata against the exact staged pricing blob", () => {const workflow = fs.readFileSync(workflowPath, "utf8");const prepareStep = workflow.slice(@@ -164,9 +275,7 @@ test("validates metadata against the exact staged pricing blob", () => {stageIndex < metadataIndex,"metadata validation must run after staging",);- assert.match(- prepareStep,- /CURRENT_PRICING_FILE="\$staged_pricing_file"/,- );+ assert.match(prepareStep, /CURRENT_PRICING_FILE="\$staged_pricing_file"/);+ assert.match(prepareStep, /CURRENT_TYPES_FILE="\$staged_types_file"/);assert.doesNotMatch(prepareStep, /checkout -b "\$BOT_BRANCH"/);});diff --git a/scripts/model-price-audit/validate-and-render-audit-output.mjs b/scripts/model-price-audit/validate-and-render-audit-output.mjsnew file mode 100644index 0000000000..94e895ce54--- /dev/null+++ b/scripts/model-price-audit/validate-and-render-audit-output.mjs@@ -0,0 +1,53 @@+#!/usr/bin/env node++import fs from "node:fs";+import {+ reconcileAuditOutput,+ renderAuditSummary,+} from "./audit-output-contract.mjs";++process.on("uncaughtException", (error) => {+ console.error(`::error::${error.message}`);+ process.exit(1);+});++const requiredEnvironment = [+ "BASE_PRICING_FILE",+ "BASE_TYPES_FILE",+ "CURRENT_PRICING_FILE",+ "CURRENT_TYPES_FILE",+ "HAS_REPOSITORY_DIFF",+ "MEMORY_SNAPSHOT_ONLY",+ "STRUCTURED_OUTPUT",+ "STRUCTURED_OUTPUT_PATH",+ "TYPES_DIFF_FILE",+];+for (const name of requiredEnvironment) {+ if (!process.env[name]) {+ throw new Error(`Missing required environment variable: ${name}`);+ }+}++const rawOutput = JSON.parse(process.env.STRUCTURED_OUTPUT);+const basePrices = JSON.parse(+ fs.readFileSync(process.env.BASE_PRICING_FILE, "utf8"),+);+const currentPrices = JSON.parse(+ fs.readFileSync(process.env.CURRENT_PRICING_FILE, "utf8"),+);+const baseTypes = fs.readFileSync(process.env.BASE_TYPES_FILE, "utf8");+const currentTypes = fs.readFileSync(process.env.CURRENT_TYPES_FILE, "utf8");+const typesDiff = fs.readFileSync(process.env.TYPES_DIFF_FILE, "utf8");+const { output, warnings } = reconcileAuditOutput(rawOutput, {+ basePrices,+ currentPrices,+ baseTypes,+ currentTypes,+ hasRepositoryDiff: process.env.HAS_REPOSITORY_DIFF === "true",+ memorySnapshotOnly: process.env.MEMORY_SNAPSHOT_ONLY === "true",+ typesDiff,+});++fs.writeFileSync(process.env.STRUCTURED_OUTPUT_PATH, JSON.stringify(output));+for (const warning of warnings) console.error(`::warning::${warning}`);+process.stdout.write(renderAuditSummary(output));