Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(pricing): reconcile model audit output (#17074)

Hassieb Pakzad40507586a6fb
diff --git a/.github/workflows/model-price-audit.yml b/.github/workflows/model-price-audit.yml
index af24478865..c2c94bb30c 100644
--- a/.github/workflows/model-price-audit.yml
+++ b/.github/workflows/model-price-audit.yml
@@ -70,6 +70,9 @@ jobs:
node-version: "24"
package-manager-cache: false
+ - name: Test audit guardrails
+ run: node --test scripts/model-price-audit/*.test.mjs
+
- name: Validate workflow inputs
id: validate-inputs
env:
@@ -203,7 +206,7 @@ jobs:
6. Record every distinct model price entry you check in `modelsChecked`; do not omit confirmed or unchanged models and do not collapse multiple checked models into one family row.
7. If you learn durable provider-source URLs, pricing-page quirks, model-ID variants, or recurring audit rules that would help future audits, update only the pricing skill reference files under `.agents/skills/add-model-price/references/*.md`.
8. You may replace the optional snapshot in `.agents/skills/add-model-price/references/model-audit-memory.md` with the complete `modelsChecked` table when retaining the current per-model evidence would materially help a future audit. Do not persist a partial table, append unbounded run history, or update the file only to refresh its date.
- 9. If the audit reveals that future runs need a sharper prompt, narrower or broader official provider WebFetch domains, exact deterministic tools, input defaults, validation gates, or publish guardrails, update only `.github/workflows/model-price-audit.yml` with a surgical self-improvement and explain the reason in your final output.
+ 9. If the audit reveals that future runs need a sharper prompt, exact deterministic tools, input defaults, validation gates, or publish guardrails, update only `.github/workflows/model-price-audit.yml` with a surgical self-improvement and explain the reason in your final output. Report required provider-domain changes as unresolved so the fetch and deterministic source allowlists can be reviewed together.
10. Re-run the deterministic validation script before finishing.
11. If you change any `matchPattern`, run the bundled match-pattern tester with representative accepted and rejected model IDs before finishing.
@@ -222,7 +225,7 @@ jobs:
- Do not edit `.agents/skills/add-model-price/SKILL.md` or `.agents/skills/add-model-price/scripts/**`.
- Workflow self-improvements must preserve the schedule/manual triggers, repo guard, read-only audit permissions, explicit read-only GitHub token, separate publish job, secret names, input validation, diff allowlist, hook-disabled commit path, staged-blob checks, artifact handoff, and non-fatal reviewer request.
- Do not grant the audit job write permissions, `id-token: write`, package-manager tools, arbitrary shell tools, arbitrary network tools, `gh`, or `git push`.
- - If official provider source domains change, keep the WebFetch tool allowlist and the audit report's `officialSourceHosts` validation list synchronized.
+ - Do not change official provider WebFetch domains. Report a required domain change as unresolved so the workflow and deterministic source validation can be updated together.
- Keep the first entry in every selectable model array unchanged unless the audit explicitly changes the default model.
- Use the allowed exact `date -u +%Y-%m-%dT00:00:00.000Z` command if you need the current timestamp.
@@ -287,7 +290,7 @@ jobs:
usageKeysChecked: "Not checked",
usageKeyCoverageConfirmed: "no",
tieringChecked: "Not checked",
- tieringCorrect: "not_applicable",
+ tieringCorrect: "no",
change: "none",
officialSources: [],
comments: "Claude and provider-source checks were skipped in dry-run mode.",
@@ -323,6 +326,19 @@ jobs:
echo "JSON"
} >> "$GITHUB_OUTPUT"
+ - name: Re-test audit guardrails after agent edits
+ run: node --test scripts/model-price-audit/*.test.mjs
+
+ - name: Discard pricing-only formatting changes
+ run: |
+ set -euo pipefail
+
+ base_pricing_file="$RUNNER_TEMP/default-model-prices-before-audit.json"
+ git show "HEAD:worker/src/constants/default-model-prices.json" > "$base_pricing_file"
+ node scripts/model-price-audit/discard-formatting-only-pricing-diff.mjs \
+ --base "$base_pricing_file" \
+ --current worker/src/constants/default-model-prices.json
+
- name: Write audit summary
env:
STRUCTURED_OUTPUT: ${{ steps.claude-audit.outputs.structured_output || steps.mock-claude-audit.outputs.structured_output }}
@@ -335,167 +351,27 @@ jobs:
git ls-files --others --exclude-standard
} | sort -u
)
+ HAS_REPOSITORY_DIFF=false
+ if [ "${#changed_files[@]}" -gt 0 ]; then
+ HAS_REPOSITORY_DIFF=true
+ fi
+
MEMORY_SNAPSHOT_ONLY=false
if [ "${#changed_files[@]}" -eq 1 ] && [ "${changed_files[0]}" = ".agents/skills/add-model-price/references/model-audit-memory.md" ]; then
MEMORY_SNAPSHOT_ONLY=true
fi
+ BASE_PRICING_FILE="$RUNNER_TEMP/default-model-prices-before-audit.json"
+ BASE_TYPES_FILE="$RUNNER_TEMP/llm-types-before-audit.ts"
+ git show "HEAD:packages/shared/src/server/llm/types.ts" > "$BASE_TYPES_FILE"
+ CURRENT_PRICING_FILE="worker/src/constants/default-model-prices.json"
+ CURRENT_TYPES_FILE="packages/shared/src/server/llm/types.ts"
STRUCTURED_OUTPUT_PATH="$RUNNER_TEMP/claude-model-price-audit.json"
- export MEMORY_SNAPSHOT_ONLY STRUCTURED_OUTPUT_PATH
- node <<'NODE' | tee "$RUNNER_TEMP/claude-model-price-audit.txt"
- const fs = require("node:fs");
- const output = JSON.parse(process.env.STRUCTURED_OUTPUT);
- if (
- process.env.MEMORY_SNAPSHOT_ONLY === "true" &&
- output.pullRequestTitle.trim() === ""
- ) {
- output.pullRequestTitle =
- "chore(pricing): record model price audit snapshot";
- }
- fs.writeFileSync(process.env.STRUCTURED_OUTPUT_PATH, JSON.stringify(output));
- const escapeCell = (value) =>
- String(value ?? "")
- .replace(/&/g, "&amp;")
- .replace(/</g, "&lt;")
- .replace(/>/g, "&gt;")
- .replace(/\\/g, "\\\\")
- .replace(/\|/g, "\\|")
- .replace(/`/g, "\\`")
- .replace(/\[/g, "\\[")
- .replace(/\]/g, "\\]")
- .replace(/\r?\n/g, "<br>");
- const list = (items) =>
- Array.isArray(items) && items.length > 0
- ? items.map((item) => `- ${String(item).replace(/\r?\n/g, " ")}`).join("\n")
- : "- None";
- const officialSourceHosts = [
- "ai.google.dev",
- "aws.amazon.com",
- "azure.microsoft.com",
- "cloud.google.com",
- "developers.openai.com",
- "docs.anthropic.com",
- "platform.claude.com",
- ];
- const sourceLinks = (sources, model) => {
- if (!Array.isArray(sources)) {
- throw new Error(`officialSources must be an array for ${model}`);
- }
-
- return sources.length > 0
- ? sources
- .map((source, index) => {
- const url = new URL(source);
- if (url.protocol !== "https:") {
- throw new Error(`Unsupported source URL protocol for ${model}: ${source}`);
- }
- if (
- !officialSourceHosts.some(
- (host) => url.hostname === host || url.hostname.endsWith(`.${host}`),
- )
- ) {
- throw new Error(`Source URL is not on an approved official domain for ${model}: ${source}`);
- }
- const href = url.href.replace(/\(/g, "%28").replace(/\)/g, "%29");
- return `[${index + 1}](${href})`;
- })
- .join(" ")
- : "—";
- };
-
- if (!Array.isArray(output.modelsChecked) || output.modelsChecked.length === 0) {
- throw new Error("modelsChecked must contain every model price checked during the audit");
- }
-
- const modelKeys = new Set();
- const modelRows = output.modelsChecked.map((item) => {
- const key = `${item.provider}\u0000${item.model}`.toLowerCase();
- if (modelKeys.has(key)) {
- throw new Error(`Duplicate modelsChecked row: ${item.provider} / ${item.model}`);
- }
- modelKeys.add(key);
-
- if (
- (item.priceConfirmed === "yes" ||
- item.usageKeyCoverageConfirmed === "yes" ||
- item.tieringCorrect === "yes") &&
- item.officialSources.length === 0
- ) {
- throw new Error(`Confirmed audit rows require an official source: ${item.model}`);
- }
- if (
- (item.priceConfirmed === "no" ||
- item.usageKeyCoverageConfirmed === "no" ||
- item.tieringCorrect === "no") &&
- !item.comments.trim()
- ) {
- throw new Error(`Unconfirmed audit rows require comments: ${item.model}`);
- }
- if (
- ["added", "updated"].includes(item.change) &&
- item.usageKeyCoverageConfirmed !== "yes"
- ) {
- throw new Error(`Changed model rows require confirmed usage-key coverage: ${item.model}`);
- }
-
- const priceConfirmed = item.priceConfirmed === "yes" ? "Yes" : "No";
- const usageKeyCoverageConfirmed =
- item.usageKeyCoverageConfirmed === "yes" ? "Yes" : "No";
- const tieringCorrect =
- item.tieringCorrect === "not_applicable"
- ? "N/A"
- : item.tieringCorrect === "yes"
- ? "Yes"
- : "No";
- const changeLabels = {
- none: "None",
- updated: "Updated",
- added: "Added",
- unresolved: "Unresolved",
- };
-
- return `| ${escapeCell(item.provider)} | ${escapeCell(item.model)} | ${escapeCell(item.pricingChecked)} | ${priceConfirmed} | ${escapeCell(item.usageKeysChecked)} | ${usageKeyCoverageConfirmed} | ${escapeCell(item.tieringChecked)} | ${tieringCorrect} | ${changeLabels[item.change]} | ${sourceLinks(item.officialSources, item.model)} | ${escapeCell(item.comments) || "—"} |`;
- });
- const proposedTitle = output.pullRequestTitle
- ? `\`${output.pullRequestTitle.replace(/`/g, "\\`")}\``
- : "_No repository diff proposed_";
-
- process.stdout.write(
- [
- `**Audit date:** ${output.auditDate}`,
- `**Proposed pull request title:** ${proposedTitle}`,
- "",
- output.summary,
- "",
- "### Models checked",
- "",
- "| Provider | Model / pricing entry | Pricing checked | Price confirmed | Usage keys checked | Usage-key coverage | Tiering checked | Tiering correct | Change | Official source(s) | Comments |",
- "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",
- ...modelRows,
- "",
- "### Changed models",
- "",
- list(output.changedModels),
- "",
- "### Skill reference updates",
- "",
- list(output.skillReferenceUpdates),
- "",
- "### Workflow updates",
- "",
- list(output.workflowUpdates),
- "",
- "### Unresolved findings",
- "",
- list(output.unresolvedFindings),
- "",
- "### Validation",
- "",
- list(output.validation),
- "",
- ].join("\n"),
- );
- NODE
+ TYPES_DIFF_FILE="$RUNNER_TEMP/llm-types-before-prepare.diff"
+ git diff --unified=0 -- "packages/shared/src/server/llm/types.ts" > "$TYPES_DIFF_FILE"
+ export BASE_PRICING_FILE BASE_TYPES_FILE CURRENT_PRICING_FILE CURRENT_TYPES_FILE HAS_REPOSITORY_DIFF MEMORY_SNAPSHOT_ONLY STRUCTURED_OUTPUT_PATH TYPES_DIFF_FILE
+ node scripts/model-price-audit/validate-and-render-audit-output.mjs \
+ | tee "$RUNNER_TEMP/claude-model-price-audit.txt"
{
echo "## Model price audit summary"
@@ -503,16 +379,6 @@ jobs:
cat "$RUNNER_TEMP/claude-model-price-audit.txt"
} >> "$GITHUB_STEP_SUMMARY"
- - name: Discard pricing-only formatting changes
- run: |
- set -euo pipefail
-
- base_pricing_file="$RUNNER_TEMP/default-model-prices-before-audit.json"
- git show "HEAD:worker/src/constants/default-model-prices.json" > "$base_pricing_file"
- node scripts/model-price-audit/discard-formatting-only-pricing-diff.mjs \
- --base "$base_pricing_file" \
- --current worker/src/constants/default-model-prices.json
-
- name: Validate audit diff
run: |
set -euo pipefail
@@ -646,15 +512,20 @@ jobs:
staged_pricing_file="$RUNNER_TEMP/default-model-prices-staged.json"
git show ":worker/src/constants/default-model-prices.json" > "$staged_pricing_file"
+ staged_types_file="$RUNNER_TEMP/llm-types-staged.ts"
+ git show ":packages/shared/src/server/llm/types.ts" > "$staged_types_file"
git diff --cached --binary > "$RUNNER_TEMP/model-price-audit-diagnostic.patch"
test -s "$RUNNER_TEMP/model-price-audit-diagnostic.patch"
CURRENT_PRICING_FILE="$staged_pricing_file"
+ BASE_TYPES_FILE="$RUNNER_TEMP/llm-types-base.ts"
+ git show "HEAD:packages/shared/src/server/llm/types.ts" > "$BASE_TYPES_FILE"
+ CURRENT_TYPES_FILE="$staged_types_file"
TYPES_DIFF_FILE="$RUNNER_TEMP/llm-types.diff"
git diff --cached --unified=0 -- "packages/shared/src/server/llm/types.ts" > "$TYPES_DIFF_FILE"
STRUCTURED_OUTPUT_PATH="$RUNNER_TEMP/claude-model-price-audit.json"
PR_TITLE_PATH="$RUNNER_TEMP/model-price-audit-pr-title.txt"
- export BASE_PRICING_FILE CHANGED_MODEL_TYPES CHANGED_PRICING_JSON CURRENT_PRICING_FILE STRUCTURED_OUTPUT_PATH TYPES_DIFF_FILE
+ export BASE_PRICING_FILE BASE_TYPES_FILE CHANGED_MODEL_TYPES CHANGED_PRICING_JSON CURRENT_PRICING_FILE CURRENT_TYPES_FILE STRUCTURED_OUTPUT_PATH TYPES_DIFF_FILE
node scripts/model-price-audit/prepare-metadata.mjs > "$PR_TITLE_PATH"
PR_TITLE="$(cat "$PR_TITLE_PATH")"
diff --git a/scripts/model-price-audit/audit-output-contract.mjs b/scripts/model-price-audit/audit-output-contract.mjs
new file mode 100644
index 0000000000..367835a7b8
--- /dev/null
+++ b/scripts/model-price-audit/audit-output-contract.mjs
@@ -0,0 +1,510 @@
+const officialSourceHosts = [
+ "ai.google.dev",
+ "aws.amazon.com",
+ "azure.microsoft.com",
+ "cloud.google.com",
+ "developers.openai.com",
+ "docs.anthropic.com",
+ "platform.claude.com",
+];
+
+const normalize = (value) => value.trim().toLowerCase();
+
+export const normalizeReportedModel = (value) =>
+ normalize(value).replace(/\s+\((?:also\s+)?matches\b[^)]*\)$/u, "");
+
+const canonicalize = (value) => {
+ if (Array.isArray(value)) return value.map(canonicalize);
+ if (value !== null && typeof value === "object") {
+ return Object.fromEntries(
+ Object.keys(value)
+ .sort()
+ .map((key) => [key, canonicalize(value[key])]),
+ );
+ }
+ return value;
+};
+
+const indexUniquePricingModels = (prices, label) => {
+ const modelsByName = new Map();
+ for (const item of prices) {
+ const key = normalize(item.modelName);
+ if (modelsByName.has(key)) {
+ throw new Error(
+ `${label} pricing contains duplicate normalized modelName: ${item.modelName}`,
+ );
+ }
+ modelsByName.set(key, item);
+ }
+ return modelsByName;
+};
+
+export function collectPricingModelChanges(basePrices, currentPrices) {
+ const baseByName = indexUniquePricingModels(basePrices, "Base");
+ const currentByName = indexUniquePricingModels(currentPrices, "Current");
+ const modelNames = Array.from(baseByName.keys()).concat(
+ Array.from(currentByName.keys()),
+ );
+ const changes = [];
+
+ for (const modelName of new Set(modelNames)) {
+ const before = baseByName.get(modelName);
+ const after = currentByName.get(modelName);
+ if (!after) {
+ changes.push({ modelName: before.modelName, expectedChange: "removed" });
+ } else if (!before) {
+ changes.push({ modelName: after.modelName, expectedChange: "added" });
+ } else if (
+ JSON.stringify(canonicalize(before)) !==
+ JSON.stringify(canonicalize(after))
+ ) {
+ changes.push({ modelName: after.modelName, expectedChange: "updated" });
+ }
+ }
+
+ return changes;
+}
+
+const selectableModelArrays = [
+ {
+ name: "openAIModels",
+ pattern: /export const openAIModels = \[([\s\S]*?)\] as const;/u,
+ provider: "OpenAI",
+ },
+ {
+ name: "anthropicModels",
+ pattern: /export const anthropicModels = \[([\s\S]*?)\] as const;/u,
+ provider: "Anthropic",
+ },
+ {
+ name: "vertexAIModels",
+ pattern: /export const vertexAIModels = \[([\s\S]*?)\] as const;/u,
+ provider: "Google",
+ },
+ {
+ name: "googleAIStudioModels",
+ pattern: /export const googleAIStudioModels = \[([\s\S]*?)\] as const;/u,
+ provider: "Google",
+ },
+];
+
+const readSelectableModelArrays = (source, label) =>
+ new Map(
+ selectableModelArrays.map(({ name, pattern, provider }) => {
+ const match = source.match(pattern);
+ if (!match) {
+ throw new Error(`${label} types file is missing ${name}`);
+ }
+ const models = match[1]
+ .split("\n")
+ .map((line) => line.trim())
+ .filter((line) => line && !line.startsWith("//"))
+ .map((line) => {
+ const modelMatch = line.match(/^"([^"]+)",(?:\s*\/\/.*)?$/u);
+ if (!modelMatch) {
+ throw new Error(
+ `${label} ${name} contains an unsupported entry: ${line}`,
+ );
+ }
+ return modelMatch[1];
+ });
+ const normalizedModels = models.map(normalize);
+ if (new Set(normalizedModels).size !== normalizedModels.length) {
+ throw new Error(`${label} ${name} contains duplicate model entries`);
+ }
+ return [name, { models, provider }];
+ }),
+ );
+
+export function collectTypeModelChanges(baseTypes, currentTypes, typesDiff) {
+ const baseArrays = readSelectableModelArrays(baseTypes, "Base");
+ const currentArrays = readSelectableModelArrays(currentTypes, "Current");
+ const changes = [];
+
+ for (const { name } of selectableModelArrays) {
+ const before = baseArrays.get(name);
+ const after = currentArrays.get(name);
+ const currentModelNames = new Set(after.models.map(normalize));
+ const removedModels = before.models.filter(
+ (modelName) => !currentModelNames.has(normalize(modelName)),
+ );
+ if (removedModels.length > 0) {
+ throw new Error(
+ `Automated selectable-model removal is not allowed: ${removedModels.join(", ")}`,
+ );
+ }
+ if (before.models[0] !== after.models[0]) {
+ throw new Error(`${name} must keep its first default model unchanged`);
+ }
+
+ let baseIndex = 0;
+ const baseModelNames = new Set(before.models.map(normalize));
+ for (const modelName of after.models) {
+ if (normalize(before.models[baseIndex] ?? "") === normalize(modelName)) {
+ baseIndex += 1;
+ } else if (baseModelNames.has(normalize(modelName))) {
+ throw new Error(`${name} may not reorder existing model entries`);
+ } else {
+ changes.push({
+ arrayName: name,
+ expectedChange: "added",
+ modelName,
+ provider: after.provider,
+ });
+ }
+ }
+ }
+
+ const additionsFromDiff = [];
+ for (const line of typesDiff.split("\n")) {
+ if (line.startsWith("+++") || line.startsWith("---")) continue;
+ if (!line.startsWith("+") && !line.startsWith("-")) continue;
+
+ const match = line.match(/^([+-])\s*"([^"]+)",(?:\s*\/\/.*)?$/u);
+ if (!match) {
+ throw new Error(
+ `Selectable-model diff contains an unsupported changed line: ${line}`,
+ );
+ }
+
+ const [, sign, modelName] = match;
+ if (sign === "-") {
+ throw new Error(
+ `Automated selectable-model removal is not allowed: ${modelName}`,
+ );
+ }
+ additionsFromDiff.push(normalize(modelName));
+ }
+
+ const semanticAdditions = changes.map((change) =>
+ normalize(change.modelName),
+ );
+ additionsFromDiff.sort();
+ semanticAdditions.sort();
+ if (JSON.stringify(additionsFromDiff) !== JSON.stringify(semanticAdditions)) {
+ throw new Error(
+ "Selectable-model diff does not match additive changes in the approved model arrays",
+ );
+ }
+
+ return changes;
+}
+
+export function mergeModelChanges(pricingChanges, typeModelChanges) {
+ const changesByModel = new Map();
+ for (const change of typeModelChanges) {
+ changesByModel.set(normalize(change.modelName), change);
+ }
+ for (const change of pricingChanges) {
+ changesByModel.set(normalize(change.modelName), change);
+ }
+ return Array.from(changesByModel.values());
+}
+
+export function validateOfficialSources(sources, model) {
+ if (!Array.isArray(sources)) {
+ throw new Error(`officialSources must be an array for ${model}`);
+ }
+
+ for (const source of sources) {
+ const url = new URL(source);
+ if (url.protocol !== "https:") {
+ throw new Error(
+ `Unsupported source URL protocol for ${model}: ${source}`,
+ );
+ }
+ if (
+ !officialSourceHosts.some(
+ (host) => url.hostname === host || url.hostname.endsWith(`.${host}`),
+ )
+ ) {
+ throw new Error(
+ `Source URL is not on an approved official domain for ${model}: ${source}`,
+ );
+ }
+ }
+}
+
+export function validateChangedModelRow(row) {
+ if (row.priceConfirmed !== "yes") {
+ throw new Error(
+ `Changed model rows require confirmed prices: ${row.model}`,
+ );
+ }
+ if (row.usageKeyCoverageConfirmed !== "yes") {
+ throw new Error(
+ `Changed model rows require confirmed usage-key coverage: ${row.model}`,
+ );
+ }
+ if (!["yes", "not_applicable"].includes(row.tieringCorrect)) {
+ throw new Error(
+ `Changed model rows require confirmed or inapplicable tiering: ${row.model}`,
+ );
+ }
+ if (row.officialSources.length === 0) {
+ throw new Error(
+ `Changed model rows require an official source: ${row.model}`,
+ );
+ }
+}
+
+const appendComment = (row, message) => {
+ const separator = row.comments.trim() ? " " : "";
+ row.comments = `${row.comments.trim()}${separator}${message}`;
+};
+
+export function reconcileAuditOutput(
+ rawOutput,
+ {
+ basePrices,
+ currentPrices,
+ baseTypes,
+ currentTypes,
+ hasRepositoryDiff,
+ memorySnapshotOnly,
+ typesDiff,
+ },
+) {
+ const output = structuredClone(rawOutput);
+ const warnings = [];
+
+ if (
+ !Array.isArray(output.modelsChecked) ||
+ output.modelsChecked.length === 0
+ ) {
+ throw new Error(
+ "modelsChecked must contain every model price checked during the audit",
+ );
+ }
+ if (!Array.isArray(output.unresolvedFindings)) {
+ throw new Error("unresolvedFindings must be an array");
+ }
+
+ if (memorySnapshotOnly && output.pullRequestTitle.trim() === "") {
+ output.pullRequestTitle =
+ "chore(pricing): record model price audit snapshot";
+ }
+
+ const pricingChanges = collectPricingModelChanges(basePrices, currentPrices);
+ const typeModelChanges = collectTypeModelChanges(
+ baseTypes,
+ currentTypes,
+ typesDiff,
+ );
+ const removedPricingModels = pricingChanges.filter(
+ (change) => change.expectedChange === "removed",
+ );
+ if (removedPricingModels.length > 0) {
+ throw new Error(
+ `Automated pricing-entry removal is not allowed: ${removedPricingModels
+ .map((change) => change.modelName)
+ .join(", ")}`,
+ );
+ }
+ const expectedModelChanges = mergeModelChanges(
+ pricingChanges,
+ typeModelChanges,
+ );
+ const changedModelKeys = new Set(
+ expectedModelChanges.map((change) => normalize(change.modelName)),
+ );
+
+ const modelKeys = new Set();
+ for (const row of output.modelsChecked) {
+ const key = `${row.provider}\u0000${row.model}`.toLowerCase();
+ if (modelKeys.has(key)) {
+ throw new Error(
+ `Duplicate modelsChecked row: ${row.provider} / ${row.model}`,
+ );
+ }
+ modelKeys.add(key);
+ validateOfficialSources(row.officialSources, row.model);
+
+ const unsupportedConfirmations = [];
+ if (row.officialSources.length === 0) {
+ if (row.priceConfirmed === "yes") unsupportedConfirmations.push("price");
+ if (row.usageKeyCoverageConfirmed === "yes") {
+ unsupportedConfirmations.push("usage-key coverage");
+ }
+ if (["yes", "not_applicable"].includes(row.tieringCorrect)) {
+ unsupportedConfirmations.push("tiering applicability");
+ }
+ }
+
+ if (unsupportedConfirmations.length > 0) {
+ if (changedModelKeys.has(normalizeReportedModel(row.model))) {
+ throw new Error(
+ `Confirmed audit rows require an official source: ${row.model}`,
+ );
+ }
+ if (row.priceConfirmed === "yes") row.priceConfirmed = "no";
+ if (row.usageKeyCoverageConfirmed === "yes") {
+ row.usageKeyCoverageConfirmed = "no";
+ }
+ if (["yes", "not_applicable"].includes(row.tieringCorrect)) {
+ row.tieringCorrect = "no";
+ }
+ const warning = `${row.model}: ${unsupportedConfirmations.join(
+ ", ",
+ )} confirmation downgraded because no approved official source was provided.`;
+ appendComment(row, warning);
+ output.unresolvedFindings.push(warning);
+ warnings.push(warning);
+ }
+
+ if (
+ (row.priceConfirmed === "no" ||
+ row.usageKeyCoverageConfirmed === "no" ||
+ row.tieringCorrect === "no") &&
+ !row.comments.trim()
+ ) {
+ throw new Error(`Unconfirmed audit rows require comments: ${row.model}`);
+ }
+ }
+
+ const rowsByModel = new Map();
+ for (const row of output.modelsChecked) {
+ const key = normalizeReportedModel(row.model);
+ rowsByModel.set(key, [...(rowsByModel.get(key) ?? []), row]);
+ }
+ const expectedChangesByModel = new Map(
+ expectedModelChanges.map((change) => [normalize(change.modelName), change]),
+ );
+
+ for (const change of typeModelChanges) {
+ const matchingRows = rowsByModel.get(normalize(change.modelName)) ?? [];
+ if (
+ matchingRows.length === 1 &&
+ normalize(matchingRows[0].provider) !== normalize(change.provider)
+ ) {
+ throw new Error(
+ `${change.arrayName} additions require provider ${change.provider}: ${change.modelName}`,
+ );
+ }
+ }
+
+ for (const change of expectedChangesByModel.values()) {
+ const matchingRows = rowsByModel.get(normalize(change.modelName)) ?? [];
+ if (matchingRows.length === 0) {
+ throw new Error(
+ `modelsChecked must report the actual ${change.expectedChange} model entry: ${change.modelName}`,
+ );
+ }
+ if (matchingRows.length > 1) {
+ throw new Error(
+ `modelsChecked must report exactly one row for changed model: ${change.modelName}`,
+ );
+ }
+ const [row] = matchingRows;
+ row.change = change.expectedChange;
+ validateChangedModelRow(row);
+ }
+
+ for (const row of output.modelsChecked) {
+ if (!["added", "updated"].includes(row.change)) continue;
+ if (expectedChangesByModel.has(normalizeReportedModel(row.model))) continue;
+
+ if (hasRepositoryDiff) {
+ throw new Error(
+ `modelsChecked reports a model without a matching pricing or selectable-model diff: ${row.model}`,
+ );
+ }
+ warnings.push(
+ `${row.model}: reported ${row.change} without a repository diff; change reset to none.`,
+ );
+ row.change = "none";
+ }
+
+ output.changedModels = expectedModelChanges.map(
+ (change) => `${change.modelName} (${change.expectedChange})`,
+ );
+
+ return { output, pricingChanges, typeModelChanges, warnings };
+}
+
+const escapeCell = (value) =>
+ String(value ?? "")
+ .replace(/&/g, "&amp;")
+ .replace(/</g, "&lt;")
+ .replace(/>/g, "&gt;")
+ .replace(/\\/g, "\\\\")
+ .replace(/\|/g, "\\|")
+ .replace(/`/g, "\\`")
+ .replace(/\[/g, "\\[")
+ .replace(/\]/g, "\\]")
+ .replace(/\r?\n/g, "<br>");
+
+const list = (items) =>
+ Array.isArray(items) && items.length > 0
+ ? items.map((item) => `- ${String(item).replace(/\r?\n/g, " ")}`).join("\n")
+ : "- None";
+
+const sourceLinks = (sources) =>
+ sources.length > 0
+ ? sources
+ .map((source, index) => {
+ const href = new URL(source).href
+ .replace(/\(/g, "%28")
+ .replace(/\)/g, "%29");
+ return `[${index + 1}](${href})`;
+ })
+ .join(" ")
+ : "—";
+
+export function renderAuditSummary(output) {
+ const changeLabels = {
+ none: "None",
+ updated: "Updated",
+ added: "Added",
+ unresolved: "Unresolved",
+ };
+ const modelRows = output.modelsChecked.map((row) => {
+ const priceConfirmed = row.priceConfirmed === "yes" ? "Yes" : "No";
+ const usageKeyCoverageConfirmed =
+ row.usageKeyCoverageConfirmed === "yes" ? "Yes" : "No";
+ const tieringCorrect =
+ row.tieringCorrect === "not_applicable"
+ ? "N/A"
+ : row.tieringCorrect === "yes"
+ ? "Yes"
+ : "No";
+ return `| ${escapeCell(row.provider)} | ${escapeCell(row.model)} | ${escapeCell(row.pricingChecked)} | ${priceConfirmed} | ${escapeCell(row.usageKeysChecked)} | ${usageKeyCoverageConfirmed} | ${escapeCell(row.tieringChecked)} | ${tieringCorrect} | ${changeLabels[row.change]} | ${sourceLinks(row.officialSources)} | ${escapeCell(row.comments) || "—"} |`;
+ });
+ const proposedTitle = output.pullRequestTitle
+ ? `\`${output.pullRequestTitle.replace(/\\/g, "\\\\").replace(/`/g, "\\`")}\``
+ : "_No repository diff proposed_";
+
+ return [
+ `**Audit date:** ${output.auditDate}`,
+ `**Proposed pull request title:** ${proposedTitle}`,
+ "",
+ output.summary,
+ "",
+ "### Models checked",
+ "",
+ "| Provider | Model / pricing entry | Pricing checked | Price confirmed | Usage keys checked | Usage-key coverage | Tiering checked | Tiering correct | Change | Official source(s) | Comments |",
+ "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",
+ ...modelRows,
+ "",
+ "### Changed models",
+ "",
+ list(output.changedModels),
+ "",
+ "### Skill reference updates",
+ "",
+ list(output.skillReferenceUpdates),
+ "",
+ "### Workflow updates",
+ "",
+ list(output.workflowUpdates),
+ "",
+ "### Unresolved findings",
+ "",
+ list(output.unresolvedFindings),
+ "",
+ "### Validation",
+ "",
+ list(output.validation),
+ "",
+ ].join("\n");
+}
diff --git a/scripts/model-price-audit/audit-output-contract.test.mjs b/scripts/model-price-audit/audit-output-contract.test.mjs
new file mode 100644
index 0000000000..459c70dab0
--- /dev/null
+++ b/scripts/model-price-audit/audit-output-contract.test.mjs
@@ -0,0 +1,497 @@
+import assert from "node:assert/strict";
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { spawnSync } from "node:child_process";
+import test from "node:test";
+import {
+ collectTypeModelChanges,
+ renderAuditSummary,
+} from "./audit-output-contract.mjs";
+
+const scriptPath = path.join(
+ import.meta.dirname,
+ "validate-and-render-audit-output.mjs",
+);
+const workflowPath = path.join(
+ import.meta.dirname,
+ "../../.github/workflows/model-price-audit.yml",
+);
+const repositoryTypesPath = path.join(
+ import.meta.dirname,
+ "../../packages/shared/src/server/llm/types.ts",
+);
+
+const officialSource = "https://developers.openai.com/api/docs/pricing";
+
+const pricingEntry = (modelName, inputPrice = 1) => ({
+ modelName,
+ pricingTiers: [{ prices: { input: inputPrice } }],
+});
+
+const auditRow = ({
+ model,
+ change = "none",
+ priceConfirmed = "yes",
+ usageKeyCoverageConfirmed = "yes",
+ tieringCorrect = "not_applicable",
+ officialSources = [officialSource],
+ comments = "Confirmed from the official pricing page.",
+}) => ({
+ provider: "OpenAI",
+ model,
+ pricingChecked: "Input pricing",
+ priceConfirmed,
+ usageKeysChecked: "input",
+ usageKeyCoverageConfirmed,
+ tieringChecked: "No tiering applies",
+ tieringCorrect,
+ change,
+ officialSources,
+ comments,
+});
+
+const structuredOutput = (modelsChecked) => ({
+ summary: "Audit complete.",
+ auditDate: "2026-09-04",
+ pullRequestTitle: "",
+ modelsChecked,
+ changedModels: [],
+ skillReferenceUpdates: [],
+ workflowUpdates: [],
+ unresolvedFindings: [],
+ validation: [],
+});
+
+const typesSource = ({
+ anthropic = ["claude-3"],
+ googleAIStudio = ["gemini-2"],
+ openAI = ["gpt-4o"],
+ vertexAI = ["gemini-2"],
+} = {}) => `
+export const openAIModels = [
+${openAI.map((model) => ` "${model}",`).join("\n")}
+] as const;
+export const anthropicModels = [
+${anthropic.map((model) => ` "${model}",`).join("\n")}
+] as const;
+export const vertexAIModels = [
+${vertexAI.map((model) => ` "${model}",`).join("\n")}
+] as const;
+export const googleAIStudioModels = [
+${googleAIStudio.map((model) => ` "${model}",`).join("\n")}
+] as const;
+`;
+
+function runContract({
+ baseTypes = typesSource(),
+ basePrices = [],
+ currentTypes = baseTypes,
+ currentPrices = basePrices,
+ output,
+ typesDiff = "",
+}) {
+ const fixtureDirectory = fs.mkdtempSync(
+ path.join(os.tmpdir(), "model-price-audit-output-"),
+ );
+ const basePath = path.join(fixtureDirectory, "base.json");
+ const currentPath = path.join(fixtureDirectory, "current.json");
+ const baseTypesPath = path.join(fixtureDirectory, "base-types.ts");
+ const currentTypesPath = path.join(fixtureDirectory, "current-types.ts");
+ const normalizedOutputPath = path.join(fixtureDirectory, "output.json");
+ const typesDiffPath = path.join(fixtureDirectory, "types.diff");
+ fs.writeFileSync(basePath, JSON.stringify(basePrices));
+ fs.writeFileSync(currentPath, JSON.stringify(currentPrices));
+ fs.writeFileSync(baseTypesPath, baseTypes);
+ fs.writeFileSync(currentTypesPath, currentTypes);
+ fs.writeFileSync(typesDiffPath, typesDiff);
+
+ const result = spawnSync(process.execPath, [scriptPath], {
+ encoding: "utf8",
+ env: {
+ ...process.env,
+ BASE_PRICING_FILE: basePath,
+ BASE_TYPES_FILE: baseTypesPath,
+ CURRENT_PRICING_FILE: currentPath,
+ CURRENT_TYPES_FILE: currentTypesPath,
+ HAS_REPOSITORY_DIFF: String(
+ JSON.stringify(basePrices) !== JSON.stringify(currentPrices) ||
+ typesDiff.length > 0,
+ ),
+ MEMORY_SNAPSHOT_ONLY: "false",
+ STRUCTURED_OUTPUT: JSON.stringify(output),
+ STRUCTURED_OUTPUT_PATH: normalizedOutputPath,
+ TYPES_DIFF_FILE: typesDiffPath,
+ },
+ });
+ const normalizedOutput = fs.existsSync(normalizedOutputPath)
+ ? JSON.parse(fs.readFileSync(normalizedOutputPath, "utf8"))
+ : null;
+ fs.rmSync(fixtureDirectory, { force: true, recursive: true });
+ return { normalizedOutput, result };
+}
+
+test("downgrades unsupported confirmation claims on a no-diff audit", () => {
+ const model = "gpt-4o-2024-05-13";
+ const { normalizedOutput, result } = runContract({
+ output: structuredOutput([
+ auditRow({
+ model,
+ priceConfirmed: "no",
+ usageKeyCoverageConfirmed: "yes",
+ officialSources: [],
+ comments: "Not independently re-fetched during this run.",
+ }),
+ ]),
+ });
+
+ assert.equal(result.status, 0, result.stderr);
+ assert.equal(
+ normalizedOutput.modelsChecked[0].usageKeyCoverageConfirmed,
+ "no",
+ );
+ assert.equal(normalizedOutput.modelsChecked[0].tieringCorrect, "no");
+ assert.match(
+ normalizedOutput.modelsChecked[0].comments,
+ /downgraded because no approved official source was provided/i,
+ );
+ assert.match(normalizedOutput.unresolvedFindings[0], new RegExp(model));
+ assert.match(result.stderr, /::warning::/);
+ assert.match(result.stdout, /gpt-4o-2024-05-13/);
+});
+
+test("rejects a changed pricing entry without confirmed prices", () => {
+ const model = "gpt-6-astra";
+ const { result } = runContract({
+ basePrices: [],
+ currentPrices: [pricingEntry(model)],
+ output: {
+ ...structuredOutput([
+ auditRow({ model, change: "added", priceConfirmed: "no" }),
+ ]),
+ pullRequestTitle: "chore(pricing): add gpt-6-astra pricing",
+ },
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Changed model rows require confirmed prices: gpt-6-astra/,
+ );
+});
+
+test("derives changed pricing metadata from the semantic diff", () => {
+ const model = "gpt-6-astra";
+ const { normalizedOutput, result } = runContract({
+ basePrices: [],
+ currentPrices: [pricingEntry(model)],
+ output: {
+ ...structuredOutput([auditRow({ model, change: "none" })]),
+ pullRequestTitle: "chore(pricing): add gpt-6-astra pricing",
+ },
+ });
+
+ assert.equal(result.status, 0, result.stderr);
+ assert.equal(normalizedOutput.modelsChecked[0].change, "added");
+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (added)`]);
+});
+
+test("rejects automated pricing-entry removal", () => {
+ const model = "gpt-4o";
+ const { result } = runContract({
+ basePrices: [pricingEntry(model)],
+ currentPrices: [],
+ output: {
+ ...structuredOutput([auditRow({ model, change: "updated" })]),
+ pullRequestTitle: "chore(pricing): remove gpt-4o pricing",
+ },
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Automated pricing-entry removal is not allowed: gpt-4o/,
+ );
+});
+
+test("ignores object-key ordering when deriving changed pricing metadata", () => {
+ const unchangedBefore = {
+ modelName: "gpt-4o",
+ pricingTiers: [{ prices: { input: 1, output: 2 }, priority: 0 }],
+ };
+ const unchangedAfter = {
+ pricingTiers: [{ priority: 0, prices: { output: 2, input: 1 } }],
+ modelName: "gpt-4o",
+ };
+ const changedBefore = pricingEntry("gpt-6-astra", 1);
+ const changedAfter = pricingEntry("gpt-6-astra", 2);
+ const { normalizedOutput, result } = runContract({
+ basePrices: [unchangedBefore, changedBefore],
+ currentPrices: [unchangedAfter, changedAfter],
+ output: {
+ ...structuredOutput([
+ auditRow({ model: "gpt-4o" }),
+ auditRow({ model: "gpt-6-astra", change: "updated" }),
+ ]),
+ pullRequestTitle: "chore(pricing): update gpt-6-astra pricing",
+ },
+ });
+
+ assert.equal(result.status, 0, result.stderr);
+ assert.deepEqual(normalizedOutput.changedModels, ["gpt-6-astra (updated)"]);
+});
+
+test("reconciles a selectable-model-only addition", () => {
+ const model = "gpt-6-astra";
+ const { normalizedOutput, result } = runContract({
+ basePrices: [pricingEntry(model)],
+ currentTypes: typesSource({ openAI: ["gpt-4o", model] }),
+ output: {
+ ...structuredOutput([auditRow({ model, change: "none" })]),
+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",
+ },
+ typesDiff: `+ "${model}",\n`,
+ });
+
+ assert.equal(result.status, 0, result.stderr);
+ assert.equal(normalizedOutput.modelsChecked[0].change, "added");
+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (added)`]);
+});
+
+test("rejects changing a selectable array default model", () => {
+ const model = "gpt-6-astra";
+ const { result } = runContract({
+ basePrices: [pricingEntry(model)],
+ currentTypes: typesSource({ openAI: [model, "gpt-4o"] }),
+ output: {
+ ...structuredOutput([auditRow({ model, change: "added" })]),
+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",
+ },
+ typesDiff: `+ "${model}",\n`,
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /openAIModels must keep its first default model unchanged/,
+ );
+});
+
+test("rejects a selectable model added to the wrong provider array", () => {
+ const model = "gpt-6-astra";
+ const { result } = runContract({
+ basePrices: [pricingEntry(model)],
+ currentTypes: typesSource({ anthropic: ["claude-3", model] }),
+ output: {
+ ...structuredOutput([auditRow({ model, change: "added" })]),
+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",
+ },
+ typesDiff: `+ "${model}",\n`,
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /anthropicModels additions require provider Anthropic: gpt-6-astra/,
+ );
+});
+
+test("uses the pricing delta when the same model also becomes selectable", () => {
+ const model = "gpt-6-astra";
+ const { normalizedOutput, result } = runContract({
+ basePrices: [pricingEntry(model, 1)],
+ currentPrices: [pricingEntry(model, 2)],
+ currentTypes: typesSource({ openAI: ["gpt-4o", model] }),
+ output: {
+ ...structuredOutput([auditRow({ model, change: "added" })]),
+ pullRequestTitle: "chore(pricing): update and expose gpt-6-astra",
+ },
+ typesDiff: `+ "${model}",\n`,
+ });
+
+ assert.equal(result.status, 0, result.stderr);
+ assert.equal(normalizedOutput.modelsChecked[0].change, "updated");
+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (updated)`]);
+});
+
+test("rejects automated selectable-model removal", () => {
+ const model = "gpt-4o";
+ const { result } = runContract({
+ baseTypes: typesSource({ openAI: ["gpt-4.1", model, "gpt-5"] }),
+ basePrices: [pricingEntry(model)],
+ currentTypes: typesSource({ openAI: ["gpt-4.1", "gpt-5"] }),
+ output: {
+ ...structuredOutput([auditRow({ model, change: "updated" })]),
+ pullRequestTitle: "chore(pricing): remove gpt-4o from playground",
+ },
+ typesDiff: `- "${model}",\n`,
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Automated selectable-model removal is not allowed: gpt-4o/,
+ );
+});
+
+test("rejects remove-plus-add selectable-model edits", () => {
+ const model = "gpt-4o";
+ const { result } = runContract({
+ basePrices: [pricingEntry(model)],
+ output: {
+ ...structuredOutput([auditRow({ model, change: "updated" })]),
+ pullRequestTitle: "chore(pricing): reorder gpt-4o in playground",
+ },
+ typesDiff: `- "${model}",\n+ "${model}",\n`,
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Automated selectable-model removal is not allowed: gpt-4o/,
+ );
+});
+
+test("rejects unrelated changes in the selectable-model file", () => {
+ const model = "gpt-6-astra";
+ const { result } = runContract({
+ basePrices: [pricingEntry(model)],
+ currentTypes: typesSource({ openAI: ["gpt-4o", model] }),
+ output: {
+ ...structuredOutput([auditRow({ model, change: "added" })]),
+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",
+ },
+ typesDiff: `+ "${model}",\n+export const unrelated = true;\n`,
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Selectable-model diff contains an unsupported changed line/,
+ );
+});
+
+test("rejects case-insensitive pricing model collisions", () => {
+ const model = "gpt-4o";
+ const { result } = runContract({
+ basePrices: [pricingEntry(model)],
+ currentPrices: [pricingEntry(model), pricingEntry("GPT-4O")],
+ output: {
+ ...structuredOutput([auditRow({ model, change: "updated" })]),
+ pullRequestTitle: "chore(pricing): update gpt-4o pricing",
+ },
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Current pricing contains duplicate normalized modelName: GPT-4O/,
+ );
+});
+
+test("rejects ambiguous evidence rows for a changed model", () => {
+ const model = "gpt-6-astra";
+ const { result } = runContract({
+ currentPrices: [pricingEntry(model)],
+ output: {
+ ...structuredOutput([
+ auditRow({ model }),
+ { ...auditRow({ model }), provider: "Azure OpenAI" },
+ ]),
+ pullRequestTitle: "chore(pricing): add gpt-6-astra pricing",
+ },
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /modelsChecked must report exactly one row for changed model: gpt-6-astra/,
+ );
+});
+
+test("clears model-controlled changedModels for non-model diffs", () => {
+ const model = "gpt-4o";
+ const { normalizedOutput, result } = runContract({
+ basePrices: [pricingEntry(model)],
+ output: {
+ ...structuredOutput([
+ auditRow({
+ model,
+ officialSources: [],
+ priceConfirmed: "no",
+ usageKeyCoverageConfirmed: "yes",
+ }),
+ ]),
+ changedModels: ["invented-model (added)"],
+ pullRequestTitle: "chore(pricing): tighten audit workflow contract",
+ },
+ typesDiff: "diff --git a/reference.md b/reference.md\n",
+ });
+
+ assert.equal(result.status, 0, result.stderr);
+ assert.deepEqual(normalizedOutput.changedModels, []);
+ assert.equal(
+ normalizedOutput.modelsChecked[0].usageKeyCoverageConfirmed,
+ "no",
+ );
+});
+
+test("parses the checked-in selectable-model arrays", () => {
+ const types = fs.readFileSync(repositoryTypesPath, "utf8");
+ assert.deepEqual(collectTypeModelChanges(types, types, ""), []);
+});
+
+test("tolerates pre-existing comments inside selectable-model arrays", () => {
+ const types = typesSource().replace(
+ ' "gpt-4o",',
+ ' // Current flagship\n "gpt-4o",',
+ );
+ assert.deepEqual(collectTypeModelChanges(types, types, ""), []);
+});
+
+test("escapes backslashes and backticks in the proposed title", () => {
+ const output = {
+ ...structuredOutput([auditRow({ model: "gpt-4o" })]),
+ pullRequestTitle: "chore(pricing): check \\ path and `code`",
+ };
+
+ assert.match(
+ renderAuditSummary(output),
+ /chore\(pricing\): check \\\\ path and \\`code\\`/,
+ );
+});
+
+test("runs formatting cleanup before the extracted output contract", () => {
+ const workflow = fs.readFileSync(workflowPath, "utf8");
+ const cleanupIndex = workflow.indexOf(
+ " - name: Discard pricing-only formatting changes",
+ );
+ const summaryIndex = workflow.indexOf(" - name: Write audit summary");
+
+ assert.ok(cleanupIndex >= 0, "workflow must clean up formatting-only diffs");
+ assert.ok(
+ cleanupIndex < summaryIndex,
+ "semantic cleanup must run before report reconciliation",
+ );
+ assert.match(
+ workflow,
+ /node scripts\/model-price-audit\/validate-and-render-audit-output\.mjs/,
+ );
+ assert.match(
+ workflow,
+ /run: node --test scripts\/model-price-audit\/\*\.test\.mjs/,
+ );
+ assert.equal(
+ workflow.match(
+ /run: node --test scripts\/model-price-audit\/\*\.test\.mjs/g,
+ ).length,
+ 2,
+ );
+ assert.ok(
+ workflow.lastIndexOf(
+ "run: node --test scripts/model-price-audit/*.test.mjs",
+ ) > workflow.indexOf(" - name: Run Claude price audit"),
+ "guardrail tests must rerun after permitted workflow self-edits",
+ );
+ assert.doesNotMatch(workflow, /node <<'NODE' \| tee/);
+});
diff --git a/scripts/model-price-audit/prepare-metadata.mjs b/scripts/model-price-audit/prepare-metadata.mjs
index 3fba18281a..fa677177c2 100644
--- a/scripts/model-price-audit/prepare-metadata.mjs
+++ b/scripts/model-price-audit/prepare-metadata.mjs
@@ -2,6 +2,14 @@
import { createHash } from "node:crypto";
import fs from "node:fs";
+import {
+ collectPricingModelChanges,
+ collectTypeModelChanges,
+ mergeModelChanges,
+ normalizeReportedModel,
+ validateChangedModelRow,
+ validateOfficialSources,
+} from "./audit-output-contract.mjs";
process.on("uncaughtException", (error) => {
console.error(`::error::${error.message}`);
@@ -10,7 +18,9 @@ process.on("uncaughtException", (error) => {
const requiredEnvironment = [
"BASE_PRICING_FILE",
+ "BASE_TYPES_FILE",
"CURRENT_PRICING_FILE",
+ "CURRENT_TYPES_FILE",
"STRUCTURED_OUTPUT_PATH",
"TYPES_DIFF_FILE",
];
@@ -26,8 +36,6 @@ const output = JSON.parse(
);
const title = output.pullRequestTitle.trim();
const normalize = (value) => value.trim().toLowerCase();
-const normalizeReportedModel = (value) =>
- normalize(value).replace(/\s+\((?:also\s+)?matches\b[^)]*\)$/u, "");
if (!/^chore\(pricing\): [^\r\n]+$/.test(title) || title.length > 100) {
throw new Error(
@@ -46,44 +54,33 @@ if (
);
}
-const basePricingText = fs.readFileSync(
- process.env.BASE_PRICING_FILE,
- "utf8",
-);
+const basePricingText = fs.readFileSync(process.env.BASE_PRICING_FILE, "utf8");
const currentPricingText = fs.readFileSync(
process.env.CURRENT_PRICING_FILE,
"utf8",
);
const basePrices = JSON.parse(basePricingText);
const currentPrices = JSON.parse(currentPricingText);
-const basePricesByName = new Map(
- basePrices.map((item) => [normalize(item.modelName), item]),
+const pricingModelChanges = collectPricingModelChanges(
+ basePrices,
+ currentPrices,
);
-const currentPricesByName = new Map(
- currentPrices.map((item) => [normalize(item.modelName), item]),
+const removedPricingModels = pricingModelChanges.filter(
+ (change) => change.expectedChange === "removed",
);
-const pricingModelChanges = [];
-for (const modelName of new Set([
- ...basePricesByName.keys(),
- ...currentPricesByName.keys(),
-])) {
- const before = basePricesByName.get(modelName);
- const after = currentPricesByName.get(modelName);
- if (JSON.stringify(before) !== JSON.stringify(after)) {
- pricingModelChanges.push({
- modelName: after?.modelName ?? before.modelName,
- expectedChange: before ? "updated" : "added",
- });
- }
+if (removedPricingModels.length > 0) {
+ throw new Error(
+ `Automated pricing-entry removal is not allowed: ${removedPricingModels
+ .map((change) => change.modelName)
+ .join(", ")}`,
+ );
}
-const typeModelChanges = new Set();
-for (const line of fs
- .readFileSync(process.env.TYPES_DIFF_FILE, "utf8")
- .split("\n")) {
- const match = line.match(/^[+-]\s*"([^"]+)"(?:,|:)/);
- if (match) typeModelChanges.add(match[1]);
-}
+const typeModelChanges = collectTypeModelChanges(
+ fs.readFileSync(process.env.BASE_TYPES_FILE, "utf8"),
+ fs.readFileSync(process.env.CURRENT_TYPES_FILE, "utf8"),
+ fs.readFileSync(process.env.TYPES_DIFF_FILE, "utf8"),
+);
if (
process.env.CHANGED_PRICING_JSON === "true" &&
@@ -97,7 +94,10 @@ if (
`base sha256: ${digest(basePricingText)}, current sha256: ${digest(currentPricingText)})`,
);
}
-if (process.env.CHANGED_MODEL_TYPES === "true" && typeModelChanges.size === 0) {
+if (
+ process.env.CHANGED_MODEL_TYPES === "true" &&
+ typeModelChanges.length === 0
+) {
throw new Error(
"Selectable-model types changed without a concrete model identifier change",
);
@@ -109,27 +109,33 @@ const changedModelRows = output.modelsChecked.filter((item) =>
const changedRowsByModel = new Map(
changedModelRows.map((item) => [normalizeReportedModel(item.model), item]),
);
-for (const change of pricingModelChanges) {
+for (const change of typeModelChanges) {
+ const row = changedRowsByModel.get(normalize(change.modelName));
+ if (row && normalize(row.provider) !== normalize(change.provider)) {
+ throw new Error(
+ `${change.arrayName} additions require provider ${change.provider}: ${change.modelName}`,
+ );
+ }
+}
+const expectedModelChanges = mergeModelChanges(
+ pricingModelChanges,
+ typeModelChanges,
+);
+for (const change of expectedModelChanges) {
const row = changedRowsByModel.get(normalize(change.modelName));
if (!row || row.change !== change.expectedChange) {
throw new Error(
- `modelsChecked must report the actual ${change.expectedChange} pricing entry: ${change.modelName}`,
+ `modelsChecked must report the actual ${change.expectedChange} model entry: ${change.modelName}`,
);
}
+ validateOfficialSources(row.officialSources, row.model);
+ validateChangedModelRow(row);
}
const affectedModels = new Map();
-for (const change of pricingModelChanges) {
+for (const change of expectedModelChanges) {
affectedModels.set(normalize(change.modelName), change.modelName);
}
-for (const modelName of typeModelChanges) {
- affectedModels.set(normalize(modelName), modelName);
- if (!changedRowsByModel.has(normalize(modelName))) {
- throw new Error(
- `modelsChecked must report the selectable-model change: ${modelName}`,
- );
- }
-}
for (const row of changedModelRows) {
if (!affectedModels.has(normalizeReportedModel(row.model))) {
throw new Error(
diff --git a/scripts/model-price-audit/prepare-metadata.test.mjs b/scripts/model-price-audit/prepare-metadata.test.mjs
index bcedc0b376..f3c75919e6 100644
--- a/scripts/model-price-audit/prepare-metadata.test.mjs
+++ b/scripts/model-price-audit/prepare-metadata.test.mjs
@@ -11,8 +11,30 @@ const workflowPath = path.join(
"../../.github/workflows/model-price-audit.yml",
);
+const typesSource = ({
+ anthropic = ["claude-3"],
+ googleAIStudio = ["gemini-2"],
+ openAI = ["gpt-4o"],
+ vertexAI = ["gemini-2"],
+} = {}) => `
+export const openAIModels = [
+${openAI.map((model) => ` "${model}",`).join("\n")}
+] as const;
+export const anthropicModels = [
+${anthropic.map((model) => ` "${model}",`).join("\n")}
+] as const;
+export const vertexAIModels = [
+${vertexAI.map((model) => ` "${model}",`).join("\n")}
+] as const;
+export const googleAIStudioModels = [
+${googleAIStudio.map((model) => ` "${model}",`).join("\n")}
+] as const;
+`;
+
function runValidator({
+ baseTypes = typesSource(),
basePrices = [],
+ currentTypes = baseTypes,
currentPrices = basePrices,
output,
typesDiff = "",
@@ -24,12 +46,16 @@ function runValidator({
);
const files = {
base: path.join(fixtureDirectory, "base.json"),
+ baseTypes: path.join(fixtureDirectory, "base-types.ts"),
current: path.join(fixtureDirectory, "current.json"),
+ currentTypes: path.join(fixtureDirectory, "current-types.ts"),
output: path.join(fixtureDirectory, "output.json"),
typesDiff: path.join(fixtureDirectory, "types.diff"),
};
fs.writeFileSync(files.base, JSON.stringify(basePrices));
+ fs.writeFileSync(files.baseTypes, baseTypes);
fs.writeFileSync(files.current, JSON.stringify(currentPrices));
+ fs.writeFileSync(files.currentTypes, currentTypes);
fs.writeFileSync(files.output, JSON.stringify(output));
fs.writeFileSync(files.typesDiff, typesDiff);
@@ -38,9 +64,11 @@ function runValidator({
env: {
...process.env,
BASE_PRICING_FILE: files.base,
+ BASE_TYPES_FILE: files.baseTypes,
CHANGED_MODEL_TYPES: String(changedModelTypes),
CHANGED_PRICING_JSON: String(changedPricingJson),
CURRENT_PRICING_FILE: files.current,
+ CURRENT_TYPES_FILE: files.currentTypes,
STRUCTURED_OUTPUT_PATH: files.output,
TYPES_DIFF_FILE: files.typesDiff,
},
@@ -50,6 +78,15 @@ function runValidator({
}
const modelName = "gpt-5.5-2026-04-23";
+const confirmedChange = (model, change) => ({
+ provider: "OpenAI",
+ model,
+ change,
+ priceConfirmed: "yes",
+ usageKeyCoverageConfirmed: "yes",
+ tieringCorrect: "not_applicable",
+ officialSources: ["https://developers.openai.com/api/docs/pricing"],
+});
const changedPrices = {
before: [{ modelName, pricingTiers: [{ prices: { input: 5 } }] }],
after: [{ modelName, pricingTiers: [{ prices: { input: 10 } }] }],
@@ -66,7 +103,7 @@ test("accepts an added pricing entry from the staged snapshot", () => {
changedPricingJson: true,
output: {
pullRequestTitle: "chore(pricing): add OpenAI gpt-6-astra pricing",
- modelsChecked: [{ model: addedModel.modelName, change: "added" }],
+ modelsChecked: [confirmedChange(addedModel.modelName, "added")],
},
});
@@ -81,10 +118,7 @@ test("accepts a unique trailing match-alias annotation", () => {
output: {
pullRequestTitle: "chore(pricing): update gpt-5.5 large-context tier",
modelsChecked: [
- {
- model: `${modelName} (also matches gpt-5.5)`,
- change: "updated",
- },
+ confirmedChange(`${modelName} (also matches gpt-5.5)`, "updated"),
],
},
});
@@ -96,6 +130,22 @@ test("accepts a unique trailing match-alias annotation", () => {
);
});
+test("accepts a selectable-model-only addition with confirmed evidence", () => {
+ const selectableModel = "gpt-6-astra";
+ const result = runValidator({
+ basePrices: [{ modelName: selectableModel }],
+ changedModelTypes: true,
+ currentTypes: typesSource({ openAI: ["gpt-4o", selectableModel] }),
+ output: {
+ pullRequestTitle: "chore(pricing): expose gpt-6-astra in playground",
+ modelsChecked: [confirmedChange(selectableModel, "added")],
+ },
+ typesDiff: `+ "${selectableModel}",\n`,
+ });
+
+ assert.equal(result.status, 0, result.stderr);
+});
+
test("accepts a concrete title for a reference-only diff", () => {
const result = runValidator({
output: {
@@ -127,7 +177,7 @@ test("rejects an unrelated decorated model row", () => {
assert.notEqual(result.status, 0);
assert.match(
result.stderr,
- /modelsChecked must report the actual updated pricing entry/,
+ /modelsChecked must report the actual updated model entry/,
);
});
@@ -146,6 +196,67 @@ test("rejects a generic pull request title", () => {
);
});
+test("revalidates changed-entry evidence from the staged snapshot", () => {
+ const result = runValidator({
+ basePrices: changedPrices.before,
+ currentPrices: changedPrices.after,
+ changedPricingJson: true,
+ output: {
+ pullRequestTitle: "chore(pricing): update gpt-5.5 pricing",
+ modelsChecked: [
+ {
+ ...confirmedChange(modelName, "updated"),
+ priceConfirmed: "no",
+ },
+ ],
+ },
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Changed model rows require confirmed prices: gpt-5.5-2026-04-23/,
+ );
+});
+
+test("rejects pricing-entry removal from the staged snapshot", () => {
+ const result = runValidator({
+ basePrices: changedPrices.before,
+ currentPrices: [],
+ changedPricingJson: true,
+ output: {
+ pullRequestTitle: "chore(pricing): remove gpt-5.5 pricing",
+ modelsChecked: [confirmedChange(modelName, "updated")],
+ },
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Automated pricing-entry removal is not allowed: gpt-5.5-2026-04-23/,
+ );
+});
+
+test("rejects selectable-model removal from the staged snapshot", () => {
+ const result = runValidator({
+ baseTypes: typesSource({ openAI: ["gpt-4o", modelName] }),
+ basePrices: changedPrices.before,
+ changedModelTypes: true,
+ currentTypes: typesSource(),
+ output: {
+ pullRequestTitle: "chore(pricing): remove gpt-5.5 from playground",
+ modelsChecked: [confirmedChange(modelName, "updated")],
+ },
+ typesDiff: `- "${modelName}",\n`,
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(
+ result.stderr,
+ /Automated selectable-model removal is not allowed: gpt-5.5-2026-04-23/,
+ );
+});
+
test("validates metadata against the exact staged pricing blob", () => {
const workflow = fs.readFileSync(workflowPath, "utf8");
const prepareStep = workflow.slice(
@@ -164,9 +275,7 @@ test("validates metadata against the exact staged pricing blob", () => {
stageIndex < metadataIndex,
"metadata validation must run after staging",
);
- assert.match(
- prepareStep,
- /CURRENT_PRICING_FILE="\$staged_pricing_file"/,
- );
+ assert.match(prepareStep, /CURRENT_PRICING_FILE="\$staged_pricing_file"/);
+ assert.match(prepareStep, /CURRENT_TYPES_FILE="\$staged_types_file"/);
assert.doesNotMatch(prepareStep, /checkout -b "\$BOT_BRANCH"/);
});
diff --git a/scripts/model-price-audit/validate-and-render-audit-output.mjs b/scripts/model-price-audit/validate-and-render-audit-output.mjs
new file mode 100644
index 0000000000..94e895ce54
--- /dev/null
+++ b/scripts/model-price-audit/validate-and-render-audit-output.mjs
@@ -0,0 +1,53 @@
+#!/usr/bin/env node
+
+import fs from "node:fs";
+import {
+ reconcileAuditOutput,
+ renderAuditSummary,
+} from "./audit-output-contract.mjs";
+
+process.on("uncaughtException", (error) => {
+ console.error(`::error::${error.message}`);
+ process.exit(1);
+});
+
+const requiredEnvironment = [
+ "BASE_PRICING_FILE",
+ "BASE_TYPES_FILE",
+ "CURRENT_PRICING_FILE",
+ "CURRENT_TYPES_FILE",
+ "HAS_REPOSITORY_DIFF",
+ "MEMORY_SNAPSHOT_ONLY",
+ "STRUCTURED_OUTPUT",
+ "STRUCTURED_OUTPUT_PATH",
+ "TYPES_DIFF_FILE",
+];
+for (const name of requiredEnvironment) {
+ if (!process.env[name]) {
+ throw new Error(`Missing required environment variable: ${name}`);
+ }
+}
+
+const rawOutput = JSON.parse(process.env.STRUCTURED_OUTPUT);
+const basePrices = JSON.parse(
+ fs.readFileSync(process.env.BASE_PRICING_FILE, "utf8"),
+);
+const currentPrices = JSON.parse(
+ fs.readFileSync(process.env.CURRENT_PRICING_FILE, "utf8"),
+);
+const baseTypes = fs.readFileSync(process.env.BASE_TYPES_FILE, "utf8");
+const currentTypes = fs.readFileSync(process.env.CURRENT_TYPES_FILE, "utf8");
+const typesDiff = fs.readFileSync(process.env.TYPES_DIFF_FILE, "utf8");
+const { output, warnings } = reconcileAuditOutput(rawOutput, {
+ basePrices,
+ currentPrices,
+ baseTypes,
+ currentTypes,
+ hasRepositoryDiff: process.env.HAS_REPOSITORY_DIFF === "true",
+ memorySnapshotOnly: process.env.MEMORY_SNAPSHOT_ONLY === "true",
+ typesDiff,
+});
+
+fs.writeFileSync(process.env.STRUCTURED_OUTPUT_PATH, JSON.stringify(output));
+for (const warning of warnings) console.error(`::warning::${warning}`);
+process.stdout.write(renderAuditSummary(output));