Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(ci): let the model price audit list the rows its validator expects (#18514)

Hassieb Pakzada36654dcff05
diff --git a/.github/workflows/model-price-audit.yml b/.github/workflows/model-price-audit.yml
index 5b733ea7c8..b1a9bc05ec 100644
--- a/.github/workflows/model-price-audit.yml
+++ b/.github/workflows/model-price-audit.yml
@@ -231,6 +231,7 @@ jobs:
- Never remove a pricing entry from `default-model-prices.json`. Removal is only allowed for selectable model array entries in `types.ts`.
- Do not reorder selectable model arrays. Keep the first entry in every array unchanged, because it is the default model and the connection test model. The only exception is when that first model itself meets the removal criteria; then the next remaining entry becomes the default. If that next entry is itself deprecated with an announced retirement date, report it as unresolved so a human can choose a new default.
- Use the allowed exact `date -u +%Y-%m-%dT00:00:00.000Z` command if you need the current timestamp.
+ - After your last edit to `default-model-prices.json` or `types.ts`, run the allowed exact `node scripts/model-price-audit/list-required-audit-rows.mjs` command. It prints every model ID the output validator expects a changed row for. Report each printed ID in exactly one `modelsChecked` row with that exact `model`, `change`, and `provider`.
Final response:
- Always return one `modelsChecked` row for every distinct model price entry reviewed, including confirmed and unchanged entries. Set `model` to the exact `modelName` from `default-model-prices.json` for existing entries; put aliases and match-pattern details in `comments`, never in `model`.
@@ -238,7 +239,7 @@ jobs:
- Set `usageKeyCoverageConfirmed` to `yes` only after comparing the provider usage object, Langfuse ingestion normalization, and the provider usage-key matrix. Use `no` for a gap or unresolved capability and do not change that model entry.
- Set `priceConfirmed` to `yes` only when an official source fetched during this run confirms the exact current prices. Otherwise set it to `no` and explain why in `comments`.
- Set `tieringCorrect` to `yes` only when all applicable price tiers, thresholds, and tier-specific prices are confirmed. Use `not_applicable` only when the provider has no applicable tiering dimension, otherwise use `no` and explain the gap in `comments`.
- - For every model removed from a selectable model array, add exactly one row with `change` set to `removed`, the provider's own retirement, shutdown, or model-listing URL in `officialSources`, and a comment naming the arrays it was removed from, its retirement status or date, and its successor if one exists. Price, usage-key, and tiering confirmations may be `no` for removed rows. A model listed in several arrays gets one row. Set `model` to the exact string from the selectable array, even when its pricing `modelName` is spelled differently (for example `claude-3-7-sonnet-20250219` in the array versus `claude-3.7-sonnet-20250219` in the pricing file), and set `provider` to `OpenAI`, `Anthropic`, or `Google` to match the array.
+ - For every model removed from a selectable model array, add exactly one row with `change` set to `removed`, the provider's own retirement, shutdown, or model-listing URL in `officialSources`, and a comment naming the arrays it was removed from, its retirement status or date, and its successor if one exists. Price, usage-key, and tiering confirmations may be `no` for removed rows. A model ID listed in several arrays gets one row, but every distinct removed ID gets its own row: an alias and its dated snapshot (for example `o1-preview` and `o1-preview-2024-09-12`) are two rows, not one row with the snapshot named in `comments`. Set `model` to the exact string from the selectable array, even when its pricing `modelName` is spelled differently (for example `claude-3-7-sonnet-20250219` in the array versus `claude-3.7-sonnet-20250219` in the pricing file), and set `provider` to `OpenAI`, `Anthropic`, or `Google` to match the array.
- Put the official evidence URLs used for each model in that row. Comments should capture price units and conversions, tier thresholds, corrections, uncertainty, or why no change was made.
- Treat every allowed repository edit as a diff, including an audit snapshot written to `model-audit-memory.md` when no model prices changed. If there is no repository diff at all, set `pullRequestTitle` to an empty string, say that no pricing changes were made, and list notable unresolved findings.
- If there is a diff, set `pullRequestTitle` to a Conventional Commit title starting with `chore(pricing): ` that names the affected model families or the specific audit behavior changed, for example `chore(pricing): remove retired gemini-2.0 models from pickers`. Never use a generic title such as `update default model prices`.
@@ -248,7 +249,7 @@ jobs:
--max-turns ${{ steps.validate-inputs.outputs.max_turns }}
--max-budget-usd ${{ steps.validate-inputs.outputs.max_budget_usd }}
--no-session-persistence
- --allowedTools "Read(/.github/workflows/model-price-audit.yml),Read(/worker/src/constants/default-model-prices.json),Read(/packages/shared/src/server/llm/types.ts),Read(/packages/shared/src/server/otel/OtelIngestionProcessor.ts),Read(/.agents/skills/add-model-price/SKILL.md),Read(/.agents/skills/add-model-price/references/*.md),Edit(/.github/workflows/model-price-audit.yml),Edit(/worker/src/constants/default-model-prices.json),Edit(/packages/shared/src/server/llm/types.ts),Edit(/.agents/skills/add-model-price/references/*.md),Write(/.agents/skills/add-model-price/references/*.md),WebFetch(domain:platform.claude.com),WebFetch(domain:docs.anthropic.com),WebFetch(domain:developers.openai.com),WebFetch(domain:ai.google.dev),WebFetch(domain:cloud.google.com),WebFetch(domain:aws.amazon.com),WebFetch(domain:azure.microsoft.com),WebFetch(domain:typesafe.ai),WebFetch(domain:docs.typesafe.ai),Bash(date -u +%Y-%m-%dT00:00:00.000Z),Bash(uuidgen),Bash(node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs:*),Bash(node .agents/skills/add-model-price/scripts/test-match-pattern.mjs:*)"
+ --allowedTools "Read(/.github/workflows/model-price-audit.yml),Read(/worker/src/constants/default-model-prices.json),Read(/packages/shared/src/server/llm/types.ts),Read(/packages/shared/src/server/otel/OtelIngestionProcessor.ts),Read(/.agents/skills/add-model-price/SKILL.md),Read(/.agents/skills/add-model-price/references/*.md),Edit(/.github/workflows/model-price-audit.yml),Edit(/worker/src/constants/default-model-prices.json),Edit(/packages/shared/src/server/llm/types.ts),Edit(/.agents/skills/add-model-price/references/*.md),Write(/.agents/skills/add-model-price/references/*.md),WebFetch(domain:platform.claude.com),WebFetch(domain:docs.anthropic.com),WebFetch(domain:developers.openai.com),WebFetch(domain:ai.google.dev),WebFetch(domain:cloud.google.com),WebFetch(domain:aws.amazon.com),WebFetch(domain:azure.microsoft.com),WebFetch(domain:typesafe.ai),WebFetch(domain:docs.typesafe.ai),Bash(date -u +%Y-%m-%dT00:00:00.000Z),Bash(uuidgen),Bash(node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs:*),Bash(node .agents/skills/add-model-price/scripts/test-match-pattern.mjs:*),Bash(node scripts/model-price-audit/list-required-audit-rows.mjs)"
--json-schema '{"type":"object","additionalProperties":false,"properties":{"summary":{"type":"string"},"auditDate":{"type":"string","pattern":"^[0-9]{4}-[0-9]{2}-[0-9]{2}$"},"pullRequestTitle":{"type":"string","maxLength":100},"modelsChecked":{"type":"array","minItems":1,"items":{"type":"object","additionalProperties":false,"properties":{"provider":{"type":"string","minLength":1},"model":{"type":"string","minLength":1},"pricingChecked":{"type":"string","minLength":1},"priceConfirmed":{"type":"string","enum":["yes","no"]},"usageKeysChecked":{"type":"string","minLength":1},"usageKeyCoverageConfirmed":{"type":"string","enum":["yes","no"]},"tieringChecked":{"type":"string","minLength":1},"tieringCorrect":{"type":"string","enum":["yes","no","not_applicable"]},"change":{"type":"string","enum":["none","updated","added","removed","unresolved"]},"officialSources":{"type":"array","items":{"type":"string"}},"comments":{"type":"string"}},"required":["provider","model","pricingChecked","priceConfirmed","usageKeysChecked","usageKeyCoverageConfirmed","tieringChecked","tieringCorrect","change","officialSources","comments"]}},"changedModels":{"type":"array","items":{"type":"string"}},"skillReferenceUpdates":{"type":"array","items":{"type":"string"}},"workflowUpdates":{"type":"array","items":{"type":"string"}},"unresolvedFindings":{"type":"array","items":{"type":"string"}},"validation":{"type":"array","items":{"type":"string"}}},"required":["summary","auditDate","pullRequestTitle","modelsChecked","changedModels","skillReferenceUpdates","workflowUpdates","unresolvedFindings","validation"]}'
- name: Mock Claude price audit
diff --git a/scripts/model-price-audit/audit-output-contract.test.mjs b/scripts/model-price-audit/audit-output-contract.test.mjs
index 49372ed6f9..6f4d47ce6f 100644
--- a/scripts/model-price-audit/audit-output-contract.test.mjs
+++ b/scripts/model-price-audit/audit-output-contract.test.mjs
@@ -725,4 +725,13 @@ test("runs formatting cleanup before the extracted output contract", () => {
"guardrail tests must rerun after permitted workflow self-edits",
);
assert.doesNotMatch(workflow, /node <<'NODE' \| tee/);
+ assert.match(
+ workflow,
+ /Bash\(node scripts\/model-price-audit\/list-required-audit-rows\.mjs\)/,
+ "the audit must be allowed to list the rows the validator expects",
+ );
+ assert.match(
+ workflow,
+ /run the allowed exact `node scripts\/model-price-audit\/list-required-audit-rows\.mjs` command/,
+ );
});
diff --git a/scripts/model-price-audit/list-required-audit-rows.mjs b/scripts/model-price-audit/list-required-audit-rows.mjs
new file mode 100644
index 0000000000..ff7e2f8ca8
--- /dev/null
+++ b/scripts/model-price-audit/list-required-audit-rows.mjs
@@ -0,0 +1,58 @@
+#!/usr/bin/env node
+
+// Lists every model ID whose pricing entry or selectable-model entry differs
+// from HEAD, using the same rules as the audit output validator. Each listed
+// ID needs exactly one changed `modelsChecked` row with that exact `model`.
+
+import { execFileSync } from "node:child_process";
+import fs from "node:fs";
+import {
+ collectPricingModelChanges,
+ collectTypeModelChanges,
+ mergeModelChanges,
+} from "./audit-output-contract.mjs";
+
+const pricingPath = "worker/src/constants/default-model-prices.json";
+const typesPath = "packages/shared/src/server/llm/types.ts";
+
+const git = (...args) => execFileSync("git", args, { encoding: "utf8" });
+
+const typeChanges = collectTypeModelChanges(
+ git("show", `HEAD:${typesPath}`),
+ fs.readFileSync(typesPath, "utf8"),
+ git("diff", "--unified=0", "--", typesPath),
+);
+const changes = mergeModelChanges(
+ collectPricingModelChanges(
+ JSON.parse(git("show", `HEAD:${pricingPath}`)),
+ JSON.parse(fs.readFileSync(pricingPath, "utf8")),
+ ),
+ typeChanges,
+);
+
+if (changes.length === 0) {
+ console.log(
+ "No model entries changed; no changed modelsChecked rows needed.",
+ );
+ process.exit(0);
+}
+
+const arraysByModel = new Map();
+for (const change of typeChanges) {
+ const arrays = arraysByModel.get(change.modelName) ?? [];
+ arraysByModel.set(change.modelName, [...arrays, change]);
+}
+
+console.log(
+ "Each line needs exactly one modelsChecked row with this exact model, change, and provider:",
+);
+for (const change of changes) {
+ const arrays = arraysByModel.get(change.modelName) ?? [];
+ const location =
+ arrays.length > 0
+ ? `${arrays.map((entry) => entry.arrayName).join(", ")}; provider ${arrays[0].provider}`
+ : "default-model-prices.json";
+ console.log(
+ `- model=${change.modelName} change=${change.expectedChange} (${location})`,
+ );
+}