Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

ci(model-price-audit): keep bot pull request bodies to changes, reasons, and sources (#18673)

Hassieb Pakzad8a5326997e75
diff --git a/.github/workflows/model-price-audit.yml b/.github/workflows/model-price-audit.yml
index bc5c2c646f..124c9b2f34 100644
--- a/.github/workflows/model-price-audit.yml
+++ b/.github/workflows/model-price-audit.yml
@@ -240,7 +240,7 @@ jobs:
- Set `priceConfirmed` to `yes` only when an official source fetched during this run confirms the exact current prices. Otherwise set it to `no` and explain why in `comments`.
- Set `tieringCorrect` to `yes` only when all applicable price tiers, thresholds, and tier-specific prices are confirmed. Use `not_applicable` only when the provider has no applicable tiering dimension, otherwise use `no` and explain the gap in `comments`.
- For every model removed from a selectable model array, add exactly one row with `change` set to `removed`, the provider's own retirement, shutdown, or model-listing URL in `officialSources`, and a comment naming the arrays it was removed from, its retirement status or date, and its successor if one exists. Price, usage-key, and tiering confirmations may be `no` for removed rows. A model ID listed in several arrays gets one row, but every distinct removed ID gets its own row: an alias and its dated snapshot (for example `o1-preview` and `o1-preview-2024-09-12`) are two rows, not one row with the snapshot named in `comments`. Set `model` to the exact string from the selectable array, even when its pricing `modelName` is spelled differently (for example `claude-3-7-sonnet-20250219` in the array versus `claude-3.7-sonnet-20250219` in the pricing file), and set `provider` to `OpenAI`, `Anthropic`, or `Google` to match the array.
- - Put the official evidence URLs used for each model in that row. Comments should capture price units and conversions, tier thresholds, corrections, uncertainty, or why no change was made.
+ - Put the official evidence URLs used for each model in that row. Comments should capture price units and conversions, tier thresholds, corrections, uncertainty, or why no change was made. For `added`, `updated`, and `removed` rows, start `comments` with the old and new values and the reason for the change, because the pull request body shows only these rows' `pricingChecked`, `comments`, and `officialSources`.
- Treat every allowed repository edit as a diff, including an audit snapshot written to `model-audit-memory.md` when no model prices changed. If there is no repository diff at all, set `pullRequestTitle` to an empty string, say that no pricing changes were made, and list notable unresolved findings.
- If there is a diff, set `pullRequestTitle` to a Conventional Commit title starting with `chore(pricing): ` that names the affected model families or the specific audit behavior changed, for example `chore(pricing): remove retired gemini-2.0 models from pickers`. Never use a generic title such as `update default model prices`.
- If there is a diff, also list every changed model, every workflow or skill-reference improvement, and the validation commands run.
@@ -548,37 +548,14 @@ jobs:
test -s "$RUNNER_TEMP/model-price-audit.patch"
{
- echo "Automated default model price audit."
- echo
if [ "$DRY_RUN_MODE" != "disabled" ]; then
echo "**Dry run mode:** \`$DRY_RUN_MODE\`"
echo
echo "This artifact was generated to debug post-agent workflow steps. The publish job is skipped for dry runs."
echo
fi
- echo "## Scope"
- echo
- echo "- Audits default model pricing data with official provider evidence."
- echo "- Edits are restricted to pricing JSON, shared selectable model lists, pricing skill reference docs, and this audit workflow."
- echo "- Selectable models may be added, or removed with official retirement evidence; pricing entries are never removed."
- echo
- echo "## Validation"
- echo
- echo "- pricing structure and changed-entry usage-key coverage validation"
- echo "- \`node scripts/agents/sync-agent-shims.mjs --check\`"
- echo "- \`git diff --check\`"
- echo "- staged pricing blob validation"
- echo "- patch artifact generated with \`git format-patch -1\`"
- echo
- echo "## Diff Stat"
- echo
- echo '```text'
- git show --stat --oneline HEAD
- echo '```'
- echo
- echo "## Agent Summary"
- echo
- cat "$RUNNER_TEMP/claude-model-price-audit.txt"
+ AUDIT_REPORT_URL="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \
+ node scripts/model-price-audit/render-pull-request-body.mjs
} > "$RUNNER_TEMP/model-price-audit-pr-body.md"
echo "has_changes=true" >> "$GITHUB_OUTPUT"
diff --git a/scripts/model-price-audit/audit-output-contract.mjs b/scripts/model-price-audit/audit-output-contract.mjs
index e4f6b7ed5b..ececf09fbb 100644
--- a/scripts/model-price-audit/audit-output-contract.mjs
+++ b/scripts/model-price-audit/audit-output-contract.mjs
@@ -584,3 +584,74 @@ export function renderAuditSummary(output) {
"",
].join("\n");
}
+
+const inlineText = (value) =>
+ String(value ?? "")
+ .replace(/\r?\n+/g, " ")
+ .trim();
+
+const inlineCode = (value) => `\`${inlineText(value).replace(/`/g, "")}\``;
+
+// Pull request bodies stay limited to what changed, why, and the sources; the
+// full per-model report belongs in the workflow run summary.
+export function renderPullRequestBody(output, { reportUrl } = {}) {
+ const changeLabels = {
+ added: "added",
+ updated: "updated",
+ removed: "removed",
+ };
+ const changedRows = output.modelsChecked.filter(
+ (row) => row.change in changeLabels,
+ );
+ const otherUpdates = [
+ ...(output.skillReferenceUpdates ?? []),
+ ...(output.workflowUpdates ?? []),
+ ];
+ const unresolvedCount = output.unresolvedFindings?.length ?? 0;
+
+ const sections = [
+ `Automated default model price audit from ${inlineText(output.auditDate)}.`,
+ "",
+ "## What changed",
+ "",
+ ];
+
+ if (changedRows.length === 0) {
+ sections.push("No model prices or selectable models changed.", "");
+ }
+ for (const row of changedRows) {
+ sections.push(
+ `### ${inlineCode(row.model)} (${inlineText(row.provider)}): ${changeLabels[row.change]}`,
+ "",
+ `**Pricing:** ${inlineText(row.pricingChecked)}`,
+ "",
+ `**Why:** ${inlineText(row.comments) || "No reason reported."}`,
+ "",
+ "**Sources:**",
+ "",
+ ...row.officialSources.map((source) => `- ${new URL(source).href}`),
+ "",
+ );
+ }
+
+ if (otherUpdates.length > 0) {
+ sections.push(
+ "## Reference and workflow updates",
+ "",
+ list(otherUpdates),
+ "",
+ );
+ }
+
+ const reportLink = reportUrl
+ ? `[workflow run summary](${reportUrl})`
+ : "workflow run summary";
+ sections.push(
+ "---",
+ "",
+ `Checked ${output.modelsChecked.length} model entries; ${unresolvedCount} unresolved finding${unresolvedCount === 1 ? "" : "s"}. The full audit report is in the ${reportLink}.`,
+ "",
+ );
+
+ return sections.join("\n");
+}
diff --git a/scripts/model-price-audit/audit-output-contract.test.mjs b/scripts/model-price-audit/audit-output-contract.test.mjs
index 6f4d47ce6f..f32dc1416d 100644
--- a/scripts/model-price-audit/audit-output-contract.test.mjs
+++ b/scripts/model-price-audit/audit-output-contract.test.mjs
@@ -7,6 +7,7 @@ import test from "node:test";
import {
collectTypeModelChanges,
renderAuditSummary,
+ renderPullRequestBody,
} from "./audit-output-contract.mjs";
const scriptPath = path.join(
@@ -692,6 +693,56 @@ test("escapes backslashes and backticks in the proposed title", () => {
);
});
+test("limits the pull request body to changed rows, reasons, and sources", () => {
+ const output = {
+ ...structuredOutput([
+ auditRow({ model: "gpt-4o", comments: "Unchanged row detail." }),
+ auditRow({
+ model: "gpt-5",
+ change: "updated",
+ comments: "Cache read lowered from $0.20 to $0.10/MTok.",
+ }),
+ ]),
+ skillReferenceUpdates: ["provider-sources-and-price-keys.md - noted fix"],
+ unresolvedFindings: ["Bedrock SKU still ambiguous"],
+ validation: ["validate-pricing-file.mjs passed"],
+ };
+
+ const body = renderPullRequestBody(output, {
+ reportUrl: "https://github.com/langfuse/langfuse/actions/runs/1",
+ });
+
+ assert.match(body, /### `gpt-5` \(OpenAI\): updated/);
+ assert.match(body, /\*\*Why:\*\* Cache read lowered from \$0\.20/);
+ assert.ok(body.includes(`- ${officialSource}`));
+ assert.ok(body.includes("provider-sources-and-price-keys.md - noted fix"));
+ assert.ok(
+ body.includes(
+ "[workflow run summary](https://github.com/langfuse/langfuse/actions/runs/1)",
+ ),
+ );
+ assert.match(body, /Checked 2 model entries; 1 unresolved finding\./);
+ for (const omitted of [
+ "Unchanged row detail.",
+ "Bedrock SKU still ambiguous",
+ "validate-pricing-file.mjs passed",
+ "| Provider |",
+ ]) {
+ assert.ok(!body.includes(omitted), `body must omit ${omitted}`);
+ }
+
+ const workflow = fs.readFileSync(workflowPath, "utf8");
+ const bodyStep = workflow.slice(
+ workflow.lastIndexOf(
+ "{",
+ workflow.indexOf('> "$RUNNER_TEMP/model-price-audit-pr-body.md"'),
+ ),
+ workflow.indexOf('> "$RUNNER_TEMP/model-price-audit-pr-body.md"'),
+ );
+ assert.match(bodyStep, /render-pull-request-body\.mjs/);
+ assert.doesNotMatch(bodyStep, /claude-model-price-audit\.txt/);
+});
+
test("runs formatting cleanup before the extracted output contract", () => {
const workflow = fs.readFileSync(workflowPath, "utf8");
const cleanupIndex = workflow.indexOf(
diff --git a/scripts/model-price-audit/render-pull-request-body.mjs b/scripts/model-price-audit/render-pull-request-body.mjs
new file mode 100644
index 0000000000..305b88ee25
--- /dev/null
+++ b/scripts/model-price-audit/render-pull-request-body.mjs
@@ -0,0 +1,22 @@
+#!/usr/bin/env node
+
+import fs from "node:fs";
+import { renderPullRequestBody } from "./audit-output-contract.mjs";
+
+process.on("uncaughtException", (error) => {
+ console.error(`::error::${error.message}`);
+ process.exit(1);
+});
+
+if (!process.env.STRUCTURED_OUTPUT_PATH) {
+ throw new Error(
+ "Missing required environment variable: STRUCTURED_OUTPUT_PATH",
+ );
+}
+
+const output = JSON.parse(
+ fs.readFileSync(process.env.STRUCTURED_OUTPUT_PATH, "utf8"),
+);
+process.stdout.write(
+ renderPullRequestBody(output, { reportUrl: process.env.AUDIT_REPORT_URL }),
+);