🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
ci(model-price-audit): keep bot pull request bodies to changes, reasons, and sources (#18673)
Hassieb Pakzad8a5326997e75
diff --git a/.github/workflows/model-price-audit.yml b/.github/workflows/model-price-audit.ymlindex bc5c2c646f..124c9b2f34 100644--- a/.github/workflows/model-price-audit.yml+++ b/.github/workflows/model-price-audit.yml@@ -240,7 +240,7 @@ jobs:- Set `priceConfirmed` to `yes` only when an official source fetched during this run confirms the exact current prices. Otherwise set it to `no` and explain why in `comments`.- Set `tieringCorrect` to `yes` only when all applicable price tiers, thresholds, and tier-specific prices are confirmed. Use `not_applicable` only when the provider has no applicable tiering dimension, otherwise use `no` and explain the gap in `comments`.- For every model removed from a selectable model array, add exactly one row with `change` set to `removed`, the provider's own retirement, shutdown, or model-listing URL in `officialSources`, and a comment naming the arrays it was removed from, its retirement status or date, and its successor if one exists. Price, usage-key, and tiering confirmations may be `no` for removed rows. A model ID listed in several arrays gets one row, but every distinct removed ID gets its own row: an alias and its dated snapshot (for example `o1-preview` and `o1-preview-2024-09-12`) are two rows, not one row with the snapshot named in `comments`. Set `model` to the exact string from the selectable array, even when its pricing `modelName` is spelled differently (for example `claude-3-7-sonnet-20250219` in the array versus `claude-3.7-sonnet-20250219` in the pricing file), and set `provider` to `OpenAI`, `Anthropic`, or `Google` to match the array.- - Put the official evidence URLs used for each model in that row. Comments should capture price units and conversions, tier thresholds, corrections, uncertainty, or why no change was made.+ - Put the official evidence URLs used for each model in that row. Comments should capture price units and conversions, tier thresholds, corrections, uncertainty, or why no change was made. For `added`, `updated`, and `removed` rows, start `comments` with the old and new values and the reason for the change, because the pull request body shows only these rows' `pricingChecked`, `comments`, and `officialSources`.- Treat every allowed repository edit as a diff, including an audit snapshot written to `model-audit-memory.md` when no model prices changed. If there is no repository diff at all, set `pullRequestTitle` to an empty string, say that no pricing changes were made, and list notable unresolved findings.- If there is a diff, set `pullRequestTitle` to a Conventional Commit title starting with `chore(pricing): ` that names the affected model families or the specific audit behavior changed, for example `chore(pricing): remove retired gemini-2.0 models from pickers`. Never use a generic title such as `update default model prices`.- If there is a diff, also list every changed model, every workflow or skill-reference improvement, and the validation commands run.@@ -548,37 +548,14 @@ jobs:test -s "$RUNNER_TEMP/model-price-audit.patch"{- echo "Automated default model price audit."- echoif [ "$DRY_RUN_MODE" != "disabled" ]; thenecho "**Dry run mode:** \`$DRY_RUN_MODE\`"echoecho "This artifact was generated to debug post-agent workflow steps. The publish job is skipped for dry runs."echofi- echo "## Scope"- echo- echo "- Audits default model pricing data with official provider evidence."- echo "- Edits are restricted to pricing JSON, shared selectable model lists, pricing skill reference docs, and this audit workflow."- echo "- Selectable models may be added, or removed with official retirement evidence; pricing entries are never removed."- echo- echo "## Validation"- echo- echo "- pricing structure and changed-entry usage-key coverage validation"- echo "- \`node scripts/agents/sync-agent-shims.mjs --check\`"- echo "- \`git diff --check\`"- echo "- staged pricing blob validation"- echo "- patch artifact generated with \`git format-patch -1\`"- echo- echo "## Diff Stat"- echo- echo '```text'- git show --stat --oneline HEAD- echo '```'- echo- echo "## Agent Summary"- echo- cat "$RUNNER_TEMP/claude-model-price-audit.txt"+ AUDIT_REPORT_URL="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \+ node scripts/model-price-audit/render-pull-request-body.mjs} > "$RUNNER_TEMP/model-price-audit-pr-body.md"echo "has_changes=true" >> "$GITHUB_OUTPUT"diff --git a/scripts/model-price-audit/audit-output-contract.mjs b/scripts/model-price-audit/audit-output-contract.mjsindex e4f6b7ed5b..ececf09fbb 100644--- a/scripts/model-price-audit/audit-output-contract.mjs+++ b/scripts/model-price-audit/audit-output-contract.mjs@@ -584,3 +584,74 @@ export function renderAuditSummary(output) {"",].join("\n");}++const inlineText = (value) =>+ String(value ?? "")+ .replace(/\r?\n+/g, " ")+ .trim();++const inlineCode = (value) => `\`${inlineText(value).replace(/`/g, "")}\``;++// Pull request bodies stay limited to what changed, why, and the sources; the+// full per-model report belongs in the workflow run summary.+export function renderPullRequestBody(output, { reportUrl } = {}) {+ const changeLabels = {+ added: "added",+ updated: "updated",+ removed: "removed",+ };+ const changedRows = output.modelsChecked.filter(+ (row) => row.change in changeLabels,+ );+ const otherUpdates = [+ ...(output.skillReferenceUpdates ?? []),+ ...(output.workflowUpdates ?? []),+ ];+ const unresolvedCount = output.unresolvedFindings?.length ?? 0;++ const sections = [+ `Automated default model price audit from ${inlineText(output.auditDate)}.`,+ "",+ "## What changed",+ "",+ ];++ if (changedRows.length === 0) {+ sections.push("No model prices or selectable models changed.", "");+ }+ for (const row of changedRows) {+ sections.push(+ `### ${inlineCode(row.model)} (${inlineText(row.provider)}): ${changeLabels[row.change]}`,+ "",+ `**Pricing:** ${inlineText(row.pricingChecked)}`,+ "",+ `**Why:** ${inlineText(row.comments) || "No reason reported."}`,+ "",+ "**Sources:**",+ "",+ ...row.officialSources.map((source) => `- ${new URL(source).href}`),+ "",+ );+ }++ if (otherUpdates.length > 0) {+ sections.push(+ "## Reference and workflow updates",+ "",+ list(otherUpdates),+ "",+ );+ }++ const reportLink = reportUrl+ ? `[workflow run summary](${reportUrl})`+ : "workflow run summary";+ sections.push(+ "---",+ "",+ `Checked ${output.modelsChecked.length} model entries; ${unresolvedCount} unresolved finding${unresolvedCount === 1 ? "" : "s"}. The full audit report is in the ${reportLink}.`,+ "",+ );++ return sections.join("\n");+}diff --git a/scripts/model-price-audit/audit-output-contract.test.mjs b/scripts/model-price-audit/audit-output-contract.test.mjsindex 6f4d47ce6f..f32dc1416d 100644--- a/scripts/model-price-audit/audit-output-contract.test.mjs+++ b/scripts/model-price-audit/audit-output-contract.test.mjs@@ -7,6 +7,7 @@ import test from "node:test";import {collectTypeModelChanges,renderAuditSummary,+ renderPullRequestBody,} from "./audit-output-contract.mjs";const scriptPath = path.join(@@ -692,6 +693,56 @@ test("escapes backslashes and backticks in the proposed title", () => {);});+test("limits the pull request body to changed rows, reasons, and sources", () => {+ const output = {+ ...structuredOutput([+ auditRow({ model: "gpt-4o", comments: "Unchanged row detail." }),+ auditRow({+ model: "gpt-5",+ change: "updated",+ comments: "Cache read lowered from $0.20 to $0.10/MTok.",+ }),+ ]),+ skillReferenceUpdates: ["provider-sources-and-price-keys.md - noted fix"],+ unresolvedFindings: ["Bedrock SKU still ambiguous"],+ validation: ["validate-pricing-file.mjs passed"],+ };++ const body = renderPullRequestBody(output, {+ reportUrl: "https://github.com/langfuse/langfuse/actions/runs/1",+ });++ assert.match(body, /### `gpt-5` \(OpenAI\): updated/);+ assert.match(body, /\*\*Why:\*\* Cache read lowered from \$0\.20/);+ assert.ok(body.includes(`- ${officialSource}`));+ assert.ok(body.includes("provider-sources-and-price-keys.md - noted fix"));+ assert.ok(+ body.includes(+ "[workflow run summary](https://github.com/langfuse/langfuse/actions/runs/1)",+ ),+ );+ assert.match(body, /Checked 2 model entries; 1 unresolved finding\./);+ for (const omitted of [+ "Unchanged row detail.",+ "Bedrock SKU still ambiguous",+ "validate-pricing-file.mjs passed",+ "| Provider |",+ ]) {+ assert.ok(!body.includes(omitted), `body must omit ${omitted}`);+ }++ const workflow = fs.readFileSync(workflowPath, "utf8");+ const bodyStep = workflow.slice(+ workflow.lastIndexOf(+ "{",+ workflow.indexOf('> "$RUNNER_TEMP/model-price-audit-pr-body.md"'),+ ),+ workflow.indexOf('> "$RUNNER_TEMP/model-price-audit-pr-body.md"'),+ );+ assert.match(bodyStep, /render-pull-request-body\.mjs/);+ assert.doesNotMatch(bodyStep, /claude-model-price-audit\.txt/);+});+test("runs formatting cleanup before the extracted output contract", () => {const workflow = fs.readFileSync(workflowPath, "utf8");const cleanupIndex = workflow.indexOf(diff --git a/scripts/model-price-audit/render-pull-request-body.mjs b/scripts/model-price-audit/render-pull-request-body.mjsnew file mode 100644index 0000000000..305b88ee25--- /dev/null+++ b/scripts/model-price-audit/render-pull-request-body.mjs@@ -0,0 +1,22 @@+#!/usr/bin/env node++import fs from "node:fs";+import { renderPullRequestBody } from "./audit-output-contract.mjs";++process.on("uncaughtException", (error) => {+ console.error(`::error::${error.message}`);+ process.exit(1);+});++if (!process.env.STRUCTURED_OUTPUT_PATH) {+ throw new Error(+ "Missing required environment variable: STRUCTURED_OUTPUT_PATH",+ );+}++const output = JSON.parse(+ fs.readFileSync(process.env.STRUCTURED_OUTPUT_PATH, "utf8"),+);+process.stdout.write(+ renderPullRequestBody(output, { reportUrl: process.env.AUDIT_REPORT_URL }),+);