🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
chore(pricing): let the model price audit verify jev pricing against TypeSafe docs (#17859)
langfuse-botf21a5bc8e203
diff --git a/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md b/.agents/skills/add-model-price/references/provider-sources-and-price-keys.mdindex dc6f09bb4f..8293c96126 100644--- a/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md+++ b/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md@@ -12,6 +12,7 @@ Always fetch pricing from the provider's official docs before editing.| Google Gemini (Vertex AI) | `https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models` || AWS Bedrock | `https://aws.amazon.com/bedrock/pricing/` || Azure OpenAI | `https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/` |+| TypeSafe (Jev) | `https://docs.typesafe.ai/models` |### Known source quirks (as of 2026-06)@@ -790,6 +791,23 @@ Formula:price_per_token = price_per_mtok / 1_000_000```+- **TypeSafe Jev pricing (documented September 24 2026)** — `jev` is+ TypeSafe's decision model, not an LLM from the other covered providers. Its+ official price table is `https://docs.typesafe.ai/models` ("Price (per Btok /+ per Mtok) $42 / $0.042"; "Charged per input token. Output tokens are free."),+ and the `https://typesafe.ai/` homepage repeats "$42 Per Billion input+ tokens". Convert per billion tokens: `$42 / 1_000_000_000 = 4.2e-8` for+ `input`, with `output` at `0`. The entry keeps a single Standard tier: the+ models page documents no cache, batch, context-size, or modality tiers.+ Jev's usage object has only `input_tokens` and `output_tokens`, which+ Langfuse stores as `input` and `output`. Verify against TypeSafe's own docs+ only; Vercel AI Gateway and OpenRouter can proxy Jev (`TYPESAFE_UPSTREAMS`+ in `types.ts`), but their resale prices are not official evidence. Versioned+ IDs (`jev-1.13.0`) and the `jev-latest`/`jev-preview` aliases listed on the+ models page are already covered by the `matchPattern`. `typeSafeModels` in+ `types.ts` is not one of the selectable arrays the audit may edit; report a+ newly released Jev version that should become selectable as unresolved.+## Provider Usage KeysUse [provider-usage-key-matrix.md](provider-usage-key-matrix.md) as the singlediff --git a/.agents/skills/add-model-price/references/provider-usage-key-matrix.md b/.agents/skills/add-model-price/references/provider-usage-key-matrix.mdindex d5480d0c40..776b7f8460 100644--- a/.agents/skills/add-model-price/references/provider-usage-key-matrix.md+++ b/.agents/skills/add-model-price/references/provider-usage-key-matrix.md@@ -88,6 +88,13 @@ different response and billing dimensions.- State why cache, reasoning, or modality families are not applicable.- Do not reuse Claude cache keys or rates without provider evidence.+## TypeSafe Jev++TypeSafe's usage object reports only `input_tokens` and `output_tokens`, which+Langfuse stores as `input` and `output`. Price `input` per token and set+`output` to `0`, since TypeSafe does not charge for output tokens. Jev has no+cache, reasoning, or modality dimensions, so those key families do not apply.+## Deterministic validationThe pricing validator always checks structural catalog invariants. It checksdiff --git a/.github/workflows/model-price-audit.yml b/.github/workflows/model-price-audit.ymlindex 623929423d..1f6f3470b6 100644--- a/.github/workflows/model-price-audit.yml+++ b/.github/workflows/model-price-audit.yml@@ -226,6 +226,7 @@ jobs:- Workflow self-improvements must preserve the schedule/manual triggers, repo guard, read-only audit permissions, explicit read-only GitHub token, separate publish job, secret names, input validation, diff allowlist, hook-disabled commit path, staged-blob checks, artifact handoff, and non-fatal reviewer request.- Do not grant the audit job write permissions, `id-token: write`, package-manager tools, arbitrary shell tools, arbitrary network tools, `gh`, or `git push`.- Do not change official provider WebFetch domains. Report a required domain change as unresolved so the workflow and deterministic source validation can be updated together.+ - Only add or change a pricing entry for a model served by a covered provider (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Vertex AI, or TypeSafe for its Jev models), confirmed via an allowed WebFetch domain. Verify `jev` against TypeSafe's own docs (`https://docs.typesafe.ai/models`), not gateway or reseller prices. Do not add or modify a pricing entry for any other internal, partner-routed, or gateway-only model without official evidence from an allowed provider domain; report it as unresolved instead.- Keep the first entry in every selectable model array unchanged unless the audit explicitly changes the default model.- Use the allowed exact `date -u +%Y-%m-%dT00:00:00.000Z` command if you need the current timestamp.@@ -244,7 +245,7 @@ jobs:--max-turns ${{ steps.validate-inputs.outputs.max_turns }}--max-budget-usd ${{ steps.validate-inputs.outputs.max_budget_usd }}--no-session-persistence- --allowedTools "Read(/.github/workflows/model-price-audit.yml),Read(/worker/src/constants/default-model-prices.json),Read(/packages/shared/src/server/llm/types.ts),Read(/packages/shared/src/server/otel/OtelIngestionProcessor.ts),Read(/.agents/skills/add-model-price/SKILL.md),Read(/.agents/skills/add-model-price/references/*.md),Edit(/.github/workflows/model-price-audit.yml),Edit(/worker/src/constants/default-model-prices.json),Edit(/packages/shared/src/server/llm/types.ts),Edit(/.agents/skills/add-model-price/references/*.md),Write(/.agents/skills/add-model-price/references/*.md),WebFetch(domain:platform.claude.com),WebFetch(domain:docs.anthropic.com),WebFetch(domain:developers.openai.com),WebFetch(domain:ai.google.dev),WebFetch(domain:cloud.google.com),WebFetch(domain:aws.amazon.com),WebFetch(domain:azure.microsoft.com),Bash(date -u +%Y-%m-%dT00:00:00.000Z),Bash(uuidgen),Bash(node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs:*),Bash(node .agents/skills/add-model-price/scripts/test-match-pattern.mjs:*)"+ --allowedTools "Read(/.github/workflows/model-price-audit.yml),Read(/worker/src/constants/default-model-prices.json),Read(/packages/shared/src/server/llm/types.ts),Read(/packages/shared/src/server/otel/OtelIngestionProcessor.ts),Read(/.agents/skills/add-model-price/SKILL.md),Read(/.agents/skills/add-model-price/references/*.md),Edit(/.github/workflows/model-price-audit.yml),Edit(/worker/src/constants/default-model-prices.json),Edit(/packages/shared/src/server/llm/types.ts),Edit(/.agents/skills/add-model-price/references/*.md),Write(/.agents/skills/add-model-price/references/*.md),WebFetch(domain:platform.claude.com),WebFetch(domain:docs.anthropic.com),WebFetch(domain:developers.openai.com),WebFetch(domain:ai.google.dev),WebFetch(domain:cloud.google.com),WebFetch(domain:aws.amazon.com),WebFetch(domain:azure.microsoft.com),WebFetch(domain:typesafe.ai),WebFetch(domain:docs.typesafe.ai),Bash(date -u +%Y-%m-%dT00:00:00.000Z),Bash(uuidgen),Bash(node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs:*),Bash(node .agents/skills/add-model-price/scripts/test-match-pattern.mjs:*)"--json-schema '{"type":"object","additionalProperties":false,"properties":{"summary":{"type":"string"},"auditDate":{"type":"string","pattern":"^[0-9]{4}-[0-9]{2}-[0-9]{2}$"},"pullRequestTitle":{"type":"string","maxLength":100},"modelsChecked":{"type":"array","minItems":1,"items":{"type":"object","additionalProperties":false,"properties":{"provider":{"type":"string","minLength":1},"model":{"type":"string","minLength":1},"pricingChecked":{"type":"string","minLength":1},"priceConfirmed":{"type":"string","enum":["yes","no"]},"usageKeysChecked":{"type":"string","minLength":1},"usageKeyCoverageConfirmed":{"type":"string","enum":["yes","no"]},"tieringChecked":{"type":"string","minLength":1},"tieringCorrect":{"type":"string","enum":["yes","no","not_applicable"]},"change":{"type":"string","enum":["none","updated","added","unresolved"]},"officialSources":{"type":"array","items":{"type":"string"}},"comments":{"type":"string"}},"required":["provider","model","pricingChecked","priceConfirmed","usageKeysChecked","usageKeyCoverageConfirmed","tieringChecked","tieringCorrect","change","officialSources","comments"]}},"changedModels":{"type":"array","items":{"type":"string"}},"skillReferenceUpdates":{"type":"array","items":{"type":"string"}},"workflowUpdates":{"type":"array","items":{"type":"string"}},"unresolvedFindings":{"type":"array","items":{"type":"string"}},"validation":{"type":"array","items":{"type":"string"}}},"required":["summary","auditDate","pullRequestTitle","modelsChecked","changedModels","skillReferenceUpdates","workflowUpdates","unresolvedFindings","validation"]}'- name: Mock Claude price auditdiff --git a/scripts/model-price-audit/audit-output-contract.mjs b/scripts/model-price-audit/audit-output-contract.mjsindex 367835a7b8..b2197ce715 100644--- a/scripts/model-price-audit/audit-output-contract.mjs+++ b/scripts/model-price-audit/audit-output-contract.mjs@@ -6,6 +6,7 @@ const officialSourceHosts = ["developers.openai.com","docs.anthropic.com","platform.claude.com",+ "typesafe.ai",];const normalize = (value) => value.trim().toLowerCase();diff --git a/scripts/model-price-audit/audit-output-contract.test.mjs b/scripts/model-price-audit/audit-output-contract.test.mjsindex 459c70dab0..ffe4a75440 100644--- a/scripts/model-price-audit/audit-output-contract.test.mjs+++ b/scripts/model-price-audit/audit-output-contract.test.mjs@@ -196,6 +196,44 @@ test("derives changed pricing metadata from the semantic diff", () => {assert.deepEqual(normalizedOutput.changedModels, [`${model} (added)`]);});+test("accepts TypeSafe docs as the official source for a jev price update", () => {+ const model = "jev";+ const { normalizedOutput, result } = runContract({+ basePrices: [pricingEntry(model, 4.2e-8)],+ currentPrices: [pricingEntry(model, 5e-8)],+ output: {+ ...structuredOutput([+ {+ ...auditRow({+ model,+ change: "updated",+ officialSources: ["https://docs.typesafe.ai/models"],+ }),+ provider: "TypeSafe",+ },+ ]),+ pullRequestTitle: "chore(pricing): update jev pricing",+ },+ });++ assert.equal(result.status, 0, result.stderr);+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (updated)`]);+});++test("rejects lookalike TypeSafe source hosts", () => {+ const { result } = runContract({+ output: structuredOutput([+ auditRow({+ model: "jev",+ officialSources: ["https://docs.typesafe.ai.example.com/models"],+ }),+ ]),+ });++ assert.notEqual(result.status, 0);+ assert.match(result.stderr, /not on an approved official domain for jev/);+});+test("rejects automated pricing-entry removal", () => {const model = "gpt-4o";const { result } = runContract({