Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

chore(pricing): let the model price audit verify jev pricing against TypeSafe docs (#17859)

langfuse-botf21a5bc8e203
diff --git a/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md b/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md
index dc6f09bb4f..8293c96126 100644
--- a/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md
+++ b/.agents/skills/add-model-price/references/provider-sources-and-price-keys.md
@@ -12,6 +12,7 @@ Always fetch pricing from the provider's official docs before editing.
| Google Gemini (Vertex AI) | `https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models` |
| AWS Bedrock | `https://aws.amazon.com/bedrock/pricing/` |
| Azure OpenAI | `https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/` |
+| TypeSafe (Jev) | `https://docs.typesafe.ai/models` |
### Known source quirks (as of 2026-06)
@@ -790,6 +791,23 @@ Formula:
price_per_token = price_per_mtok / 1_000_000
```
+- **TypeSafe Jev pricing (documented September 24 2026)** — `jev` is
+ TypeSafe's decision model, not an LLM from the other covered providers. Its
+ official price table is `https://docs.typesafe.ai/models` ("Price (per Btok /
+ per Mtok) $42 / $0.042"; "Charged per input token. Output tokens are free."),
+ and the `https://typesafe.ai/` homepage repeats "$42 Per Billion input
+ tokens". Convert per billion tokens: `$42 / 1_000_000_000 = 4.2e-8` for
+ `input`, with `output` at `0`. The entry keeps a single Standard tier: the
+ models page documents no cache, batch, context-size, or modality tiers.
+ Jev's usage object has only `input_tokens` and `output_tokens`, which
+ Langfuse stores as `input` and `output`. Verify against TypeSafe's own docs
+ only; Vercel AI Gateway and OpenRouter can proxy Jev (`TYPESAFE_UPSTREAMS`
+ in `types.ts`), but their resale prices are not official evidence. Versioned
+ IDs (`jev-1.13.0`) and the `jev-latest`/`jev-preview` aliases listed on the
+ models page are already covered by the `matchPattern`. `typeSafeModels` in
+ `types.ts` is not one of the selectable arrays the audit may edit; report a
+ newly released Jev version that should become selectable as unresolved.
+
## Provider Usage Keys
Use [provider-usage-key-matrix.md](provider-usage-key-matrix.md) as the single
diff --git a/.agents/skills/add-model-price/references/provider-usage-key-matrix.md b/.agents/skills/add-model-price/references/provider-usage-key-matrix.md
index d5480d0c40..776b7f8460 100644
--- a/.agents/skills/add-model-price/references/provider-usage-key-matrix.md
+++ b/.agents/skills/add-model-price/references/provider-usage-key-matrix.md
@@ -88,6 +88,13 @@ different response and billing dimensions.
- State why cache, reasoning, or modality families are not applicable.
- Do not reuse Claude cache keys or rates without provider evidence.
+## TypeSafe Jev
+
+TypeSafe's usage object reports only `input_tokens` and `output_tokens`, which
+Langfuse stores as `input` and `output`. Price `input` per token and set
+`output` to `0`, since TypeSafe does not charge for output tokens. Jev has no
+cache, reasoning, or modality dimensions, so those key families do not apply.
+
## Deterministic validation
The pricing validator always checks structural catalog invariants. It checks
diff --git a/.github/workflows/model-price-audit.yml b/.github/workflows/model-price-audit.yml
index 623929423d..1f6f3470b6 100644
--- a/.github/workflows/model-price-audit.yml
+++ b/.github/workflows/model-price-audit.yml
@@ -226,6 +226,7 @@ jobs:
- Workflow self-improvements must preserve the schedule/manual triggers, repo guard, read-only audit permissions, explicit read-only GitHub token, separate publish job, secret names, input validation, diff allowlist, hook-disabled commit path, staged-blob checks, artifact handoff, and non-fatal reviewer request.
- Do not grant the audit job write permissions, `id-token: write`, package-manager tools, arbitrary shell tools, arbitrary network tools, `gh`, or `git push`.
- Do not change official provider WebFetch domains. Report a required domain change as unresolved so the workflow and deterministic source validation can be updated together.
+ - Only add or change a pricing entry for a model served by a covered provider (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Vertex AI, or TypeSafe for its Jev models), confirmed via an allowed WebFetch domain. Verify `jev` against TypeSafe's own docs (`https://docs.typesafe.ai/models`), not gateway or reseller prices. Do not add or modify a pricing entry for any other internal, partner-routed, or gateway-only model without official evidence from an allowed provider domain; report it as unresolved instead.
- Keep the first entry in every selectable model array unchanged unless the audit explicitly changes the default model.
- Use the allowed exact `date -u +%Y-%m-%dT00:00:00.000Z` command if you need the current timestamp.
@@ -244,7 +245,7 @@ jobs:
--max-turns ${{ steps.validate-inputs.outputs.max_turns }}
--max-budget-usd ${{ steps.validate-inputs.outputs.max_budget_usd }}
--no-session-persistence
- --allowedTools "Read(/.github/workflows/model-price-audit.yml),Read(/worker/src/constants/default-model-prices.json),Read(/packages/shared/src/server/llm/types.ts),Read(/packages/shared/src/server/otel/OtelIngestionProcessor.ts),Read(/.agents/skills/add-model-price/SKILL.md),Read(/.agents/skills/add-model-price/references/*.md),Edit(/.github/workflows/model-price-audit.yml),Edit(/worker/src/constants/default-model-prices.json),Edit(/packages/shared/src/server/llm/types.ts),Edit(/.agents/skills/add-model-price/references/*.md),Write(/.agents/skills/add-model-price/references/*.md),WebFetch(domain:platform.claude.com),WebFetch(domain:docs.anthropic.com),WebFetch(domain:developers.openai.com),WebFetch(domain:ai.google.dev),WebFetch(domain:cloud.google.com),WebFetch(domain:aws.amazon.com),WebFetch(domain:azure.microsoft.com),Bash(date -u +%Y-%m-%dT00:00:00.000Z),Bash(uuidgen),Bash(node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs:*),Bash(node .agents/skills/add-model-price/scripts/test-match-pattern.mjs:*)"
+ --allowedTools "Read(/.github/workflows/model-price-audit.yml),Read(/worker/src/constants/default-model-prices.json),Read(/packages/shared/src/server/llm/types.ts),Read(/packages/shared/src/server/otel/OtelIngestionProcessor.ts),Read(/.agents/skills/add-model-price/SKILL.md),Read(/.agents/skills/add-model-price/references/*.md),Edit(/.github/workflows/model-price-audit.yml),Edit(/worker/src/constants/default-model-prices.json),Edit(/packages/shared/src/server/llm/types.ts),Edit(/.agents/skills/add-model-price/references/*.md),Write(/.agents/skills/add-model-price/references/*.md),WebFetch(domain:platform.claude.com),WebFetch(domain:docs.anthropic.com),WebFetch(domain:developers.openai.com),WebFetch(domain:ai.google.dev),WebFetch(domain:cloud.google.com),WebFetch(domain:aws.amazon.com),WebFetch(domain:azure.microsoft.com),WebFetch(domain:typesafe.ai),WebFetch(domain:docs.typesafe.ai),Bash(date -u +%Y-%m-%dT00:00:00.000Z),Bash(uuidgen),Bash(node .agents/skills/add-model-price/scripts/validate-pricing-file.mjs:*),Bash(node .agents/skills/add-model-price/scripts/test-match-pattern.mjs:*)"
--json-schema '{"type":"object","additionalProperties":false,"properties":{"summary":{"type":"string"},"auditDate":{"type":"string","pattern":"^[0-9]{4}-[0-9]{2}-[0-9]{2}$"},"pullRequestTitle":{"type":"string","maxLength":100},"modelsChecked":{"type":"array","minItems":1,"items":{"type":"object","additionalProperties":false,"properties":{"provider":{"type":"string","minLength":1},"model":{"type":"string","minLength":1},"pricingChecked":{"type":"string","minLength":1},"priceConfirmed":{"type":"string","enum":["yes","no"]},"usageKeysChecked":{"type":"string","minLength":1},"usageKeyCoverageConfirmed":{"type":"string","enum":["yes","no"]},"tieringChecked":{"type":"string","minLength":1},"tieringCorrect":{"type":"string","enum":["yes","no","not_applicable"]},"change":{"type":"string","enum":["none","updated","added","unresolved"]},"officialSources":{"type":"array","items":{"type":"string"}},"comments":{"type":"string"}},"required":["provider","model","pricingChecked","priceConfirmed","usageKeysChecked","usageKeyCoverageConfirmed","tieringChecked","tieringCorrect","change","officialSources","comments"]}},"changedModels":{"type":"array","items":{"type":"string"}},"skillReferenceUpdates":{"type":"array","items":{"type":"string"}},"workflowUpdates":{"type":"array","items":{"type":"string"}},"unresolvedFindings":{"type":"array","items":{"type":"string"}},"validation":{"type":"array","items":{"type":"string"}}},"required":["summary","auditDate","pullRequestTitle","modelsChecked","changedModels","skillReferenceUpdates","workflowUpdates","unresolvedFindings","validation"]}'
- name: Mock Claude price audit
diff --git a/scripts/model-price-audit/audit-output-contract.mjs b/scripts/model-price-audit/audit-output-contract.mjs
index 367835a7b8..b2197ce715 100644
--- a/scripts/model-price-audit/audit-output-contract.mjs
+++ b/scripts/model-price-audit/audit-output-contract.mjs
@@ -6,6 +6,7 @@ const officialSourceHosts = [
"developers.openai.com",
"docs.anthropic.com",
"platform.claude.com",
+ "typesafe.ai",
];
const normalize = (value) => value.trim().toLowerCase();
diff --git a/scripts/model-price-audit/audit-output-contract.test.mjs b/scripts/model-price-audit/audit-output-contract.test.mjs
index 459c70dab0..ffe4a75440 100644
--- a/scripts/model-price-audit/audit-output-contract.test.mjs
+++ b/scripts/model-price-audit/audit-output-contract.test.mjs
@@ -196,6 +196,44 @@ test("derives changed pricing metadata from the semantic diff", () => {
assert.deepEqual(normalizedOutput.changedModels, [`${model} (added)`]);
});
+test("accepts TypeSafe docs as the official source for a jev price update", () => {
+ const model = "jev";
+ const { normalizedOutput, result } = runContract({
+ basePrices: [pricingEntry(model, 4.2e-8)],
+ currentPrices: [pricingEntry(model, 5e-8)],
+ output: {
+ ...structuredOutput([
+ {
+ ...auditRow({
+ model,
+ change: "updated",
+ officialSources: ["https://docs.typesafe.ai/models"],
+ }),
+ provider: "TypeSafe",
+ },
+ ]),
+ pullRequestTitle: "chore(pricing): update jev pricing",
+ },
+ });
+
+ assert.equal(result.status, 0, result.stderr);
+ assert.deepEqual(normalizedOutput.changedModels, [`${model} (updated)`]);
+});
+
+test("rejects lookalike TypeSafe source hosts", () => {
+ const { result } = runContract({
+ output: structuredOutput([
+ auditRow({
+ model: "jev",
+ officialSources: ["https://docs.typesafe.ai.example.com/models"],
+ }),
+ ]),
+ });
+
+ assert.notEqual(result.status, 0);
+ assert.match(result.stderr, /not on an approved official domain for jev/);
+});
+
test("rejects automated pricing-entry removal", () => {
const model = "gpt-4o";
const { result } = runContract({