🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
fix(api): advance unstable evaluator sunset (#16716)
Tobias Wochinger74ded7eda13b
diff --git a/fern/apis/server/definition/unstable/evaluation-rules.yml b/fern/apis/server/definition/unstable/evaluation-rules.ymlindex ccf0d16187..26240d0f13 100644--- a/fern/apis/server/definition/unstable/evaluation-rules.yml+++ b/fern/apis/server/definition/unstable/evaluation-rules.yml@@ -11,7 +11,7 @@ service:create:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |Create an evaluation rule.@@ -241,7 +241,7 @@ service:list:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |List evaluation rules in the authenticated project.@@ -269,7 +269,7 @@ service:get:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |Get one evaluation rule by its identifier.@@ -293,7 +293,7 @@ service:update:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |Update an evaluation rule.@@ -337,7 +337,7 @@ service:delete:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |Delete an evaluation rule.diff --git a/fern/apis/server/definition/unstable/evaluators.yml b/fern/apis/server/definition/unstable/evaluators.ymlindex 622adf79c1..86f64bee17 100644--- a/fern/apis/server/definition/unstable/evaluators.yml+++ b/fern/apis/server/definition/unstable/evaluators.yml@@ -11,7 +11,7 @@ service:create:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |Create an evaluator in the authenticated project.@@ -262,7 +262,7 @@ service:list:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |List the evaluators available to the authenticated project.@@ -292,7 +292,7 @@ service:get:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |Get one evaluator by `id`.@@ -316,7 +316,7 @@ service:delete:availability:status: deprecated- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."docs: |Delete an evaluator.diff --git a/web/public/generated/api/openapi.yml b/web/public/generated/api/openapi.ymlindex 46477fb5de..e6541a1766 100644--- a/web/public/generated/api/openapi.yml+++ b/web/public/generated/api/openapi.yml@@ -8572,7 +8572,7 @@ paths:post:description: >-**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on November 16, 2026. Use the stable+ and will be removed on September 4, 2026. Use the stable`/api/public/v2/evaluation-rules` API instead. Self-hosted deploymentsare unaffected by this date; the endpoint becomes unavailable when theyupgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade@@ -8745,7 +8745,7 @@ paths:get:description: >-**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on November 16, 2026. Use the stable+ and will be removed on September 4, 2026. Use the stable`/api/public/v2/evaluation-rules` API instead. Self-hosted deploymentsare unaffected by this date; the endpoint becomes unavailable when theyupgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade@@ -8827,7 +8827,7 @@ paths:get:description: >-**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on November 16, 2026. Use the stable+ and will be removed on September 4, 2026. Use the stable`/api/public/v2/evaluation-rules` API instead. Self-hosted deploymentsare unaffected by this date; the endpoint becomes unavailable when theyupgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade@@ -8901,7 +8901,7 @@ paths:patch:description: >-**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on November 16, 2026. Use the stable+ and will be removed on September 4, 2026. Use the stable`/api/public/v2/evaluation-rules` API instead. Self-hosted deploymentsare unaffected by this date; the endpoint becomes unavailable when theyupgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade@@ -9028,7 +9028,7 @@ paths:delete:description: >-**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on November 16, 2026. Use the stable+ and will be removed on September 4, 2026. Use the stable`/api/public/v2/evaluation-rules` API instead. Self-hosted deploymentsare unaffected by this date; the endpoint becomes unavailable when theyupgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade@@ -9100,7 +9100,7 @@ paths:post:description: >-**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on November 16, 2026. Use the stable+ and will be removed on September 4, 2026. Use the stable`/api/public/v2/evaluators` API instead. Self-hosted deployments areunaffected by this date; the endpoint becomes unavailable when theyupgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade@@ -9250,7 +9250,7 @@ paths:deprecated: trueget:description: |-- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).+ **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).List the evaluators available to the authenticated project.@@ -9325,7 +9325,7 @@ paths:get:description: >-**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on November 16, 2026. Use the stable+ and will be removed on September 4, 2026. Use the stable`/api/public/v2/evaluators` API instead. Self-hosted deployments areunaffected by this date; the endpoint becomes unavailable when theyupgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade@@ -9398,7 +9398,7 @@ paths:delete:description: >-**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated- and will be removed on November 16, 2026. Use the stable+ and will be removed on September 4, 2026. Use the stable`/api/public/v2/evaluators` API instead. Self-hosted deployments areunaffected by this date; the endpoint becomes unavailable when theyupgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade@@ -13693,8 +13693,8 @@ components:$ref: '#/components/schemas/EvaluationRuleAssignment'description: >-All modern and legacy evaluation-rule assignments in- newest-assignment-first order, including each assigned rule ID and- its stored mapping.+ newest-assignment-first order. Rule-specific mappings are exposed as+ `variableMappingOverride`; inherited defaults are omitted.createdAt:type: stringformat: date-timediff --git a/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts b/web/src/__tests__/server/unit/openapi-deprecations.servertest.tsindex 0d49cd4d1d..991af5cb04 100644--- a/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts+++ b/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts@@ -35,6 +35,18 @@ const openApiPath = path.resolve();const MESSAGE = "Use `GET /api/public/v2/observations?from=<from>` instead.";+const UNSTABLE_EVALS_SUNSET_HUMAN = "September 4, 2026";+const UNSTABLE_EVAL_OPERATIONS = new Set([+ "POST /api/public/unstable/evaluators",+ "GET /api/public/unstable/evaluators",+ "GET /api/public/unstable/evaluators/{evaluatorId}",+ "DELETE /api/public/unstable/evaluators/{evaluatorId}",+ "POST /api/public/unstable/evaluation-rules",+ "GET /api/public/unstable/evaluation-rules",+ "GET /api/public/unstable/evaluation-rules/{evaluationRuleId}",+ "PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}",+ "DELETE /api/public/unstable/evaluation-rules/{evaluationRuleId}",+]);/** Spec fixture in the shape `fern export` produces, aliased `security` included. */const SPEC = `openapi: 3.0.1@@ -126,9 +138,12 @@ describe("OpenAPI deprecations", () => {for (const { method, endpointPath, message } of getFernDeprecatedOperations(definitionDirectory,)) {- expect(message, `${method.toUpperCase()} ${endpointPath}`).toContain(- `will be removed on ${V3_SUNSET_HUMAN}.`,- );+ const operation = `${method.toUpperCase()} ${endpointPath}`;+ const sunsetDate = UNSTABLE_EVAL_OPERATIONS.has(operation)+ ? UNSTABLE_EVALS_SUNSET_HUMAN+ : V3_SUNSET_HUMAN;++ expect(message, operation).toContain(`will be removed on ${sunsetDate}.`);}});