Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(api): advance unstable evaluator sunset (#16716)

Tobias Wochinger74ded7eda13b
diff --git a/fern/apis/server/definition/unstable/evaluation-rules.yml b/fern/apis/server/definition/unstable/evaluation-rules.yml
index ccf0d16187..26240d0f13 100644
--- a/fern/apis/server/definition/unstable/evaluation-rules.yml
+++ b/fern/apis/server/definition/unstable/evaluation-rules.yml
@@ -11,7 +11,7 @@ service:
create:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
Create an evaluation rule.
@@ -241,7 +241,7 @@ service:
list:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
List evaluation rules in the authenticated project.
@@ -269,7 +269,7 @@ service:
get:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
Get one evaluation rule by its identifier.
@@ -293,7 +293,7 @@ service:
update:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
Update an evaluation rule.
@@ -337,7 +337,7 @@ service:
delete:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluation-rules` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
Delete an evaluation rule.
diff --git a/fern/apis/server/definition/unstable/evaluators.yml b/fern/apis/server/definition/unstable/evaluators.yml
index 622adf79c1..86f64bee17 100644
--- a/fern/apis/server/definition/unstable/evaluators.yml
+++ b/fern/apis/server/definition/unstable/evaluators.yml
@@ -11,7 +11,7 @@ service:
create:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
Create an evaluator in the authenticated project.
@@ -262,7 +262,7 @@ service:
list:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
List the evaluators available to the authenticated project.
@@ -292,7 +292,7 @@ service:
get:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
Get one evaluator by `id`.
@@ -316,7 +316,7 @@ service:
delete:
availability:
status: deprecated
- message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
+ message: "On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4."
docs: |
Delete an evaluator.
diff --git a/web/public/generated/api/openapi.yml b/web/public/generated/api/openapi.yml
index 46477fb5de..e6541a1766 100644
--- a/web/public/generated/api/openapi.yml
+++ b/web/public/generated/api/openapi.yml
@@ -8572,7 +8572,7 @@ paths:
post:
description: >-
**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on November 16, 2026. Use the stable
+ and will be removed on September 4, 2026. Use the stable
`/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
are unaffected by this date; the endpoint becomes unavailable when they
upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
@@ -8745,7 +8745,7 @@ paths:
get:
description: >-
**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on November 16, 2026. Use the stable
+ and will be removed on September 4, 2026. Use the stable
`/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
are unaffected by this date; the endpoint becomes unavailable when they
upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
@@ -8827,7 +8827,7 @@ paths:
get:
description: >-
**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on November 16, 2026. Use the stable
+ and will be removed on September 4, 2026. Use the stable
`/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
are unaffected by this date; the endpoint becomes unavailable when they
upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
@@ -8901,7 +8901,7 @@ paths:
patch:
description: >-
**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on November 16, 2026. Use the stable
+ and will be removed on September 4, 2026. Use the stable
`/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
are unaffected by this date; the endpoint becomes unavailable when they
upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
@@ -9028,7 +9028,7 @@ paths:
delete:
description: >-
**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on November 16, 2026. Use the stable
+ and will be removed on September 4, 2026. Use the stable
`/api/public/v2/evaluation-rules` API instead. Self-hosted deployments
are unaffected by this date; the endpoint becomes unavailable when they
upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
@@ -9100,7 +9100,7 @@ paths:
post:
description: >-
**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on November 16, 2026. Use the stable
+ and will be removed on September 4, 2026. Use the stable
`/api/public/v2/evaluators` API instead. Self-hosted deployments are
unaffected by this date; the endpoint becomes unavailable when they
upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
@@ -9250,7 +9250,7 @@ paths:
deprecated: true
get:
description: |-
- **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on November 16, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
+ **Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated and will be removed on September 4, 2026. Use the stable `/api/public/v2/evaluators` API instead. Self-hosted deployments are unaffected by this date; the endpoint becomes unavailable when they upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4).
List the evaluators available to the authenticated project.
@@ -9325,7 +9325,7 @@ paths:
get:
description: >-
**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on November 16, 2026. Use the stable
+ and will be removed on September 4, 2026. Use the stable
`/api/public/v2/evaluators` API instead. Self-hosted deployments are
unaffected by this date; the endpoint becomes unavailable when they
upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
@@ -9398,7 +9398,7 @@ paths:
delete:
description: >-
**Deprecated:** On Langfuse Cloud, this unstable endpoint is deprecated
- and will be removed on November 16, 2026. Use the stable
+ and will be removed on September 4, 2026. Use the stable
`/api/public/v2/evaluators` API instead. Self-hosted deployments are
unaffected by this date; the endpoint becomes unavailable when they
upgrade to Langfuse v4. See the [Langfuse v3 to v4 upgrade
@@ -13693,8 +13693,8 @@ components:
$ref: '#/components/schemas/EvaluationRuleAssignment'
description: >-
All modern and legacy evaluation-rule assignments in
- newest-assignment-first order, including each assigned rule ID and
- its stored mapping.
+ newest-assignment-first order. Rule-specific mappings are exposed as
+ `variableMappingOverride`; inherited defaults are omitted.
createdAt:
type: string
format: date-time
diff --git a/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts b/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts
index 0d49cd4d1d..991af5cb04 100644
--- a/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts
+++ b/web/src/__tests__/server/unit/openapi-deprecations.servertest.ts
@@ -35,6 +35,18 @@ const openApiPath = path.resolve(
);
const MESSAGE = "Use `GET /api/public/v2/observations?from=<from>` instead.";
+const UNSTABLE_EVALS_SUNSET_HUMAN = "September 4, 2026";
+const UNSTABLE_EVAL_OPERATIONS = new Set([
+ "POST /api/public/unstable/evaluators",
+ "GET /api/public/unstable/evaluators",
+ "GET /api/public/unstable/evaluators/{evaluatorId}",
+ "DELETE /api/public/unstable/evaluators/{evaluatorId}",
+ "POST /api/public/unstable/evaluation-rules",
+ "GET /api/public/unstable/evaluation-rules",
+ "GET /api/public/unstable/evaluation-rules/{evaluationRuleId}",
+ "PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}",
+ "DELETE /api/public/unstable/evaluation-rules/{evaluationRuleId}",
+]);
/** Spec fixture in the shape `fern export` produces, aliased `security` included. */
const SPEC = `openapi: 3.0.1
@@ -126,9 +138,12 @@ describe("OpenAPI deprecations", () => {
for (const { method, endpointPath, message } of getFernDeprecatedOperations(
definitionDirectory,
)) {
- expect(message, `${method.toUpperCase()} ${endpointPath}`).toContain(
- `will be removed on ${V3_SUNSET_HUMAN}.`,
- );
+ const operation = `${method.toUpperCase()} ${endpointPath}`;
+ const sunsetDate = UNSTABLE_EVAL_OPERATIONS.has(operation)
+ ? UNSTABLE_EVALS_SUNSET_HUMAN
+ : V3_SUNSET_HUMAN;
+
+ expect(message, operation).toContain(`will be removed on ${sunsetDate}.`);
}
});